Building a Production-Ready File Download System with GoFrame

Introduction 👋

Hey devs! If you're building web applications with Go, you've probably encountered the need to handle file downloads. While it might seem straightforward at first, implementing production-ready file downloads comes with its own set of challenges. In this guide, I'll share practical insights on building robust file download functionality using GoFrame.

TL;DR: We'll cover everything from basic file downloads to production-ready implementations with GoFrame, including handling large files, implementing resume support, and optimizing performance.

What We'll Cover 📝

Setting up basic file downloads
Handling large files without memory issues
Implementing resume/partial downloads
Adding security measures
Optimizing for production
Real-world examples and gotchas

Why GoFrame for File Downloads? 🤔

Before we dive in, you might be wondering why choose GoFrame for handling file downloads. Here's the deal: while you could implement file downloads with standard Go libraries, GoFrame provides some nice abstractions that make our lives easier:

Stream-based processing (goodbye memory issues!)
Built-in middleware support
Clean error handling
Progress tracking capabilities

Getting Started 🌱

First things first, let's set up our environment. Make sure you have Go installed (1.16+), then grab GoFrame:

go get -u github.com/gogf/gf/v2@latest

Basic Implementation

Let's start with a simple file download handler. This is your entry point to understanding how GoFrame handles downloads:

package handler

import (
    "github.com/gogf/gf/v2/net/ghttp"
)

func SimpleDownload(r *ghttp.Request) {
    filePath := r.Get("file").String()

    // Quick security check
    if !isValidFile(filePath) {
        r.Response.WriteStatus(403)
        return
    }

    // Set headers and serve
    r.Response.Header().Set("Content-Type", "application/octet-stream")
    r.Response.ServeFileDownload(filePath)
}

Pretty straightforward, right? But wait - there's more to consider when building for production! 🛠️

Security First! 🔒

Before we get too excited about serving files, let's talk security. Here's a robust validation function to prevent common security issues:

var allowedExtensions = map[string]bool{
    ".txt":  true,
    ".pdf":  true,
    ".doc":  true,
    ".xlsx": true,
}

func isValidFile(filePath string) bool {
    // 🚫 Path traversal check
    if strings.Contains(filePath, "..") {
        return false
    }

    // 📁 File existence & type check
    fileInfo, err := os.Stat(filePath)
    if err != nil || !fileInfo.Mode().IsRegular() {
        return false
    }

    // 📏 Size check (100MB limit)
    if fileInfo.Size() > 100*1024*1024 {
        return false
    }

    // 🔍 Extension check
    ext := strings.ToLower(filepath.Ext(filePath))
    return allowedExtensions[ext]
}

Handling Large Files Like a Pro 🏋️‍♂️

Now, here's where things get interesting. When dealing with large files, you don't want to load everything into memory. Here's a streaming implementation that won't make your server cry:

func StreamDownload(r *ghttp.Request) {
    const bufSize = 32 * 1024 // 32KB chunks

    file := r.Get("file").String()
    fd, err := os.Open(file)
    if err != nil {
        r.Response.WriteStatus(500)
        return
    }
    defer fd.Close() // Don't forget this! 😉

    info, _ := fd.Stat()
    r.Response.Header().Set("Content-Length", gconv.String(info.Size()))

    // Stream in chunks
    buf := make([]byte, bufSize)
    for {
        n, err := fd.Read(buf)
        if n > 0 {
            r.Response.Write(buf[:n])
        }
        if err == io.EOF {
            break
        }
    }
}

Resume Support: Because Downloads Sometimes Fail 😅

Let's be real - downloads can fail, especially for large files. Here's how to implement resume support:

func ResumeDownload(r *ghttp.Request) {
    rangeHeader := r.Header.Get("Range")
    if rangeHeader != "" {
        // Handle range request
        // ... (previous range parsing code)

        r.Response.Header().Set("Content-Range",
            fmt.Sprintf("bytes %d-%d/%d", start, end, fileSize))
        r.Response.WriteStatus(206) // Partial Content

        // Serve the requested chunk
        streamFileContent(r, fd, end-start+1)
    }
}

Real-World Scenarios and Solutions 🎯

Let's look at some common scenarios you might encounter and how to handle them:

1. Handling Different File Types 📁

Different file types often need different handling. Here's a practical example:

func SmartDownload(r *ghttp.Request) {
    filePath := r.Get("file").String()
    fileExt := strings.ToLower(filepath.Ext(filePath))

    switch fileExt {
    case ".pdf":
        // For PDFs, we might want to allow preview
        r.Response.Header().Set("Content-Type", "application/pdf")
        r.Response.Header().Set("Content-Disposition", "inline")
    case ".csv":
        // For CSVs, we might want to add BOM for Excel compatibility
        r.Response.Header().Set("Content-Type", "text/csv")
        r.Response.Write("\xEF\xBB\xBF") // Add BOM
    case ".mp4":
        // For videos, support range requests for streaming
        handleVideoStream(r)
    default:
        // Default download behavior
        r.Response.Header().Set("Content-Type", "application/octet-stream")
        r.Response.Header().Set("Content-Disposition", "attachment")
    }

    r.Response.ServeFileDownload(filePath)
}

2. Browser Compatibility Handling 🌐

Different browsers handle downloads differently. Here's how to handle it:

func BrowserAwareDownload(r *ghttp.Request) {
    filename := r.Get("file").String()
    userAgent := r.Header.Get("User-Agent")

    // Handle filename encoding for different browsers
    if strings.Contains(userAgent, "MSIE") || 
       strings.Contains(userAgent, "Edge") {
        // URL encode for IE/Edge
        filename = url.QueryEscape(filename)
    } else {
        // RFC 5987 encoding for others
        filename = fmt.Sprintf("UTF-8''%s", 
            url.QueryEscape(filename))
    }

    disposition := fmt.Sprintf("attachment; filename*=%s", filename)
    r.Response.Header().Set("Content-Disposition", disposition)
}

3. Download Progress Monitoring 📊

Here's how to implement progress monitoring with WebSocket updates:

type Progress struct {
    Total     int64   `json:"total"`
    Current   int64   `json:"current"`
    Speed     float64 `json:"speed"`
    Remaining int     `json:"remaining"`
}

func ProgressDownload(r *ghttp.Request) {
    ws, err := r.WebSocket()
    if err != nil {
        return
    }

    file := r.Get("file").String()
    info, _ := os.Stat(file)
    total := info.Size()

    // Create a custom reader that reports progress
    reader := &ProgressReader{
        Reader: bufio.NewReader(file),
        Total:  total,
        OnProgress: func(current int64) {
            progress := Progress{
                Total:     total,
                Current:   current,
                Speed:    calculateSpeed(current),
                Remaining: calculateRemaining(current, total),
            }
            ws.WriteJSON(progress)
        },
    }

    io.Copy(r.Response.Writer, reader)
}

Production Tips and Best Practices 💡

After implementing file downloads in numerous production environments, here are some battle-tested tips:

Always Rate Limit 🚦

var downloadLimiter = rate.NewLimiter(rate.Limit(100), 200)

Implement Caching 🗄️

func CachedDownload(r *ghttp.Request) {
    if data := memCache.Get(file); data != nil {
        return serveContent(r, data)
    }
    // ... fallback to disk
}

Monitor Everything 📊

Here's a practical monitoring implementation:

type DownloadMetrics struct {
    ActiveDownloads    *atomic.Int64
    TotalBytes        *atomic.Int64
    ErrorCount        *atomic.Int64
    DownloadDurations *metrics.Histogram
}

func NewDownloadMetrics() *DownloadMetrics {
    return &DownloadMetrics{
        ActiveDownloads:    atomic.NewInt64(0),
        TotalBytes:        atomic.NewInt64(0),
        ErrorCount:        atomic.NewInt64(0),
        DownloadDurations: metrics.NewHistogram(metrics.HistogramOpts{
            Buckets: []float64{.1, .5, 1, 2.5, 5, 10, 30},
        }),
    }
}

func (m *DownloadMetrics) Track(r *ghttp.Request) func() {
    start := time.Now()
    m.ActiveDownloads.Add(1)

    return func() {
        m.ActiveDownloads.Add(-1)
        duration := time.Since(start).Seconds()
        m.DownloadDurations.Observe(duration)
    }
}

Implement Circuit Breakers 🔌

Protect your system with circuit breakers:

type DownloadBreaker struct {
    breaker *gobreaker.CircuitBreaker
}

func NewDownloadBreaker() *DownloadBreaker {
    return &DownloadBreaker{
        breaker: gobreaker.NewCircuitBreaker(gobreaker.Settings{
            Name:        "download-breaker",
            MaxRequests: 100,
            Interval:    10 * time.Second,
            Timeout:     30 * time.Second,
            ReadyToTrip: func(counts gobreaker.Counts) bool {
                failureRatio := float64(counts.TotalFailures) / float64(counts.Requests)
                return counts.Requests >= 10 && failureRatio >= 0.6
            },
        }),
    }
}

Implement Graceful Shutdown 🛑

Handle shutdowns properly:

type DownloadManager struct {
    activeDownloads sync.WaitGroup
    shutdownCh     chan struct{}
}

func (dm *DownloadManager) HandleDownload(r *ghttp.Request) {
    dm.activeDownloads.Add(1)
    defer dm.activeDownloads.Done()

    // Create download context with shutdown signal
    ctx, cancel := context.WithCancel(r.Context())
    defer cancel()

    go func() {
        select {
        case <-dm.shutdownCh:
            // Save progress and cleanup
            cancel()
        case <-ctx.Done():
            return
        }
    }()

    // Proceed with download...
}

func (dm *DownloadManager) Shutdown(timeout time.Duration) error {
    close(dm.shutdownCh)

    // Wait for active downloads with timeout
    c := make(chan struct{})
    go func() {
        dm.activeDownloads.Wait()
        close(c)
    }()

    select {
    case <-c:
        return nil
    case <-time.After(timeout):
        return errors.New("shutdown timeout")
    }
}

Advanced Features and Examples 🚀

1. Zip Download on the Fly 📦

Need to create ZIP archives dynamically? Here's how:

func ZipDownload(r *ghttp.Request) {
    files := r.GetArray("files")

    r.Response.Header().Set("Content-Type", "application/zip")
    r.Response.Header().Set("Content-Disposition", 
        "attachment; filename=archive.zip")

    zw := zip.NewWriter(r.Response.Writer)
    defer zw.Close()

    for _, file := range files {
        // Add file to zip
        f, err := os.Open(file)
        if err != nil {
            continue
        }
        defer f.Close()

        // Create zip entry
        header := &zip.FileHeader{
            Name:   filepath.Base(file),
            Method: zip.Deflate,
        }

        writer, err := zw.CreateHeader(header)
        if err != nil {
            continue
        }

        io.Copy(writer, f)
    }
}

2. Cloud Storage Integration 🌥️

Here's an example integrating with S3-compatible storage:

func CloudDownload(r *ghttp.Request) {
    bucket := "my-bucket"
    key := r.Get("key").String()

    // Get object from S3
    input := &s3.GetObjectInput{
        Bucket: aws.String(bucket),
        Key:    aws.String(key),
    }

    result, err := s3Client.GetObject(input)
    if err != nil {
        r.Response.WriteStatus(500)
        return
    }
    defer result.Body.Close()

    // Set headers
    r.Response.Header().Set("Content-Type", *result.ContentType)
    r.Response.Header().Set("Content-Length", 
        fmt.Sprintf("%d", *result.ContentLength))

    // Stream the response
    io.Copy(r.Response.Writer, result.Body)
}

3. Download Queue Implementation 📋

For managing large numbers of downloads:

type DownloadQueue struct {
    queue    chan DownloadJob
    workers  int
    metrics  *DownloadMetrics
}

type DownloadJob struct {
    ID       string
    FilePath string
    Priority int
    Callback func(error)
}

func NewDownloadQueue(workers int) *DownloadQueue {
    dq := &DownloadQueue{
        queue:   make(chan DownloadJob, 1000),
        workers: workers,
        metrics: NewDownloadMetrics(),
    }

    for i := 0; i < workers; i++ {
        go dq.worker()
    }

    return dq
}

func (dq *DownloadQueue) worker() {
    for job := range dq.queue {
        // Process download job
        err := processDownload(job)
        if job.Callback != nil {
            job.Callback(err)
        }
    }
}

func (dq *DownloadQueue) Enqueue(job DownloadJob) error {
    select {
    case dq.queue <- job:
        return nil
    default:
        return errors.New("queue full")
    }
}

Comprehensive Error Handling Patterns 🎯

Let's look at how to handle various error scenarios robustly:

// Custom error types for better error handling
type DownloadError struct {
    Code    int
    Message string
    Err     error
}

func (e *DownloadError) Error() string {
    if e.Err != nil {
        return fmt.Sprintf("%s: %v", e.Message, e.Err)
    }
    return e.Message
}

// Error codes
const (
    ErrFileNotFound = iota + 1000
    ErrPermissionDenied
    ErrQuotaExceeded
    ErrInvalidFileType
    ErrFileTooLarge
)

// Robust download handler with error handling
func RobustDownload(r *ghttp.Request) {
    defer func() {
        if err := recover(); err != nil {
            // Log the stack trace
            debug.PrintStack()
            // Return 500 error to client
            r.Response.WriteStatus(500)
        }
    }()

    file := r.Get("file").String()

    // Validate request
    if err := validateDownloadRequest(r); err != nil {
        handleDownloadError(r, err)
        return
    }

    // Check user quota
    if err := checkUserQuota(r); err != nil {
        handleDownloadError(r, &DownloadError{
            Code:    ErrQuotaExceeded,
            Message: "Download quota exceeded",
            Err:     err,
        })
        return
    }

    // Attempt file download
    if err := streamFileWithRetry(r, file); err != nil {
        handleDownloadError(r, err)
        return
    }
}

// Error handler for different scenarios
func handleDownloadError(r *ghttp.Request, err error) {
    var downloadErr *DownloadError
    if errors.As(err, &downloadErr) {
        switch downloadErr.Code {
        case ErrFileNotFound:
            r.Response.WriteStatus(404)
            r.Response.WriteJson(g.Map{
                "error": "File not found",
                "details": downloadErr.Message,
            })
        case ErrPermissionDenied:
            r.Response.WriteStatus(403)
            r.Response.WriteJson(g.Map{
                "error": "Access denied",
                "details": downloadErr.Message,
            })
        case ErrQuotaExceeded:
            r.Response.WriteStatus(429)
            r.Response.WriteJson(g.Map{
                "error": "Quota exceeded",
                "details": downloadErr.Message,
            })
        default:
            r.Response.WriteStatus(500)
            r.Response.WriteJson(g.Map{
                "error": "Internal server error",
                "reference": uuid.New().String(),
            })
        }
        return
    }

    // Handle generic errors
    r.Response.WriteStatus(500)
}

// Retry mechanism for transient failures
func streamFileWithRetry(r *ghttp.Request, file string) error {
    const maxRetries = 3
    const baseDelay = 100 * time.Millisecond

    var lastErr error
    for attempt := 0; attempt < maxRetries; attempt++ {
        err := streamFile(r, file)
        if err == nil {
            return nil
        }

        lastErr = err

        // Don't retry on certain errors
        if errors.Is(err, os.ErrNotExist) || 
           errors.Is(err, os.ErrPermission) {
            return err
        }

        // Exponential backoff
        delay := baseDelay * time.Duration(math.Pow(2, float64(attempt)))
        time.Sleep(delay)
    }

    return fmt.Errorf("failed after %d attempts: %w", maxRetries, lastErr)
}

Specific Use Case Examples 🔍

1. Video Streaming with HLS Support 🎥

func VideoStreamHandler(r *ghttp.Request) {
    videoPath := r.Get("video").String()

    // Check if requesting manifest
    if strings.HasSuffix(videoPath, ".m3u8") {
        serveHLSManifest(r, videoPath)
        return
    }

    // Check if requesting segment
    if strings.HasSuffix(videoPath, ".ts") {
        serveHLSSegment(r, videoPath)
        return
    }

    // Serve video file directly with range support
    serveVideoWithRange(r, videoPath)
}

func serveHLSManifest(r *ghttp.Request, path string) {
    r.Response.Header().Set("Content-Type", "application/vnd.apple.mpegurl")
    r.Response.Header().Set("Cache-Control", "max-age=5")

    // Read and serve manifest
    content, err := ioutil.ReadFile(path)
    if err != nil {
        r.Response.WriteStatus(404)
        return
    }

    r.Response.Write(content)
}

func serveVideoWithRange(r *ghttp.Request, path string) {
    info, err := os.Stat(path)
    if err != nil {
        r.Response.WriteStatus(404)
        return
    }

    file, err := os.Open(path)
    if err != nil {
        r.Response.WriteStatus(500)
        return
    }
    defer file.Close()

    rangeHeader := r.Header.Get("Range")
    if rangeHeader != "" {
        ranges, err := parseRange(rangeHeader, info.Size())
        if err != nil {
            r.Response.WriteStatus(416)
            return
        }

        if len(ranges) > 0 {
            start, end := ranges[0][0], ranges[0][1]
            r.Response.Header().Set("Content-Range",
                fmt.Sprintf("bytes %d-%d/%d", start, end, info.Size()))
            r.Response.Header().Set("Content-Length",
                fmt.Sprintf("%d", end-start+1))
            r.Response.WriteStatus(206)

            file.Seek(start, 0)
            io.CopyN(r.Response.Writer, file, end-start+1)
            return
        }
    }

    r.Response.Header().Set("Content-Length", fmt.Sprintf("%d", info.Size()))
    r.Response.Header().Set("Content-Type", "video/mp4")
    io.Copy(r.Response.Writer, file)
}

2. Large Excel File Generation 📊

func ExcelDownloadHandler(r *ghttp.Request) {
    // Create a new file
    f := excelize.NewFile()
    defer f.Close()

    // Create buffered writer
    buf := new(bytes.Buffer)
    writer := bufio.NewWriter(buf)

    // Start progress tracking
    progress := 0
    total := 1000000 // Example: 1 million rows

    // Stream data writing
    for i := 0; i < total; i++ {
        // Write row data
        row := []interface{}{
            fmt.Sprintf("Data %d", i),
            time.Now(),
            rand.Float64(),
        }
        cell, _ := excelize.CoordinatesToCellName(1, i+1)
        f.SetSheetRow("Sheet1", cell, &row)

        // Update progress every 1%
        currentProgress := (i * 100) / total
        if currentProgress > progress {
            progress = currentProgress
            // Send progress through WebSocket if needed
            sendProgress(r, progress)
        }
    }

    // Set headers for Excel file
    r.Response.Header().Set("Content-Type", 
        "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet")
    r.Response.Header().Set("Content-Disposition", 
        "attachment; filename=large-report.xlsx")

    // Save to response writer
    if err := f.Write(r.Response.Writer); err != nil {
        r.Response.WriteStatus(500)
        return
    }
}

func sendProgress(r *ghttp.Request, progress int) {
    // Implementation depends on your WebSocket setup
    // Example using gorilla/websocket
    if ws, ok := r.GetCtxVar("ws").(*websocket.Conn); ok {
        ws.WriteJSON(map[string]interface{}{
            "progress": progress,
        })
    }
}

3. PDF Generation and Download 📄

func PDFDownloadHandler(r *ghttp.Request) {
    // Create PDF document
    pdf := gofpdf.New("P", "mm", "A4", "")
    pdf.AddPage()

    // Add content
    pdf.SetFont("Arial", "B", 16)
    pdf.Cell(40, 10, "Generated Report")

    // Add table
    pdf.SetFont("Arial", "", 12)
    data := [][]string{
        {"Column 1", "Column 2", "Column 3"},
        {"Data 1", "Data 2", "Data 3"},
        // ... more rows
    }

    for i, row := range data {
        for j, col := range row {
            pdf.Cell(40, 10, col)
            if j == len(row)-1 {
                pdf.Ln(-1)
            }
        }
        if i == 0 {
            pdf.Ln(-1)
        }
    }

    // Set headers
    r.Response.Header().Set("Content-Type", "application/pdf")
    r.Response.Header().Set("Content-Disposition", 
        "attachment; filename=report.pdf")

    // Write to response
    if err := pdf.Output(r.Response.Writer); err != nil {
        r.Response.WriteStatus(500)
        return
    }
}

Common Gotchas to Avoid 🚨

Memory Leaks
- Always use defer for cleanup
- Don't read entire files into memory
Security Issues
- Validate file paths
- Check file permissions
- Limit file types
Performance Problems
- Use buffered reading
- Implement proper caching
- Don't forget to close files

Real-World Example: A Complete Download Service 🌟

Here's a production-ready download service combining all the concepts we've discussed:

type DownloadService struct {
    limiter  *rate.Limiter
    cache    *Cache
    metrics  *Metrics
}

func (s *DownloadService) ServeDownload(r *ghttp.Request) {
    // 1. Rate limiting
    if err := s.limiter.Wait(r.Context()); err != nil {
        r.Response.WriteStatus(429)
        return
    }

    // 2. Try cache
    if data := s.cache.Get(r.Get("file").String()); data != nil {
        s.metrics.RecordHit()
        return s.serveContent(r, data)
    }

    // 3. Stream file with resume support
    s.streamWithResume(r)
}

Testing Your Implementation 🧪

Don't forget to test! Here's a quick test case to get you started:

func TestDownload(t *testing.T) {
    s := g.Server()
    s.BindHandler("/download", DownloadHandler)

    client := g.Client()
    r, err := client.Get("/download?file=test.txt")

    assert.Nil(t, err)
    assert.Equal(t, 200, r.StatusCode)
}

Wrapping Up 🎁

Building production-ready file downloads with GoFrame isn't just about calling ServeFileDownload. It's about:

Handling security properly
Managing resources efficiently
Supporting resume capabilities
Monitoring and maintaining performance

What's Next?

Add progress tracking for downloads
Implement cloud storage integration
Add support for file preprocessing
Explore streaming compression

Resources 📚

Let me know in the comments if you have any questions or want to share your own experiences with file downloads in Go! 💬

Like this article? Follow me for more Go development tips and tutorials! 🚀

Building a Production-Ready File Download System with GoFrame

Introduction 👋

What We'll Cover 📝

Why GoFrame for File Downloads? 🤔

Getting Started 🌱

Basic Implementation

Security First! 🔒

Handling Large Files Like a Pro 🏋️‍♂️

Resume Support: Because Downloads Sometimes Fail 😅

Real-World Scenarios and Solutions 🎯

1. Handling Different File Types 📁

2. Browser Compatibility Handling 🌐

3. Download Progress Monitoring 📊

Production Tips and Best Practices 💡

Advanced Features and Examples 🚀

1. Zip Download on the Fly 📦

2. Cloud Storage Integration 🌥️

3. Download Queue Implementation 📋

Comprehensive Error Handling Patterns 🎯

Specific Use Case Examples 🔍

1. Video Streaming with HLS Support 🎥

2. Large Excel File Generation 📊

3. PDF Generation and Download 📄

Common Gotchas to Avoid 🚨

Real-World Example: A Complete Download Service 🌟

Testing Your Implementation 🧪

Wrapping Up 🎁

What's Next?

Resources 📚

Comments (0)

Read More

#reading

#popular

Building a Production-Ready File Download System with GoFrame

Introduction 👋

What We'll Cover 📝

Why GoFrame for File Downloads? 🤔

Getting Started 🌱

Basic Implementation

Security First! 🔒

Handling Large Files Like a Pro 🏋️‍♂️

Resume Support: Because Downloads Sometimes Fail 😅

Real-World Scenarios and Solutions 🎯

1. Handling Different File Types 📁

2. Browser Compatibility Handling 🌐

3. Download Progress Monitoring 📊

Production Tips and Best Practices 💡

Advanced Features and Examples 🚀

1. Zip Download on the Fly 📦

2. Cloud Storage Integration 🌥️

3. Download Queue Implementation 📋

Comprehensive Error Handling Patterns 🎯

Specific Use Case Examples 🔍

1. Video Streaming with HLS Support 🎥

2. Large Excel File Generation 📊

3. PDF Generation and Download 📄

Common Gotchas to Avoid 🚨

Real-World Example: A Complete Download Service 🌟

Testing Your Implementation 🧪

Wrapping Up 🎁

What's Next?

Resources 📚

Comments (0)

Read More

⚛️ Build a Simple Todo App with React Store - a Tiny React State Manager

How to manage large env files?

Top 8 Open-Source Tools for Web Application Development

Encrypted Chat Application with web option

#reading

#popular