docker-compose.yml (local dev)

t is a data transfer problem. Optimizing for memory and network resilience requires moving transfer logic out of the application tier entirely, using cloud-native upload primitives, and implementing client-side chunking with integrity verification.

Core Solution

Scaling file uploads requires a three-tier architecture: client-side chunking, server-side presigned URL orchestration, and async storage/processing pipelines. Below is a production-ready implementation pattern.

1. Client-Side Chunking & Integrity Generation

The client splits files into fixed-size chunks, computes hashes for verification, and requests presigned URLs per chunk.

// client/chunker.ts
import { createHash } from 'crypto';

const CHUNK_SIZE = 5 * 1024 * 1024; // 5MB

export async function prepareChunks(file: File) {
  const chunks: { index: number; data: Blob; hash: string }[] = [];
  let offset = 0;

  while (offset < file.size) {
    const slice = file.slice(offset, offset + CHUNK_SIZE);
    const buffer = await slice.arrayBuffer();
    const hash = createHash('sha256').update(Buffer.from(buffer)).digest('hex');
    
    chunks.push({
      index: Math.floor(offset / CHUNK_SIZE),
      data: slice,
      hash
    });
    offset += CHUNK_SIZE;
  }

  return chunks;
}

export async function uploadChunk(chunk: { index: number; data: Blob; hash: string }, uploadId: string) {
  const res = await fetch(`/api/uploads/presign`, {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({ uploadId, partNumber: chunk.index + 1, size: chunk.data.size })
  });
  const { url } = await res.json();

  const uploadRes = await fetch(url, {
    method: 'PUT',
    body: chunk.data,
    headers: { 'x-amz-checksum-sha256': chunk.hash }
  });

  if (!uploadRes.ok) throw new Error(`Chunk ${chunk.index} failed: ${uploadRes.status}`);
  
  return {
    ETag: uploadRes.headers.get('etag')?.replace(/"/g, ''),
    PartNumber: chunk.index + 1
  };
}

2. Server-Side Presigned URL Orchestration

The server generates presigned URLs without touching the file data. It initializes multipart uploads and tracks state.

// server/routes/uploads.ts
import { S3Client, CreateMultipartUploadCommand, UploadPartCommand } from '@aws-sdk/client-s3';
import { getSignedUrl } from '@aws-sdk/s3-request-presigner';

const s3 = new S3Client({ region: process.env.AWS_REGION });

export async function createUploadSession(fileName: string) {
  const cmd = new CreateMultipartUploadCommand({
    Bucket: process.env.S3_BUCKET,
    Key: `uploads/${crypto.randomUUID()}/${fileName}`,
    Metadata: { 'client-id': 'web-app' }
  });
  const res = await s3.send(cmd);
  return { uploadId: res.UploadId!, key: res.Key! };
}

export async function generatePresignedUrl(uploadId: string, partNumber: number, size: number) {
  const cmd = new UploadPartCommand({
    Bucket: process.env.S3_BUCKET,
    Key: `uploads/${uploadId}/temp`, // Simplified; production should map uploadId to key
    UploadId: uploadId,
    PartNumber: partNumber,
    ContentLength: size
  });
  const url = await getSignedUrl(s3, cmd, { expiresIn: 300 });
  return { url };
}

3. Async Assembly & Processing Pipeline

Once all chunks upload, the client signals completion. A queue worker assembles the file and triggers downstream processing.

// server/workers/assembly.ts
import { Queue, Worker } from 'bullmq';
import { S3Client, CompleteMultipartUploadCommand } from '@aws-sdk/client-s3';

const s3 = new S3Client({ region: process.env.AWS_REGION });
const assemblyQueue = new Queue('upload-assembly', { connection: { host: 'localhost', port: 6379 } });

assemblyQueue.add('assemble', {
  uploadId: 'abc-123',
  key: 'uploads/abc-123/document.pdf',
  parts: [{ ETag: '"etag1"', PartNumber: 1 }, { ETag: '"etag2"', PartNumber: 2 }]
});

const worker = new Worker('upload-assembly', async (job) => {
  const { uploadId, key, parts } = job.data;
  
  await s3.send(new CompleteMultipartUploadCommand({
    Bucket: process.env.S3_BUCKET,
    Key: key,
    UploadId: uploadId,
    MultipartUpload: { Parts: parts }
  }));

  // Trigger downstream processing (image resize, virus scan, metadata extraction)
  await fetch(`${process.env.PROCESSING_API}/trigger`, {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({ key, uploadId })
  });
}, { connection: { host: 'localhost', port: 6379 } });

Architecture Decisions & Rationale

• 5MB Chunk Size: Aligns with S3/GCS multipart limits (5MB min, 10GB max per part). Balances network overhead with retry granularity.
• SHA-256 Per Chunk: Enables integrity verification without re-downloading. Cloud providers validate checksums during upload, failing fast on corruption.
• Presigned URL TTL (300s): Limits exposure window while allowing sufficient time for chunk upload. Shorter TTLs require dynamic regeneration, increasing server load.
• Async Assembly: Decouples transfer from processing. Application servers never block on I/O. Queue workers scale independently based on storage write latency.
• Separate Processing Pipeline: Virus scanning, metadata extraction, and format conversion run in isolated containers. Failures in processing never corrupt the uploaded artifact.

Pitfall Guide

1. Buffering Entire Files in Application Memory Treating req.file as a safe abstraction leads to heap exhaustion. Always stream or bypass application memory entirely. Use direct-to-storage patterns or pipe to temporary disk with strict size limits.

2. Ignoring Cloud Provider Chunk Limits S3 requires minimum 5MB parts (except the final part). GCS enforces similar constraints. Uploading 1MB chunks triggers EntityTooSmall errors and wastes API calls. Validate chunk size before generation.

3. Skipping Multipart Completion Validation Failing to verify ETag parity during CompleteMultipartUpload results in silent corruption. Cloud providers return mismatched checksums if parts are reordered or duplicated. Always validate part lists server-side before completion.

4. Synchronous Post-Upload Processing Blocking the upload response path with image resizing, PDF parsing, or virus scanning creates timeout cascades. Process asynchronously. Return a 202 Accepted with a job ID, then notify clients via WebSocket or polling.

5. Misconfigured CORS or Presigned Header Requirements Browsers reject direct uploads if presigned URLs lack Access-Control-Allow-Origin or if client headers don't match signed headers exactly. Ensure x-amz-checksum-sha256 and custom metadata are included in the signature scope.

6. Orphaned Multipart Uploads Abandoned uploads consume storage and accrue costs. Implement lifecycle policies or scheduled jobs to abort incomplete multipart uploads older than 24-48 hours.

7. Inadequate Rate Limiting & File Validation Upload endpoints are prime abuse vectors. Validate MIME types, scan for executable payloads, and enforce per-user rate limits before generating presigned URLs. Never trust client-provided filenames or extensions.

Production Best Practices:

• Implement exponential backoff with jitter for chunk retries.
• Monitor upload latency distribution, not just success rates.
• Use idempotency keys to prevent duplicate assembly jobs.
• Store upload metadata in a relational database for audit trails and resume state tracking.

Production Bundle

Action Checklist

• Configure chunk size to 5-8MB and validate against cloud provider limits
• Implement SHA-256 hashing per chunk and pass checksums to presigned URL scope
• Set presigned URL TTL to 300-600 seconds with dynamic regeneration on expiry
• Decouple assembly and processing into async queue workers (BullMQ, SQS, or Pub/Sub)
• Deploy lifecycle rules to abort orphaned multipart uploads after 24 hours
• Add per-user rate limiting and MIME/type validation before presign generation
• Instrument upload success rate, chunk retry count, and assembly latency metrics
• Test network interruption recovery with throttled connections and forced disconnects

Decision Matrix

Scenario	Recommended Approach	Why	Cost Impact
Low volume (<100 uploads/day), trusted network	Direct-to-App with streaming	Simplicity outweighs resilience needs	Low infra, high dev time
High concurrency, mobile clients	Chunked + Resumable + Direct	Handles packet loss, scales independently	Moderate infra, low support cost
Strict compliance (HIPAA, SOC2)	Direct-to-Storage + KMS + Async Scan	Data never touches app servers, audit trail intact	High storage cost, low risk
Cost-sensitive, predictable traffic	Presigned URL (Single Part)	Minimal server load, straightforward implementation	Low infra, moderate retry cost
Real-time preview required	Chunked + Direct + Edge Processing	Low latency assembly, CDN-ready artifacts	High edge compute, fast UX

Configuration Template

# docker-compose.yml (local dev)
version: '3.8'
services:
  redis:
    image: redis:7-alpine
    ports: ["6379:6379"]
  minio:
    image: minio/minio:latest
    command: server /data --console-address ":9001"
    environment:
      MINIO_ROOT_USER: dev
      MINIO_ROOT_PASSWORD: dev123456
    ports: ["9000:9000", "9001:9001"]
    volumes: ["minio_data:/data"]

volumes:
  minio_data:

# .env
AWS_REGION=us-east-1
AWS_ACCESS_KEY_ID=dev
AWS_SECRET_ACCESS_KEY=dev123456
S3_BUCKET=app-uploads
S3_ENDPOINT=http://localhost:9000
REDIS_URL=redis://localhost:6379
CHUNK_SIZE_MB=5
PRESIGN_TTL_SECONDS=300
MAX_UPLOAD_SIZE_MB=2048

// server/config/s3.ts
import { S3Client } from '@aws-sdk/client-s3';

export const s3Client = new S3Client({
  region: process.env.AWS_REGION,
  credentials: {
    accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
    secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!
  },
  ...(process.env.S3_ENDPOINT && { endpoint: process.env.S3_ENDPOINT, forcePathStyle: true })
});

Quick Start Guide

1. Initialize dependencies: npm i @aws-sdk/client-s3 @aws-sdk/s3-request-presigner bullmq crypto
2. Start local infrastructure: docker compose up -d (MinIO + Redis)
3. Configure environment: Copy .env template, set credentials to match local MinIO, adjust chunk size if needed.
4. Run server & worker: ts-node server/index.ts (API) + ts-node server/workers/assembly.ts (queue consumer)
5. Test upload flow: Use the client chunker script with a test file. Verify chunks upload to MinIO, assembly completes, and metadata triggers downstream processing.

Scaling file uploads is not about optimizing HTTP handlers. It is about designing resilient data pipelines that treat network instability as a first-class constraint, delegate transfer to infrastructure built for it, and isolate processing from the critical path. Implement chunked direct uploads, enforce integrity at the edge, and let async workers handle the rest. The architecture scales linearly, costs drop predictably, and failure recovery becomes deterministic.