import express, { Request, Response, NextFunction } from 'express';
import multer, { StorageEngine, FileFilterCallback } from 'multer';
import path from 'path';
import crypto from 'crypto';
import fs from 'fs';

const app = express();
app.use(express.json());

// Ensure upload directory exists
const UPLOAD_DIR = path.join(__dirname, '../storage/assets');
if (!fs.existsSync(UPLOAD_DIR)) {
  fs.mkdirSync(UPLOAD_DIR, { recursive: true });
}

const assetStorage: StorageEngine = multer.diskStorage({
  destination: (_req: Request, _file: Express.Multer.File, callback: (error: Error | null, destination: string) => void) => {
    callback(null, UPLOAD_DIR);
  },
  filename: (_req: Request, file: Express.Multer.File, callback: (error: Error | null, filename: string) => void) => {
    const uniquePrefix = crypto.randomBytes(8).toString('hex');
    const extension = path.extname(file.originalname).toLowerCase();
    callback(null, `${uniquePrefix}${extension}`);
  }
});

const MAX_PAYLOAD_SIZE = 10 * 1024 * 1024; // 10MB
const ALLOWED_MIMETYPES = new Set([
  'image/jpeg', 'image/png', 'image/webp', 
  'application/pdf', 'application/zip'
]);

const assetFilter = (_req: Request, file: Express.Multer.File, callback: FileFilterCallback) => {
  if (ALLOWED_MIMETYPES.has(file.mimetype)) {
    callback(null, true);
  } else {
    callback(new Error(`Rejected: ${file.mimetype} is not permitted`), false);
  }

};

const assetProcessor = multer({ storage: assetStorage, limits: { fileSize: MAX_PAYLOAD_SIZE }, fileFilter: assetFilter });

**Architecture Rationale:**
- `Set` lookup provides O(1) validation performance.
- Limits are enforced at the stream level, terminating the request before the full payload reaches disk.
- Filtering runs synchronously to block invalid streams early.

### Step 3: Implement Route Handlers & Error Boundaries

Multer middleware must be wrapped in an error boundary because it throws asynchronously. We separate success logic from failure handling.

```typescript
app.post('/api/v1/assets', assetProcessor.single('document'), (req: Request, res: Response) => {
  if (!req.file) {
    return res.status(400).json({ error: 'No valid file attached to request' });
  }

  res.status(201).json({
    id: req.file.filename,
    originalName: req.file.originalname,
    mimeType: req.file.mimetype,
    sizeBytes: req.file.size,
    storedAt: req.file.path
  });
});

// Centralized ingestion error handler
app.use((error: Error, _req: Request, res: Response, _next: NextFunction) => {
  if (error instanceof multer.MulterError) {
    const status = error.code === 'LIMIT_FILE_SIZE' ? 413 : 400;
    return res.status(status).json({ error: `Ingestion limit exceeded: ${error.message}` });
  }
  if (error.message.startsWith('Rejected:')) {
    return res.status(415).json({ error: error.message });
  }
  res.status(500).json({ error: 'Internal ingestion pipeline failure' });
});

Architecture Rationale:

• multer.MulterError provides structured error codes for programmatic handling.
• HTTP 413 (Payload Too Large) and 415 (Unsupported Media Type) align with REST semantics.
• Error middleware catches exceptions thrown during stream parsing, preventing unhandled promise rejections.

Step 4: Serve & Secure Assets

Static serving requires explicit path mapping. In production, this layer should sit behind authentication or CDN distribution.

app.use('/assets', express.static(UPLOAD_DIR, {
  dotfiles: 'ignore',
  index: false,
  maxAge: '1d'
}));

Architecture Rationale:

• dotfiles: 'ignore' prevents exposure of hidden system files.
• index: false disables directory listing, mitigating information disclosure.
• maxAge enables browser caching without compromising security.

Pitfall Guide

1. Missing enctype on Client Forms

Explanation: Browsers default to application/x-www-form-urlencoded for forms. Without enctype="multipart/form-data", binary data is base64-encoded or stripped entirely, causing silent upload failures. Fix: Always declare enctype="multipart/form-data" on HTML forms or set Content-Type: multipart/form-data with proper boundary generation in API clients.

2. Default Memory Storage Overflow

Explanation: Omitting storage configuration forces Multer to buffer files in RAM. Large uploads trigger V8 heap exhaustion and crash the Node process. Fix: Explicitly configure multer.diskStorage() or stream directly to object storage. Never rely on memory buffers in production.

3. MIME Type Spoofing

Explanation: file.mimetype is derived from client headers, not file content. Attackers can rename executables as .jpg to bypass validation. Fix: Implement server-side magic number verification using libraries like file-type or mmmagic. Validate content signatures after initial header checks.

4. Path Traversal in Custom Filenames

Explanation: Concatenating file.originalname directly into storage paths allows ../../../etc/passwd injection, overwriting system files. Fix: Strip directory components using path.basename(), enforce alphanumeric extensions, and generate server-side identifiers as demonstrated in the core solution.

5. Unhandled MulterError Exceptions

Explanation: Multer throws synchronously during stream parsing. If not caught, these errors bypass Express error middleware and terminate the worker. Fix: Wrap Multer middleware in a try/catch block or rely on Express error-handling middleware that explicitly checks instanceof multer.MulterError.

6. Orphaned Files on Validation Failure

Explanation: When fileFilter rejects a file after partial disk write, temporary fragments remain in the upload directory, consuming storage over time. Fix: Implement a cleanup hook in the error boundary that removes req.file?.path when validation fails or business logic throws.

7. Direct Public Exposure of Uploads

Explanation: Serving the upload directory without access control allows unauthorized enumeration and hotlinking. Fix: Route static serving through authentication middleware, generate expiring signed URLs, or proxy requests through a CDN with token-based access.

Production Bundle

Action Checklist

• Verify client-side enctype configuration matches multipart expectations
• Configure diskStorage with server-generated identifiers and extension sanitization
• Implement MIME type validation using both header checks and magic number verification
• Set explicit limits.fileSize thresholds aligned with infrastructure capacity
• Attach centralized error middleware that distinguishes MulterError from application errors
• Implement orphaned file cleanup in rejection and failure pathways
• Restrict static serving with directory listing disabled and dotfiles ignored
• Plan migration path to object storage for horizontal scaling requirements

Decision Matrix

Scenario	Recommended Approach	Why	Cost Impact
Internal admin dashboard with low volume	Local disk storage + Multer	Simplicity, low latency, minimal dependencies	Near-zero infrastructure cost
Public-facing media platform	Direct-to-S3 presigned URLs	Eliminates server bandwidth, scales infinitely	Higher CDN/storage costs, lower compute
High-throughput API gateway	Streaming to temporary storage + async processor	Decouples ingestion from business logic, prevents request timeouts	Moderate compute cost for queue workers
Compliance-heavy environment (HIPAA/GDPR)	Encrypted disk storage + immediate cloud replication	Meets data residency requirements while maintaining control	Increased storage and encryption overhead

Configuration Template

import multer from 'multer';
import path from 'path';
import crypto from 'crypto';
import fs from 'fs';

export const createIngestionMiddleware = (config: {
  directory: string;
  maxSizeBytes: number;
  allowedTypes: string[];
}) => {
  const targetDir = path.resolve(config.directory);
  fs.mkdirSync(targetDir, { recursive: true });

  return multer({
    storage: multer.diskStorage({
      destination: (_req, _file, cb) => cb(null, targetDir),
      filename: (_req, file, cb) => {
        const hash = crypto.randomBytes(12).toString('hex');
        const ext = path.extname(file.originalname).replace(/[^a-z0-9.]/gi, '');
        cb(null, `${hash}${ext}`);
      }
    }),
    limits: { fileSize: config.maxSizeBytes },
    fileFilter: (_req, file, cb) => {
      const valid = config.allowedTypes.includes(file.mimetype);
      cb(valid ? null : new Error('Type not permitted'), valid);
    }
  });
};

Quick Start Guide

1. Install dependencies: npm install express multer @types/express @types/multer
2. Create the ingestion module: Copy the configuration template into src/middleware/assetProcessor.ts
3. Mount the middleware: Import and attach to your Express route using .single('fieldName') or .array('fieldName', limit)
4. Add error handling: Register the error middleware after all routes to catch MulterError and validation rejections
5. Test with curl: curl -X POST http://localhost:3000/api/v1/assets -F "document=@./test.pdf" -F "category=report"

File ingestion in Express is rarely about parsing; it's about resource management, security boundaries, and lifecycle control. By treating uploads as a first-class pipeline rather than an afterthought, you eliminate memory leaks, prevent storage exhaustion, and build a foundation that scales from local development to distributed cloud deployments.