Building an Image Optimization Pipeline in Node.js with ShortPixel

ervice class to manage configuration and provide type-safe methods. This pattern promotes reusability and simplifies testing.

import { ShortPixelClient } from '@shortpixel-com/shortpixel';
import type { OptimizationConfig } from '@shortpixel-com/shortpixel';

export interface MediaPipelineConfig {
  apiKey: string;
  defaultLossyLevel: 0 | 1 | 2;
  outputDir: string;
}

export class MediaPipeline {
  private client: ShortPixelClient;
  private config: MediaPipelineConfig;

  constructor(config: MediaPipelineConfig) {
    this.config = config;
    this.client = new ShortPixelClient({ apiKey: config.apiKey });
  }

  /**
   * Optimizes a single local file or buffer.
   * Returns the download promise for the result.
   */
  async transformAsset(
    source: string | Buffer,
    options: Partial<OptimizationConfig> = {}
  ) {
    const result = await this.client.optimize(source, {
      lossy: options.lossy ?? this.config.defaultLossyLevel,
      convertto: options.convertto ?? '+webp',
      ...options,
    });

    return result.downloadTo(this.config.outputDir);
  }

  /**
   * Processes a batch of files with parallel polling.
   */
  async processBatch(
    sources: string[],
    options: Partial<OptimizationConfig> = {}
  ) {
    const batchResult = await this.client.fromFiles(sources, {
      lossy: options.lossy ?? this.config.defaultLossyLevel,
      convertto: options.convertto ?? '+webp',
      ...options,
    });

    return batchResult.downloadTo(this.config.outputDir);
  }
}

Architecture Rationale:

• Encapsulation: Wrapping the client allows centralized configuration management and easier mocking in unit tests.
• Type Safety: Using Partial<OptimizationConfig> ensures callers can override defaults without losing type checking.
• Default Flags: +webp retains the original format alongside WebP. Use webp to replace the original entirely. Adjust lossy levels based on asset type (e.g., 0 for logos, 1 for photos).

2. Express Integration

For real-time uploads, the SDK provides middleware that intercepts requests, processes assets, and mutates the request object. This eliminates manual download steps in route handlers.

import express from 'express';
import multer from 'multer';
import { ShortPixelExpress } from '@shortpixel-com/shortpixel';

const router = express.Router();
const upload = multer({ storage: multer.memoryStorage() });

router.post(
  '/ingest',
  upload.single('media_asset'),
  ShortPixelExpress({
    apiKey: process.env.SHORTPIXEL_API_KEY!,
    lossy: 1,
    convertto: '+webp',
    passthrough: false,
  }),
  (req, res) => {
    const optimizedBuffer = req.file?.buffer;
    const metadata = req.shortPixel?.files[0];

    if (!optimizedBuffer) {
      return res.status(500).json({ error: 'Optimization failed' });
    }

    res.json({
      status: 'processed',
      originalSize: req.file?.size,
      optimizedSize: optimizedBuffer.length,
      formats: metadata?.formats,
      filename: metadata?.outputFilename,
    });
  }
);

Key Behaviors:

• In-Place Mutation: The middleware replaces req.file.buffer with the optimized content. No additional await is required in the handler.
• Metadata Access: req.shortPixel contains normalized results, including format details and output filenames.
• Passthrough Mode: Set passthrough: true to allow unmatched routes to proceed without processing.

3. Advanced Operations

The SDK includes intent-named helpers for common transformations. These wrappers set specific parameters and forward additional options.

// Upscale with format conversion
await pipeline.client.upscale('product.jpg', 2, { convertto: '+webp' });

// Smart crop to exact dimensions
await pipeline.client.smartCrop('avatar.jpg', 300, 200, { convertto: '+webp' });

// Background removal (returns transparent WebP)
await pipeline.client.backgroundRemove('shoe.jpg', { convertto: '+webp' });

// Background replacement with color
await pipeline.client.backgroundChange('shoe.jpg', '#00ff0080', { convertto: '+webp' });

Note: Operations like backgroundRemove are computationally intensive. Increase the polling budget to prevent timeouts.

pipeline.client.set('poll', {
  enabled: true,
  interval: 2000,
  maxAttempts: 30,
});

Pitfall Guide

Production deployments must address common failure modes. The following pitfalls and fixes are derived from real-world usage patterns.

1. ESM Configuration Mismatch

   • Issue: Import errors due to CommonJS/ESM incompatibility.
   • Fix: Ensure package.json contains "type": "module". If using TypeScript, configure module: "NodeNext" in tsconfig.json.

2. Blocking Polling on Heavy Operations

   • Issue: Background removal or large batches cause request timeouts due to default polling limits.
   • Fix: Adjust poll.maxAttempts and poll.interval via client.set(). For background removal, set maxAttempts to at least 30.

3. Misinterpreting convertto Flags

   • Issue: Losing original formats when both are needed.
   • Fix: Use +webp or +avif to retain the original alongside the modern format. Use webp alone to replace the original.

4. Generic Error Handling

   • Issue: Catching all errors as generic exceptions, losing diagnostic context.
   • Fix: Leverage typed errors (ShortPixelAuthError, ShortPixelQuotaError, ShortPixelBatchError). Inspect err.spCode and err.httpStatus for precise branching.

5. Batch Failure Aggregation

   • Issue: Assuming batch success when partial failures occur.
   • Fix: Catch ShortPixelBatchError and inspect error.details for per-item status. Log failures for retry or manual review.

6. Memory Pressure in Express

   • Issue: Large uploads exhaust server memory when using memoryStorage.
   • Fix: Configure multer with limits: { fileSize: 10 * 1024 * 1024 }. For very large assets, consider streaming to temporary storage before processing.

7. Hardcoded API Keys

   • Issue: Security risks from embedding credentials.
   • Fix: Always load keys from environment variables. Validate presence at startup and fail fast if missing.

Production Bundle

Action Checklist

• Verify Node.js version is 20+ and package.json has "type": "module".
• Set SHORTPIXEL_API_KEY in environment variables; validate at startup.
• Configure polling limits for background removal or large batches.
• Implement typed error handling using spCode and httpStatus.
• Use +webp or +avif flags to retain original formats when required.
• Add multer file size limits in Express routes to prevent memory exhaustion.
• Log batch failure details via ShortPixelBatchError.details for observability.
• Test with malformed inputs to verify error branching.

Decision Matrix

Scenario	Recommended Approach	Why	Cost/Performance Impact
Single User Upload	Express Middleware	Low latency; inline processing	Fast; minimal overhead
Bulk Import Job	fromFiles Batch	Parallel polling; atomic results	Efficient; reduced API calls
Background Removal	backgroundRemove Helper	API offload; high quality	Higher cost; longer duration
High Traffic Ingestion	Queue + Batch Worker	Decouples processing from request	Scalable; improved UX
Format Preservation	convertto: '+webp'	Retains original + modern format	Slightly larger storage

Configuration Template

Use this template to bootstrap a robust pipeline configuration.

// config/media-pipeline.ts
import { MediaPipeline, MediaPipelineConfig } from '../services/media-pipeline';

export const mediaPipelineConfig: MediaPipelineConfig = {
  apiKey: process.env.SHORTPIXEL_API_KEY!,
  defaultLossyLevel: 1,
  outputDir: './processed_assets',
};

// Adjust polling for heavy operations
export function configurePolling(client: MediaPipeline['client']) {
  client.set('poll', {
    enabled: true,
    interval: 2000,
    maxAttempts: 30,
  });
}

Quick Start Guide

1. Install Dependencies:

   npm i @shortpixel-com/shortpixel express multer
   

2. Set Environment:

   export SHORTPIXEL_API_KEY="your_api_key_here"
   

3. Run Script:

   import { MediaPipeline } from './services/media-pipeline';
   import { mediaPipelineConfig } from './config/media-pipeline';
   
   const pipeline = new MediaPipeline(mediaPipelineConfig);
   await pipeline.transformAsset('./assets/sample.png');
   console.log('Optimization complete.');
   

4. Verify Output: Check the processed_assets directory for optimized files. Confirm format conversion and size reduction.

This pipeline structure ensures that image optimization is integrated directly into the backend flow, reducing manual intervention and improving application performance. By leveraging the SDK's abstractions, teams can focus on business logic while maintaining reliable media processing.