t. The idempotency store must be separate from the primary business database to avoid coupling and contention. 2. Key Generation: Keys should be generated by the client to allow retries across network hops. The key must be unique per intent. A common strategy is a UUID v4 or a hash of the request payload combined with a client nonce. 3. Atomicity: The check-and-set operation must be atomic. A race condition where two threads simultaneously see the key as missing will result in duplicate processing. This requires atomic primitives like SET NX or Lua scripts. 4. Result Caching: To handle the scenario where processing succeeds but the response is lost (network drop), the result must be stored. This allows subsequent retries to return the exact same response, ensuring the client receives the confirmation it missed.

TypeScript Implementation

The following implementation provides a robust idempotency middleware using a decorator pattern for Express/Fastify style handlers. It uses a Lua script for atomic check-and-set operations in Redis.

import { Redis } from 'ioredis';
import { Request, Response, NextFunction } from 'express';

// Lua script for atomic check-and-set with result caching
const IDEMPOTENCY_LUA_SCRIPT = `
  local key = KEYS[1]
  local ttl = tonumber(ARGV[1])
  local payload_hash = ARGV[2]
  
  local current = redis.call('GET', key)
  if current then
    return current
  end
  
  -- Set key with payload hash as initial value
  -- We use a placeholder state to prevent race conditions during processing
  redis.call('SET', key, 'PROCESSING', 'EX', ttl)
  return 'PROCESSING'
`;

interface IdempotencyResult {
  statusCode: number;
  body: any;
  headers?: Record<string, string>;
}

export class IdempotencyMiddleware {
  private redis: Redis;
  private keyPrefix: string;
  private ttlSeconds: number;

  constructor(redis: Redis, options: { keyPrefix?: string; ttlSeconds?: number }) {
    this.redis = redis;
    this.keyPrefix = options.keyPrefix || 'idempotency:';
    this.ttlSeconds = options.ttlSeconds || 86400; // Default 24h
  }

  // Middleware factory
  public enforce() {
    return async (req: Request, res: Response, next: NextFunction) => {
      const idempotencyKey = req.headers['x-idempotency-key'] as string;
      
      if (!idempotencyKey) {
        // Fallback: Generate key for safe retries? 
        // Recommendation: Reject or generate server-side key for specific endpoints
        return next(new Error('Missing Idempotency-Key header'));
      }

      const storageKey = `${this.keyPrefix}${idempotencyKey}`;
      
      // Normalize payload for validation (optional but recommended)
      const payloadHash = this.hashPayload(req.body);

      try {
        // Atomic check
        const status = await this.redis.eval(
          IDEMPOTENCY_LUA_SCRIPT,
          1,
          storageKey,
          this.ttlSeconds,
          payloadHash
        );

        if (status === 'PROCESSING') {
          // First request: Proceed to handler
          // Wrap res.json to capture result
          this.interceptResponse(res, storageKey);
          return next();
        } else {
          // Duplicate request: Return cached result
          const cachedResult: IdempotencyResult = JSON.parse(status as string);
          res.status(cachedResult.statusCode);
          if (cachedResult.headers) {
            res.set(cachedResult.headers);
          }
          return res.json(cachedResult.body);
        }
      } catch (error) {
        // Fail-open or fail-closed? 
        // Recommendation: Fail-closed for financial ops, fail-open for read-heavy ops
        console.error('Idempotency check failed:', error);
        return next(error);
      }
    };
  }

  private interceptResponse(res: Response, storageKey: string) {
    const originalJson = res.json.bind(res);
    
    res.json = (body: any) => {
      const result: IdempotencyResult = {
        statusCode: res.statusCode,
        body,
        headers: res.getHeaders() as Record<string, string>
      };

      // Store result asynchronously
      this.redis.setex(
        storageKey,
        this.ttlSeconds,
        JSON.stringify(result)
      ).catch(err => console.error('Failed to cache idempotency result:', err));

      return originalJson(body);
    };
  }

  private hashPayload(body: any): string {
    // Implementation of deterministic hashing (e.g., SHA-256)
    // Must handle object key ordering consistently
    return JSON.stringify(body); // Simplified for example
  }
}

Usage Example

const app = express();
const redis = new Redis();
const idempotency = new IdempotencyMiddleware(redis);

app.post('/api/payments', 
  express.json(),
  idempotency.enforce(),
  async (req: Request, res: Response) => {
    // Business logic executes only once per unique key
    const payment = await paymentService.process(req.body);
    res.status(201).json(payment);
  }
);

Pitfall Guide

1. Race Conditions via Non-Atomic Checks

Mistake: Using separate GET and SET commands. Explanation: Two concurrent requests may both see the key as missing, proceed to process, and both store results, causing duplicates. Fix: Always use atomic operations like SET NX, SETNX, or Lua scripts that combine check and set.

2. Payload Mutation and Key Collision

Mistake: Using only a client-generated UUID without validating payload integrity. Explanation: A client might reuse a key with a modified payload (e.g., changing the amount). The server returns the old result, causing data inconsistency. Fix: Implement payload hashing. Store the hash of the normalized request body with the key. On retry, compare the hash; if it differs, return a 409 Conflict error.

3. Ignoring Side Effects

Mistake: Making the database operation idempotent but failing to guard external side effects like emails or webhooks. Explanation: The DB update is suppressed, but the email service is called twice. Fix: Idempotency must wrap the entire transaction boundary, including side effects. Use an outbox pattern or ensure side effects are also keyed and deduplicated.

4. Storage Explosion via Missing TTLs

Mistake: Storing idempotency keys indefinitely. Explanation: High-traffic systems can generate millions of keys per day. Without TTL, Redis memory consumption grows linearly, leading to eviction or OOM crashes. Fix: Enforce strict TTLs. For financial transactions, 24-72 hours is sufficient. For audit requirements, move keys to cold storage asynchronously rather than keeping them in the hot cache.

5. The 201 vs. 200 Trap

Mistake: Returning 200 OK for a duplicate POST that originally returned 201 Created. Explanation: REST semantics dictate that POST creating a resource returns 201. Returning 200 on retry breaks client expectations and can cause parsing errors in strict clients. Fix: Cache the full response metadata, including status code and headers. Return the exact same status code on retries.

6. Idempotency on Non-Idempotent Operations

Mistake: Applying idempotency keys to operations that are inherently non-idempotent, such as "Add Item to Cart" where quantity increments. Explanation: Retrying "Add 1 Item" should result in +1, not +2. However, idempotency keys suppress the second addition, leaving the cart under-filled. Fix: Redesign the API to be idempotent by intent. Use "Set Cart Item Quantity" instead of "Add." Or, use idempotency keys only for creation/initialization operations, not accumulation operations.

7. Result Loss on Partial Failure

Mistake: Processing succeeds, but the process crashes before storing the result. Explanation: The client retries, the key check fails (key not stored), and the operation runs again. Fix: Ensure the result storage is part of the same transaction as the business logic, or use a Write-Ahead Log pattern. Alternatively, accept the risk for low-impact operations, but for financial data, the result must be persisted before acknowledging success.

Production Bundle

Action Checklist

• Audit Endpoints: Identify all POST, PATCH, and PUT endpoints that mutate state and require idempotency.
• Implement Atomic Checks: Ensure all idempotency checks use atomic Redis operations or database constraints (UNIQUE indexes).
• Define TTL Policy: Configure TTLs based on business requirements. Default to 24 hours for retries; extend only if required by client retry windows.
• Payload Normalization: Implement deterministic hashing for request bodies to detect payload tampering on retries.
• Result Caching: Capture and store full response metadata (status, headers, body) to handle response-loss scenarios.
• Monitoring: Add metrics for idempotency hits, misses, and conflicts. Alert on high conflict rates indicating client bugs.
• Client SDK Updates: Ensure client libraries automatically generate and attach X-Idempotency-Key headers for retryable requests.
• Load Testing: Simulate retry storms and network partitions to verify idempotency behavior under concurrency.

Decision Matrix

Scenario	Recommended Approach	Why	Cost Impact
Payment Processing	Idempotency Key + Result Caching + Journal	Requires strict consistency, auditability, and protection against double charges.	Medium (Redis storage + Journal overhead)
High-Throughput Metrics	Idempotency Key + Memory Store	Latency is critical; duplicates are acceptable within a small window.	Low (Memory only, no persistence)
Complex Workflow/Saga	Idempotency Key + Compensating Transactions	Long-running processes need state tracking; simple keys insufficient.	High (Orchestration overhead)
Internal Microservice RPC	Idempotency Key + DB Unique Index	Low latency; leveraging existing DB constraints reduces external dependencies.	Low (DB index overhead)
User Registration	Idempotency Key + DB Unique Constraint	Prevents duplicate accounts; DB constraint provides ultimate safety net.	Low

Configuration Template

# idempotency-config.yaml
idempotency:
  storage:
    type: redis
    url: "redis://idempotency-store:6379"
    key_prefix: "idem:"
    ttl_seconds: 86400
    max_memory_policy: "allkeys-lru"
  
  validation:
    payload_hash_algorithm: "sha256"
    strict_mode: true  # Reject retries with modified payloads
    allow_missing_key: false
  
  endpoints:
    - path: "/api/v1/payments"
      method: "POST"
      required: true
      ttl_override: 2592000 # 30 days for financial audit
    - path: "/api/v1/users"
      method: "POST"
      required: true
    - path: "/api/v1/metrics"
      method: "POST"
      required: false
      ttl_override: 3600

Quick Start Guide

1. Initialize Storage: Deploy a Redis instance dedicated to idempotency or configure a separate DB namespace. Apply the configuration template.
2. Add Middleware: Import the IdempotencyMiddleware class in your application entry point. Initialize with Redis connection and config.
3. Attach to Routes: Apply the middleware to sensitive routes. Ensure express.json() or body parser runs before the middleware if payload hashing is enabled.
4. Client Integration: Update client code to generate a UUID v4 for each logical operation and attach it as the X-Idempotency-Key header. Enable retry logic in HTTP clients to reuse the header on failure.
5. Verify: Send a request with a key. Capture the response. Resend the request with the same key. Verify the response is identical and the backend logs show a cache hit. Check that no duplicate records exist in the database.