API request deduplication

te(JSON.stringify(req.body)) .digest('hex') .slice(0, 16);

const dedupKey = `${keyPrefix}:${req.method}:${req.path}:${clientKey || bodyHash}`;

// 2. Check Cache
const cachedResult = await redis.get(dedupKey);
if (cachedResult) {
  try {
    const parsed = JSON.parse(cachedResult);
    return res.status(parsed.status).json(parsed.body);
  } catch {
    // Corrupted cache entry; proceed to processing
  }
}

// 3. Wrap Response to Capture Result
const originalJson = res.json.bind(res);
const originalSend = res.send.bind(res);

const captureResponse = (statusCode: number, body: any) => {
  const responsePayload = { status: statusCode, body };
  const serialized = JSON.stringify(responsePayload);

  if (serialized.length <= maxResponseSizeBytes) {
    // Store result with TTL
    // Use SET with NX to avoid overwriting if race condition occurs
    redis.set(dedupKey, serialized, 'EX', ttlSeconds, 'NX');
  }
};

res.json = (body: any) => {
  captureResponse(res.statusCode, body);
  return originalJson(body);
};

res.send = (body: any) => {
  captureResponse(res.statusCode, body);
  return originalSend(body);
};

// 4. Handle Errors: Clear cache on failure to allow retry
const originalEnd = res.end.bind(res);
res.end = function (...args: any[]) {
  if (res.statusCode >= 500) {
    redis.del(dedupKey).catch(() => {});
  }
  return originalEnd(...args);
};

next();

}; };

#### Architecture Decisions and Rationale

*   **Response Caching vs. Rejection:** Rejecting duplicates with a `409 Conflict` forces the client to handle errors. Caching the response allows the client to receive the success result transparently, which is critical for mobile apps where the initial success response may have been lost in transit.
*   **Hash Slicing:** The body hash is sliced to reduce key size and storage overhead. SHA-256 provides sufficient collision resistance for API payloads; slicing to 16 characters balances uniqueness with storage efficiency.
*   **TTL Strategy:** The TTL must exceed the client's maximum retry duration. If the client retries after the TTL expires, the request is processed again, which is acceptable behavior for expired idempotency windows.
*   **Size Limits:** Caching responses without limits can exhaust Redis memory. Enforcing `maxResponseSizeBytes` ensures that only typical API responses are cached; large payloads should be fetched via subsequent GET requests.

### Pitfall Guide

1.  **Storing Responses in Primary Database:** Using a relational table for deduplication storage introduces write amplification and index bloat. This degrades performance for high-write workloads and complicates schema migrations.
    *   *Best Practice:* Use a dedicated cache layer with TTLs. Archive metadata to a data warehouse if audit trails are required.
2.  **Ignoring Partial Failures:** If a request processes partially (e.g., payment captured but notification fails) and the client retries, naive deduplication may block the retry.
    *   *Best Practice:* Implement idempotency keys that support "in-flight" states or use compensating transactions. Ensure the deduplication key covers the entire transaction scope.
3.  **Weak Hashing Algorithms:** Using MD5 or CRC32 for body hashing increases collision probability, leading to false deduplication of distinct requests.
    *   *Best Practice:* Always use SHA-256 or SHA-3 for content hashing.
4.  **TTL Mismatch:** Setting a TTL shorter than the client's retry policy causes duplicates to be processed after the cache expires.
    *   *Best Practice:* Align TTL with the maximum retry window. Document the idempotency window in API specs.
5.  **Caching PII in Cache:** Storing request/response bodies in Redis may violate data retention policies if PII is cached without encryption or scrubbing.
    *   *Best Practice:* Implement a response scrubber that removes sensitive fields before caching. Use Redis ACLs and encryption at rest.
6.  **Cross-Service Deduplication:** In microservices, a duplicate request may hit different services. Service-level deduplication does not prevent duplicate downstream calls.
    *   *Best Practice:* Propagate the idempotency key via headers (e.g., `X-Request-ID`) and implement deduplication at the orchestration layer or use distributed tracing context.
7.  **Mutable Payloads:** If the request body contains timestamps or nonces that change per retry, hashing the body will generate different keys, defeating deduplication.
    *   *Best Practice:* Require clients to use stable idempotency keys for mutable payloads. Do not rely solely on body hashing for such requests.

### Production Bundle

#### Action Checklist

- [ ] **Generate Client Keys:** Ensure all clients generate and transmit `X-Idempotency-Key` headers for mutating operations.
- [ ] **Deploy Deduplication Store:** Provision a Redis cluster with sufficient memory and configure persistence policies appropriate for ephemeral data.
- [ ] **Implement Middleware:** Deploy the deduplication middleware to the API gateway or service layer, configuring TTL based on retry policies.
- [ ] **Configure Response Limits:** Set `maxResponseSizeBytes` to prevent cache exhaustion from large payloads.
- [ ] **Add Observability:** Instrument metrics for `dedup.hit_count`, `dedup.miss_count`, and `dedup.cache_size` to monitor effectiveness.
- [ ] **Audit PII:** Review cached responses for sensitive data and implement scrubbing if necessary.
- [ ] **Test Retry Scenarios:** Validate deduplication behavior under network partition simulation using chaos engineering tools.

#### Decision Matrix

| Scenario | Recommended Approach | Why | Cost Impact |
| :--- | :--- | :--- | :--- |
| **Financial Transactions** | Distributed Cache + Response Cache | Guarantees no double charges; returns cached result on retry; high integrity. | Medium (Redis nodes) |
| **High-Throughput Logging** | Client Debounce + Server Rate Limit | Deduplication overhead is unjustified; focus on volume reduction at edge. | Low |
| **Public API with Mobile Clients** | Server-Side Idempotency Key | Mobile networks cause frequent retries; response caching improves UX significantly. | Low-Medium |
| **Legacy Monolith** | DB Unique Constraint + Retry Logic | Minimal architectural change; constraints prevent data corruption; acceptable latency. | Low (DB load) |
| **Event Processing Pipeline** | Distributed Deduplication + State Store | Ensures exactly-once semantics; handles out-of-order delivery and consumer rebalancing. | High (State storage) |

#### Configuration Template

Use this configuration interface to standardize deduplication settings across services.

```typescript
// dedup.config.ts
export const dedupConfig = {
  redis: {
    host: process.env.REDIS_HOST || 'localhost',
    port: parseInt(process.env.REDIS_PORT || '6379'),
    password: process.env.REDIS_PASSWORD,
    keyPrefix: 'dedup:v1',
  },
  ttlSeconds: parseInt(process.env.IDEMPOTENCY_TTL || '86400'), // 24 hours default
  maxResponseSizeBytes: 1024 * 1024, // 1MB limit
  hashAlgorithm: 'sha256',
  enabled: process.env.DEDUP_ENABLED === 'true',
  // Paths to exclude (e.g., health checks, webhooks)
  excludePaths: ['/health', '/webhooks/*'],
};

Quick Start Guide

1. Install Dependencies:

   npm install express ioredis
   

2. Add Middleware: Import createDedupMiddleware and apply it to your Express app before route definitions.

   import { createDedupMiddleware } from './dedup-middleware';
   import { dedupConfig } from './dedup.config';
   
   app.use(createDedupMiddleware({
     redis: new Redis(dedupConfig.redis),
     ttlSeconds: dedupConfig.ttlSeconds,
     keyPrefix: dedupConfig.redis.keyPrefix,
     maxResponseSizeBytes: dedupConfig.maxResponseSizeBytes,
   }));
   

3. Configure Client Headers: Update client SDKs to generate UUIDs and attach X-Idempotency-Key headers to all POST/PUT/PATCH requests.
4. Verify Behavior: Send a request, capture the response, and resend the identical request within the TTL. Confirm the second request returns the cached response with negligible latency and no backend processing logs.
5. Monitor: Check Redis keyspace hits/misses and application logs to ensure deduplication is active and effective.