ibuted transaction complexity, cross-shard queries, and rebalancing overhead. Start with read replicas, query optimization, and connection pooling. Shard only when a single table exceeds 2–5TB or write throughput saturates a single primary node. Use logical partitioning (e.g., tenant_id, region) when sharding becomes necessary.

Step 3: Multi-Layer Caching with Stampede Prevention

Caching reduces database load but introduces consistency and concurrency risks. Implement a cache-aside pattern with TTL, cache locking, and request coalescing to prevent thundering herds.

// cache-layer.ts
import Redis from 'ioredis';
import { EventEmitter } from 'events';

const redis = new Redis('redis://cache.internal:6379');
const emitters = new Map<string, EventEmitter>();

export async function getCachedOrFetch<T>(
  key: string,
  ttl: number,
  fetchFn: () => Promise<T>
): Promise<T> {
  const cached = await redis.get(key);
  if (cached) return JSON.parse(cached);

  // Prevent cache stampede: coalesce concurrent fetches
  if (emitters.has(key)) {
    return new Promise<T>((resolve) => {
      emitters.get(key)!.once('result', resolve);
    });
  }

  const emitter = new EventEmitter();
  emitters.set(key, emitter);

  try {
    const data = await fetchFn();
    await redis.set(key, JSON.stringify(data), 'EX', ttl);
    emitter.emit('result', data);
    return data;
  } finally {
    emitters.delete(key);
  }
}

Architecture Decision: Cache hot reads, not writes. Use CDN for static assets, application-level cache for computed results, and Redis for session/data caching. Always set explicit TTLs. Implement cache invalidation via event pub/sub rather than manual deletion to maintain consistency across distributed instances.

Step 4: Observability and Metric-Driven Auto-Scaling

Auto-scaling based on CPU/memory is insufficient for modern applications. Scale based on business and latency metrics: request queue depth, database connection utilization, and p95 latency. Integrate OpenTelemetry for distributed tracing and Prometheus for metric aggregation.

# autoscaling-policy.yaml (Kubernetes HPA example)
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: api-deployment
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api-service
  minReplicas: 3
  maxReplicas: 50
  metrics:
    - type: Pods
      pods:
        metric:
          name: http_request_duration_p95
        target:
          type: AverageValue
          averageValue: "250ms"
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 65

Architecture Decision: Never auto-scale below 3 replicas. Two replicas create split-brain risks during rolling updates. Set cooldown periods (300s minimum) to prevent thrashing. Correlate scaling events with deployment timestamps to distinguish traffic spikes from code regressions.

Pitfall Guide

1. Premature Database Sharding Sharding before reaching connection or storage limits introduces cross-shard joins, distributed transactions, and operational complexity. Wait until a single primary node hits 80% sustained write utilization or table size exceeds 3TB. Use read replicas and query optimization first.

2. Unbounded Connection Pools Default connection pool settings often allow hundreds of connections per instance. At 1M users, 10 instances × 100 connections = 1,000 concurrent DB connections. Most databases throttle or crash beyond 500–800 active connections. Cap pools, monitor active/idle ratios, and implement queueing when pools are exhausted.

3. Synchronous Service Chains Service A calls Service B calls Service C. A 200ms latency in C becomes 600ms in A. At scale, timeout propagation causes cascading failures. Replace synchronous calls with events or async polling. Implement circuit breakers with fallback responses for non-critical paths.

4. Cache Stampedes and Thundering Herds When a popular cache key expires, thousands of requests hit the database simultaneously. Use request coalescing (as shown in the cache-layer example), lock-based fetching, or probabilistic early expiration to stagger refreshes.

5. Missing Idempotency Keys Retries, network partitions, and broker redeliveries cause duplicate processing. Without idempotency keys, users get charged twice, notifications duplicate, and state corrupts. Generate UUIDs at the edge, store processed keys in Redis with TTL, and skip duplicates before business logic execution.

6. Skipping Progressive Load Testing Testing at 10k requests/sec and deploying to production guarantees failure at 1M users. Implement synthetic load testing at 2x expected peak. Test failure modes: kill replicas, simulate broker downtime, inject network latency. Validate degradation paths, not just happy paths.

7. Treating Consistency as Absolute Strong consistency across distributed services kills throughput. Accept eventual consistency for non-critical data (analytics, notifications, feed generation). Use sagas or outbox patterns for transactional consistency where required. Document consistency guarantees per data domain.

Production Bundle

Action Checklist

• Route reads to replicas and writes to primary: Implement connection routing with explicit isWrite flags to prevent replica lag from blocking user actions.
• Enforce connection pool limits: Cap max connections per service, monitor utilization, and implement request queueing when pools are saturated.
• Deploy event-driven write paths: Replace synchronous microservice calls with durable event publishing and consumer groups for async processing.
• Implement multi-layer caching with stampede prevention: Use Redis for hot data, coalesce concurrent fetches, and set explicit TTLs with event-based invalidation.
• Generate idempotency keys at the edge: Attach UUIDs to all user-facing requests and broker messages; deduplicate before business logic execution.
• Configure metric-driven auto-scaling: Scale on p95 latency and queue depth, not just CPU/memory; set minimum replicas to 3 and cooldown to 300s.
• Run progressive load tests: Validate at 2x expected peak, inject failures, and verify graceful degradation before production rollout.

Decision Matrix

Configuration Template

// scaling-config.ts
export const ScalingConfig = {
  database: {
    primary: { host: process.env.DB_PRIMARY_HOST, max: 20, idleTimeout: 30000 },
    replica: { host: process.env.DB_REPLICA_HOST, max: 50, idleTimeout: 30000 },
    connectionTimeout: 2000,
    queryTimeout: 5000,
  },
  cache: {
    redis: { url: process.env.REDIS_URL, maxRetriesPerRequest: 3, enableReadyCheck: false },
    ttl: { hot: 300, warm: 1800, cold: 3600 },
    stampedePrevention: true,
  },
  eventBroker: {
    url: process.env.NATS_URL,
    jetStream: { maxMsgs: 1000000, maxBytes: '10GB', storage: 'file' },
    consumerGroup: 'api-workers',
    ackWait: 30,
    maxDeliver: 5,
  },
  observability: {
    otel: { serviceName: 'api-service', exportInterval: 5000, samplingRate: 0.1 },
    metrics: { p95LatencyThreshold: 250, cpuTarget: 65, cooldownSeconds: 300 },
  },
  rateLimiting: {
    windowMs: 60000,
    maxRequests: 120,
    skipIpHeader: 'x-forwarded-for',
    keyGenerator: (req: any) => req.user?.id || req.ip,
  },
};

Quick Start Guide

1. Deploy the event broker and cache layer: Run docker compose up -d nats redis to spin up NATS JetStream and Redis. Verify connectivity with nats server check jetstream and redis-cli ping.
2. Configure database routing: Point your application to primary and replica hosts. Apply the db-router.ts pattern and enforce isWrite flags on all mutations.
3. Enable observability and auto-scaling: Instrument OpenTelemetry, export metrics to Prometheus, and apply the HPA YAML to your Kubernetes cluster. Validate scaling triggers with a synthetic load test.
4. Run progressive load validation: Use k6 or autocannon to simulate 2x expected peak traffic. Monitor p95 latency, connection pool utilization, and event consumer lag. Adjust pool sizes and TTLs based on observed bottlenecks before production rollout.