consumer-scaling-config.yaml

g, and lag-driven autoscaling. The implementation below uses a TypeScript consumer pattern that enforces these principles. While written for Kafka-style brokers, the mechanics apply to RabbitMQ, NATS JetStream, and cloud-native queues.

Step 1: Align Consumers with Partition Topology

Each partition can be consumed by only one consumer in a group. Scaling beyond the partition count yields zero throughput gain and increases coordination overhead. Map your consumer pool to max(consumer_count, partition_count).

Step 2: Enable Cooperative Rebalancing

Eager rebalancing revokes all assignments and pauses consumption during group changes. Cooperative rebalancing allows consumers to keep unrevoked partitions while only contested partitions are reassigned. This eliminates stop-the-world pauses.

Step 3: Tune Prefetch and Enforce Backpressure

Prefetch determines how many messages a broker delivers before waiting for acknowledgments. Unbounded prefetch causes memory exhaustion and hides processing bottlenecks. Set prefetch to target_concurrent_processing * average_message_size. Implement explicit backpressure by pausing partition consumption when in-flight count exceeds a threshold.

Step 4: Implement Lag-Based Autoscaling

Consumer lag is the only reliable scaling trigger. Monitor lag = end_offset - committed_offset. Scale consumers when lag exceeds a threshold for a sustained window. Scale down when lag drops below a safety margin.

TypeScript Implementation

import { Kafka, logLevel, Consumer, EachMessagePayload } from 'kafkajs';

const kafka = new Kafka({
  clientId: 'scale-aware-consumer',
  brokers: [process.env.BROKER_URL || 'localhost:9092'],
  logLevel: logLevel.WARN,
});

const consumer = kafka.consumer({
  groupId: 'processing-group',
  // KIP-429 cooperative rebalancing
  rebalanceProtocol: 'CooperativeSticky',
  // Prefetch limits in-flight messages per partition
  maxBytesPerPartition: 1_048_576, // 1MB
  maxWaitTimeInMs: 500,
});

const MAX_IN_FLIGHT = 50;
let inFlight = 0;

async function processMessage(payload: EachMessagePayload): Promise<void> {
  // Simulate I/O bound work
  await new Promise(res => setTimeout(res, 120));
}

async function run() {
  await consumer.connect();
  await consumer.subscribe({ topic: 'events', fromBeginning: false });

  // Run with explicit session timeout and cooperative rebalance
  await consumer.run({
    eachMessage: async (payload) => {
      // Backpressure: pause partition if in-flight threshold exceeded
      if (inFlight >= MAX_IN_FLIGHT) {
        consumer.pause([{ topic: payload.topic, partitions: [payload.partition] }]);
        // Wait until drain callback fires
        await new Promise<void>(resolve => {
          const check = setInterval(() => {
            if (inFlight < MAX_IN_FLIGHT) {
              consumer.resume([{ topic: payload.topic, partitions: [payload.partition] }]);
              clearInterval(check);
              resolve();
            }
          }, 50);
        });
      }

      inFlight++;
      try {
        await processMessage(payload);
        // Explicit commit after successful processing
        await consumer.commitOffsets([
          { topic: payload.topic, partition: payload.partition, offset: (Number(payload.message.offset) + 1).toString() }
        ]);
      } catch (err) {
        // Route to DLQ or nack; do not commit
        console.error(`Processing failed: ${err.message}`);
      } finally {
        inFlight--;
      }
    },
  });
}

// Graceful shutdown to prevent offset loss
process.on('SIGTERM', async () => {
  await consumer.disconnect();
  process.exit(0);
});

run().catch(console.error);

Architecture Decisions & Rationale

• CooperativeSticky rebalancing: Reduces partition movement during deployments. Only contested partitions are reassigned, preserving throughput stability.
• Explicit offset commits: Automatic commits risk duplicate processing on failure. Manual commits after successful work guarantee at-least-once semantics with controllable retry boundaries.
• In-flight throttling: Prevents memory exhaustion and hides processing latency from the broker. The pause/resume cycle implements backpressure without blocking the event loop.
• Partition-aware scaling: HPA policies must reference partition count. Scaling consumers beyond partitions wastes resources and increases rebalance frequency.

Pitfall Guide

1. Scaling consumers beyond partition count Partitions are the hard concurrency limit. Adding consumers beyond N partitions leaves N-1 consumers idle and increases group coordination overhead. Always align consumer count with partition count or use consumer pooling per partition.

2. Unbounded prefetch buffers Default prefetch values often deliver thousands of messages per connection. When processing latency spikes, memory usage grows linearly. Set maxBytesPerPartition and maxWaitTimeInMs to match your processing SLA. Use explicit backpressure instead of relying on broker-side limits.

3. Blocking the event loop in async consumers CPU-bound work inside eachMessage blocks the I/O thread, preventing ack commits and triggering session timeouts. Offload heavy computation to worker threads or external services. Keep consumer handlers strictly I/O-bound and non-blocking.

4. Ignoring rebalance storms during deployments Rolling deployments trigger eager rebalancing in older clients. Partitions are revoked, processing halts, and lag spikes. Migrate to cooperative rebalancing protocols and stagger deployments to allow incremental assignment.

5. Silent ack failures and poison pills Failing to handle processing errors causes repeated redelivery of malformed messages. Implement dead-letter queues (DLQ), exponential backoff, and circuit breakers. Never commit offsets on failure.

6. Scaling brokers without tuning I/O schedulers Adding broker nodes does not fix consumer-side bottlenecks. Broker CPU, disk I/O, and network throughput are constrained by OS page cache, journaling, and TCP backlog. Tune log.flush.interval.messages, socket.send.buffer.bytes, and use noop or deadline I/O schedulers for consistent latency.

7. Scaling without lag observability CPU and memory metrics are lagging indicators. Consumer lag is the leading indicator of scaling needs. Instrument lag = end_offset - committed_offset per partition. Trigger autoscaling only when lag exceeds threshold for a sustained window (e.g., 30s).

Best Practices from Production

• Use partition-aware Horizontal Pod Autoscalers (HPA) that read broker lag metrics via Prometheus adapters.
• Set prefetch to target_concurrency * average_processing_time.
• Implement cooperative rebalancing in all consumer groups.
• Route failed messages to DLQs with metadata retention for replay.
• Monitor rebalance frequency; spikes indicate unstable consumer sessions or network partitions.

Production Bundle

Action Checklist

• Map partition count to consumer capacity; never scale beyond partition limit
• Enable cooperative rebalancing protocol in consumer configuration
• Set explicit prefetch limits matching processing latency and memory budget
• Implement backpressure via partition pause/resume when in-flight threshold is reached
• Route processing failures to dead-letter queues with retry metadata
• Instrument consumer lag per partition; trigger autoscaling on sustained lag breaches
• Stagger deployments to minimize rebalance storms; verify assignment stability post-rollout

Decision Matrix

Scenario	Recommended Approach	Why	Cost Impact
Bursty traffic with predictable peaks	Partition-aligned consumers + lag-based HPA	Scales only when lag breaches threshold; avoids idle resource waste	Moderate (autoscaling overhead)
Steady high-throughput ingestion	Fixed consumer pool matching partition count + tuned prefetch	Eliminates rebalance overhead; maximizes steady-state throughput	Low (predictable baseline)
Latency-sensitive processing	Cooperative rebalance + strict backpressure + dedicated DLQ	Prevents consumer starvation; guarantees p99 stability under load	High (requires monitoring infra)
Cost-constrained edge deployments	Single consumer with multiplexed partition polling + aggressive prefetch tuning	Minimizes node count; relies on broker-side batching	Low (sacrifices parallelism)

Configuration Template

# consumer-scaling-config.yaml
kafka:
  client:
    id: "scale-aware-consumer"
    session_timeout_ms: 30000
    heartbeat_interval_ms: 10000
  consumer:
    group: "processing-group"
    rebalance_protocol: "CooperativeSticky"
    max_bytes_per_partition: 1048576
    max_wait_time_in_ms: 500
    fetch_min_bytes: 1024
    fetch_max_bytes: 10485760
  backpressure:
    max_in_flight: 50
    pause_threshold_ms: 2000
  dlq:
    topic: "events-dlq"
    max_retries: 3
    retry_backoff_ms: 1000
  autoscaling:
    metric: "consumer_lag"
    target_lag: 10000
    scale_up_cooldown: 30
    scale_down_cooldown: 300
    min_replicas: 2
    max_replicas: 16

Quick Start Guide

1. Provision partitions matching expected concurrency: Create topics with N partitions where N equals your target consumer count. Use kafka-topics --create --partitions N --topic events.
2. Deploy the cooperative consumer: Apply the TypeScript runtime or equivalent client with CooperativeSticky rebalancing and explicit prefetch limits. Verify session stability under load.
3. Instrument lag metrics: Expose consumer_lag per partition to Prometheus. Configure the HPA to scale when lag exceeds target_lag for scale_up_cooldown seconds.
4. Validate backpressure and DLQ routing: Inject malformed messages and verify they route to the dead-letter queue without blocking the main consumer group. Monitor in-flight count and pause/resume cycles.
5. Stagger deployments: Roll out new consumer versions using rolling updates with maxSurge: 1. Monitor rebalance frequency and partition assignment stability. Adjust prefetch and cooldown values based on observed lag recovery time.