Message queue comparison (RabbitMQ vs Kafka)

th require idempotency because network partitions can cause redelivery.

RabbitMQ Consumer

export async function consumeTasks(channel: amqp.Channel) {
  channel.prefetch(10); // Limits unacknowledged messages
  channel.consume('tasks', async (msg) => {
    if (!msg) return;
    try {
      const payload = JSON.parse(msg.content.toString());
      await processTask(payload);
      channel.ack(msg);
    } catch (error) {
      channel.nack(msg, false, false); // Drop or route to dead-letter
    }
  });
}

Kafka Consumer

export async function consumeEvents(topic: string) {
  await consumer.connect();
  await consumer.subscribe({ topic, fromBeginning: false });

  await consumer.run({
    eachMessage: async ({ topic, partition, message }) => {
      const payload = JSON.parse(message.value?.toString() || '{}');
      await processEvent(payload);
      // Kafka auto-commits by default; manual commit recommended for exactly-once semantics
    },
  });
}

Architecture Decisions and Rationale

1. Routing vs Partitioning: RabbitMQ uses exchanges and bindings to route messages to queues. This is ideal for domain-driven routing (e.g., order.created → inventory-service, billing-service). Kafka uses partitions to distribute load and enable parallel consumption. This is ideal for throughput scaling and stateful stream processing.
2. Acknowledgment Model: RabbitMQ’s ack/nack model supports immediate retry or dead-letter routing. Kafka’s offset model supports replay and consumer group rebalancing. Choose ack for task queues; choose offset for event logs.
3. State Management: RabbitMQ stores messages in memory with optional disk persistence. Kafka writes to sequential segment files with zero-copy OS page cache utilization. Kafka’s design minimizes I/O overhead at scale; RabbitMQ’s design minimizes latency for small batches.
4. Schema Evolution: Both systems transmit raw bytes. Implement a schema registry (Avro/Protobuf) to prevent consumer breakage. Kafka integrates natively with Confluent Schema Registry; RabbitMQ requires external validation or middleware.

Pitfall Guide

1. Treating Kafka as a Traditional Queue: Kafka does not guarantee FIFO delivery across partitions. If you consume without consumer groups or ignore partition keys, you will experience out-of-order processing and duplicate handling. Best practice: Use consistent partition keys for related events and design consumers to be idempotent.
2. Overcomplicating RabbitMQ Routing: Binding dozens of queues to a single topic exchange creates management debt and increases routing latency. Best practice: Limit exchange depth to 2-3 hops. Use direct or header exchanges for deterministic routing. Reserve topic exchanges for fan-out patterns.
3. Misconfiguring Acknowledgment Modes: Auto-ack in RabbitMQ deletes messages before processing completes, causing data loss on crashes. Auto-commit in Kafka commits offsets before processing finishes, causing duplicate processing on rebalances. Best practice: Use manual ack/commit and wrap processing in try/catch with explicit failure handling.
4. Ignoring Partition Skew in Kafka: Uneven key distribution causes hot partitions, starving consumers and throttling throughput. Best practice: Profile key distribution early. Use murmur2 hashing defaults. Monitor partition lag via kafka-consumer-groups.sh or Prometheus metrics.
5. Neglecting Schema Evolution: JSON payloads without versioning or schema validation cause silent consumer failures. Best practice: Implement schema versioning (event_v1, event_v2). Use backward-compatible changes. Validate at producer and consumer boundaries.
6. Underestimating Monitoring Requirements: Both systems require visibility into queue depth, consumer lag, disk I/O, and network saturation. Best practice: Export metrics to Prometheus. Alert on consumer lag > threshold, queue depth > memory limit, and broker disk usage > 80%.
7. Mixing Broker Responsibilities in a Single Service: A service that both publishes to Kafka and consumes from RabbitMQ without clear boundaries becomes a coupling bottleneck. Best practice: Separate event producers from task consumers. Use dedicated worker services for each broker domain.

Production Bundle

Action Checklist

• Define data lifecycle: transient task vs persistent event
• Select broker based on latency vs throughput requirements
• Implement idempotency at consumer layer regardless of broker
• Configure manual acknowledgment or offset commit
• Establish schema versioning and validation pipeline
• Deploy monitoring for queue depth, consumer lag, and broker health
• Design dead-letter or error-handling strategy for failed messages
• Test partition skew and routing complexity under load

Decision Matrix

Configuration Template

Docker Compose (RabbitMQ + Kafka)

version: '3.8'
services:
  rabbitmq:
    image: rabbitmq:3.12-management
    ports:
      - "5672:5672"
      - "15672:15672"
    environment:
      RABBITMQ_DEFAULT_USER: admin
      RABBITMQ_DEFAULT_PASS: secret
    volumes:
      - rabbitmq_data:/var/lib/rabbitmq

  zookeeper:
    image: confluentinc/cp-zookeeper:7.5.0
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
    ports:
      - "2181:2181"

  kafka:
    image: confluentinc/cp-kafka:7.5.0
    depends_on:
      - zookeeper
    ports:
      - "9092:9092"
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_LOG_RETENTION_HOURS: 168
    volumes:
      - kafka_data:/var/lib/kafka/data

volumes:
  rabbitmq_data:
  kafka_data:

Kafka Consumer Group Configuration (kafkajs)

export const consumerConfig = {
  groupId: 'production-processor',
  sessionTimeout: 30000,
  rebalanceTimeout: 60000,
  heartbeatInterval: 5000,
  maxBytesPerPartition: 1048576,
  autoCommit: false,
  autoCommitThreshold: 100,
};

RabbitMQ Channel Configuration (amqplib)

export const channelConfig = {
  prefetch: 20,
  durable: true,
  exclusive: false,
  autoDelete: false,
  arguments: {
    'x-dead-letter-exchange': 'dlx',
    'x-message-ttl': 60000,
  },
};

Quick Start Guide

1. Launch Infrastructure: Run docker-compose up -d to start RabbitMQ, Zookeeper, and Kafka locally. Verify endpoints at localhost:5672 (RabbitMQ) and localhost:9092 (Kafka).
2. Initialize Client Connections: Install amqplib and kafkajs. Create connection modules matching the Core Solution examples. Ensure environment variables point to local endpoints.
3. Create Topics/Queues: For RabbitMQ, call channel.assertQueue('tasks', { durable: true }). For Kafka, run kafka-topics.sh --create --topic events --partitions 3 --replication-factor 1 --bootstrap-server localhost:9092.
4. Run Producer/Consumer: Execute the TypeScript producer script to publish 100 messages. Start the consumer script. Verify RabbitMQ queue depth drops to zero and Kafka consumer lag reaches zero.
5. Validate Idempotency: Restart the consumer mid-processing. Confirm no duplicate side effects occur. Check broker metrics for acknowledgment/offset behavior.

This architecture eliminates broker ambiguity by aligning technology choice with data lifecycle requirements. RabbitMQ handles transient work with routing precision. Kafka handles persistent events with throughput and replay capability. Implementing either correctly requires respecting its native semantics, not forcing it to mimic the other.