ement idempotency keys derived from event metadata (event_id or aggregate_id + event_type). Store processed keys in a fast lookup store (Redis, DynamoDB, or local cache with TTL).

func (c *Consumer) HandleEvent(ctx context.Context, msg kafka.Message) error {
    eventID := string(msg.Headers["event_id"])
    
    // Idempotency check
    if c.idempotencyStore.Exists(ctx, eventID) {
        return nil // Safe no-op
    }
    
    // Validate schema before unmarshaling
    if err := c.schemaRegistry.Validate(msg.Topic, msg.Value); err != nil {
        return fmt.Errorf("schema validation failed: %w", err)
    }
    
    var event Event
    if err := json.Unmarshal(msg.Value, &event); err != nil {
        return err
    }
    
    // Business logic
    if err := c.process(ctx, event); err != nil {
        return err
    }
    
    // Mark as processed
    return c.idempotencyStore.Set(ctx, eventID, true, 24*time.Hour)
}

Step 5: Architecture Decisions

• Delivery Semantics: Prefer at-least-once + idempotency over exactly-once. Exactly-once requires complex broker-side state machines and transactional IDs that rarely justify the operational overhead.
• Partitioning: Route events using aggregate_id as the partition key. This guarantees ordering for a single entity while allowing parallel processing across entities.
• Backpressure: Implement consumer-side prefetch limits and exponential backoff. Never allow unbounded goroutine/thread spawning.
• Versioning: Adopt backward-compatible schema evolution. Add fields as optional, never remove or rename. Use a schema_version header for routing legacy consumers.

Pitfall Guide

1. Treating Events as Commands: Events are historical facts. Commands are instructions. Mixing them breaks idempotency and forces consumers to mutate state based on assumptions rather than confirmed state changes.
2. Missing Idempotency Keys: Broker retries, network partitions, and consumer rebalances guarantee duplicate deliveries. Without deterministic idempotency, state corruption is inevitable.
3. Schema Evolution Without Contracts: Dropping fields, changing types, or renaming payloads without registry enforcement causes silent consumer failures. Always version and validate.
4. Synchronous Event Publishing: Blocking business logic on broker acknowledgment defeats the purpose of EDA and couples service latency to infrastructure health. Use the outbox pattern.
5. Neglecting Dead Letter Queues: Poison messages will crash consumers if not isolated. Route malformed or repeatedly failing events to a DLQ with alerting and manual replay capabilities.
6. Over-Reliance on Global Ordering: Kafka/RabbitMQ only guarantee ordering within a partition. Assuming global ordering across topics or partitions introduces artificial bottlenecks and false assumptions.
7. Inadequate Distributed Tracing: Events lack request/response correlation by default. Inject trace_id and span_id into event headers. Without this, async flows are untraceable in production.

Production Bundle

Action Checklist

• Implement transactional outbox table with CDC or poller
• Register all event schemas in a centralized registry with compatibility checks
• Add deterministic idempotency keys to every consumer pipeline
• Configure DLQ routing with alert thresholds and replay tooling
• Inject distributed tracing headers (trace_id, parent_span_id) into all events
• Load-test consumer backpressure with 10x expected throughput spikes
• Deploy schema drift detection dashboard with automatic CI/CD pipeline gates

Decision Matrix

Configuration Template

// producer.go - Production-grade event publisher with outbox integration
package eventbus

import (
	"context"
	"fmt"
	"time"
	
	"github.com/confluentinc/confluent-kafka-go/kafka"
)

type Producer struct {
	broker     *kafka.Producer
	outboxRepo OutboxRepository
	schemaReg  SchemaValidator
}

func (p *Producer) Publish(ctx context.Context, event Event) error {
	// 1. Validate schema
	if err := p.schemaReg.Validate(event.Type, event.Payload); err != nil {
		return fmt.Errorf("invalid event schema: %w", err)
	}
	
	// 2. Write to outbox within business transaction
	if err := p.outboxRepo.Insert(ctx, event); err != nil {
		return fmt.Errorf("outbox insert failed: %w", err)
	}
	
	// Note: Actual broker publish happens via CDC/poller
	// This guarantees database + event consistency
	return nil
}

// consumer.go - Idempotent consumer with DLQ routing
type Consumer struct {
	broker        *kafka.Consumer
	idempotency   IdempotencyStore
	dlqProducer   *kafka.Producer
	schemaReg     SchemaValidator
}

func (c *Consumer) Consume(ctx context.Context) error {
	for {
		msg, err := c.broker.ReadMessage(100 * time.Millisecond)
		if err != nil {
			continue // Timeout is expected
		}
		
		eventID := string(msg.Headers["event_id"])
		if c.idempotency.Exists(ctx, eventID) {
			continue
		}
		
		if err := c.schemaReg.Validate(msg.Topic, msg.Value); err != nil {
			c.routeToDLQ(msg, "schema_validation_failed")
			continue
		}
		
		var event Event
		if err := json.Unmarshal(msg.Value, &event); err != nil {
			c.routeToDLQ(msg, "unmarshal_failed")
			continue
		}
		
		if err := c.process(ctx, event); err != nil {
			c.routeToDLQ(msg, "processing_failed")
			continue
		}
		
		c.idempotency.Set(ctx, eventID, true, 24*time.Hour)
		c.broker.CommitMessage(msg)
	}
}

Quick Start Guide

1. Define Event Contracts: List domain mutations. Draft JSON/Avro schemas. Register in schema registry with BACKWARD_TRANSITIVE compatibility.
2. Deploy Outbox Infrastructure: Create outbox_events table. Configure Debezium or implement a lightweight Go poller that reads PENDING rows and publishes to Kafka/Redpanda.
3. Wire Idempotent Consumers: Add event_id header to all published events. Implement Redis-backed idempotency store with TTL. Route failures to DLQ.
4. Instrument Observability: Inject trace_id into event headers. Export consumer lag, DLQ depth, and schema validation failure rates to Prometheus/Grafana. Set alert thresholds at 5% error rate.
5. Validate Under Load: Run synthetic traffic with 3x expected volume. Verify partition ordering, idempotency enforcement, and DLQ routing. Iterate before production rollout.

EDA succeeds when treated as a contract discipline, not a messaging convenience. Enforce schemas, guarantee idempotency, isolate failures, and trace every mutation. The architecture will scale; the operations will stabilize.