docker-compose.yml for event-driven observability stack

N Schema or Protobuf. Each event carries:

• event_type: metric, log, trace, error
• trace_id, span_id, parent_span_id for correlation
• timestamp (ISO 8601 or epoch nanoseconds)
• attributes: key-value pairs matching OTel conventions
• payload: type-specific data (value, message, stack trace)

Register schemas in a schema registry (Confluent, Karapace, or local JSON validation) to enforce backward compatibility and prevent silent drift.

Step 2: Instrument Producers with Async Emitters

Applications emit telemetry events asynchronously. The emitter must handle context propagation, schema validation, and backpressure tolerance without blocking the request path.

import { context, propagation } from '@opentelemetry/api';
import { Kafka, Producer, logLevel } from 'kafkajs';
import { validateTelemetryEvent } from './schema-validator';

export class EventDrivenTelemetryEmitter {
  private producer: Producer;
  private topic: string;
  private retryAttempts: number = 3;
  private baseDelayMs: number = 100;

  constructor(brokers: string[], topic: string) {
    const kafka = new Kafka({
      brokers,
      logLevel: logLevel.WARN,
      retry: { retries: 5, initialRetryTime: 100 }
    });
    this.producer = kafka.producer({ idempotent: true });
    this.topic = topic;
  }

  async init() {
    await this.producer.connect();
  }

  async emit(eventType: 'metric' | 'log' | 'trace' | 'error', payload: Record<string, unknown>): Promise<void> {
    const traceCtx = context.active();
    const spanContext = propagation.getSpan(traceCtx)?.spanContext();

    const telemetryEvent = {
      event_type: eventType,
      trace_id: spanContext?.traceId || '00000000000000000000000000000000',
      span_id: spanContext?.spanId || '0000000000000000',
      timestamp: Date.now() * 1_000_000,
      attributes: { service: process.env.SERVICE_NAME || 'unknown' },
      payload
    };

    if (!validateTelemetryEvent(telemetryEvent)) {
      throw new Error('Telemetry event failed schema validation');
    }

    const message = JSON.stringify(telemetryEvent);
    let attempt = 0;

    while (attempt < this.retryAttempts) {
      try {
        await this.producer.send({
          topic: this.topic,
          messages: [{ value: message, headers: { trace_id: telemetryEvent.trace_id } }]
        });
        return;
      } catch (err) {
        attempt++;
        if (attempt === this.retryAttempts) throw err;
        await new Promise(res => setTimeout(res, this.baseDelayMs * Math.pow(2, attempt - 1)));
      }
    }
  }

  async shutdown() {
    await this.producer.disconnect();
  }
}

Step 3: Deploy Stream Router with Partitioning Strategy

Use a streaming platform that supports high-throughput, partitioned logs, and consumer group coordination. Redpanda or Apache Kafka are standard. Partition telemetry events by service_name or trace_id to ensure ordering within service boundaries while enabling parallel processing. Configure retention policies based on SLOs, not storage limits.

Step 4: Implement Stream Processors

Stream processors consume raw telemetry events, apply transformations, and route to backends. Key responsibilities:

• Context enrichment (lookup service metadata, environment tags)
• Adaptive sampling (drop low-value traces, retain error paths)
• Metric aggregation (counter/summary rollups)
• Dead-letter queue routing for malformed events

Processors should run as stateful services with exactly-once semantics where available, or idempotent writes to sinks.

Step 5: Sink to Observability Backends

Route processed events to destination systems. Metrics sink to Prometheus-compatible receivers or TimescaleDB. Logs sink to Loki or Elasticsearch. Traces sink to Jaeger, Tempo, or New Relic. Use the OpenTelemetry Collector in stream mode rather than HTTP push mode to consume from Kafka topics directly.

Architecture Decisions & Rationale

• Decoupled Emission: Applications never block on telemetry delivery. The emitter runs fire-and-forget with idempotent retries, preserving request latency.
• Schema-First Validation: Prevents downstream parser crashes and enables safe schema evolution. Breaking changes are caught at the edge.
• Partitioned Correlation: Routing by trace_id or service_name maintains causal ordering without global locks.
• Stream-Native Processing: Replaces heavy sidecar collectors with lightweight consumers that scale horizontally with partition count.
• Backpressure Tolerance: Streaming platforms expose consumer lag metrics. Auto-scaling processors based on lag prevents pipeline saturation.

Pitfall Guide

1. Schema Drift Without Registry

Telemetry events evolve. Without schema validation, downstream parsers fail silently or corrupt dashboards. Implement a schema registry or inline JSON Schema validation. Enforce versioning in event payloads (schema_version: "1.2").

2. Unbounded Event Volume

Emitting every log line or metric tick floods the pipeline. Apply adaptive sampling: retain 100% of error traces, sample healthy traces at 10–20%, and aggregate counters server-side. Define cardinality budgets per service.

3. Lost Trace Context in Async Boundaries

Event-driven systems break span continuity when context isn't propagated. Always inject trace_id and span_id into message headers. Consumers must extract context before creating child spans. Use OpenTelemetry context propagation interceptors in message clients.

4. Treating Raw Logs as Telemetry Events

Unstructured logs cannot be correlated or queried efficiently. Convert logs to structured telemetry events with OTel attributes. Include severity, module, user_id, and request_id. Avoid free-text dumps in event payloads.

5. Missing Dead-Letter Handling

Malformed or oversized events block consumers. Configure a dead-letter topic with alerting. Implement replay jobs for transient failures. Monitor DLQ depth as a pipeline health signal.

6. Assuming Message Broker Equals Observability Platform

Kafka/Redpanda routes events; it does not visualize, alert, or correlate. Pair streaming infrastructure with OTel Collector stream receivers, dashboard tools, and alerting engines. Observability is a pipeline, not a transport.

7. Ignoring Pipeline Self-Monitoring

The observability pipeline itself requires observability. Emit metrics for consumer lag, schema validation failures, emitter retry rates, and sink delivery latency. Alert on pipeline degradation before application telemetry drops.

Production Best Practices:

• Enforce OpenTelemetry semantic conventions across all event types.
• Implement circuit breakers in emitters to prevent cascade failures during broker outages.
• Use idempotent producers and exactly-once consumer configurations where supported.
• Version telemetry schemas and maintain backward compatibility windows.
• Monitor cardinality budgets and enforce limits at the emitter level.
• Document event contracts and schema evolution policies for all teams.

Production Bundle

Action Checklist

• Schema Registry Setup: Deploy schema validation layer with versioned telemetry contracts and backward compatibility rules.
• Context Propagation: Inject trace/span IDs into all async message headers and extract in consumers before span creation.
• Adaptive Sampling: Configure trace sampling rates by severity, retain 100% of error paths, aggregate counters server-side.
• Dead-Letter Queue: Route malformed events to DLQ with alerting, implement replay jobs, monitor DLQ depth.
• Pipeline Self-Monitoring: Emit consumer lag, emitter retry rates, schema validation failures, and sink delivery latency as metrics.
• Cost Controls: Enforce cardinality budgets, set retention policies aligned with SLOs, compress event payloads where applicable.
• Fallback Mechanisms: Implement local buffer fallback when brokers are unreachable, with periodic flush and deduplication.

Decision Matrix

Scenario	Recommended Approach	Why	Cost Impact
High-throughput microservices (>10k RPS)	Kafka/Redpanda + OTel Stream Collector	Native partitioning, consumer groups, and idempotent delivery handle scale without collector bottlenecks	-35% infrastructure cost vs HTTP collectors
IoT/Edge telemetry with intermittent connectivity	MQTT + Local Buffer + Async Kafka Sink	Edge devices cannot sustain HTTP keep-alives; local buffering prevents data loss	+12% storage cost for edge buffers, -28% network egress
Compliance-heavy audit logging	Pulsar + Immutable Sinks + Schema Registry	Pulsar's tiered storage and strict retention policies meet audit requirements without custom tooling	+18% licensing/storage, -40% audit response time
Small team with limited ops capacity	Sync OTel Push + Managed Backend	Event-driven pipeline requires stream ops expertise; sync push reduces initial complexity	+25% collector cost, -60% initial setup time

Configuration Template

# docker-compose.yml for event-driven observability stack
version: '3.8'
services:
  redpanda:
    image: docker.redpanda.com/redpandadata/redpanda:v23.3.1
    command:
      - redpanda start
      - --smp 1
      - --memory 1G
      - --overprovisioned
      - --node-id 0
      - --kafka-addr PLAINTEXT://0.0.0.0:9092
      - --advertise-kafka-addr PLAINTEXT://redpanda:9092
    ports:
      - "9092:9092"
    environment:
      - REDPANDA_ENABLE_TRANSACTIONS=true
      - REDPANDA_AUTO_CREATE_TOPICS_ENABLE=false

  otel-collector:
    image: otel/opentelemetry-collector-contrib:0.92.0
    command: ["--config=/etc/otel-collector-config.yaml"]
    volumes:
      - ./otel-config.yaml:/etc/otel-collector-config.yaml
    depends_on:
      - redpanda
    ports:
      - "4317:4317"
      - "8888:8888"

  stream-processor:
    build: ./processor
    environment:
      - KAFKA_BROKERS=redpanda:9092
      - TELEMETRY_TOPIC=telemetry.raw
      - ENRICHED_TOPIC=telemetry.enriched
      - DLQ_TOPIC=telemetry.dlq
    depends_on:
      - redpanda

# otel-config.yaml
receivers:
  kafka:
    brokers: ["redpanda:9092"]
    protocol_version: "3.0.0"
    topic: "telemetry.enriched"
    encoding: "otlp_json"
    initial_offset: "latest"

exporters:
  prometheus:
    endpoint: "0.0.0.0:8889"
  otlp/jaeger:
    endpoint: "jaeger:14250"
    tls:
      insecure: true

service:
  pipelines:
    metrics:
      receivers: [kafka]
      exporters: [prometheus]
    traces:
      receivers: [kafka]
      exporters: [otlp/jaeger]

Quick Start Guide

1. Initialize project and install dependencies:

   npm init -y && npm install kafkajs @opentelemetry/api zod
   

2. Create schema validator using Zod for telemetry events:

   import { z } from 'zod';
   export const telemetrySchema = z.object({
     event_type: z.enum(['metric', 'log', 'trace', 'error']),
     trace_id: z.string().length(32),
     span_id: z.string().length(16),
     timestamp: z.number().int().positive(),
     attributes: z.record(z.unknown()),
     payload: z.unknown()
   });
   export const validateTelemetryEvent = (data: unknown) => telemetrySchema.safeParse(data).success;
   

3. Start Redpanda locally and create topics:

   docker run -d -p 9092:9092 docker.redpanda.com/redpandadata/redpanda:v23.3.1
   rpk topic create telemetry.raw telemetry.enriched telemetry.dlq
   

4. Emit first event and verify ingestion:

   const emitter = new EventDrivenTelemetryEmitter(['localhost:9092'], 'telemetry.raw');
   await emitter.init();
   await emitter.emit('log', { message: 'Service started', module: 'main' });
   console.log('Telemetry event emitted successfully');
   

   Monitor telemetry.raw via rpk topic consume telemetry.raw to confirm structured delivery.