# app.py
import os
from fastapi import FastAPI
from opentelemetry import trace, metrics
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.resources import Resource
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.exporter.otlp.proto.grpc.metric_exporter import OTLPMetricExporter
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader

# Resource attributes (essential for service identification)
resource = Resource.create({
    "service.name": os.getenv("OTEL_SERVICE_NAME", "payment-service"),
    "service.version": os.getenv("OTEL_VERSION", "1.0.0"),
    "deployment.environment": os.getenv("OTEL_ENV", "production")
})

# Tracer setup
trace_provider = TracerProvider(resource=resource)
trace_provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))
trace.set_tracer_provider(trace_provider)

# Meter setup
metric_reader = PeriodicExportingMetricReader(OTLPMetricExporter())
metric_provider = MeterProvider(resource=resource, metric_readers=[metric_reader])
metrics.set_meter_provider(metric_provider)

tracer = trace.get_tracer("payment.tracer")
meter = metrics.get_meter("payment.meter")
request_counter = meter.create_counter("http.server.requests", unit="1")

app = FastAPI()

@app.get("/process/{payment_id}")
async def process_payment(payment_id: str):
    with tracer.start_as_current_span("process_payment") as span:
        span.set_attribute("payment.id", payment_id)
        span.set_attribute("payment.type", "credit_card")
        
        # Business logic with child span
        with tracer.start_as_current_span("validate_payment") as child_span:
            child_span.set_attribute("validation.result", "success")
            # Simulate validation
            import time; time.sleep(0.05)
            
        request_counter.add(1, {"method": "GET", "status": "200"})
        return {"status": "processed", "id": payment_id}

telemetry data plane. It receives OTLP, applies processors, and exports to backends. Production deployments should run collectors as sidecars or daemonsets, never as single points of failure.

# otel-collector-config.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: "0.0.0.0:4317"
      http:
        endpoint: "0.0.0.0:4318"

processors:
  batch:
    timeout: 5s
    send_batch_size: 1024
  memory_limiter:
    check_interval: 1s
    limit_mib: 512
    spike_limit_mib: 128
  resource:
    attributes:
      - key: k8s.cluster.name
        value: "prod-us-east-1"
        action: upsert

exporters:
  otlp/traces:
    endpoint: "tempo:4317"
    tls:
      insecure: true
  prometheus:
    endpoint: "0.0.0.0:8889"
    namespace: "otel"
  otlp/logs:
    endpoint: "loki:3100/otlp"
    tls:
      insecure: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch, resource]
      exporters: [otlp/traces]
    metrics:
      receivers: [otlp]
      processors: [batch, resource]
      exporters: [prometheus]
    logs:
      receivers: [otlp]
      processors: [batch, resource]
      exporters: [otlp/logs]

3. Context Propagation & Correlation

Distributed tracing requires propagating context across service boundaries. OTel uses W3C Trace Context headers (traceparent, tracestate). HTTP clients and message brokers must be instrumented to inject/extract context automatically.

# Example: HTTP client propagation
from opentelemetry.propagators.textmap import TextMapPropagator
from opentelemetry.trace.propagation.tracecontext import TraceContextTextMapPropagator
import requests

propagator = TraceContextTextMapPropagator()
headers = {}
propagator.inject(headers)  # Adds traceparent to outgoing request
response = requests.get("http://inventory-service/check", headers=headers)

4. Backend Integration

OTel exports via OTLP to any compliant backend. Common combinations include:

• Traces: Tempo, Jaeger, Honeycomb, Datadog
• Metrics: Prometheus, VictoriaMetrics, New Relic
• Logs: Loki, Elasticsearch, Splunk

Query examples (PromQL for metrics, LogQL for logs, Tempo for traces) become trivial once semantic conventions are enforced. Use service.name, http.status_code, and k8s.pod.name as primary dimensions.

Pitfall Guide

1. Over-Instrumentation Without Sampling

Risk: Capturing every span for high-throughput services overwhelms storage and degrades P99 latency. Mitigation: Implement probabilistic sampling at the SDK level (OTEL_TRACES_SAMPLER=parentbased_traceidratio, OTEL_TRACES_SAMPLER_ARG=0.1). Use head-based sampling for traces and tail-based sampling in the Collector for error-only retention.

2. Ignoring Cardinality Limits

Risk: Unbounded labels (e.g., user IDs, request UUIDs) in metrics cause Prometheus/Tempo to OOM or reject data. Mitigation: Restrict metric labels to low-cardinality attributes (service, method, status, region). Use span attributes for high-cardinality data. Enforce limits via Collector filter processor or backend admission controllers.

3. Mixing Vendor SDKs with OTel

Risk: Running Datadog APM agent, New Relic SDK, and OTel Collector simultaneously creates duplicate spans, conflicts context propagation, and inflates costs. Mitigation: Standardize on OTel as the single instrumentation layer. Use vendor-specific exporters only in the Collector. Remove legacy agents before deployment.

4. Neglecting Log-Trace Correlation

Risk: Logs and traces exist in separate systems without shared identifiers, forcing manual cross-referencing during incidents. Mitigation: Inject trace_id and span_id into log records using OTel log bridge or framework-specific appenders. Configure backends to index correlation fields. Use opentelemetry-instrumentation-logging for automatic enrichment.

5. Misconfigured Collector Topology

Risk: Deploying a single collector instance creates a bottleneck and single point of failure. Running collectors in-process blocks application threads. Mitigation: Use sidecar containers for per-pod isolation or daemonsets for node-level aggregation. Deploy collectors as stateless deployments with horizontal scaling. Always configure memory_limiter and batch processors.

6. Skipping Semantic Conventions

Risk: Custom attribute names (user_id vs enduser.id, http_url vs http.url) break dashboards, prevent cross-service correlation, and require constant query rewrites. Mitigation: Enforce CNCF semantic conventions via SDK configuration, linting tools, and code review checklists. Use opentelemetry-semantic-conventions package. Document exceptions in an internal observability playbook.

Production Bundle

Checklist

• SDK version matches latest stable release (check opentelemetry-python or language-specific repo)
• Environment variables configured via CI/CD pipeline (not hardcoded)
• Resource attributes include service.name, service.version, deployment.environment
• Sampling strategy defined (head-based for traces, rate-limited for metrics)
• Collector deployed as sidecar or daemonset with resource limits
• memory_limiter and batch processors configured
• OTLP endpoints use TLS in production; mTLS for internal mesh
• Cardinality policy enforced (max 10-15 labels per metric)
• Log-trace correlation fields injected and indexed
• Backend retention policies aligned with SLA/SLO requirements
• Alerting rules based on OTel metrics, not raw logs
• Runbook includes OTel-specific failure modes (exporter timeout, collector OOM, context leak)

Decision Matrix

Scenario	Recommended Approach	Rationale
Legacy app, minimal code changes allowed	Auto-instrumentation + OTel Agent	Zero code modifications; framework patching covers 80% of HTTP/DB calls
Critical payment/auth service	Manual instrumentation + custom spans	Precise control over business logic spans, error handling, and attribute enrichment
Kubernetes cluster with 50+ services	OTel Collector Operator + DaemonSet	Centralized management, automatic config reloading, resource governance
Multi-cloud hybrid (on-prem + AWS/GCP)	Collector as gateway + OTLP over mTLS	Unified data plane, secure cross-network export, consistent processing
Budget-constrained startup	Prometheus + Tempo + Loki (open source)	Zero licensing, community support, scales to 10k RPS with proper tuning
Enterprise compliance (SOC2, HIPAA)	Commercial backend (Datadog/Honeycomb) + OTel SDK	Built-in audit trails, data residency controls, vendor support SLAs

Config Template (OTel Collector Production-Ready)

# otel-prod.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: "0.0.0.0:4317"
        max_recv_msg_size_mib: 32
      http:
        endpoint: "0.0.0.0:4318"
        cors:
          allowed_origins:
            - "https://*.yourdomain.com"
          allowed_headers:
            - "Authorization"

processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 768
    spike_limit_mib: 192
  batch:
    timeout: 10s
    send_batch_max_size: 2048
    send_batch_size: 1024
  resource:
    attributes:
      - key: k8s.cluster.name
        value: "${K8S_CLUSTER_NAME}"
        action: upsert
      - key: deployment.environment
        value: "${OTEL_ENV}"
        action: upsert
  filter:
    metrics:
      include:
        match_type: regexp
        metric_names:
          - "^http\\.server\\..*"
          - "^rpc\\.client\\..*"
          - "^process\\.runtime\\..*"

exporters:
  otlp/traces:
    endpoint: "${TRACE_BACKEND}:4317"
    tls:
      insecure: false
      ca_file: "/etc/ssl/certs/ca-bundle.crt"
    retry_on_failure:
      enabled: true
      initial_interval: 5s
      max_interval: 30s
      max_elapsed_time: 300s
  prometheus:
    endpoint: "0.0.0.0:8889"
    namespace: "prod"
    send_timestamps: true
    metric_expiration: 180m

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch, resource, filter]
      exporters: [otlp/traces]
    metrics:
      receivers: [otlp]
      processors: [batch, resource]
      exporters: [prometheus]
  telemetry:
    logs:
      level: "info"
      development: false
    metrics:
      address: "0.0.0.0:8888"

Quick Start

1. Initialize SDK in your service:

   pip install opentelemetry-api opentelemetry-sdk opentelemetry-exporter-otlp-proto-grpc
   export OTEL_SERVICE_NAME="demo-app"
   export OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4317"
   

2. Run Collector locally:

   docker run -d --name otel-collector \
     -p 4317:4317 -p 4318:4318 -p 8889:8889 \
     -v $(pwd)/otel-collector-config.yaml:/etc/otel-collector-config.yaml \
     otel/opentelemetry-collector-contrib:latest \
     --config=/etc/otel-collector-config.yaml
   

3. Instrument & run application:

   # Ensure OTel providers are initialized before app startup
   # Run FastAPI/Uvicorn normally; OTel will auto-export
   uvicorn app:app --host 0.0.0.0 --port 8000
   

4. Verify data flow:

   # Check metrics endpoint
   curl http://localhost:8889/metrics | grep http_server_requests
   
   # Query traces (if Tempo is connected)
   curl http://localhost:3200/api/search?service=demo-app
   

5. Add to CI/CD:

   • Inject OTEL_* variables via secrets manager
   • Validate collector health probes (/health)
   • Run integration tests with OTEL_TRACES_SAMPLER=always_on
   • Monitor collector memory/CPU via Prometheus metrics

OpenTelemetry is not a monitoring tool; it is an instrumentation standard. Treat it as critical infrastructure. Enforce conventions, govern cardinality, decouple export from collection, and align telemetry with SLOs. When implemented correctly, OTel transforms observability from a cost center into a reliability engine.