# main.py
from opentelemetry import trace, metrics
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.resources import Resource
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.exporter.otlp.proto.grpc.metric_exporter import OTLPMetricExporter
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
from opentelemetry.instrumentation.flask import FlaskInstrumentor
from flask import Flask, request
import os

# Resource attributes for service identification
resource = Resource.create({
    "service.name": os.getenv("SERVICE_NAME", "checkout-service"),
    "service.version": os.getenv("SERVICE_VERSION", "1.0.0"),
    "deployment.environment": os.getenv("ENV", "production")
})

# Tracing setup
trace.set_tracer_provider(TracerProvider(resource=resource))
span_exporter = OTLPSpanExporter(endpoint="otel-collector:4317", insecure=True)
trace.get_tracer_provider().add_span_processor(BatchSpanProcessor(span_exporter))

# Metrics setup
metric_reader = PeriodicExportingMetricReader(
    OTLPMetricExporter(endpoint="otel-collector:4317", insecure=True)
)
metrics.set_meter_provider(MeterProvider(resource=resource, metric_readers=[metric_reader]))

app = Flask(__name__)
FlaskInstrumentor().instrument_app(app)
tracer = trace.get_tracer(__name__)

@app.route("/checkout", methods=["POST"])
def checkout():
    with tracer.start_as_current_span("process_payment") as span:
        span.set_attribute("payment.method", request.json.get("method"))
        # Business logic here
        return {"status": "success"}, 200

if __name__ == "__main__":
    app.run(host="0.0.0.0", port=8080)

node --require @opentelemetry/auto-instrumentations-node/register server.js

# Producer
import opentelemetry.propagate as propagate
from opentelemetry.trace.propagation.tracecontext import TraceContextTextMapPropagator

carrier = {}
propagate.inject(carrier)
# Send carrier as message headers to Kafka/RabbitMQ

# Consumer
from opentelemetry.context import extract
ctx = extract(carrier)
with tracer.start_as_current_span("consume_order", context=ctx):
    # Process message
    pass

import logging
from opentelemetry import trace

class OTelLogFormatter(logging.Formatter):
    def format(self, record):
        span = trace.get_c

urrent_span() if span and span.is_recording(): ctx = span.get_span_context() record.trace_id = format(ctx.trace_id, '032x') record.span_id = format(ctx.span_id, '016x') else: record.trace_id = "00000000000000000000000000000000" record.span_id = "0000000000000000" return super().format(record)

handler = logging.StreamHandler() handler.setFormatter(OTelLogFormatter("%(asctime)s [%(trace_id)s] %(message)s")) logging.getLogger().addHandler(handler)

### 4. Kubernetes Deployment (Collector)

The OpenTelemetry Collector acts as a vendor-neutral proxy. Deploy it as a DaemonSet or Deployment:

```yaml
# otel-collector-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: otel-collector-conf
data:
  config.yaml: |
    receivers:
      otlp:
        protocols:
          grpc:
          http:
    processors:
      batch:
      memory_limiter:
        check_interval: 1s
        limit_mib: 1024
    exporters:
      otlp/jaeger:
        endpoint: "jaeger-collector:14250"
        tls:
          insecure: true
      prometheus:
        endpoint: "0.0.0.0:8889"
      loki:
        endpoint: "http://loki:3100/loki/api/v1/push"
    service:
      pipelines:
        traces:
          receivers: [otlp]
          processors: [batch, memory_limiter]
          exporters: [otlp/jaeger]
        metrics:
          receivers: [otlp]
          processors: [batch]
          exporters: [prometheus]
        logs:
          receivers: [otlp]
          processors: [batch]
          exporters: [loki]

5. Query & Visualization

Grafana unifies Prometheus (metrics), Loki (logs), and Tempo (traces). Example PromQL for error rate by service:

sum(rate(http_server_duration_seconds_count{status=~"5.."}[5m])) by (service_name)
/
sum(rate(http_server_duration_seconds_count[5m])) by (service_name)

In Grafana, enable "Explore" mode to pivot from a latency spike in metrics → filter logs by trace_id → open the exact failing trace in Tempo. This closed-loop workflow eliminates tool-switching and accelerates incident response.

Pitfall Guide

1. High-Cardinality Explosion

   • Description: Tagging metrics or traces with user IDs, email addresses, or request UUIDs causes index blowouts and storage costs to scale linearly with traffic.
   • Root Cause: Misunderstanding cardinality limits in Prometheus/Loki/Tempo.
   • Mitigation: Enforce low-cardinality attributes (service, endpoint, region, status). Use OTel processors to drop or hash high-cardinality keys. Implement cardinality guardrails in the Collector.

2. Siloed Observability Stack

   • Description: Running separate tools for logs, metrics, and traces without correlation capabilities forces engineers to manually cross-reference data.
   • Root Cause: Legacy tool adoption or vendor-driven procurement without architectural alignment.
   • Mitigation: Standardize on OpenTelemetry, deploy a unified query layer (Grafana), and mandate trace_id/span_id in all log lines. Use Tempo/Loki/Prometheus or commercial equivalents with native cross-pillar linking.

3. Naive Sampling Strategies

   • Description: Recording 100% of traces in production consumes excessive storage and network bandwidth, while random sampling drops critical failure paths.
   • Root Cause: Lack of sampling policy design during early adoption.
   • Mitigation: Implement head sampling for latency/error thresholds, tail sampling for complex decision-making, and adaptive sampling that scales with traffic. OTel's probabilistic_sampler and rate_limiting processors handle this efficiently.

4. Treating Observability as Monitoring

   • Description: Building static dashboards and alerting on fixed thresholds without enabling exploratory analysis or SLO tracking.
   • Root Cause: Cultural inertia; teams expect observability to replace PagerDuty without changing workflows.
   • Mitigation: Define SLOs first. Use error budgets to govern release velocity. Train engineers on query-driven debugging. Shift from "alert fatigue" to "signal-driven investigation."

5. Ignoring Security & Compliance

   • Description: Traces and logs inadvertently capture PII, secrets, or health data, violating GDPR/HIPAA and creating liability.
   • Root Cause: Over-instrumentation without data classification or redaction policies.
   • Mitigation: Implement OTel processors to mask/strip sensitive fields. Classify telemetry data streams. Apply RBAC in query interfaces. Audit sampling rules for compliance alignment.

6. Lack of Service Dependency Mapping

   • Description: Teams cannot visualize how services interact, leading to blind spots during cascading failures or capacity planning.
   • Root Cause: Traces are collected but not processed into topology graphs.
   • Mitigation: Enable OTel's servicegraph processor. Export to Jaeger/Grafana Service Map. Correlate with Kubernetes networking policies. Use dependency maps to identify single points of failure and optimize sync/async boundaries.

Production Bundle

Checklist

Pre-Deployment

• Define SLOs/SLIs for each critical service (latency, error rate, availability)
• Standardize on OpenTelemetry semantic conventions across teams
• Implement low-cardinality tagging policy; document allowed attributes
• Configure sampling strategy (head/tail/adaptive) aligned with storage budget
• Redact PII/secrets in instrumentation or Collector processors
• Set up RBAC and audit logging for observability platforms

Runtime

• Verify trace context propagation across sync/async boundaries
• Monitor Collector health (memory, CPU, export queue depth)
• Validate log-trace-metric correlation in query interface
• Run chaos experiments to verify alerting on emergent failures
• Review cardinality metrics weekly; prune unused labels

Post-Deployment & Governance

• Integrate observability data into incident postmortems
• Automate dashboard generation from SLO definitions
• Conduct quarterly tooling review (cost, performance, vendor lock-in)
• Train on-call engineers on query-driven debugging workflows
• Document observability runbooks and escalation paths

Decision Matrix

Criteria	OpenTelemetry + Open Source (Prometheus/Loki/Tempo/Grafana)	Commercial APM (Datadog/New Relic/Dynatrace)	Cloud-Native (AWS CloudWatch/Azure Monitor/GCP Operations)
Cost	Low (infrastructure only); scales with retention	High (per-GB/per-host pricing); predictable but expensive	Medium-High (pay-as-you-go); can spike with high volume
Vendor Lock-in	Minimal (OTel standard, pluggable backends)	High (proprietary SDKs, custom query languages)	Medium (cloud-specific APIs, but OTel exporters available)
Scalability	High (horizontal scaling, Kubernetes-native)	High (managed, but cost-prohibitive at scale)	High (fully managed, but limited cross-cloud)
Ease of Use	Medium (requires SRE expertise, self-managed)	High (out-of-box dashboards, AI insights)	Medium-High (managed, but fragmented across services)
Compliance	Full control (on-prem, air-gapped, custom retention)	Vendor-dependent (SOC2, HIPAA, GDPR certified)	Cloud-region dependent; strong enterprise certifications
Best For	Engineering teams with SRE maturity, cost-sensitive, multi-cloud	Fast-moving teams, limited ops resources, budget-flexible	Single-cloud deployments, regulated industries, managed-service preference

Config Template

# otel-collector-prod.yaml
receivers:
  otlp:
    protocols:
      grpc:
        max_recv_msg_size_mib: 32
      http:
        cors:
          allowed_origins: ["*"]
  prometheus:
    config:
      scrape_configs:
        - job_name: 'kubernetes-pods'
          kubernetes_sd_configs:
            - role: pod
          relabel_configs:
            - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
              action: keep
              regex: true

processors:
  batch:
    timeout: 5s
    send_batch_max_size: 2000
  memory_limiter:
    check_interval: 1s
    limit_mib: 2048
    spike_limit_mib: 512
  attributes:
    actions:
      - key: "http.user_agent"
        action: delete
      - key: "user.email"
        action: update
        value: "***REDACTED***"
  tail_sampling:
    policies:
      - name: error-policy
        type: status_code
        status_code: { status_codes: ["ERROR"] }
      - name: latency-policy
        type: latency
        latency: { threshold_ms: 500 }
      - name: probabilistic
        type: probabilistic
        probabilistic: { sampling_percentage: 10 }

exporters:
  otlp/jaeger:
    endpoint: "jaeger-collector.monitoring.svc:14250"
    tls:
      insecure: true
  prometheusremotewrite:
    endpoint: "http://victoriametrics:8428/api/v1/write"
    tls:
      insecure: true
  loki:
    endpoint: "http://loki.monitoring.svc:3100/loki/api/v1/push"
    batch:
      max_size: 2000
      timeout: 5s

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch, tail_sampling]
      exporters: [otlp/jaeger]
    metrics:
      receivers: [otlp, prometheus]
      processors: [memory_limiter, batch]
      exporters: [prometheusremotewrite]
    logs:
      receivers: [otlp]
      processors: [memory_limiter, batch, attributes]
      exporters: [loki]

Quick Start

1. Clone the baseline stack:

   git clone https://github.com/open-telemetry/opentelemetry-demo.git
   cd opentelemetry-demo
   

2. Launch observability infrastructure:

   docker compose -f docker-compose.yml -f docker-compose.override.yml up -d jaeger prometheus grafana loki
   

3. Instrument a sample service (Python):

   pip install opentelemetry-sdk opentelemetry-exporter-otlp opentelemetry-instrumentation-flask
   export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
   export OTEL_SERVICE_NAME=quickstart-service
   python main.py
   

4. Verify data flow:
   • Send traffic: curl -X POST http://localhost:8080/checkout -H "Content-Type: application/json" -d '{"method":"card"}'
   • Open Grafana (http://localhost:3000) → Explore → Query metrics, logs, and traces using service_name="quickstart-service"
   • Confirm trace context appears in log lines and metric labels
5. Productionize:
   • Replace Docker with Kubernetes manifests
   • Apply the otel-collector-prod.yaml config
   • Enforce cardinality policies and sampling
   • Integrate with CI/CD for automated SLO validation

Observability is not a destination; it's a continuous feedback loop between system behavior and engineering decisions. By standardizing on OpenTelemetry, enforcing data hygiene, and aligning telemetry with SLOs, teams transform microservices complexity from a liability into a competitive advantage. The tools are mature. The patterns are proven. The only remaining variable is organizational commitment to instrument, observe, and iterate.