Difficulty

Intermediate

Read Time

11 min

Monitoring Kubernetes Clusters with OpenTelemetry Collector: The Agent + Gateway Pattern Explained

By Codcompass Team·2026-07-04·11 min read

Architecting Resilient Telemetry Pipelines: Decoupling Collection from Delivery in Kubernetes

Current Situation Analysis

Modern observability stacks are frequently deployed with a single, monolithic OpenTelemetry Collector instance per node. This approach appears efficient during initial setup: one binary, one configuration, direct forwarding to Tempo, Loki, or Prometheus. The architecture collapses under production scale because it conflates two fundamentally incompatible operational profiles. Local signal collection requires minimal resource footprint and high availability. Remote signal delivery requires stateful buffering, cryptographic credential management, and aggressive batching.

The primary failure mode manifests as connection exhaustion. When every application pod or sidecar exporter opens a direct gRPC channel to a central ingest endpoint, the backend receives thousands of concurrent connections. Observability backends like Grafana Tempo or Loki are optimized for aggregated streams, not unbounded fan-out. Each gRPC connection maintains keepalive state, negotiates TLS handshakes, and consumes file descriptors. At 20+ nodes with autoscaling workloads, connection counts easily exceed 500. The ingest layer does not degrade linearly; it hits concurrency thresholds and begins returning RESOURCE_EXHAUSTED or UNAVAILABLE status codes. Most SDK exporters log the error once, discard the payload, and continue. Traces and logs vanish silently.

Resource contention compounds the problem. A collector attempting to scrape kubeletstats, tail container logs, and maintain retry queues for remote backends will experience memory pressure during scrape spikes. The batch processor accumulates payloads while the prometheusreceiver or filelog receiver consumes CPU and RAM. Without strict isolation, the collector pod exceeds its memory limit, triggers an OOMKill, and restarts. During the 30–90 second restart window, node-level telemetry goes dark. When the pod returns, the retry queue is empty. Data loss becomes a feature of the deployment model rather than an exception.

Teams overlook this architectural flaw because official documentation often presents single-pipeline examples for simplicity. Tutorials rarely demonstrate how backpressure propagates through the collector's internal queues. The assumption that "more memory solves buffering problems" ignores the reality of node-level resource constraints. Kubernetes schedules pods based on requests/limits, not actual runtime behavior. A collector configured with a 512Mi memory limit on a busy node will compete with application workloads, trigger eviction policies, and destabilize the very telemetry it was meant to collect.

WOW Moment: Key Findings

Decoupling collection from delivery transforms an unbounded fan-out problem into a controlled, predictable stream. The following comparison illustrates the operational divergence between monolithic and split architectures:

Approach	Upstream Connections	Memory Profile	Failure Mode	Security Blast Radius	Tail Sampling Capability
Monolithic Collector	Unbounded (scales with pod count)	High (requires persistent queues)	OOMKill loops, silent data drops	Full cluster API + backend credentials	Not possible (spans fragmented across nodes)
Agent + Gateway	Bounded (3–5 per gateway replica)	Low (agent: 200–300Mi, gateway: scalable)	Graceful degradation, controlled drops	Strictly isolated (agent: read-only, gateway: network-only)	Native (all spans converge at gateway)

This finding matters because it shifts telemetry infrastructure from a reactive, resource-draining model to a proactive, capacity-planning model. The gateway acts as a pressure valve: it absorbs connection spikes, applies sampling policies, and maintains a single authenticated channel to external backends. The agent remains lightweight, node-local, and disposable. When a backend experiences latency, the gateway's retry queue absorbs the delay while agents continue accepting local OTLP traffic. The system degrades predictably instead of collapsing under connection limits.

Core Solution

The architecture separates telemetry handling into two distinct Kubernetes workloads: a node-scoped DaemonSet for collection and a cluster-scoped Deployment for delivery. Each component operates under explicit resource boundaries and security contexts.

Step 1: Deploy the Node Agent (DaemonSet)

The agent runs exactly once per node. It listens on loopback interfaces to receive OTLP traffic from local SDKs and sidecars. It scrapes node-level metrics and tails container logs. Its configuration enforces strict memory limits and avoids persistent storage.

Architecture Decision: Use memory_limiter as the first processor in every pipeline. The collector evaluates memory pressure before payloads enter the batch queue. Placing it later allows memory to exhaust before the li

miter can act, causing OOMKills.

Architecture Decision: Disable persistent queues on agents. Node storage is ephemeral and shared with application workloads. If the gateway becomes unreachable, the agent should drop data rather than accumulate gigabytes of retry state. This is a deliberate trade-off: prioritize node stability over temporary buffering.

Step 2: Deploy the Delivery Gateway (Deployment)

The gateway runs as a highly available Deployment (minimum 2 replicas) behind a ClusterIP service. It receives aggregated OTLP streams from all agents, applies tail-based sampling, batches payloads, and manages TLS credentials for external backends.

Architecture Decision: Centralize backend credentials in the gateway. Agents never store API keys, certificates, or remote write URLs. This eliminates credential sprawl and ensures that a compromised agent cannot exfiltrate data to unauthorized endpoints.

Architecture Decision: Implement tail-based sampling at the gateway. Tail sampling requires visibility into complete trace IDs before deciding whether to retain or discard spans. Since agents only see local spans, sampling decisions must occur after fan-in.

Step 3: Network & Service Discovery

Agents forward to the gateway using internal Kubernetes DNS. The gateway service name remains static regardless of pod restarts or scaling events. Agents do not perform service discovery; they rely on cluster DNS resolution.

Step 4: RBAC Isolation

The agent ServiceAccount requires read-only access to Kubernetes API resources for metadata enrichment (pods, nodes, endpoints, services). The gateway ServiceAccount requires zero Kubernetes API permissions. It only needs network egress to external observability backends. This separation ensures that a misconfigured gateway cannot enumerate cluster topology, and a compromised agent cannot reach remote credentials.

Implementation Code (New Examples)

Agent Configuration (telemetry-node-agent.yaml)

apiVersion: v1
kind: ConfigMap
metadata:
  name: telemetry-node-agent
  namespace: observability
data:
  collector.yaml: |
    extensions:
      health_check:
        endpoint: "0.0.0.0:13133"

    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: "0.0.0.0:4317"
          http:
            endpoint: "0.0.0.0:4318"
      kubeletstats:
        collection_interval: 30s
        auth_type: serviceAccount
        endpoint: "https://${env:K8S_NODE_IP}:10250"
        insecure_skip_verify: true
        metric_groups: [node, pod, container]
      hostmetrics:
        collection_interval: 30s
        scrapers:
          cpu: {}
          memory: {}
          disk: {}
          filesystem:
            exclude_mount_points:
              mount_points: [/dev, /proc, /sys, /run/containerd]
              match_type: strict
          network: {}
          load: {}
      filelog:
        include: [/var/log/pods/*/*/*.log]
        include_file_path: true
        operators:
          - type: router
            id: detect_format
            routes:
              - output: json_parser
                expr: 'body matches "^\\{"'
              - output: cri_parser
                expr: 'body matches "^[^ Z]+ "'
          - type: json_parser
            id: json_parser
            output: enrich
          - type: regex_parser
            id: cri_parser
            regex: '^(?P<time>[^ Z]+) (?P<stream>stdout|stderr) (?P<flags>[^ ]*) ?(?P<log>.*)$'
            output: enrich
          - type: move
            id: enrich
            from: attributes["log"]
            to: body

    processors:
      memory_limiter:
        check_interval: 1s
        limit_mib: 240
        spike_limit_mib: 50
      batch:
        send_batch_size: 1024
        timeout: 5s
        send_batch_max_size: 2048
      resourcedetection:
        detectors: [env, k8snode]
        timeout: 5s
      k8sattributes:
        auth_type: serviceAccount
        passthrough: false
        filter:
          node_from_env_var: K8S_NODE_NAME
        extract:
          metadata:
            - k8s.pod.name
            - k8s.pod.uid
            - k8s.deployment.name
            - k8s.namespace.name
            - k8s.node.name
            - k8s.container.name

    exporters:
      otlp/gateway:
        endpoint: "telemetry-aggregator.observability.svc.cluster.local:4317"
        tls:
          insecure: true

    service:
      extensions: [health_check]
      pipelines:
        traces:
          receivers: [otlp]
          processors: [memory_limiter, k8sattributes, batch]
          exporters: [otlp/gateway]
        metrics:
          receivers: [otlp, kubeletstats, hostmetrics]
          processors: [memory_limiter, resourcedetection, batch]
          exporters: [otlp/gateway]
        logs:
          receivers: [otlp, filelog]
          processors: [memory_limiter, k8sattributes, batch]
          exporters: [otlp/gateway]

Gateway Configuration (telemetry-aggregator.yaml)

apiVersion: v1
kind: ConfigMap
metadata:
  name: telemetry-aggregator
  namespace: observability
data:
  collector.yaml: |
    extensions:
      health_check:
        endpoint: "0.0.0.0:13133"
      basicauth/backend:
        client_auth:
          username: "${env:BACKEND_USER}"
          password: "${env:BACKEND_PASS}"

    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: "0.0.0.0:4317"
          http:
            endpoint: "0.0.0.0:4318"

    processors:
      memory_limiter:
        check_interval: 1s
        limit_mib: 512
        spike_limit_mib: 100
      batch:
        send_batch_size: 4096
        timeout: 10s
        send_batch_max_size: 8192
      tail_sampling:
        policies:
          - name: error-filter
            type: status_code
            status_code: { status_codes: [ERROR] }
          - name: latency-filter
            type: latency
            latency: { threshold_ms: 500 }
          - name: probabilistic
            type: probabilistic
            probabilistic: { sampling_percentage: 25 }

    exporters:
      otlp/tempo:
        endpoint: "${env:TEMPO_ENDPOINT}"
        tls:
          cert_file: /etc/secrets/tls/client.crt
          key_file: /etc/secrets/tls/client.key
        auth:
          authenticator: basicauth/backend
      prometheusremotewrite:
        endpoint: "${env:PROMETHEUS_RW_URL}"
        tls:
          insecure: true
      loki:
        endpoint: "${env:LOKI_PUSH_URL}"
        tls:
          insecure: true
        auth:
          authenticator: basicauth/backend

    service:
      extensions: [health_check, basicauth/backend]
      pipelines:
        traces:
          receivers: [otlp]
          processors: [memory_limiter, tail_sampling, batch]
          exporters: [otlp/tempo]
        metrics:
          receivers: [otlp]
          processors: [memory_limiter, batch]
          exporters: [prometheusremotewrite]
        logs:
          receivers: [otlp]
          processors: [memory_limiter, batch]
          exporters: [loki]

Pitfall Guide

1. Misordered Memory Limiter

Explanation: Placing memory_limiter after batch or k8sattributes allows payloads to accumulate in memory before pressure is evaluated. The collector exceeds its limit, triggers OOMKill, and restarts. Fix: Always declare memory_limiter as the first processor in every pipeline. It must evaluate memory before any downstream processor allocates buffers.

2. Persistent Queues on Node Agents

Explanation: Enabling persistent_queue on DaemonSet agents consumes node disk space, competes with application workloads, and survives pod restarts. During backend outages, agents accumulate gigabytes of retry data, eventually filling node storage and triggering Kubernetes eviction. Fix: Use memory-only queues on agents. Accept controlled data loss during partitions. Configure retry_on_failure with short backoff windows. Reserve persistent queues exclusively for the gateway.

3. Shared ServiceAccounts Across Components

Explanation: Using a single ServiceAccount for both agents and gateways grants unnecessary Kubernetes API access to the gateway and backend credential access to the agent. A misconfiguration or container escape exposes cluster topology or remote write tokens. Fix: Create isolated ServiceAccounts. Agent role: get/list/watch on pods, nodes, endpoints. Gateway role: zero Kubernetes permissions. Mount backend secrets only into gateway pods.

4. Ignoring Collector Self-Metrics

Explanation: Teams monitor application telemetry but forget to scrape the collector's own /metrics endpoint. When the collector experiences backpressure, queue saturation, or exporter timeouts, there is no visibility until data loss occurs. Fix: Expose the collector's internal metrics via Prometheus receiver or Prometheus Operator ServiceMonitor. Alert on otelcol_processor_refused_spans, otelcol_exporter_send_failed_spans, and otelcol_memory_limiter_capacity_mib.

5. Hardcoding Backend URLs in Agent Configs

Explanation: Embedding Tempo, Loki, or Prometheus URLs in agent ConfigMaps forces rolling updates across every node when endpoints change. It also exposes backend infrastructure to every node's filesystem. Fix: Centralize all remote endpoints in the gateway configuration. Use environment variables or Kubernetes Secrets for dynamic resolution. Agents only need the internal gateway DNS name.

6. Skipping Tail-Based Sampling

Explanation: Head-based sampling (dropping spans at the SDK level) discards traces before context is available. Critical error traces or high-latency requests may be randomly dropped, reducing observability value. Fix: Implement tail sampling at the gateway. The gateway receives complete trace IDs from all nodes, enabling policy-based retention (error filtering, latency thresholds, probabilistic sampling). This preserves high-value telemetry while reducing backend costs.

7. Over-Provisioning Agent CPU Requests

Explanation: Agents are I/O bound, not CPU bound. Setting high CPU requests wastes node capacity and triggers scheduler fragmentation. Agents spend most of their time waiting on network sockets and file descriptors. Fix: Set conservative CPU requests (e.g., 50m–100m) and allow burstable performance. Use memory as the primary scaling metric. Monitor otelcol_processor_batch_batch_send_size to validate throughput without CPU saturation.

Production Bundle

Action Checklist

Deploy agent DaemonSet with strict memory limits (200–300Mi) and no persistent queues
Deploy gateway Deployment (2+ replicas) with centralized backend credentials and TLS termination
Verify memory_limiter is the first processor in every pipeline configuration
Isolate RBAC: agent gets read-only Kubernetes API access, gateway gets zero API access
Configure tail-based sampling policies at the gateway for error/latency filtering
Expose collector internal metrics and alert on queue saturation/exporter failures
Test partition resilience: simulate gateway downtime and verify agents drop gracefully without OOMKills
Validate backend connection count: ensure gateway maintains 3–5 persistent connections regardless of cluster size

Decision Matrix

Scenario	Recommended Approach	Why	Cost Impact
Cluster < 10 nodes, low telemetry volume	Single Collector Deployment	Simplicity outweighs complexity; connection limits rarely hit	Low operational overhead, minimal infrastructure cost
Cluster 10–50 nodes, mixed workloads	Agent + Gateway Pattern	Prevents connection exhaustion, enables tail sampling, isolates credentials	Moderate infrastructure cost (gateway replicas), high reliability ROI
Cluster > 50 nodes, regulated/compliance	Agent + Gateway + Dedicated Ingest	Gateway handles sampling/batching; dedicated ingest layer meets audit requirements for credential isolation	Higher infrastructure cost, reduced compliance risk, predictable backend scaling
Multi-tenant platform, strict isolation	Per-tenant Gateway + Shared Agents	Agents remain node-local; gateways enforce tenant boundaries and separate backend routing	Increased gateway footprint, simplified tenant billing and data segregation

Configuration Template

Gateway Deployment Manifest (gateway-deployment.yaml)

apiVersion: apps/v1
kind: Deployment
metadata:
  name: telemetry-aggregator
  namespace: observability
spec:
  replicas: 3
  selector:
    matchLabels:
      app: telemetry-aggregator
  template:
    metadata:
      labels:
        app: telemetry-aggregator
    spec:
      serviceAccountName: telemetry-gateway-sa
      containers:
        - name: collector
          image: otel/opentelemetry-collector-contrib:0.96.0
          args: ["--config", "/conf/collector.yaml"]
          ports:
            - containerPort: 4317
              name: grpc
            - containerPort: 4318
              name: http
          env:
            - name: BACKEND_USER
              valueFrom:
                secretKeyRef:
                  name: backend-credentials
                  key: username
            - name: BACKEND_PASS
              valueFrom:
                secretKeyRef:
                  name: backend-credentials
                  key: password
            - name: TEMPO_ENDPOINT
              value: "tempo.observability.svc.cluster.local:4317"
            - name: LOKI_PUSH_URL
              value: "http://loki-distributor.observability.svc.cluster.local:3100/loki/api/v1/push"
            - name: PROMETHEUS_RW_URL
              value: "http://prometheus-server.observability.svc.cluster.local:9090/api/v1/write"
          volumeMounts:
            - name: config
              mountPath: /conf
            - name: tls-secrets
              mountPath: /etc/secrets/tls
              readOnly: true
          resources:
            requests:
              cpu: 200m
              memory: 512Mi
            limits:
              cpu: 1000m
              memory: 1Gi
      volumes:
        - name: config
          configMap:
            name: telemetry-aggregator
        - name: tls-secrets
          secret:
            secretName: backend-tls-cert

Quick Start Guide

Create isolated ServiceAccounts and RBAC roles for the agent (read-only Kubernetes API) and gateway (network-only). Apply them to their respective manifests.
Deploy the gateway Deployment with backend credentials injected via Kubernetes Secrets. Verify the ClusterIP service is reachable within the cluster.
Deploy the agent DaemonSet using the provided ConfigMap template. Ensure memory_limiter is first in all pipelines and persistent queues are disabled.
Configure application SDKs to export OTLP to localhost:4317 (gRPC) or localhost:4318 (HTTP). No backend URLs should exist in application code.
Validate the pipeline by generating test telemetry, checking gateway logs for successful batching, and confirming backend connection counts remain stable at 3–5 per replica.

🎉 Mid-Year Sale — Unlock Full Article

Base plan from just $4.99/mo or $49/yr

7-day free trial · Cancel anytime · 30-day money-back