Network monitoring guide

. Deploy Kernel-Level Agents: Use eBPF-based agents (e.g., Cilium, Pixie, or custom bpftrace scripts) deployed as node agents or sidecars. These agents hook into kernel functions to capture TCP, UDP, DNS, and TLS events without modifying application code.

*Architecture Decision:* Prefer node-level agents over sidecar proxies to reduce resource contention and avoid the complexity of traffic interception. Node agents capture all traffic originating from or destined to the host, providing a complete view.

2. Instrument Network Metrics: Collect standard RED (Rate, Errors, Duration) and USE (Utilization, Saturation, Errors) metrics adapted for network protocols. * TCP: tcp_retransmits, tcp_active_opens, tcp_established, tcp_drops. * DNS: dns_query_duration, dns_failures, dns_cache_hits. * TLS: tls_handshake_duration, tls_errors, cipher_suite_distribution. * Application: http_request_duration correlated with network_latency.

3. Integrate with Distributed Tracing: Inject network spans into distributed traces. When a service calls a downstream dependency, the trace should include a span for the network hop, capturing DNS lookup time, TCP connection time, TLS handshake time, and data transfer time. This allows visualization of the network cost within the request lifecycle.

4. Configure Prometheus Exporters: Expose metrics in Prometheus format. Ensure labels include service name, namespace, source/destination pods, and protocol.

Code Example: TypeScript Network Health Exporter

While eBPF handles kernel collection, application teams must expose network health context. The following TypeScript example demonstrates a custom Prometheus metric exporter that tracks application-level network failures, enabling correlation with kernel metrics.

import { PrometheusClient, Registry, Counter, Histogram } from 'prom-client';
import { createServer } from 'http';

const register = new Registry();

// Track network errors by type and destination
const networkErrorsTotal = new Counter({
  name: 'app_network_errors_total',
  help: 'Total number of network errors by type and destination',
  labelNames: ['error_type', 'destination_service', 'status_code'],
  registers: [register],
});

// Track latency distribution for network calls
const networkLatencyHistogram = new Histogram({
  name: 'app_network_request_duration_seconds',
  help: 'Duration of network requests in seconds',
  labelNames: ['destination_service', 'method'],
  buckets: [0.01, 0.05, 0.1, 0.5, 1, 2, 5],
  registers: [register],
});

// Wrapper function to instrument fetch calls
export async function instrumentedFetch(url: string, options: RequestInit = {}) {
  const destination = new URL(url).hostname;
  const start = process.hrtime.bigint();
  let timer = networkLatencyHistogram.startTimer({ destination_service: destination, method: options.method || 'GET' });

  try {
    const response = await fetch(url, options);
    timer();

    if (!response.ok) {
      networkErrorsTotal.inc({
        error_type: 'http_error',
        destination_service: destination,
        status_code: response.status.toString(),
      });
    }
    return response;
  } catch (error: any) {
    timer();
    const errorType = error.code === 'ENOTFOUND' ? 'dns_failure' : 
                      error.code === 'ECONNREFUSED' ? 'connection_refused' : 
                      'network_error';

    networkErrorsTotal.inc({
      error_type: errorType,
      destination_service: destination,
      status_code: '0',
    });
    throw error;
  }
}

// Expose metrics endpoint
createServer(async (req, res) => {
  if (req.url === '/metrics') {
    res.setHeader('Content-Type', register.contentType);
    res.end(await register.metrics());
  } else {
    res.writeHead(404);
    res.end();
  }
}).listen(9090, () => console.log('Metrics server running on :9090'));

Architecture Decisions and Rationale

• eBPF over Packet Capture: Packet capture (tcpdump/Wireshark) is essential for deep forensics but unsustainable for continuous monitoring due to storage and CPU costs. eBPF filters events in the kernel and exports only aggregated metrics or sampled events, reducing overhead to <1%.
• Push vs. Pull: Use Prometheus pull model for metrics to ensure reliability and avoid backpressure. For logs and traces, use push-based pipelines (OpenTelemetry Collector) to handle high cardinality data.
• TLS Visibility: In mTLS environments, network agents cannot inspect encrypted payloads. Rely on metrics exposed by the service mesh (e.g., Envoy/Istio) or application-level instrumentation for payload analysis. Focus network monitoring on transport-layer health.

Pitfall Guide

1. Polling Intervals Mask Transient Failures

• Mistake: Configuring SNMP or exporters with 5-minute scrape intervals.
• Impact: Micro-bursts and transient DNS failures disappear between scrapes.
• Best Practice: Use streaming telemetry or sub-second eBPF metrics. Configure Prometheus scrape intervals to 15s or lower for critical network metrics.

2. Ignoring DNS Resolution Latency

• Mistake: Focusing solely on TCP/HTTP metrics and missing DNS.
• Impact: DNS cache misses or upstream resolver failures cause latency spikes indistinguishable from application slowness.
• Best Practice: Instrument DNS query duration and cache hit rates. Alert on dns_p99 > 50ms in latency-sensitive services.

3. Alerting on Bandwidth Utilization Only

• Mistake: Triggering alerts when interface utilization exceeds 80%.
• Impact: High bandwidth does not imply congestion or errors. You may miss packet loss at 10% utilization due to bufferbloat or misconfigured QoS.
• Best Practice: Alert on packet loss, retransmissions, and queue drops. Use bandwidth trends for capacity planning, not incident detection.

4. Missing Retransmission Context

• Mistake: Counting tcp_retransmits without distinguishing between loss-induced and fast-retransmits.
• Impact: False positives during normal network jitter.
• Best Practice: Correlate retransmissions with RTT variance. Use tcp_retransmits combined with tcp_rto to identify genuine congestion versus application-induced timeouts.

5. TLS Blind Spots in mTLS Meshes

• Mistake: Expecting network agents to decrypt mTLS traffic.
• Impact: Inability to detect application-level errors masked by successful TLS handshakes.
• Best Practice: Accept that payload inspection is impossible in mTLS. Rely on service mesh telemetry for HTTP-level errors and network agents for transport health. Implement mutual health checks.

6. IP Fragmentation Overhead

• Mistake: Ignoring MTU mismatches in overlay networks.
• Impact: Performance degradation due to fragmentation and reassembly overhead, especially in VXLAN/GRE tunnels.
• Best Practice: Monitor ip_fragment_fails and ip_frag_creates. Enforce consistent MTU settings across the overlay and underlay. Test Path MTU Discovery.

7. Control Plane vs. Data Plane Load

• Mistake: Monitoring only data plane traffic.
• Impact: Service mesh control plane overload can halt service discovery and policy updates, causing silent failures.
• Best Practice: Instrument control plane components (e.g., etcd latency, Istiod CPU, Envoy config update latency). Ensure control plane SLOs are monitored independently.

Production Bundle

Action Checklist

• Audit Topology: Map all network dependencies, including external APIs, DNS resolvers, and internal service meshes.
• Deploy eBPF Agents: Install kernel agents on all nodes with minimal resource quotas; verify no kernel panics.
• Define Network SLOs: Establish thresholds for latency, error rate, and retransmission per service tier.
• Integrate Traces: Ensure network spans are injected into distributed traces for all critical paths.
• Configure Alerting: Set up multi-window, multi-burn rate alerts for network metrics to avoid flapping.
• Test Failover: Simulate network partitions, DNS failures, and latency injection to validate monitoring coverage.
• Document Runbooks: Create troubleshooting guides linking network metrics to remediation steps (e.g., flush DNS, scale LB).

Decision Matrix

Scenario	Recommended Approach	Why	Cost Impact
Kubernetes Microservices	Cilium/Hubble + Prometheus	Native eBPF integration, service-aware, low overhead.	Low (Open Source)
Hybrid Cloud (Legacy + Cloud)	Datadog Network Performance Monitoring	Unified view across hybrid environments, easy onboarding.	High (SaaS License)
High-Frequency Trading	Custom eBPF + DPDK	Microsecond precision, bypass kernel stack for critical paths.	High (Engineering Effort)
Legacy Datacenter	NetFlow/sFlow + SolarWinds	Non-intrusive, works with existing hardware, standard protocols.	Medium (Tooling)
Serverless / FaaS	Application Instrumentation + Cloud Provider Metrics	No kernel access; rely on runtime metrics and provider logs.	Low (Built-in)

Configuration Template

Prometheus Alert Rule for Network Health:

groups:
  - name: network_health
    rules:
      - alert: HighTcpRetransmissionRate
        expr: rate(tcp_retransmits_total[5m]) / rate(tcp_out_segs_total[5m]) > 0.05
        for: 2m
        labels:
          severity: warning
        annotations:
          summary: "High TCP retransmission rate detected"
          description: "Service {{ $labels.service }} has a retransmission rate > 5% for 2 minutes. Investigate network congestion or MTU issues."

      - alert: DnsResolutionLatencySpike
        expr: histogram_quantile(0.95, rate(dns_query_duration_seconds_bucket[5m])) > 0.1
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "DNS resolution latency spike"
          description: "P95 DNS latency for {{ $labels.service }} is above 100ms. Check upstream resolvers and cache health."

      - alert: TlsHandshakeFailureRate
        expr: rate(tls_handshake_failures_total[5m]) > 10
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "High TLS handshake failure rate"
          description: "{{ $labels.service }} is experiencing frequent TLS failures. Verify certificates and cipher compatibility."

Cilium Hubble UI Configuration (Kubernetes):

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: allow-hubble-metrics
  namespace: kube-system
spec:
  endpointSelector:
    matchLabels:
      k8s:io.cilium.hubble/relay: "true"
  ingress:
    - fromEntities:
        - world
      toPorts:
        - ports:
            - port: "4244"
              protocol: TCP

Quick Start Guide

1. Install Agent: Deploy the eBPF agent using the vendor's Helm chart or manifest.

   helm install cilium cilium/cilium --namespace kube-system --set hubble.relay.enabled=true --set hubble.ui.enabled=true
   

2. Verify Telemetry: Check that metrics are flowing to Prometheus.

   kubectl port-forward -n kube-system svc/hubble-relay 4245:80
   curl http://localhost:4245/flow | head -n 5
   

3. Dashboard Setup: Import the pre-built network observability dashboard into Grafana. Verify service-level metrics are populating.
4. Baseline Alert: Configure a test alert for tcp_retransmits. Trigger a synthetic load test to validate alert firing and notification routing.
5. Trace Integration: Add the instrumentation library to your application code. Verify network spans appear in your trace backend (e.g., Jaeger, Tempo).

This guide provides the architectural foundation, implementation details, and operational practices required to transition network monitoring from a reactive infrastructure task to a proactive, observability-driven capability. By leveraging kernel-level telemetry and correlating network state with application performance, teams can achieve rapid detection and resolution of network-related incidents.