Infrastructure Monitoring: Architecting Resilient Systems for Modern Scale

// instrumentation.ts
import { NodeSDK } from '@opentelemetry/sdk-node';
import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node';
import { PrometheusExporter } from '@opentelemetry/exporter-prometheus';
import { Resource } from '@opentelemetry/resources';
import { SEMRESATTRS_SERVICE_NAME, SEMRESATTRS_SERVICE_VERSION } from '@opentelemetry/semantic-conventions';

// Initialize Prometheus Exporter for metrics
const promExporter = new PrometheusExporter({
  port: 9464,
  endpoint: '/metrics',
  preventServerStart: false,
});

// Initialize OTel SDK
const sdk = new NodeSDK({
  resource: new Resource({
    [SEMRESATTRS_SERVICE_NAME]: 'payment-service',
    [SEMRESATTRS_SERVICE_VERSION]: '1.2.0',
  }),
  instrumentations: [getNodeAutoInstrumentations()],
  metricReader: promExporter,
});

sdk.start(

);

// Custom Metric: Payment Processing Latency import { MeterProvider } from '@opentelemetry/sdk-metrics';

const meterProvider = new MeterProvider(); const meter = meterProvider.getMeter('payment-meter'); const paymentLatency = meter.createHistogram('payment_processing_duration', { description: 'Latency of payment processing in milliseconds', unit: 'ms', });

export function recordPaymentLatency(duration: number, currency: string) { // Currency is low cardinality; safe for metrics paymentLatency.record(duration, { currency }); }

// Graceful shutdown process.on('SIGTERM', () => { sdk.shutdown().then(() => console.log('SDK shut down successfully')); });

#### 4. Configure the Collector Pipeline
The Collector configuration defines how data flows. Use processors to handle cardinality and batching.

```yaml
# otel-collector-config.yaml
receivers:
  otlp:
    protocols:
      grpc:
      http:
  prometheus:
    config:
      scrape_configs:
        - job_name: 'node-exporter'
          static_configs:
            - targets: ['localhost:9100']

processors:
  batch:
    timeout: 10s
    send_batch_max_size: 1000
  # Critical: Drop high-cardinality labels from metrics
  filter/attributes:
    metrics:
      include:
        match_type: strict
        metric_names:
          - "http.server.duration"
      exclude:
        match_type: regexp
        attributes:
          - key: "user_id"
            value: ".*"
          - key: "request_id"
            value: ".*"

exporters:
  prometheus:
    endpoint: "0.0.0.0:8889"
  logging:
    loglevel: debug

service:
  pipelines:
    metrics:
      receivers: [otlp, prometheus]
      processors: [filter/attributes, batch]
      exporters: [prometheus]

5. Implement RED/USE Methodology

Structure dashboards and alerts around proven methodologies:

• RED Method (Services): Rate, Errors, Duration. Focus on request throughput, failure rates, and latency.
• USE Method (Infrastructure): Utilization, Saturation, Errors. Focus on resource saturation and error counts for nodes and disks.

Pitfall Guide

1. High-Cardinality Metric Explosion

Mistake: Adding user_id, session_id, or request_id as metric labels. Impact: Creates millions of unique time series, causing Prometheus to run out of memory and increasing storage costs exponentially. Best Practice: Metrics must have bounded cardinality. Use traces or logs for request-specific details. Aggregate metrics by low-cardinality attributes like service, method, and status_code.

2. Static Threshold Alerting

Mistake: Setting alerts like CPU > 80% regardless of traffic patterns. Impact: Alerts fire during legitimate traffic spikes or fail during slow-bleed degradation. Best Practice: Use dynamic thresholding or error budget burn rate alerting. Alert when the error budget is being consumed faster than sustainable over a short window.

3. Monitoring Everything, Observing Nothing

Mistake: Collecting all available metrics without defining what constitutes "healthy." Impact: Dashboards become cluttered; engineers cannot distinguish critical signals from background noise. Best Practice: Define SLIs (Service Level Indicators) and SLOs first. Monitor only what impacts user experience and business goals. Prune unused dashboards quarterly.

4. Ignoring Egress and Network Boundaries

Mistake: Focusing only on internal metrics while ignoring egress traffic, DNS resolution, and third-party API latency. Impact: Incidents caused by upstream dependencies or network partitions are detected late. Best Practice: Implement synthetic monitoring and external probes. Monitor egress bandwidth and latency to critical dependencies. Include third-party health checks in the alerting workflow.

5. Lack of Runbook Integration

Mistake: Alerts fire with a link to a dashboard but no actionable steps. Impact: MTTR increases as engineers spend time diagnosing known issues or searching for documentation. Best Practice: Every alert must link to a runbook with automated remediation steps where possible. Integrate alert metadata with incident management tools (PagerDuty, Opsgenie) to provide context.

6. No Monitoring for Monitoring

Mistake: Assuming the monitoring stack is always available. Impact: If the monitoring backend crashes during an incident, the team is flying blind. Best Practice: Monitor the health of the Collector, storage backend, and alerting pipeline. Implement redundancy for critical monitoring components. Ensure the monitoring stack can operate independently of the application network if possible.

7. Testing Gaps

Mistake: Never validating that alerts actually fire or dashboards load during an incident. Impact: False confidence in monitoring coverage. Best Practice: Conduct "Fire Drill" exercises. Use chaos engineering tools to inject failures and verify that alerts trigger, dashboards update, and runbooks are effective.

Production Bundle

Action Checklist

• Define SLOs: Establish SLOs for all Tier-1 services based on user impact.
• Deploy OTel Collector: Install Collector as DaemonSet and Gateway; configure resource detection processors.
• Enforce Cardinality Limits: Apply filter processors to drop unbounded labels; audit existing metrics for high cardinality.
• Implement RED/USE Dashboards: Create dashboards focused on Rate/Error/Duration for services and USE for infrastructure.
• Configure Burn Rate Alerts: Set up multi-window burn rate alerting rules tied to error budgets.
• Integrate Runbooks: Attach actionable runbooks to every alert rule; automate remediation for common failures.
• Monitor the Stack: Add alerts for Collector health, storage capacity, and pipeline lag.
• Conduct Fire Drill: Simulate a critical failure to validate alerting, dashboards, and runbook efficacy.

Decision Matrix

Scenario	Recommended Approach	Why	Cost Impact
Small Team / Startup	SaaS (Datadog/New Relic)	Low operational overhead; rapid setup; managed scaling.	High per-host cost; predictable OpEx.
High Scale / Cost Sensitive	Self-Hosted Prometheus + Thanos/VictoriaMetrics	Full control; no per-metric licensing fees; customizable.	High engineering effort; infrastructure costs scale with data.
Multi-Cloud / Vendor Neutral	OpenTelemetry + Backend Agnostic	Avoids lock-in; unified instrumentation across clouds.	Moderate setup cost; storage costs depend on chosen backend.
Regulatory / Data Sovereignty	On-Prem / Private Cloud Stack	Data never leaves the network; full audit control.	Highest infrastructure and maintenance cost.
Event-Driven / Serverless	OTel Push + Managed Backend	Handles ephemeral workloads; pull-based scrapers fail here.	Pay-per-use backend costs; efficient data ingestion.

Configuration Template

Copy this otel-collector-config.yaml as a baseline for a production-grade monitoring pipeline. This config includes batching, memory limits, high-cardinality filtering, and dual export for resilience.

receivers:
  otlp:
    protocols:
      grpc:
        max_recv_msg_size_mib: 32
      http:
        max_request_body_size: "32MiB"
  prometheus:
    config:
      global:
        scrape_interval: 15s
      scrape_configs:
        - job_name: 'infrastructure'
          static_configs:
            - targets: ['node-exporter:9100', 'kube-state-metrics:8080']

processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 1500
    spike_limit_mib: 500
  batch:
    timeout: 5s
    send_batch_max_size: 2000
  filter/high_cardinality:
    metrics:
      exclude:
        match_type: regexp
        attributes:
          - key: ".*id" # Drops labels ending in id
            value: ".*"
          - key: "trace_id"
            value: ".*"
          - key: "span_id"
            value: ".*"

exporters:
  prometheus:
    endpoint: "0.0.0.0:8889"
    const_labels:
      environment: "production"
  logging:
    loglevel: warn
  otlp/backup:
    endpoint: "backup-backend.internal:4317"
    tls:
      insecure: false

service:
  pipelines:
    metrics:
      receivers: [otlp, prometheus]
      processors: [memory_limiter, filter/high_cardinality, batch]
      exporters: [prometheus, logging]
    # Optional: Traces pipeline
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp/backup]

Quick Start Guide

1. Install Collector: Run the OTel Collector container locally or deploy to your cluster.

   docker run -p 4317:4317 -p 8889:8889 \
     -v $(pwd)/otel-config.yaml:/etc/otel-collector-config.yaml \
     otel/opentelemetry-collector-contrib:latest \
     --config /etc/otel-collector-config.yaml
   

2. Instrument App: Add the OTel SDK to your TypeScript application. Set environment variables to point to the Collector.

   export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
   export OTEL_SERVICE_NAME=my-service
   

3. View Metrics: Access the metrics endpoint exposed by the Collector.

   curl http://localhost:8889/metrics
   

4. Configure Grafana: Add Prometheus as a data source in Grafana pointing to http://localhost:8889. Import a standard Node.js or Kubernetes dashboard to visualize data immediately.

5. Validate Alerts: Use curl to generate load or errors on your service. Verify that metrics update in Grafana and that your alerting rules (if configured) trigger based on the defined thresholds.