Monitoring and alerting setup

erting Implement multi-window, multi-burn-rate alerting to detect both fast and slow burns of the error budget.

  - alert: SLOHighBurnRate
    expr: |
      (
        sum(rate(http_requests_total{job="payment-service", status=~"5.."}[5m]))
        /
        sum(rate(http_requests_total{job="payment-service"}[5m]))
      ) > (14.4 * (1 - 0.999))
    for: 2m
    labels:
      severity: critical
      page: "true"
    annotations:
      summary: "High error budget burn rate detected"
      description: "Error budget will be exhausted in < 1 day at current rate."
      runbook_url: "https://runbooks.internal/payment-service/error-budget-burn"

5. Alertmanager Routing and Inhibition

Configure Alertmanager to group related alerts, inhibit duplicates, and route to appropriate channels.

route:
  receiver: 'default-pagerduty'
  group_by: ['alertname', 'cluster', 'service']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - match:
        severity: critical
        page: "true"
      receiver: 'pagerduty-critical'
      continue: false
    - match:
        severity: warning
      receiver: 'slack-warnings'
      group_wait: 10s
      repeat_interval: 1h

inhibit_rules:
  - source_match:
      severity: 'critical'
    target_match:
      severity: 'warning'
    equal: ['alertname', 'cluster', 'service']

Pitfall Guide

1. Alerting on Symptoms, Not Causes:

   • Mistake: Alerting on high memory usage when the root cause is a memory leak in a specific library.
   • Fix: Alert on the business impact (e.g., increased error rate) and use traces/logs to diagnose the root cause. Symptoms change; causes are actionable.

2. Static Thresholds in Autoscaling Environments:

   • Mistake: Setting CPU > 80% as a critical alert in a Kubernetes cluster with Horizontal Pod Autoscaler (HPA).
   • Fix: Use rate-based metrics and SLOs. If HPA scales up, CPU usage should naturally drop. Alerting on absolute values ignores elasticity.

3. Cardinality Explosion:

   • Mistake: Adding high-cardinality labels (e.g., user_id, request_id) to metrics.
   • Fix: Reserve high-cardinality data for logs and traces. Metrics should have low cardinality labels (e.g., service, method, status). High cardinality causes storage bloat and query timeouts.

4. Missing Runbooks:

   • Mistake: Alerts fire without linked remediation steps.
   • Fix: Every alert must include a runbook_url annotation. Runbooks should contain diagnostic commands, rollback procedures, and escalation paths.

5. Alert Storms Due to Lack of Grouping:

   • Mistake: One underlying issue triggers hundreds of alerts for different instances.
   • Fix: Configure group_by in Alertmanager to aggregate alerts by service and cluster. Use inhibition rules to suppress warnings when critical alerts are active for the same service.

6. Ignoring Alert Lifecycle:

   • Mistake: Creating alerts and never reviewing them.
   • Fix: Implement a monthly alert review process. Archive alerts that fire less than once per quarter or have a high snooze rate. Alerts must earn their keep.

7. No Synthetic Monitoring:

   • Mistake: Relying solely on internal metrics, missing external user-facing issues.
   • Fix: Deploy synthetic checks (blackbox exporter) from multiple geographic locations to verify availability and latency from the user's perspective.

Production Bundle

Action Checklist

• Define SLOs for top 3 user journeys using the RED or USE method.
• Instrument services with OpenTelemetry; export metrics via OTLP or Prometheus exposition format.
• Create recording rules for all histogram quantiles and ratio calculations used in dashboards.
• Configure Alertmanager with grouping, inhibition, and routing rules based on severity.
• Attach runbook URLs to every alert rule; verify links are accessible.
• Set up on-call rotation with auto-escalation policies in PagerDuty/Opsgenie.
• Deploy synthetic monitors for critical endpoints with multi-region checks.
• Schedule monthly alert noise review; remove or tune low-value alerts.

Decision Matrix

Configuration Template

alertmanager.yml

global:
  resolve_timeout: 5m
  smtp_smarthost: 'smtp.internal:587'
  smtp_from: 'alerts@company.com'

route:
  receiver: 'default-slack'
  group_by: ['alertname', 'namespace', 'service']
  group_wait: 10s
  group_interval: 2m
  repeat_interval: 3h
  routes:
    - match:
        severity: critical
      receiver: 'pagerduty-critical'
      group_wait: 5s
      repeat_interval: 1h
    - match:
        severity: warning
      receiver: 'slack-warnings'
      repeat_interval: 6h
    - match:
        team: 'platform'
      receiver: 'slack-platform'

receivers:
  - name: 'pagerduty-critical'
    pagerduty_configs:
      - service_key: '<PAGERDUTY_SERVICE_KEY>'
        severity: 'critical'
        description: '{{ .GroupLabels.alertname }}: {{ .CommonAnnotations.summary }}'
  - name: 'default-slack'
    slack_configs:
      - api_url: '<SLACK_WEBHOOK_URL>'
        channel: '#monitoring-alerts'
        title: '{{ .GroupLabels.alertname }}'
        text: '{{ range .Alerts }}*{{ .Annotations.summary }}*{{ end }}'
  - name: 'slack-warnings'
    slack_configs:
      - api_url: '<SLACK_WEBHOOK_URL>'
        channel: '#monitoring-warnings'

inhibit_rules:
  - source_match:
      severity: 'critical'
    target_match:
      severity: 'warning'
    equal: ['alertname', 'namespace', 'service']

Quick Start Guide

1. Deploy Stack:

   helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
   helm install monitoring prometheus-community/kube-prometheus-stack -n monitoring --create-namespace
   

2. Add Service Monitor: Create a ServiceMonitor resource for your application to enable automatic metric scraping.

   apiVersion: monitoring.coreos.com/v1
   kind: ServiceMonitor
   metadata:
     name: my-app
   spec:
     selector:
       matchLabels:
         app: my-app
     endpoints:
       - port: metrics
         path: /metrics
         interval: 15s
   

3. Define First Alert: Apply a recording rule and alert rule via ConfigMap or PrometheusRule CRD.

   apiVersion: monitoring.coreos.com/v1
   kind: PrometheusRule
   metadata:
     name: my-app-alerts
   spec:
     groups:
       - name: my-app
         rules:
           - alert: HighErrorRate
             expr: sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) > 0.05
             for: 2m
             labels:
               severity: warning
             annotations:
               summary: "High 5xx error rate"
   

4. Verify: Access Grafana dashboard, confirm metrics ingestion, and simulate an error condition to validate alert routing to Slack/PagerDuty.