Alert routing design

e matching, and idempotent delivery. The architecture separates ingestion, context resolution, rule evaluation, and channel dispatch. This enables independent scaling, dry-run testing, and routing observability.

Step-by-Step Implementation

1. Ingest & Normalize: Accept alerts via HTTP webhook or message queue. Normalize payloads to a consistent schema (CloudEvents or OpenTelemetry alert format). Strip tool-specific wrappers.
2. Enrich Context: Resolve service ownership, team rotation, SLO breach state, and topology dependencies. Enrichment must be cached and versioned to prevent routing drift.
3. Evaluate Routing Rules: Apply priority, deduplication, grouping, and escalation logic. Rules must be deterministic, idempotent, and version-controlled.
4. Dispatch to Channels: Route to primary, secondary, and fallback channels. Implement circuit breakers and retry policies per channel.
5. Collect Feedback: Log delivery status, acknowledgment timestamps, and resolution outcomes. Feed metrics back into routing policy evaluation.

TypeScript Routing Engine

import { createHash } from 'crypto';
import { EventEmitter } from 'events';

export interface AlertPayload {
  id: string;
  severity: 'critical' | 'warning' | 'info';
  service: string;
  metric: string;
  value: number;
  timestamp: number;
  labels: Record<string, string>;
}

export interface RoutingRule {
  id: string;
  match: (alert: AlertPayload) => boolean;
  priority: number;
  dedupWindowMs: number;
  channels: string[];
  fallbackChannels: string[];
  escalationTimeoutMs: number;
}

export interface RoutingContext {
  teamOwner: string;
  rotationActive: boolean;
  sloBreached: boolean;
  topologyDepth: number;
}

class AlertRouter extends EventEmitter {
  private rules: RoutingRule[] = [];
  private dedupCache = new Map<string, number>();
  private contextResolver: (alert: AlertPayload) => Promise<RoutingContext>;

  constructor(contextResolver: (alert: AlertPayload) => Promise<RoutingContext>) {
    super();
    this.contextResolver = contextResolver;
  }

  addRule(rule: RoutingRule) {
    this.rules.push(rule);
    this.rules.sort((a, b) => a.priority - b.priority);
  }

  async route(alert: AlertPayload): Promise<void> {
    const dedupKey = createHash('sha256')
      .update(`${alert.service}-${alert.metric}-${alert.severity}`)
      .digest('hex');

    const lastSeen = this.dedupCache.get(dedupKey) || 0;
    const matchingRule = this.rules.find(r => r.match(alert));

    if (!matchingRule) return;

    if (Date.now() - lastSeen < matchingRule.dedupWindowMs) {
      this.emit('dedup', { alert, ruleId: matchingRule.id });
      return;
    }

    this.dedupCache.set(dedupKey, Date.now());

    const context = await this.contextResolver(alert);
    const channels = context.sloBreached || context.rotationActive
      ? matchingRule.channels
      : matchingRule.fallbackChannels;

    await this.dispatch(alert, channels, matchingRule);
    this.emit('routed', { alert, ruleId: matchingRule.id, channels, context });
  }

  private async dispatch(
    alert: AlertPayload,
    channels: string[],
    rule: RoutingRule
  ): Promise<void> {
    const dispatchPromises = channels.map(async channel => {
      try {
        await this.sendToChannel(channel, alert);
        this.emit('delivered', { channel, alertId: alert.id });
      } catch (err) {
        this.emit('delivery_failed', { channel, alertId: alert.id, error: err });
        throw err;
      }
    });

    await Promise.race([
      Promise.allSettled(dispatchPromises),
      new Promise((_, reject) =>
        setTimeout(() => reject(new Error('Dispatch timeout')), rule.escalationTimeoutMs)
      )
    ]);
  }

  private async sendToChannel(channel: string, alert: AlertPayload): Promise<void> {
    // Integration point: PagerDuty, Slack, Webhook, Email, etc.
    // Implement circuit breaker, retry, and idempotency per channel
    console.log(`[ROUTE] ${alert.severity.toUpperCase()} -> ${channel} | ${alert.service}:${alert.metric}`);
  }
}

Architecture Decisions & Rationale

• Stateless Rule Evaluation: Rules are pure functions. No session state is stored in the router. This enables horizontal scaling and safe zero-downtime deployments.
• Deduplication by Content Hash: Alert identity is derived from service, metric, and severity. Timestamps and values are excluded to prevent duplicate routing for sustained conditions.
• Context Injection Over Hardcoding: Team ownership, rotation state, and SLO breach status are resolved dynamically. Hardcoded channel IDs break when teams restructure or rotations change.
• Circuit Breaker & Timeout Enforcement: Dispatch respects escalationTimeoutMs. If primary channels fail, the router logs failure and triggers fallback without blocking the ingestion pipeline.
• Event-Driven Observability: The router emits dedup, routed, delivered, and delivery_failed events. These feed into OpenTelemetry traces and routing health dashboards.

Pitfall Guide

1. Overlapping Rule Precedence: Multiple rules matching the same alert cause duplicate routing. Fix: enforce strict priority ordering and mutual exclusion in rule definitions. Use a rule compiler that validates overlap at deployment time.
2. Hardcoded Channel Identifiers: Embedding Slack channel IDs or PagerDuty service keys directly in routing logic breaks when teams reorganize. Fix: resolve channels via dynamic ownership maps or configuration service with version control.
3. Ignoring Alert Grouping: Treating every alert as independent floods on-call engineers during cascading failures. Fix: implement grouping windows and severity aggregation before dispatch. Group by service topology, not individual metrics.
4. Missing Fallback Routing: When primary channels fail (rate limits, auth rotation, network partitions), alerts vanish. Fix: define explicit fallback chains with exponential backoff and dead-letter queue logging.
5. No Routing Observability: Routing is a black box until incidents slip through. Fix: instrument MTTA, delivery success rate, dedup ratio, and rule match frequency. Alert on routing degradation, not just infrastructure degradation.
6. Static Escalation Timeouts: Fixed timeouts ignore incident severity and team capacity. Fix: scale escalation windows based on severity, SLO breach state, and on-call availability. Critical alerts bypass standard timeouts.
7. Skipping Dry-Run Validation: Deploying routing rules directly to production causes immediate misrouting. Fix: implement a dry-run mode that evaluates rules against historical alert streams without dispatching. Validate match coverage and dedup behavior before promotion.

Best practice: Treat routing rules as infrastructure code. Version them, test them against replayed alert streams, and deploy them through CI/CD with rollback capabilities. Integrate routing health into the same dashboard as SLOs and on-call rotations.

Production Bundle

Action Checklist

• Define routing rule schema with explicit priority, match conditions, dedup windows, and channel fallbacks
• Implement context resolution service mapping services to team ownership, rotation state, and SLO breach flags
• Deploy stateless routing engine with event emission for dedup, delivery, and failure tracking
• Configure circuit breakers and timeout policies per downstream channel integration
• Enable dry-run mode and validate rules against 7-day alert replay before production promotion
• Instrument routing metrics: MTTA, delivery success rate, dedup ratio, rule match frequency
• Establish feedback loop from incident management system to adjust routing weights and escalation policies

Decision Matrix

Scenario	Recommended Approach	Why	Cost Impact
Small team (<10 engineers)	Static rule-based with context enrichment	Low complexity, predictable behavior, minimal overhead	Low
Multi-tenant SaaS platform	Context-aware dynamic routing	Isolates tenant-specific alerts, prevents cross-tenant noise, scales with tenant growth	Medium
High-volume metric ingestion (>10k alerts/min)	Feedback-driven adaptive routing	Collapses duplicate streams, adapts to traffic patterns, reduces channel saturation	High
Compliance-heavy environment (SOC2, HIPAA)	Static rules with audit-logged delivery	Deterministic routing, verifiable delivery trails, simplified compliance reporting	Low-Medium

Configuration Template

routing:
  version: "2.1"
  dedup:
    window_ms: 300000
    hash_fields: ["service", "metric", "severity"]
  rules:
    - id: "critical-slo-breach"
      priority: 10
      match:
        severity: "critical"
        labels.slo_breached: "true"
      dedup_window_ms: 120000
      channels: ["pagerduty-critical", "slack-oncall-urgent"]
      fallback_channels: ["email-escalation", "sms-primary"]
      escalation_timeout_ms: 90000
    - id: "warning-service-degradation"
      priority: 20
      match:
        severity: "warning"
        labels.team: "platform"
      dedup_window_ms: 600000
      channels: ["slack-platform-alerts"]
      fallback_channels: ["pagerduty-warning"]
      escalation_timeout_ms: 300000
  context:
    resolver_endpoint: "https://context-api.internal/resolve"
    cache_ttl_seconds: 60
    fallback_owner: "platform-oncall"
  delivery:
    circuit_breaker:
      failure_threshold: 5
      recovery_timeout_ms: 60000
    retry:
      max_attempts: 3
      backoff_base_ms: 1000

Quick Start Guide

1. Deploy the routing engine container with the provided TypeScript binary or Docker image. Mount the YAML configuration and set CONTEXT_RESOLVER_URL environment variable.
2. Connect your alert source (Prometheus Alertmanager, Datadog, Grafana) to the routing engine via HTTP webhook. Configure payload transformation to match AlertPayload schema.
3. Load the configuration template and run npm run validate-rules -- --dry-run --replay-window=7d. Verify dedup ratios and rule match coverage.
4. Promote to production by switching the webhook target from dry-run to live dispatch. Monitor /metrics endpoint for MTTA, delivery success, and dedup ratio. Adjust escalation timeouts based on on-call acknowledgment patterns.