Cost anomaly detection

Architecture Decisions and Rationale

1. Data Ingestion Layer: Pull cost data via cloud provider billing APIs or S3/BigQuery exports. Normalize to daily or hourly granularity depending on workload velocity.
2. Preprocessing: Apply timezone alignment, handle missing data points via forward-fill or interpolation, and filter out known maintenance windows.
3. Baseline Computation: Use exponential weighted moving average (EWMA) with seasonality decomposition. Avoid simple rolling windows that lag during sudden shifts.
4. Anomaly Scoring: Calculate z-scores relative to the dynamic baseline, adjusted for historical variance. Apply hysteresis to prevent flapping.
5. Alert Routing: Enrich alerts with cost attribution, service context, and recommended investigation steps. Route to Slack/PagerDuty with severity tiers.

Step-by-Step Implementation

1. Define Cost Data Schema

interface CostRecord {
  timestamp: Date;
  service: string;
  cost: number;
  unit: string; // e.g., "hour", "GB", "request"
  tag: Record<string, string>;
}

2. Build the Adaptive Detector

export class CostAnomalyDetector {
  private baseline: number[] = [];
  private variance: number[] = [];
  private seasonalityFactor: number;
  private readonly sensitivity: number;
  private readonly cooldownMs: number;
  private lastAlertTime: number = 0;

  constructor(
    private readonly windowSize: number = 7,
    private readonly smoothing: number = 0.3,
    private readonly thresholdZ: number = 2.5,
    seasonalityDays: number = 7
  ) {
    this.seasonalityFactor = 1 / seasonalityDays;
    this.sensitivity = thresholdZ;
    this.cooldownMs = 3600000; // 1 hour cooldown
  }

  public analyze(record: CostRecord): AnomalyResult {
    const hourOfDay = record.timestamp.getHours();
    const dayOfWeek = record.timestamp.getDay();
    
    // Apply seasonality adjustment
    const adjustedCost = record.cost / (1 + this.seasonalityFactor * (dayOfWeek + hourOfDay / 24));
    
    this.baseline.push(adjustedCost);
    if (this.baseline.length > this.windowSize) {
      this.baseline.shift();
    }

    // Compute EWMA baseline
    const ewma = this.baseline.reduce((acc, val, i) => {
      const alpha = this.smoothing;
      return i === 0 ? val : alpha * val + (1 - alpha) * acc;
    }, 0);

    // Compute rolling variance
    const variance = this.baseline.reduce((acc, val) => acc + Math.pow(val - ewma, 2), 0) / this.baseline.length;
    this.variance.push(variance);
    if (this.variance.length > this.windowSize) this.variance.shift();
    
    const currentVariance = this.variance[this.variance.length - 1] || 1;
    const stdDev = Math.sqrt(currentVariance);
    
    // Z-score calculation with variance floor to prevent division by zero
    const zScore = stdDev > 0.01 ? (adjustedCost - ewma) / stdDev : 0;
    
    const isAnomaly = Math.abs(zScore) > this.sensitivity;
    const now = Date.now();
    const suppressed = isAnomaly && (now - this.lastAlertTime < this.cooldownMs);
    
    if (isAnomaly && !suppressed) {
      this.lastAlertTime = now;
    }

    return {
      timestamp: record.timestamp,
      service: record.service,
      rawCost: record.cost,
      adjustedCost,
      baseline: ewma,
      zScore,
      isAnomaly,
      suppressed,
      severity: this.classifySeverity(zScore)
    };
  }

  private classifySeverity(z: number): 'low' | 'medium' | 'high' | 'critical' {
    const abs = Math.abs(z);
    if (abs > 5) return 'critical';
    if (abs > 3.5) return 'high';
    if (abs > 2.5) return 'medium';
    return 'low';
  }
}

export interface AnomalyResult {
  timestamp: Date;
  service: string;
  rawCost: number;
  adjustedCost: number;
  baseline: number;
  zScore: number;
  isAnomaly: boolean;
  suppressed: boolean;
  severity: 'low' | 'medium' | 'high' | 'critical';
}

3. Integration Pattern

Deploy the detector as a stateless function triggered by billing exports or a scheduled cron job. Maintain baseline state in a lightweight cache (Redis, DynamoDB, or local file for serverless cold starts). Route results through a notification pipeline that tags alerts with service, severity, and zScore.

Rationale for Statistical Over ML

Machine learning models require 30+ days of clean historical data, continuous retraining, and drift monitoring. The EWMA + z-score approach converges within 72 hours, remains fully interpretable, and handles cold-start scenarios gracefully. For cost sustainability, transparency and rapid deployment outweigh marginal accuracy gains from black-box models.

Pitfall Guide

1. Ignoring Diurnal and Weekly Seasonality Cost patterns repeat daily and weekly. Development environments idle at night; batch jobs run Sunday mornings. Failing to normalize for seasonality causes 60%+ false positives. Always apply time-based adjustment factors before scoring.

2. Alerting on Raw Currency Instead of Unit Cost A $500 spike in data transfer might be normal during a product launch, but $0.05/GB over the expected rate indicates a leak. Normalize costs to unit economics (cost_per_request, cost_per_gb, cost_per_core_hour) to isolate efficiency drift from volume growth.

3. Including Anomalies in Baseline Calculation If a runaway service inflates costs for 48 hours and that data feeds your rolling window, your baseline shifts upward. You will stop detecting the anomaly. Implement outlier rejection: discard points exceeding 3σ before updating the EWMA.

4. Missing Cost Attribution Tags Unattributed spend cannot be routed to owners. Without team, project, or environment tags, alerts become noise. Enforce tagging policies at the infrastructure-as-code level and filter detection to tagged resources only.

5. Static Cooldown Periods Causing Alert Fatigue Fixed cooldowns either suppress legitimate follow-up alerts or allow duplicate notifications during sustained spikes. Use dynamic cooldowns that scale with anomaly severity and duration. Critical anomalies should re-alert every 15 minutes; low-severity should wait 4 hours.

6. Detecting at the Wrong Granularity Account-level detection masks service-specific leaks. Resource-level detection creates thousands of alerts. Aggregate to the service + environment level for optimal signal density.

7. No Feedback Loop for False Positives Engineering teams dismiss alerts that fire during known deployments or migrations. Without a suppression mechanism tied to deployment pipelines, the detector loses credibility. Integrate with CI/CD events to temporarily raise thresholds during release windows.

Best Practice: Implement a tiered alerting strategy. Low severity → daily digest. Medium → Slack channel. High/Critical → PagerDuty with runbook link. Always include baseline, current, delta, and zScore in the payload.

Production Bundle

Action Checklist

• Ingest billing data at hourly granularity via cloud provider API or S3 export
• Implement seasonality normalization using day-of-week and hour-of-day factors
• Deploy EWMA baseline with 7-day rolling window and 0.3 smoothing coefficient
• Add outlier rejection to prevent anomaly contamination in baseline
• Configure dynamic cooldowns scaling with z-score severity
• Integrate deployment pipeline events for temporary threshold relaxation
• Route alerts through tiered channels with enriched context payloads
• Establish weekly review cadence to tune sensitivity and suppress false positives

Decision Matrix

Scenario	Recommended Approach	Why	Cost Impact
Startup / SMB (< $50k/mo)	Adaptive Statistical	Fast deployment, low maintenance, immediate ROI	Reduces waste by 15-25% within 60 days
Enterprise / Multi-Cloud	Adaptive Statistical + Centralized Aggregator	Handles heterogeneous billing formats, scales across accounts	Prevents $50k+ annual drift across environments
Batch / ML Workload Heavy	ML Forecasting (Prophet/LSTM)	Captures complex scheduling patterns and training job variance	Optimizes spot/preemptible utilization, cuts compute waste by 30%+
Regulated / Compliance-Focused	Statistical + Audit Trail	Transparent scoring meets audit requirements, no black-box models	Avoids compliance penalties, enables precise cost attribution

Configuration Template

cost_anomaly_detector:
  ingestion:
    source: "aws_cost_explorer"
    granularity: "hourly"
    timezone: "UTC"
  
  baseline:
    window_size: 7
    smoothing_factor: 0.3
    seasonality:
      enabled: true
      periods: ["daily", "weekly"]
  
  detection:
    z_threshold: 2.5
    variance_floor: 0.01
    cooldown:
      enabled: true
      base_minutes: 60
      scale_with_severity: true
  
  routing:
    low: "daily_digest"
    medium: "slack_finops_channel"
    high: "pagerduty_service"
    critical: "pagerduty_escalation"
  
  enrichment:
    required_tags: ["team", "project", "environment"]
    unit_normalization: true
    deployment_integration: true

Quick Start Guide

1. Deploy the detector function as a serverless job (AWS Lambda, GCP Cloud Run, or Azure Functions) with the TypeScript implementation.
2. Configure the billing source by granting read access to your cloud provider’s cost export bucket or API. Point the ingestion layer to the daily/hourly CSV or JSON export.
3. Set initial thresholds using the YAML template. Start with z_threshold: 2.5 and window_size: 7. Route medium severity to a dedicated Slack channel.
4. Run a dry pass using historical data (last 14 days). Validate that known promotions or migrations do not trigger critical alerts. Adjust seasonality factors if false positives exceed 15%.
5. Enable production routing and attach a runbook link to high/critical alerts. Schedule a 15-minute weekly review to tune sensitivity and suppress deployment windows.