Variance Testing in Forecasting

entation

import { acorr_ljungbox } from 'some-stats-library'; // Hypothetical stats dependency

export interface ForecastAuditConfig {
  seasonalPeriod: number;
  epsilon: number;
  ljungBoxLags: number;
  significanceLevel: number;
}

export interface AuditResult {
  metrics: {
    mape: number;
    rmse: number;
    mase: number;
    theilsU: number;
  };
  diagnostics: {
    residualMean: number;
    residualStd: number;
    ljungBoxPValue: number;
    isAutocorrelated: boolean;
  };
  recommendation: 'RETRAIN' | 'RECALIBRATE' | 'MONITOR';
}

export class ForecastValidator {
  private config: ForecastAuditConfig;

  constructor(config: ForecastAuditConfig) {
    this.config = config;
  }

  public runAudit(groundTruth: number[], predictions: number[]): AuditResult {
    if (groundTruth.length !== predictions.length) {
      throw new Error('Length mismatch between ground truth and predictions.');
    }

    const metrics = this.computeMetrics(groundTruth, predictions);
    const diagnostics = this.analyzeResiduals(groundTruth, predictions);
    const recommendation = this.deriveRecommendation(metrics, diagnostics);

    return { metrics, diagnostics, recommendation };
  }

  private computeMetrics(actuals: number[], forecasts: number[]) {
    const errors = actuals.map((a, i) => a - forecasts[i]);
    const absErrors = errors.map(Math.abs);

    // MAPE with masking for near-zero values
    const validIndices = actuals.map((a, i) => 
      Math.abs(a) > this.config.epsilon ? i : -1
    ).filter(i => i !== -1);

    const mape = validIndices.length > 0
      ? (validIndices.reduce((sum, i) => sum + absErrors[i] / Math.abs(actuals[i]), 0) / validIndices.length) * 100
      : NaN;

    // RMSE
    const rmse = Math.sqrt(errors.reduce((sum, e) => sum + e * e, 0) / errors.length);

    // MASE with seasonal naive denominator
    const naiveDenominator = this.calculateNaiveDenominator(actuals);
    const mase = (absErrors.reduce((sum, e) => sum + e, 0) / absErrors.length) / naiveDenominator;

    // Theil's U: Ratio of model RMSE to no-change naive RMSE
    const naiveRmse = this.calculateNoChangeNaiveRmse(actuals);
    const theilsU = rmse / naiveRmse;

    return { mape, rmse, mase, theilsU };
  }

  private calculateNaiveDenominator(actuals: number[]): number {
    const period = this.config.seasonalPeriod;
    let naiveErrors: number[];

    if (actuals.length > period) {
      naiveErrors = actuals.slice(period).map((val, i) => 
        Math.abs(val - actuals[i])
      );
    } else {
      // Fallback to one-step naive for short series
      naiveErrors = actuals.slice(1).map((val, i) => 
        Math.abs(val - actuals[i])
      );
    }

    return naiveErrors.reduce((sum, e) => sum + e, 0) / naiveErrors.length;
  }

  private calculateNoChangeNaiveRmse(actuals: number[]): number {
    const diffs = actuals.slice(1).map((val, i) => val - actuals[i]);
    const sumSq = diffs.reduce((sum, d) => sum + d * d, 0);
    return Math.sqrt(sumSq / diffs.length);
  }

  private analyzeResiduals(actuals: number[], forecasts: number[]) {
    const residuals = actuals.map((a, i) => a - forecasts[i]);
    const mean = residuals.reduce((s, r) => s + r, 0) / residuals.length;
    const variance = residuals.reduce((s, r) => s + (r - mean) ** 2, 0) / residuals.length;
    const std = Math.sqrt(variance);

    // Ljung-Box test for autocorrelation
    // Note: In production, use a robust stats library implementation
    const lbResult = acorr_ljungbox(residuals, this.config.ljungBoxLags);
    const pValue = lbResult.pValue;
    const isAutocorrelated = pValue < this.config.significanceLevel;

    return { residualMean: mean, residualStd: std, ljungBoxPValue: pValue, isAutocorrelated };
  }

  private deriveRecommendation(metrics: any, diagnostics: any): 'RETRAIN' | 'RECALIBRATE' | 'MONITOR' {
    // Structural failure: Worse than naive baseline
    if (metrics.mase > 1.0 || metrics.theilsU > 1.0) {
      return 'RETRAIN';
    }

    // Model misspecification: Autocorrelated residuals
    if (diagnostics.isAutocorrelated) {
      return 'RETRAIN';
    }

    // Systematic bias without structural issues
    const biasThreshold = diagnostics.residualStd * 0.5;
    if (Math.abs(diagnostics.residualMean) > biasThreshold) {
      return 'RECALIBRATE';
    }

    return 'MONITOR';
  }
}

Rationale

• MASE Calculation: The denominator uses a seasonal naive benchmark. If the series length is insufficient for a full season, the code falls back to a one-step naive difference. This prevents division by zero and ensures MASE remains interpretable across series of varying lengths.
• Theil's U: This metric compares the model's RMSE against a no-change naive forecast. A value greater than 1.0 indicates the model performs worse than simply predicting the last observed value, signaling immediate retirement or retraining.
• Residual Diagnostics: The Ljung-Box test checks the null hypothesis that residuals are white noise. A low p-value rejects this hypothesis, indicating autocorrelation. Autocorrelated residuals imply the model has failed to capture temporal dependencies, a structural flaw that bias correction cannot resolve.
• Recommendation Logic: The decision tree prioritizes structural integrity. If the model is worse than naive or has autocorrelated errors, retraining is mandatory. Bias alone triggers recalibration, which is computationally cheaper and faster to deploy.

Pitfall Guide

1. The Zero-Value Trap

   • Explanation: Calculating MAPE on series with zero actuals results in infinite errors or NaN values, skewing aggregate metrics.
   • Fix: Implement masking strategies that exclude near-zero actuals from MAPE calculation, or use alternative metrics like sMAPE for intermittent series. Always validate data distribution before metric selection.

2. Incorrect Seasonal Period in MASE

   • Explanation: Using a default seasonal period (e.g., 12) for non-monthly data invalidates the MASE denominator, making the metric meaningless.
   • Fix: Dynamically detect seasonality or require explicit configuration per time series. Validate the period against the data frequency during pipeline initialization.

3. Recalibrating Autocorrelated Models

   • Explanation: Applying bias correction to a model with autocorrelated residuals treats a symptom while ignoring the disease. The model structure remains flawed.
   • Fix: Always run residual autocorrelation tests before recalibration. If Ljung-Box indicates autocorrelation, initiate retraining with feature engineering or model architecture changes.

4. Look-Ahead Bias in Baseline Calculation

   • Explanation: Computing naive benchmarks using future data points leaks information into the evaluation, artificially inflating model performance relative to the baseline.
   • Fix: Use rolling window evaluation or strict temporal splits. Ensure naive calculations only use data available at the time of prediction.

5. Metric Gaming via Hyperparameter Tuning

   • Explanation: Optimizing hyperparameters exclusively for MAPE can drive the model toward under-forecasting bias, as the metric penalizes over-forecasts more heavily.
   • Fix: Use multi-objective optimization that balances MAPE with bias metrics and RMSE. Monitor the residual mean during training to detect directional drift.

6. Static Thresholds in Dynamic Environments

   • Explanation: Hardcoding thresholds (e.g., MASE < 1.0) without considering domain volatility can lead to false alarms or missed detections.
   • Fix: Implement adaptive thresholds based on historical baseline performance. Adjust significance levels for Ljung-Box tests based on series length and noise characteristics.

7. Ignoring Theil's U Asymmetry

   • Explanation: Theil's U is a ratio; values below 1.0 indicate improvement, but values significantly above 1.0 indicate severe degradation. Treating U=1.1 and U=5.0 as equivalent "failures" loses nuance.
   • Fix: Categorize Theil's U ranges to prioritize retraining urgency. U > 2.0 should trigger immediate model retirement, while U between 1.0 and 1.2 might warrant investigation.

Production Bundle

Action Checklist

• Define Seasonal Configuration: Map each time series to its correct seasonal period and update the validator config.
• Implement Epsilon Handling: Ensure all MAPE calculations use a masking strategy with a domain-appropriate epsilon value.
• Integrate Ljung-Box Test: Add residual autocorrelation testing to the evaluation pipeline with configurable lag selection.
• Establish Decision Logic: Deploy the recommendation engine to automate retrain/recalibrate triggers based on metric thresholds.
• Monitor Theil's U: Add Theil's U to dashboards to detect models that underperform naive baselines despite low MAPE.
• Validate Baselines: Audit naive benchmark calculations to ensure no look-ahead bias exists in the evaluation logic.
• Set Alerting Rules: Configure alerts for MASE > 1.0 or Ljung-Box p-value < 0.05 to catch structural failures early.

Decision Matrix

Scenario	Recommended Approach	Why	Cost Impact
MASE > 1.0	Retrain	Model underperforms naive baseline; structural failure.	High: Requires data pipeline rerun and compute resources.
Autocorrelated Residuals	Retrain	Model misses systematic patterns; recalibration insufficient.	High: Feature engineering and retraining required.
Bias Only (No Autocorrelation)	Recalibrate	Model structure is sound; bias correction fixes offset.	Low: Parameter update or intercept adjustment.
Theil's U > 1.0	Retrain	Model worse than no-change forecast; immediate risk.	High: Model replacement or major revision.
All Metrics Passing	Monitor	Model performs well; maintain current configuration.	None: Routine evaluation only.

Configuration Template

forecast_audit:
  metrics:
    mape:
      enabled: true
      epsilon: 1e-6
    rmse:
      enabled: true
    mase:
      enabled: true
      seasonal_period: 12
    theils_u:
      enabled: true
  diagnostics:
    ljung_box:
      enabled: true
      lags: 10
      significance_level: 0.05
  thresholds:
    mase_max: 1.0
    theils_u_max: 1.0
    bias_std_multiplier: 0.5
  actions:
    retrain_triggers:
      - mase_exceeded
      - autocorrelation_detected
      - theils_u_exceeded
    recalibrate_triggers:
      - bias_detected

Quick Start Guide

1. Install Dependencies: Add the validator class and stats library to your project. Ensure TypeScript compilation is configured.
2. Configure Audit: Create a ForecastAuditConfig object with your series' seasonal period and epsilon threshold.
3. Run Validation: Call validator.runAudit(groundTruth, predictions) with your evaluation data.
4. Parse Recommendation: Inspect the recommendation field in the result. If RETRAIN, trigger the retraining pipeline. If RECALIBRATE, apply bias correction.
5. Monitor Trends: Log metrics and diagnostics over time to detect gradual drift before thresholds are breached.