chaos-experiment.yaml (Litmus/Chaos Mesh compatible)

mplementation

1. Define the Steady State and Hypothesis Establish baseline metrics (latency p99, error rate, throughput, resource utilization). Formulate a testable hypothesis: Injecting 500ms network latency to the payment-service will not increase checkout failure rate beyond 0.5%.

2. Instrument Observability Deploy OpenTelemetry collectors, Prometheus metrics, and structured logging. Ensure trace context propagates across services. Chaos experiments must measure deviation from steady state, not just system availability.

3. Build the Fault Injection Layer Create a TypeScript chaos runner that orchestrates fault injection via Kubernetes API or HTTP proxy. The runner must support configurable blast radius, automatic rollback, and metric collection.

4. Execute with Safety Controls Run experiments against staging or canary environments first. Enforce circuit breakers that abort the experiment if SLOs are breached. Log all actions with correlation IDs for traceability.

5. Analyze and Automate Remediation Compare pre/post experiment metrics. If the hypothesis fails, generate a remediation ticket with exact failure conditions. Integrate experiment results into CI/CD gates to prevent regression.

TypeScript Chaos Runner Implementation

import { KubeConfig, AppsV1Api } from '@kubernetes/client-node';
import { createServer, IncomingMessage, ServerResponse } from 'http';
import { promisify } from 'util';

interface ChaosConfig {
  targetNamespace: string;
  targetDeployment: string;
  faultType: 'latency' | 'podKill' | 'cpuThrottle';
  maxDurationMs: number;
  abortThreshold: { errorRate: number; p99LatencyMs: number };
}

class ChaosRunner {
  private kubeConfig: KubeConfig;
  private appsApi: AppsV1Api;
  private metrics: Map<string, number[]> = new Map();

  constructor(config: ChaosConfig) {
    this.kubeConfig = new KubeConfig();
    this.kubeConfig.loadFromDefault();
    this.appsApi = this.kubeConfig.makeApiClient(AppsV1Api);
    this.config = config;
  }

  async execute(): Promise<void> {
    console.log(`[CHAOS] Starting ${this.config.faultType} experiment`);
    const baseline = await this.collectBaselineMetrics();
    
    try {
      await this.injectFault();
      await this.monitorExperiment(baseline);
    } catch (err) {
      console.error(`[CHAOS] Experiment aborted: ${(err as Error).message}`);
      await this.rollback();
      throw err;
    } finally {
      await this.cleanup();
    }
  }

  private async injectFault(): Promise<void> {
    switch (this.config.faultType) {
      case 'podKill':
        const pods = await this.appsApi.listNamespacedPod(this.config.targetNamespace);
        const targetPod = pods.body.items[0].metadata?.name;
        if (targetPod) {
          await this.appsApi.deleteNamespacedPod(targetPod, this.config.targetNamespace);
          console.log(`[CHAOS] Terminated pod: ${targetPod}`);
        }
        break;
      case 'latency':
        // Network latency injection handled via service mesh or tc-based sidecar
        console.log('[CHAOS] Latency fault injected via sidecar');
        break;
      default:
        throw new Error('Unsupported fault type');
    }
  }

  private async monitorExperiment(baseline: Record<string, number>): Promise<void> {
    const end = Date.now() + this.config.maxDurationMs;
    while (Date.now() < end) {
      const current = await this.collectBaselineMetrics();
      
      const errorRate = current.errorRate;
      const p99Latency = current.p99Latency;

      if (
        errorRate > baseline.errorRate * (1 + this.config.abortThreshold.errorRate) ||
        p99Latency > this.config.abortThreshold.p99LatencyMs
      ) {
        throw new Error(`SLO breach detected. Error: ${errorRate}, P99: ${p99Latency}`);
      }

      this.metrics.set(Date.now().toString(), [errorRate, p99Latency]);
      await new Promise(res => setTimeout(res, 5000));
    }
  }

  private async collectBaselineMetrics(): Promise<Record<string, number>> {
    // In production, integrate with Prometheus client or OpenTelemetry metrics endpoint
    return { errorRate: 0.02, p99Latency: 120, throughput: 1500 };
  }

  private async rollback(): Promise<void> {
    console.log('[CHAOS] Executing automatic rollback');
    // Restore replicas, remove tc rules, clear circuit breaker states
  }

  private async cleanup(): Promise<void> {
    console.log('[CHAOS] Experiment complete. Metrics exported.');
  }
}

export { ChaosRunner, ChaosConfig };

Architecture Decisions and Rationale

The chaos implementation follows a four-plane architecture:

• Control Plane: Orchestrator that schedules experiments, manages state, and enforces blast radius policies. Decoupled from execution to prevent cascading control failures.
• Execution Plane: Lightweight runners (TypeScript agents, Kubernetes operators, or eBPF probes) that apply faults. Designed to be idempotent and quickly reversible.
• Safety Plane: Circuit breakers, SLO monitors, and automatic rollback mechanisms. Prevents experiments from crossing production impact thresholds.
• Observability Plane: OpenTelemetry traces, Prometheus metrics, and structured logs. Provides the data foundation for hypothesis validation.

Rationale: Separation of concerns ensures that fault injection cannot compromise the control system. Safety controls are enforced at the execution boundary, not the orchestrator, reducing single points of failure. Observability is treated as a first-class dependency; without it, chaos experiments are blind guesses.

Pitfall Guide

1. Skipping Steady-State Definition Running faults without baseline metrics guarantees meaningless results. Chaos engineering measures deviation, not uptime. Always capture latency percentiles, error rates, and throughput before injection.

2. Unbounded Blast Radius Injecting faults across entire namespaces or production clusters without isolation guarantees outages. Restrict experiments to specific deployments, canary groups, or shadow traffic. Use network policies and resource quotas to contain failure domains.

3. Ignoring Business KPIs Technical metrics (CPU, memory, pod restarts) do not equal user impact. Map system degradation to business outcomes: checkout conversion, search relevance, payment success rate. Experiments must validate business continuity, not just service availability.

4. Running Without Automated Rollback Manual intervention during chaos experiments introduces latency and human error. Implement programmatic rollback triggers tied to SLO breaches. The system must self-heal or abort within seconds, not minutes.

5. Treating Chaos as a Testing Phase Chaos is not a pre-release checklist. It is a continuous practice. Schedule recurring experiments in staging, canary, and production environments. Integrate results into deployment gates to catch resilience regression early.

6. Inadequate Observability Coverage Fault injection without trace propagation and metric collection is noise. Ensure correlation IDs flow across services, logs are structured, and dashboards update in real-time. Blind experiments waste engineering time and create false confidence.

7. No Blameless Post-Experiment Analysis Focusing on who broke what instead of why the system failed misses the point. Document hypothesis, execution steps, metric deviation, and remediation actions. Treat every experiment as a reliability learning opportunity.

Best Practices from Production:

• Use feature flags to enable/disable chaos runners without redeployment.
• Start with non-critical services and low-severity faults (latency, CPU throttling) before progressing to pod kills and network partitions.
• Integrate chaos results into SLO tracking and incident post-mortems.
• Automate experiment reporting and generate remediation tickets automatically when hypotheses fail.
• Maintain an experiment registry with versioned configurations, execution history, and outcome classifications.

Production Bundle

Action Checklist

• Define steady-state metrics: Establish baseline latency, error rate, and throughput for target services before any fault injection.
• Configure blast radius limits: Restrict experiments to specific namespaces, deployments, or traffic percentages using network policies and resource quotas.
• Implement automated rollback: Program SLO-based abort conditions that trigger immediate fault removal and system restoration.
• Instrument observability: Deploy OpenTelemetry collectors, Prometheus metrics, and structured logging with cross-service trace correlation.
• Schedule recurring experiments: Run chaos tests on a fixed cadence in staging, canary, and production environments to catch resilience regression.
• Map technical faults to business impact: Validate experiments against conversion rates, payment success, and user-facing SLAs rather than infrastructure metrics alone.
• Document experiment registry: Maintain versioned configurations, execution logs, hypothesis outcomes, and remediation actions for audit and compliance.

Decision Matrix

Scenario	Recommended Approach	Why	Cost Impact
Monolith with tight coupling	Simulated fault injection via HTTP proxy/middleware	Avoids infrastructure-level disruption; validates internal dependency degradation	Low (no additional agents)
Kubernetes microservices	Operator-based chaos (Litmus/Chaos Mesh) with pod/network faults	Native integration, blast radius control, automated rollback via CRDs	Medium (operator overhead, monitoring)
Multi-cloud hybrid	Custom TypeScript runner with cloud provider SDKs	Abstracts cloud-specific APIs, enforces consistent safety policies across environments	High (cross-cloud networking, unified observability)
Regulated production (PCI/HIPAA)	Shadow traffic replay + non-destructive latency injection	Validates resilience without modifying live state or violating compliance boundaries	Medium (traffic mirroring infrastructure)

Configuration Template

# chaos-experiment.yaml (Litmus/Chaos Mesh compatible)
apiVersion: chaosmesh.org/v1alpha1
kind: NetworkChaos
metadata:
  name: payment-service-latency-test
  namespace: production
spec:
  action: delay
  mode: one
  selector:
    namespaces:
      - production
    labelSelectors:
      app: payment-service
  delay:
    latency: "300ms"
    jitter: "50ms"
    correlation: "100"
  duration: "120s"
  scheduler:
    cron: "0 */6 * * *"
---
# chaos-runner.config.ts
export const chaosConfig = {
  targetNamespace: 'production',
  targetDeployment: 'payment-service',
  faultType: 'latency',
  maxDurationMs: 120000,
  abortThreshold: {
    errorRate: 0.05,
    p99LatencyMs: 800
  },
  observability: {
    metricsEndpoint: 'http://prometheus.monitoring:9090',
    traceExporter: 'otlp',
    logLevel: 'info'
  },
  safety: {
    enableAutoRollback: true,
    blastRadiusLimit: 'deployment',
    requireApproval: false
  }
};

Quick Start Guide

1. Install the chaos runner: npm install @kubernetes/client-node prom-client opentelemetry-sdk-node
2. Configure target and thresholds: Update chaosConfig in chaos-runner.config.ts with your namespace, deployment, fault type, and SLO abort limits.
3. Deploy observability stack: Ensure Prometheus, OpenTelemetry collector, and structured logging are running in your cluster. Verify metrics are accessible at the configured endpoint.
4. Execute first experiment: Run npx ts-node chaos-runner.ts --config chaos-runner.config.ts --mode staging. Monitor the dashboard for metric deviation and confirm automatic rollback triggers on SLO breach.