Infrastructure disaster recovery test

ecture Decisions and Rationale

1. State Isolation: DR tests must never share Terraform/Pulumi state with production. Separate workspaces or stacks prevent accidental resource mutation and enable safe teardown.
2. Immutable Replay: Infrastructure is provisioned from version-controlled definitions rather than incremental updates. This guarantees that DR environments match production topology at the time of testing.
3. Validation as Code: Health checks, replication lag thresholds, DNS propagation verification, and RTO/RPO calculations are implemented in TypeScript. This enables integration with CI/CD pipelines and consistent metric collection.
4. Ephemeral Execution: DR test environments are spun up, validated, and destroyed within a single pipeline run. This eliminates stale test infrastructure and reduces cost.
5. Failback Automation: Recovery is incomplete without validated failback procedures. The pipeline includes automated promotion reversal, data synchronization verification, and traffic routing restoration.

Step-by-Step Implementation

Step 1: Define Scope and Boundaries Identify critical services, data stores, network dependencies, and DNS routing rules. Establish explicit RTO and RPO targets. Document failover triggers and rollback conditions.

Step 2: Isolate Infrastructure State Create a dedicated IaC stack for DR testing. Use separate backend configuration, variable files, and state locking. Ensure no cross-stack references to production resources.

Step 3: Automate Infrastructure Replay Trigger infrastructure provisioning in the secondary region via CI/CD. Use parameterized variables to override region, availability zones, and scaling thresholds. Apply infrastructure changes and wait for stabilization.

Step 4: Inject Synthetic Traffic and Validate Route synthetic requests through the failover endpoints. Verify service health, database replication lag, cache warming, and CDN propagation. Collect latency, error rate, and throughput metrics.

Step 5: Calculate and Enforce RTO/RPO Measure time from failover trigger to service readiness (RTO) and data loss window based on replication state (RPO). Compare against declared thresholds. Fail the pipeline if targets are breached.

Step 6: Automate Teardown and State Cleanup Destroy test infrastructure, purge synthetic data, and reset state files. Log all metrics and validation results for audit and trend analysis.

Code Example: DR Validation Runner (TypeScript)

This script orchestrates post-provisioning validation, calculates recovery metrics, and enforces policy gates.

import axios from 'axios';
import dns from 'dns';
import { promisify } from 'util';
import { performance } from 'perf_hooks';

const resolveDns = promisify(dns.resolve);

interface DRValidationConfig {
  primaryEndpoint: string;
  secondaryEndpoint: string;
  dbReplicationLagThreshold: number; // seconds
  rtoTarget: number; // minutes
  rpoTarget: number; // seconds
  healthCheckPath: string;
}

interface ValidationResult {
  rto: number;
  rpo: number;
  dnsPropagationMs: number;
  replicationLagSec: number;
  serviceHealthOk: boolean;
  passed: boolean;
}

export async function validateDR(config: DRValidationConfig): Promise<ValidationResult> {
  const failoverStart = performance.now();
  
  // 1. Verify DNS propagation to secondary region
  const dnsStart = performance.now();
  try {
    await resolveDns(config.secondaryEndpoint);
  } catch (err) {
    throw new Error(`DNS resolution failed for ${config.secondaryEndpoint}`);
  }
  const dnsPropagationMs = performance.now() - dnsStart;

  // 2. Check service health endpoints
  const healthCheck = await axios.get(`${config.secondaryEndpoint}${config.healthCheckPath}`, {
    timeout: 5000,
    validateStatus: () => true
  });
  const serviceHealthOk = healthCheck.status === 200;

  // 3. Measure replication lag (mock implementation for demonstration)
  // In production, query monitoring API or database status endpoint
  const replicationLagSec = await getReplicationLag(config.secondaryEndpoint);

  // 4. Calculate RTO/RPO
  const failoverEnd = performance.now();
  const rtoMinutes = (failoverEnd - failoverStart) / 60000;
  const rpoSeconds = replicationLagSec;

  const passed = 
    rtoMinutes <= config.rtoTarget &&
    rpoSeconds <= config.rpoTarget &&
    serviceHealthOk &&
    replicationLagSec < config.dbReplicationLagThreshold;

  return {
    rto: rtoMinutes,
    rpo: rpoSeconds,
    dnsPropagationMs,
    replicationLagSec,
    serviceHealthOk,
    passed
  };
}

async function getReplicationLag(endpoint: string): Promise<number> {
  // Replace with actual monitoring/database API call
  // Example: query CloudWatch, Datadog, or PostgreSQL pg_stat_replication
  const response = await axios.get(`${endpoint}/api/replication/lag`);
  return response.data.lag_seconds;
}

Architecture Integration

The validation script integrates into a CI/CD pipeline after IaC provisioning. Terraform or Pulumi provisions the secondary region stack. A pipeline step executes the TypeScript validator. Results are published to monitoring dashboards and stored in audit logs. If passed is false, the pipeline halts, and an incident ticket is automatically created with metric breakdowns. This enforces DR readiness as a deployment gate, not a retrospective review.

Pitfall Guide

DR testing failures rarely stem from missing infrastructure. They emerge from unvalidated assumptions, fragmented tooling, and operational friction. The following mistakes consistently degrade recovery reliability in production environments.

1. Testing Infrastructure Without Data Replication Provisioning compute and network in a secondary region does not guarantee data consistency. Database replication lag, asynchronous blob storage sync, and message queue backlog accumulation create silent RPO breaches. Validate data state post-failover using checksum comparisons, replication offset tracking, or application-level consistency checks.

2. Ignoring DNS and TTL Propagation Delays Failover mechanics fail when DNS resolvers cache stale records. Low TTL values reduce propagation time but increase DNS query load and caching instability. Test actual resolver behavior using synthetic DNS queries from multiple geographic points. Factor propagation time into RTO calculations.

3. Manual Steps in "Automated" Tests Runbooks that require manual DNS updates, database promotion commands, or credential rotation introduce human error and delay. Automate every failover action, including secret rotation, IAM role assumption, and network ACL updates. Validate automation by running tests without operator intervention.

4. Failing to Test Failback Procedures Recovery is incomplete without validated failback. Many organizations test failover but never validate returning traffic to the primary region, re-synchronizing data, or reverting DNS routing. Failback tests must run with the same rigor as failover drills, including data conflict resolution and state reconciliation.

5. Assuming Cloud Provider Redundancy Equals DR Multi-AZ deployments protect against hardware failure, not region-wide outages, control plane failures, or configuration drift. Cloud provider SLAs cover infrastructure availability, not application recovery mechanics. Validate cross-region failover independently of provider redundancy features.

6. Not Measuring Against Declared RTO/RPO Declaring targets without instrumenting actual recovery windows creates false confidence. Implement continuous metric collection during drills. Compare observed RTO/RPO against targets. Adjust architecture or thresholds based on empirical data, not theoretical capacity.

7. Running Tests in Production-Adjacent Environments Shared VPCs, overlapping IAM roles, or cross-environment state references cause test execution to impact production. Isolate DR test environments using separate accounts, VPCs, state backends, and network routing. Use synthetic data and traffic injection to avoid production contamination.

Best Practices from Production Experience

• Enforce policy-as-code gates that block deployments when DR validation fails
• Run DR tests against production-identical configurations, not scaled-down mocks
• Instrument replication, DNS, and service health with sub-minute sampling during drills
• Automate teardown to prevent stale test infrastructure from consuming resources
• Document failback procedures with explicit rollback conditions and data reconciliation steps
• Treat DR metrics as engineering KPIs, not compliance artifacts

Production Bundle

Action Checklist

• Define RTO/RPO boundaries and critical service dependencies before test execution
• Isolate DR state using separate IaC workspaces, backends, and IAM boundaries
• Automate infrastructure replay with parameterized variables for region and scaling overrides
• Implement TypeScript validation scripts to measure DNS propagation, replication lag, and service health
• Calculate actual RTO/RPO during drills and enforce pipeline gates on threshold breaches
• Automate failback procedures with data reconciliation and traffic routing restoration
• Run teardown scripts immediately after validation to eliminate stale test infrastructure
• Publish DR metrics to monitoring dashboards and audit logs for trend analysis

Decision Matrix

Scenario	Recommended Approach	Why	Cost Impact
Single-region application with periodic backups	Scheduled IaC Replay	Low complexity, predictable recovery window, minimal cross-region overhead	Low
Multi-region active-passive with async replication	Continuous DR Validation	Ensures replication lag stays within RPO, validates DNS failover mechanics	Medium
Multi-region active-active with global load balancing	Continuous DR Validation + GameDay	Tests traffic shifting, data conflict resolution, and control plane failover	High
Stateful workloads with strict compliance requirements	Continuous DR Validation + Failback Automation	Validates RTO/RPO, audit trails, and rollback procedures under regulatory constraints	Medium-High

Configuration Template

Terraform: Secondary Region DR Stack (main.tf)

terraform {
  backend "s3" {
    bucket = "dr-test-state-bucket"
    key    = "dr-environment/terraform.tfstate"
    region = "us-west-2"
    encrypt = true
  }
}

variable "primary_region" { default = "us-east-1" }
variable "secondary_region" { default = "us-west-2" }
variable "environment" { default = "dr-test" }

provider "aws" {
  region = var.secondary_region
}

module "dr_infrastructure" {
  source = "../modules/production"
  
  region            = var.secondary_region
  environment       = var.environment
  instance_count    = 1
  enable_autoscaling = false
  database_mode     = "read-replica"
  
  tags = {
    ManagedBy = "dr-test-pipeline"
    Lifecycle = "ephemeral"
  }
}

output "secondary_endpoint" {
  value = module.dr_infrastructure.app_endpoint
}

output "replication_status_endpoint" {
  value = module.dr_infrastructure.replication_api
}

TypeScript Validation Entry (dr-runner.ts)

import { validateDR, DRValidationConfig } from './dr-validator';

const config: DRValidationConfig = {
  primaryEndpoint: process.env.PRIMARY_ENDPOINT!,
  secondaryEndpoint: process.env.SECONDARY_ENDPOINT!,
  dbReplicationLagThreshold: 30,
  rtoTarget: 15,
  rpoTarget: 60,
  healthCheckPath: '/health/ready'
};

async function main() {
  try {
    const result = await validateDR(config);
    console.log(JSON.stringify(result, null, 2));
    process.exit(result.passed ? 0 : 1);
  } catch (err) {
    console.error('DR validation failed:', err);
    process.exit(1);
  }
}

main();

Quick Start Guide

1. Initialize State Backend: Create an isolated S3 bucket and DynamoDB table for Terraform state locking. Configure separate IAM roles with restricted permissions for DR test execution.
2. Deploy Secondary Stack: Run terraform init and terraform apply using the DR configuration template. Verify infrastructure provisioning in the secondary region without modifying production resources.
3. Execute Validation: Run npx ts-node dr-runner.ts with environment variables pointing to secondary endpoints. The script measures DNS propagation, replication lag, service health, and calculates RTO/RPO.
4. Enforce Gate and Teardown: If validation passes, proceed with pipeline promotion. If it fails, review metric breakdowns, adjust configuration, and retry. Execute terraform destroy immediately after testing to clean up ephemeral resources.