terraform/dr-infrastructure.tf

SClient({ region: secondaryRegion }); const route53Client = new Route53Client({ region: secondaryRegion });

// Verify cross-region replication exists const buckets = await s3Client.send(new ListBucketsCommand({})); const replicatedBuckets = buckets.Buckets?.filter(b => b.Name?.startsWith("app-dr-") ) ?? [];

if (replicatedBuckets.length === 0) { throw new Error("No replicated storage targets found in secondary region"); }

// Check database replication lag const dbInstances = await rdsClient.send(new DescribeDBInstancesCommand({})); const replica = dbInstances.DBInstances?.find(d => d.DBInstanceStatus === "available"); const replicationLag = replica?.ReplicationLag ?? 0;

// Validate health endpoint after simulated cutover DNS update const healthCheck = await axios.get(targetEndpoint, { timeout: 5000 }); const rtoElapsed = Date.now() - startTime;

return { region: secondaryRegion, status: replicationLag <= 60 && healthCheck.status === 200 ? "PASS" : "FAIL", rpoLagSeconds: replicationLag, rtoElapsedMs: rtoElapsed, healthCheckStatus: healthCheck.status, }; }

### Step 5: Integrate Chaos Engineering for Failure Injection
Schedule controlled failure scenarios: region network partition, database replica promotion, DNS TTL expiration, and IAM credential rotation. Validate that automated recovery pipelines trigger without manual intervention. Record mean time to recovery (MTTR) and compare against RTO targets.

### Architecture Decisions and Rationale
- **Declarative over imperative:** Infrastructure state must be reproducible from code. Imperative scripts introduce drift and break recovery determinism.
- **Idempotent deployments:** DR execution must handle repeated runs without data corruption or resource conflicts.
- **Health-based cutover:** Routing decisions must depend on endpoint validation, not time-based assumptions.
- **Replication lag gates:** Cutover should abort if data consistency falls outside RPO boundaries.
- **Automated rollback:** Every failover must include a verified rollback path to prevent state divergence.

## Pitfall Guide

1. **Assuming backups equal disaster recovery**
   Backups protect against data loss; DR protects against service unavailability. A backup restores files; DR restores traffic routing, state consistency, and dependency resolution. Validate recovery paths, not just backup success.

2. **Static runbooks with manual execution**
   Runbooks that require human decision-making during incidents introduce latency and error. Automate cutover logic, DNS updates, and health validation. Reserve manual intervention for architectural escalation, not routine failover.

3. **Ignoring cross-region dependency mapping**
   Applications depend on DNS, IAM roles, VPC peering, security groups, and external APIs. Replicating compute and storage without replicating network topology and identity permissions guarantees failure. Map and automate all dependency chains.

4. **No automated replication lag monitoring**
   Asynchronous replication accumulates lag under load. Without continuous monitoring, RPO targets become theoretical. Implement lag thresholds that block cutover when exceeded.

5. **Over-provisioning passive regions without utilization tracking**
   Idle infrastructure incurs cost without validation. Run continuous health checks, scheduled DR tests, and cost attribution. Treat secondary regions as active validation environments, not storage lockers.

6. **Missing DNS TTL and caching awareness**
   DNS propagation delays extend RTO beyond infrastructure recovery time. Reduce TTL to 60–300 seconds before failover events. Use health-checked routing policies (Route 53, Cloudflare Load Balancing) instead of manual record updates.

7. **Testing only during business hours with low traffic**
   DR tests under nominal load do not reflect production failure conditions. Inject traffic, simulate concurrent writes, and validate replication under stress. Recovery behavior changes significantly under load.

**Best practices from production experience:**
- Run DR validation on every major infrastructure change, not just quarterly.
- Version control all DR scripts, IaC, and configuration templates.
- Implement circuit breakers that prevent cascading failover during partial outages.
- Log every recovery step with timestamps for post-incident analysis.
- Align RTO/RPO targets with actual business impact, not engineering convenience.

## Production Bundle

### Action Checklist
- [ ] Define RTO/RPO per service tier and document dependency chains
- [ ] Migrate all infrastructure to declarative IaC with remote state replication
- [ ] Configure cross-region data replication with lag monitoring and alerting
- [ ] Build automated DR validation pipeline with health checks and rollback logic
- [ ] Reduce DNS TTL to ≤300s and implement health-based routing policies
- [ ] Schedule monthly chaos experiments simulating region failure and replica promotion
- [ ] Implement cost attribution and utilization tracking for secondary regions
- [ ] Run DR validation on every infrastructure change and major release

### Decision Matrix

| Scenario | Recommended Approach | Why | Cost Impact |
|----------|---------------------|-----|-------------|
| Legacy monolith with infrequent updates | Cold Backup + IaC snapshot | Low change velocity justifies slower recovery; minimizes idle infrastructure cost | Low ($800–$1,500/mo) |
| Customer-facing API with 15-min RTO requirement | Warm Standby with automated cutover | Balances recovery speed and cost; requires replication lag validation | Medium ($3,200–$5,500/mo) |
| Financial transaction platform with zero data loss tolerance | Hot Standby with synchronous replication | RPO <5m demands real-time consistency; automation prevents human error during failover | High ($8,500–$12,000/mo) |
| Global SaaS with multi-region traffic distribution | Active-Active with conflict resolution | Eliminates single-region dependency; requires distributed state management and consistent hashing | Very High ($15,000–$22,000/mo) |

### Configuration Template

```hcl
# terraform/dr-infrastructure.tf
resource "aws_db_instance" "primary" {
  engine               = "postgres"
  instance_class       = "db.r6g.xlarge"
  multi_az             = true
  backup_retention_period = 30
  storage_encrypted    = true
}

resource "aws_db_instance" "replica" {
  engine               = "postgres"
  instance_class       = "db.r6g.xlarge"
  replicate_source_db  = aws_db_instance.primary.id
  skip_final_snapshot  = true
  publicly_accessible  = false
}

resource "aws_s3_bucket" "primary_storage" {
  bucket = "app-primary-data"
}

resource "aws_s3_bucket_replication_configuration" "replication" {
  bucket = aws_s3_bucket.primary_storage.id
  role   = aws_iam_role.replication.arn

  rule {
    status = "Enabled"
    destination {
      bucket        = aws_s3_bucket.secondary_storage.arn
      storage_class = "STANDARD"
    }
  }
}

resource "aws_route53_health_check" "app_health" {
  fqdn              = "api.example.com"
  port              = 443
  type              = "HTTPS"
  request_interval  = 30
  failure_threshold = 3
}

resource "aws_route53_record" "api_routing" {
  zone_id = aws_route53_zone.main.zone_id
  name    = "api.example.com"
  type    = "A"

  failover_routing_policy {
    type = "PRIMARY"
  }

  set_identifier = "primary"
  health_check_id = aws_route53_health_check.app_health.id
  alias {
    name                   = aws_lb.primary.dns_name
    zone_id                = aws_lb.primary.zone_id
    evaluate_target_health = true
  }
}

Quick Start Guide

1. Install dependencies: npm install @aws-sdk/client-s3 @aws-sdk/client-rds @aws-sdk/client-route-53 axios
2. Define tier mapping: Create a recovery-tiers.json file mapping services to RTO/RPO targets and dependency lists.
3. Deploy baseline infrastructure: Apply the Terraform template to establish primary/replica resources and health-checked routing.
4. Run validation: Execute ts-node dr-validator.ts --primary us-east-1 --secondary eu-west-1 --endpoint https://api.example.com/health
5. Schedule automation: Add the validation script to your CI/CD pipeline with cron triggers (weekly dry run, monthly full failover test) and alert on status: "FAIL".