time building merge logic that fails under edge cases. The Sharded Active-Active approach offers the best risk/reward ratio for most applications: local write latency with near-zero conflicts by routing writes to a deterministic home region per entity.
Core Solution
Implementing robust cross-region sync requires a layered approach: selection of the replication mechanism, conflict resolution strategy, and idempotent consumer design.
Step 1: Select the Replication Mechanism
Avoid dual-writes. Dual-writes introduce partial failure scenarios where one region succeeds and the other fails, creating immediate inconsistency.
Recommended Pattern: Change Data Capture (CDC) via Message Broker.
- Database emits change events (PostgreSQL WAL, DynamoDB Streams, MongoDB Oplog).
- CDC connector (Debezium, AWS DMS) pushes events to a cross-region message broker (Kafka, Pub/Sub).
- Sink connectors in remote regions apply changes.
This decouples the source database from the replication process, providing durability and replay capabilities.
Step 2: Implement Conflict Resolution
For sharded architectures, conflicts are rare. For true active-active, you must implement a resolution strategy. Vector Clocks provide causality tracking superior to timestamps.
Vector Clock Implementation (TypeScript):
interface VectorClock {
[regionId: string]: number;
}
interface SyncEvent<T> {
id: string;
payload: T;
vectorClock: VectorClock;
timestamp: number; // Physical timestamp for tie-breaking
}
class ConflictResolver {
/**
* Determines if incomingEvent supersedes localState.
* Returns true if incomingEvent should overwrite localState.
*/
shouldApply<T>(
incomingEvent: SyncEvent<T>,
localClock: VectorClock
): 'APPLY' | 'SKIP' | 'CONFLICT' {
// Check causality
let dominated = true; // incoming is dominated by local
let dominates = true; // incoming dominates local
const allRegions = new Set([
...Object.keys(incomingEvent.vectorClock),
...Object.keys(localClock)
]);
for (const region of allRegions) {
const incVal = incomingEvent.vectorClock[region] || 0;
const locVal = localClock[region] || 0;
if (incVal < locVal) dominated = false;
if (incVal > locVal) dominates = false;
}
if (dominated) return 'SKIP';
if (dominates) return 'APPLY';
// Concurrent writes detected
return 'CONFLICT';
}
/**
* Resolves conflict using Last-Writer-Wins with Vector Clock merge.
* In production, replace with domain-specific logic or CRDTs.
*/
resolveConflict<T>(
eventA: SyncEvent<T>,
eventB: SyncEvent<T>
): { winner: SyncEvent<T>; mergedClock: VectorClock } {
// Tie-break by timestamp, then by region ID for determinism
const winner = (eventA.timestamp > eventB.timestamp)
|| (eventA.timestamp === eventB.timestamp && eventA.id > eventB.id)
? eventA
: eventB;
// Merge clocks: max of each component
const mergedClock: VectorClock = {};
const regions = new Set([
...Object.keys(eventA.vectorClock),
...Object.keys(eventB.vectorClock)
]);
for (const region of regions) {
mergedClock[region] = Math.max(
eventA.vectorClock[region] || 0,
eventB.vectorClock[region] || 0
);
}
return { winner, mergedClock };
}
}
Step 3: Idempotent Consumers and Schema Evolution
Sync consumers must be idempotent. Network retries or broker redeliveries will cause duplicate events.
Idempotency Pattern:
class IdempotentSyncConsumer {
private processedKeys: Set<string> = new Set();
private db: DatabaseClient;
async process(event: SyncEvent<any>): Promise<void> {
const dedupKey = `${event.id}:${event.vectorClock[this.regionId]}`;
// Check local cache first
if (this.processedKeys.has(dedupKey)) return;
// Check database for idempotency token
const exists = await this.db.checkIdempotencyToken(event.id);
if (exists) return;
try {
// Apply change
await this.db.applyChange(event.payload);
// Record token
await this.db.recordIdempotencyToken(event.id);
this.processedKeys.add(dedupKey);
} catch (err) {
if (!err.isDuplicateKey) throw err;
// Handle race condition in idempotency check
}
}
}
Architecture Decisions
- Sharding vs. Full Replication: Full replication of all data to all regions is cost-prohibitive and increases conflict surface. Use Data Partitioning: replicate reference data globally, but shard transactional data based on user geography or entity ID.
- Schema Drift Prevention: Enforce schema versioning in the sync stream. Consumers should reject events with incompatible schema versions, preventing corruption during rolling deployments.
- Backpressure Handling: Implement lag-based throttling. If a region falls behind, pause non-critical sync streams to prioritize user-facing data.
Pitfall Guide
1. Relying on NTP for Ordering
Mistake: Using physical timestamps for conflict resolution assumes synchronized clocks. NTP drift between regions can exceed 100ms, causing LWW to overwrite newer data with older data.
Best Practice: Use logical clocks (Vector Clocks, Lamport Timestamps) or hybrid logical clocks. If physical time is required, embed a monotonically increasing sequence number generated by the database.
2. Unbounded Conflict Queues
Mistake: Routing conflicts to a dead-letter queue (DLQ) without automated resolution or alerting. DLQs grow indefinitely, requiring manual intervention that scales poorly.
Best Practice: Implement automated conflict resolution policies where possible. For unresolvable conflicts, route to a reconciliation dashboard with SLA-based alerting, not just a DLQ.
3. Ignoring Data Residency
Mistake: Syncing PII to regions where the user has not consented to storage. Global tables often replicate data indiscriminately.
Best Practice: Tag data with residency requirements. Implement a Replication Policy Engine that filters events based on region compliance rules before they enter the cross-region stream.
4. Schema Evolution Without Backward Compatibility
Mistake: Deploying a schema change in Region A that breaks consumers in Region B during the rollout window.
Best Practice: Adopt Expand-Contract pattern. Add new fields first, deploy consumers to handle optional new fields, then remove old fields. Never remove fields or change types without a migration window.
5. Dual-Write Race Conditions
Mistake: Writing to two databases sequentially. If the second write fails, the system is inconsistent, and retry logic may cause duplicates or ordering issues.
Best Practice: Never dual-write. Use CDC. If dual-write is forced by legacy constraints, use a saga pattern with compensation, but recognize this is an anti-pattern for high-availability systems.
6. Testing Only in Happy Path
Mistake: Validating sync latency and throughput in stable network conditions.
Best Practice: Use chaos engineering tools (e.g., Gremlin, AWS Fault Injection Simulator) to inject latency, packet loss, and region failures. Verify that sync resumes correctly and conflicts are resolved after partition heals.
7. Egress Cost Blindness
Mistake: Replicating high-volume, low-value data (e.g., logs, telemetry) across regions.
Best Practice: Classify data by Value Density. Replicate only data required for local availability or compliance. Aggregate high-volume data in the source region and sync summaries, or use regional storage with cross-region query federation.
Production Bundle
Action Checklist
Decision Matrix
| Scenario | Recommended Approach | Why | Cost Impact |
|---|
| User Profiles (Global Users) | Sharded Active-Active | Users are distributed; home-region routing minimizes conflicts and latency. | Medium Egress |
| Financial Ledger | Active-Passive (Sync) | Zero data loss required; latency acceptable for writes. | Low Egress, High Latency |
| IoT Telemetry | Eventual / CRDT-based | High volume, mergeable metrics; conflicts acceptable or resolvable via CRDTs. | High Volume, Low Conflict Cost |
| Reference Data (Static) | Active-Passive (Async) | Infrequent updates; read-heavy; strong consistency not required. | Low Egress |
| Session State | Local Only + Replication | Sessions are ephemeral; replicate for failover but accept loss on partition. | Low Egress |
Configuration Template
Terraform: PostgreSQL Logical Replication Setup
resource "aws_db_instance" "source_region" {
engine = "postgres"
engine_version = "15.4"
instance_class = "db.r6g.xlarge"
# Enable logical replication
engine_mode = "provisioned"
publicly_accessible = false
parameter_group_name = aws_db_parameter_group.pg_replication.name
}
resource "aws_db_parameter_group" "pg_replication" {
name = "cross-region-replication"
family = "postgres15"
parameter {
name = "rds.logical_replication"
value = "1"
}
parameter {
name = "max_replication_slots"
value = "10"
}
}
# Publication on Source
resource "postgresql_publication" "global_sync" {
name = "global_data_sync"
db = aws_db_instance.source_region.db_name
all_tables = true
}
# Subscription on Target Region
# Note: Connection details must be securely managed
resource "postgresql_subscription" "region_b_sync" {
name = "sync_from_region_a"
conninfo = "host=${var.source_endpoint} dbname=${var.db_name} user=${var.replication_user} password=${var.replication_password}"
publication_names = [postgresql_publication.global_sync.name]
create_slot = true
depends_on = [aws_db_instance.target_region]
}
Quick Start Guide
- Provision Source and Target Databases: Deploy identical schema instances in two regions. Ensure network connectivity and security groups allow replication traffic.
- Enable Replication Features: On the source, enable logical replication or streams. Create a dedicated replication user with minimal privileges.
- Create Publication and Subscription: Define a publication for tables to sync. On the target, create a subscription pointing to the source. Verify initial snapshot replication.
- Monitor Lag: Query replication status (
pg_stat_replication or equivalent). Ensure lag is within acceptable bounds. Implement automated alerts for lag spikes.
- Validate Conflict Handling: Perform concurrent writes to the same record in both regions. Verify that conflict resolution logic executes correctly and data converges to the expected state.