envoy-config.yaml snippet

ice mesh philosophy.

2. Saga Orchestration vs. Choreography:

   • Decision: Use Saga Orchestration for complex workflows.
   • Rationale: Choreography (event-driven) scales well but becomes difficult to debug and manage as the number of steps increases. Orchestration provides a central point of control, clearer visibility into transaction state, and easier implementation of compensating logic for rollback scenarios.

3. Idempotency by Design:

   • Decision: Enforce idempotency keys on all event consumers.
   • Rationale: In distributed systems, retries are inevitable. Without idempotency, retries cause duplicate processing, leading to data corruption.

Step-by-Step Implementation

Step 1: Define the Saga Orchestrator

The orchestrator manages the sequence of local transactions and triggers compensating actions on failure.

// saga.types.ts
export interface SagaStep<TContext> {
  name: string;
  execute: (ctx: TContext) => Promise<void>;
  compensate: (ctx: TContext) => Promise<void>;
}

export interface SagaResult {
  success: boolean;
  completedSteps: string[];
  error?: Error;
}

// saga.orchestrator.ts
export class SagaOrchestrator<TContext> {
  private steps: SagaStep<TContext>[] = [];

  addStep(step: SagaStep<TContext>): this {
    this.steps.push(step);
    return this;
  }

  async execute(initialContext: TContext): Promise<SagaResult> {
    const result: SagaResult = { success: false, completedSteps: [] };
    const context = { ...initialContext };

    try {
      for (const step of this.steps) {
        try {
          await step.execute(context);
          result.completedSteps.push(step.name);
        } catch (err) {
          console.error(`Step ${step.name} failed. Initiating compensation.`);
          await this.compensate(result.completedSteps.reverse(), context);
          result.error = err instanceof Error ? err : new Error(String(err));
          return result;
        }
      }
      result.success = true;
      return result;
    } catch (compensationError) {
      // Critical failure during compensation
      console.error('Compensation failed. Manual intervention required.', compensationError);
      result.error = compensationError instanceof Error ? compensationError : new Error('Compensation failure');
      return result;
    }
  }

  private async compensate(stepsToCompensate: string[], context: TContext): Promise<void> {
    for (const stepName of stepsToCompensate) {
      const step = this.steps.find(s => s.name === stepName);
      if (step) {
        try {
          await step.compensate(context);
        } catch (err) {
          // Log critical error; compensation failure requires alerting
          console.error(`Compensation failed for ${stepName}:`, err);
          throw err;
        }
      }
    }
  }
}

Step 2: Implement Idempotent Event Processing

Event consumers must handle duplicate messages safely.

// idempotent.consumer.ts
import { Redis } from 'ioredis';

export class IdempotentConsumer {
  private redis: Redis;

  constructor(redisUrl: string) {
    this.redis = new Redis(redisUrl);
  }

  async process(message: { idempotencyKey: string; payload: any; handler: () => Promise<void> }): Promise<void> {
    const key = `idempotency:${message.idempotencyKey}`;
    
    // Atomic check-and-set using Redis SET NX
    const isNew = await this.redis.set(key, 'processed', 'EX', 86400, 'NX');
    
    if (isNew === 'OK') {
      try {
        await message.handler();
      } catch (err) {
        // Remove key on failure to allow retry
        await this.redis.del(key);
        throw err;
      }
    } else {
      console.log(`Message ${message.idempotencyKey} already processed. Skipping.`);
    }
  }
}

Step 3: Configure Sidecar for Traffic Management

The sidecar handles retries and circuit breaking transparently.

# envoy-config.yaml snippet
static_resources:
  listeners:
  - name: listener_0
    filter_chains:
    - filters:
      - name: envoy.filters.network.http_connection_manager
        typed_config:
          route_config:
            name: local_route
            virtual_hosts:
            - name: backend
              routes:
              - match: { prefix: "/api/orders" }
                route:
                  cluster: order_service
                  retry_policy:
                    retry_on: "5xx,reset,connect-failure"
                    num_retries: 3
                    per_try_timeout: 2s
                  circuit_breakers:
                    thresholds:
                    - priority: DEFAULT
                      max_connections: 1000
                      max_pending_requests: 1000
                      max_retries: 3

Pitfall Guide

1. The Distributed Monolith

• Mistake: Breaking a monolith into services but maintaining synchronous, tight coupling between them. Every user request triggers a chain of 5+ service calls.
• Impact: Latency multiplies; failure in one service cascades to the entire chain.
• Best Practice: Use asynchronous communication for non-critical paths. Aggregate data via CQRS rather than real-time joins. Ensure bounded contexts own their data completely.

2. Shared Database Anti-Pattern

• Mistake: Multiple microservices accessing the same database schema directly.
• Impact: Schema changes break multiple services; coupling returns; database becomes a bottleneck.
• Best Practice: Enforce database-per-service. Use the Database-per-Service pattern. Share data via APIs or events, never via direct DB access.

3. Ignoring Egress and Network Costs

• Mistake: Designing patterns that generate excessive cross-AZ or cross-region traffic without cost awareness.
• Impact: Cloud bills explode due to data transfer fees; latency increases due to network hops.
• Best Practice: Co-locate dependent services in the same availability zone. Use data locality patterns. Implement caching to reduce remote calls. Monitor egress costs in IaC pipelines.

4. Static Resource Limits in Auto-Scaling

• Mistake: Setting fixed CPU/Memory limits in Kubernetes that do not align with Horizontal Pod Autoscaler (HPA) metrics.
• Impact: Pods are throttled or evicted during traffic spikes; HPA cannot scale effectively.
• Best Practice: Define resource requests based on baseline usage and limits based on burst tolerance. Use Vertical Pod Autoscaler (VPA) in recommendation mode to tune limits. Align HPA thresholds with business SLAs.

5. Lack of Idempotency in Retries

• Mistake: Implementing retry logic without ensuring operations are idempotent.
• Impact: Duplicate charges, duplicate records, and data corruption.
• Best Practice: Design all write operations with idempotency keys. Use database constraints (unique indexes) as a safety net. Document idempotency requirements in API contracts.

6. Security Shift-Left Failure

• Mistake: Hardcoding secrets in environment variables or configuration files committed to source control.
• Impact: Credential leakage; compromised clusters.
• Best Practice: Use external secret managers (e.g., HashiCorp Vault, AWS Secrets Manager). Inject secrets via CSI drivers or sidecar injectors. Rotate credentials automatically. Never store secrets in IaC state files in plaintext.

7. Observability Debt

• Mistake: Implementing logging but neglecting distributed tracing and metrics correlation.
• Impact: Inability to diagnose cross-service failures; mean time to resolution (MTTR) remains high.
• Best Practice: Implement OpenTelemetry for standardized tracing. Correlate logs with trace IDs. Define SLOs and use error budgets to guide release velocity. Ensure metrics are exported in Prometheus format for scraping.

Production Bundle

Action Checklist

• Define Bounded Contexts: Map domain boundaries to service boundaries using Event Storming; ensure no shared data models.
• Implement Distributed Tracing: Deploy OpenTelemetry collectors; ensure every request carries a trace context across service boundaries.
• Configure Circuit Breakers: Set thresholds for error rates and latency; verify fallback mechanisms in sidecar or application code.
• Enforce Idempotency: Audit all event consumers and API endpoints; implement idempotency keys and database constraints.
• Design Compensation Logic: For every Saga step, implement and test the corresponding compensating transaction; verify rollback scenarios.
• Optimize Resource Requests: Run load tests to determine baseline resource usage; set requests/limits based on data; tune HPA metrics.
• Secure Secrets Management: Migrate all secrets to a centralized vault; implement automated rotation; audit IaC for secret leakage.
• Validate Chaos Engineering: Run failure injection tests (e.g., latency injection, pod kill) to verify resilience patterns in production-like environments.

Decision Matrix

Scenario	Recommended Approach	Why	Cost Impact
High-throughput API with variable load	Serverless + Event-Driven	Auto-scaling to zero; pay-per-invocation; decoupled processing.	Lowers cost by 40-60% for sporadic traffic.
Complex financial transaction flow	Saga Orchestration	Centralized control; clear compensation; auditability.	Moderate infra cost; reduces risk of financial loss.
Legacy app modernization	Strangler Fig + Sidecar	Incremental migration; sidecar adds resilience without code changes.	Lowers migration risk; optimizes cloud spend over time.
Real-time analytics pipeline	CQRS + Event Sourcing	Separates read/write models; immutable event log enables replay.	Higher storage cost; enables advanced analytics capabilities.
Internal CRUD tool	Modular Monolith	Simplicity; lower operational overhead; faster development.	Lowest infra cost; minimizes DevOps complexity.

Configuration Template

Kubernetes Deployment manifest demonstrating Sidecar pattern, resource management, and health probes.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: order-service
  labels:
    app: order-service
    team: commerce
spec:
  replicas: 3
  selector:
    matchLabels:
      app: order-service
  template:
    metadata:
      labels:
        app: order-service
        version: v1.2.0
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "9090"
    spec:
      containers:
      - name: app
        image: registry.internal/order-service:v1.2.0
        ports:
        - containerPort: 8080
          name: http
        resources:
          requests:
            cpu: "250m"
            memory: "256Mi"
          limits:
            cpu: "500m"
            memory: "512Mi"
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8080
          initialDelaySeconds: 15
          periodSeconds: 10
        readinessProbe:
          httpGet:
            path: /readyz
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 5
        env:
        - name: OTEL_EXPORTER_OTLP_ENDPOINT
          value: "http://otel-collector:4317"
      - name: envoy-sidecar
        image: envoyproxy/envoy:v1.28.0
        ports:
        - containerPort: 9901
          name: admin
        resources:
          requests:
            cpu: "100m"
            memory: "128Mi"
          limits:
            cpu: "200m"
            memory: "256Mi"
        volumeMounts:
        - name: envoy-config
          mountPath: /etc/envoy
      volumes:
      - name: envoy-config
        configMap:
          name: envoy-config

Quick Start Guide

1. Scaffold the Project: Initialize a TypeScript project with npm init -y and install dependencies: npm install ioredis @opentelemetry/api. Configure tsconfig.json for strict mode and ES2020.

2. Deploy Local Infrastructure: Use Docker Compose to spin up a local Redis instance and an OpenTelemetry Collector.

   # docker-compose.yml
   services:
     redis:
       image: redis:7-alpine
       ports: ["6379:6379"]
     otel-collector:
       image: otel/opentelemetry-collector:latest
       ports: ["4317:4317", "55680:55680"]
       volumes: ["./otel-config.yaml:/etc/otel/config.yaml"]
   

3. Run the Saga Orchestrator: Execute the TypeScript saga code with a mock context. Verify that compensation logic triggers on simulated failure.

   ts-node src/saga.demo.ts
   

4. Validate Observability: Check the OTEL Collector logs for trace exports. Ensure the trace contains spans for each saga step and the compensation action. Use a UI like Jaeger to visualize the distributed trace.

5. Apply IaC Template: Copy the Kubernetes configuration template. Apply it to a local cluster (Minikube/K3d) using kubectl apply -f deployment.yaml. Verify pod status and sidecar injection.

kubectl get pods -l app=order-service
kubectl logs deployment/order-service -c envoy-sidecar

This workflow establishes a baseline cloud native environment with resilience, consistency, and observability patterns implemented and validated.