Envoy Adaptive Routing Configuration

{this.healthCheckUrl}/${backend.id}/health`, { method: 'GET', signal: AbortSignal.timeout(2000) }); backend.healthScore = res.ok ? 1.0 : 0.0; backend.lastHealthCheck = Date.now(); } catch { backend.healthScore = 0.0; backend.lastHealthCheck = Date.now(); } }); await Promise.allSettled(checks); }

private calculateScore(backend: BackendState): number { if (backend.draining || backend.healthScore < 0.3) return Infinity; // Lower score = preferred backend const connectionPenalty = backend.activeConnections * 0.8; const healthBoost = (1 - backend.healthScore) * 100; const weightFactor = 1 / backend.weight; return connectionPenalty + healthBoost + weightFactor; }

selectBackend(): BackendState | null { let best: BackendState | null = null; let lowestScore = Infinity;

for (const backend of this.backends.values()) {
  const score = this.calculateScore(backend);
  if (score < lowestScore) {
    lowestScore = score;
    best = backend;
  }
}

if (best) {
  best.activeConnections++;
}
return best;

}

releaseConnection(backendId: string): void { const backend = this.backends.get(backendId); if (backend && backend.activeConnections > 0) { backend.activeConnections--; } }

markDraining(backendId: string): void { const backend = this.backends.get(backendId); if (backend) backend.draining = true; }

destroy(): void { clearInterval(this.healthCheckInterval); } }

### Architecture Decisions & Rationale

- **Application-Level vs Infrastructure-Level**: Edge proxies (Envoy/Nginx) handle L4/L7 termination but lack application context. An adaptive LB at the application layer can read request payload size, expected processing time, and backend health endpoints. The hybrid pattern pairs infra LB for connection acceptance with app-level routing for intelligent distribution.
- **Graded Health Scores**: Binary health checks cause sudden traffic shifts when instances flap. Graded scores (0.0–1.0) enable smooth traffic migration and prevent thundering herd effects during partial degradation.
- **Connection Counting**: Tracking active connections per backend provides real-time load visibility. Combined with health scoring, it prevents routing to instances that are technically "up" but saturated.
- **Drain Support**: Deployments require graceful shutdown. Marking backends as draining removes them from routing while allowing in-flight requests to complete, eliminating 502/503 spikes during rollouts.

## Pitfall Guide

### Common Mistakes

1. **Binary Health Checks**: Treating backends as strictly up/down causes traffic to swing violently when instances flap. Graded health scoring or probabilistic routing prevents sudden load redistribution.
2. **Ignoring Connection Draining**: Rolling deployments without drain support drop in-flight requests. Always implement graceful shutdown hooks and mark instances as draining before removing them from the pool.
3. **Static Weights in Auto-Scaling Environments**: Hardcoded weights fail when instance counts change dynamically. Weights should be calculated based on instance class, CPU/memory limits, or real-time capacity metrics.
4. **Overlooking HTTP/2 Multiplexing**: HTTP/2 reuses connections, making connection counts misleading. When routing over HTTP/2, track active streams or request concurrency instead of raw TCP connections.
5. **Monitoring Only Average Latency**: Averages hide tail latency. P95/P99 metrics expose routing misalignment. Optimize for tail latency, not mean throughput.
6. **Missing Circuit Breaker Integration**: Load balancers that route to degraded backends amplify failures. Pair routing logic with circuit breakers that temporarily remove instances exceeding error thresholds.
7. **Timeout Mismatches**: LB timeouts shorter than backend processing times cause premature connection drops. Align client, LB, and backend timeouts to prevent cascading retries.

### Production Best Practices

- Use health-aware scoring instead of static weights
- Implement connection draining with configurable drain timeouts
- Pair LB with service mesh or sidecar proxies for observability
- Monitor P95/P99 latency, connection variance, and drain success rates
- Align timeout configurations across the entire request path
- Test routing behavior under chaos conditions (instance failure, network partition, traffic spikes)

## Production Bundle

### Action Checklist
- [ ] Replace static routing with health-aware least-connections or adaptive scoring
- [ ] Implement connection tracking per backend instance
- [ ] Add graded health checks with configurable thresholds
- [ ] Configure connection draining for zero-downtime deployments
- [ ] Align LB, application, and database timeouts
- [ ] Integrate circuit breakers to isolate degraded backends
- [ ] Monitor P95/P99 latency and CPU utilization variance
- [ ] Validate routing behavior under simulated backend degradation

### Decision Matrix

| Scenario | Recommended Approach | Why | Cost Impact |
|----------|---------------------|-----|-------------|
| High-variability request processing times | Adaptive (Health-Aware) | Routes to least loaded healthy instance, minimizing tail latency | Reduces overprovisioning by 15-25% |
| Session-bound workloads (e.g., gaming, chat) | Consistent Hashing | Preserves affinity without application-level session stores | Increases memory usage for session replication |
| Predictable, uniform workloads | Least Connections | Simple, low overhead, effective when processing times are consistent | Neutral cost, improves resource utilization |
| Multi-tenant with tiered SLAs | Weighted + Health-Aware Hybrid | Routes premium traffic to high-capacity nodes while maintaining health awareness | Higher infra cost for tiered nodes, improves SLA compliance |

### Configuration Template

```yaml
# Envoy Adaptive Routing Configuration
static_resources:
  listeners:
    - name: main_listener
      address:
        socket_address: { address: 0.0.0.0, port_value: 8080 }
      filter_chains:
        - filters:
            - name: envoy.filters.network.http_connection_manager
              typed_config:
                "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
                stat_prefix: ingress_http
                route_config:
                  name: local_route
                  virtual_hosts:
                    - name: backend
                      domains: ["*"]
                      routes:
                        - match: { prefix: "/" }
                          route:
                            cluster: adaptive_cluster
                            timeout: 10s
                            idle_timeout: 30s
                http_filters:
                  - name: envoy.filters.http.router
                    typed_config:
                      "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router

  clusters:
    - name: adaptive_cluster
      connect_timeout: 2s
      type: STRICT_DNS
      lb_policy: LEAST_REQUEST
      health_checks:
        - timeout: 2s
          interval: 5s
          unhealthy_threshold: 2
          healthy_threshold: 2
          http_health_check:
            path: /health
      circuit_breakers:
        thresholds:
          - priority: DEFAULT
            max_connections: 1024
            max_pending_requests: 1024
            max_requests: 2048
            max_retries: 3
      outlier_detection:
        consecutive_5xx: 5
        interval: 10s
        base_ejection_time: 30s
        max_ejection_percent: 50

Quick Start Guide

1. Deploy Health Endpoints: Expose /health on all backend instances returning 200 when ready, 503 when draining or degraded.
2. Configure Edge Proxy: Apply the Envoy template above or equivalent Nginx/HAProxy config with LEAST_REQUEST and health checks enabled.
3. Integrate Application LB: Instantiate AdaptiveLoadBalancer in your Node.js/TypeScript service, passing backend registry and health check URL.
4. Hook into Request Lifecycle: Call selectBackend() before outbound requests, track active connections, and call releaseConnection() after response completion.
5. Validate Under Load: Run traffic simulation (k6/Artillery) with 20% backend degradation. Verify P99 latency stays under baseline and drain transitions complete without 5xx spikes.