Kubernetes deployment patterns

/client'; import { KubeConfig, AppsV1Api } from '@kubernetes/client-node';

export class RolloutValidator { private prometheus: PrometheusAdapter; private k8s: AppsV1Api;

constructor() { const kc = new KubeConfig(); kc.loadFromCluster(); this.k8s = kc.makeApiClient(AppsV1Api); this.prometheus = new PrometheusAdapter({ baseURL: process.env.PROMETHEUS_URL }); }

async validateCanary(namespace: string, rolloutName: string): Promise<boolean> { // Fetch error rate for canary pods over last 5 minutes const query = sum(rate(http_requests_total{namespace="${namespace}", rollout="${rolloutName}-canary", status=~"5.."}[5m])) / sum(rate(http_requests_total{namespace="${namespace}", rollout="${rolloutName}-canary"}[5m]));

const result = await this.prometheus.queryRange({
  query,
  start: Date.now() / 1000 - 300,
  end: Date.now() / 1000,
  step: 30
});

const errorRate = result.data.result[0]?.values?.at(-1)?.[1] ?? 0;
const threshold = 0.02; // 2% max error rate

if (errorRate > threshold) {
  console.warn(`Canary validation failed: error rate ${errorRate} > ${threshold}`);
  await this.triggerRollback(namespace, rolloutName);
  return false;
}

console.log(`Canary validated: error rate ${errorRate}`);
return true;

}

private async triggerRollback(namespace: string, name: string): Promise<void> { const body = { spec: { paused: true } }; await this.k8s.patchNamespacedRollout(name, namespace, body, undefined, undefined, undefined, undefined, { headers: { 'Content-Type': 'application/strategic-merge-patch+json' } }); } }

### Architecture Decisions & Rationale
- **CRD over Deployment**: `Rollout` maintains separate stable/canary service selectors, enabling traffic controllers (ALB, Istio, Nginx) to route based on service endpoints rather than pod labels.
- **Pause Steps**: Fixed-duration pauses prevent rapid promotion before metrics stabilize. Production systems require 60–120s windows for APM data to propagate.
- **Analysis Templates**: Tying promotion to `AnalysisTemplate` resources externalizes metric definitions, allowing reuse across services and version control.
- **TypeScript Validation Layer**: K8s controllers operate on state reconciliation, not time-series evaluation. A lightweight TS/Node process bridges Prometheus metrics to rollout state, enabling SLO-gated promotion without bloating the control plane.

## Pitfall Guide

1. **Treating `replicas: 1` as a canary**  
   A single pod does not isolate traffic. Without a dedicated canary service and routing controller, requests still hit the new binary based on kube-proxy round-robin. Result: unpredictable failure distribution.

2. **Missing or misconfigured readiness probes**  
   If readiness probes return `200 OK` before application initialization completes, the controller marks pods as ready and shifts traffic to unready instances. Always validate downstream dependencies (DB pools, cache connections, auth tokens) in readiness checks.

3. **Ignoring PodDisruptionBudgets during traffic shifts**  
   PDBs protect against voluntary disruptions. During canary analysis, the controller may evict old replicas to match target weights. Without PDBs, this causes simultaneous pod churn and capacity drops. Define `minAvailable: 2` or `maxUnavailable: 1` per service.

4. **Hardcoding image digests without tag immutability**  
   Using `:latest` or mutable tags breaks rollback determinism. If a tag is overwritten, the previous SHA is unrecoverable. Enforce immutable tags or SHA256 digests in CI/CD. Store tag-to-digest mappings in a registry manifest.

5. **Relying on manual promotion without metric thresholds**  
   Human-driven canary promotion introduces latency and inconsistency. Operators promote too early (missed errors) or too late (wasted canary capacity). Bind `setWeight` steps to `AnalysisTemplate` thresholds that evaluate latency, error rate, and saturation.

6. **Mixing service mesh and ingress controller responsibilities**  
   Istio, Linkerd, and cloud ALBs all support traffic splitting. Running multiple routing layers creates conflicting weight assignments and header routing loops. Choose one traffic control plane and route all progressive delivery through it.

7. **Not testing rollback paths in staging**  
   Rollbacks fail when configuration drift, secret rotation, or database migrations are not reversible. Validate rollback procedures by simulating canary failure in staging. Ensure down migrations are idempotent and feature flags can disable new behavior without redeployment.

**Production Best Practices**:
- Define SLOs per service before deployment automation.
- Use canary analysis automation, not timer-only steps.
- Enforce image signing (Cosign/Notary) and SBOM generation.
- Run chaos engineering tests on deployment pipelines quarterly.
- Separate control plane (Argo/Flux) from data plane (Ingress/Mesh).

## Production Bundle

### Action Checklist
- [ ] Replace native `Deployment` with `Rollout` CRD across production namespaces
- [ ] Configure dual services (`stable-svc`, `canary-svc`) with explicit role selectors
- [ ] Define readiness probes that validate external dependencies, not just HTTP 200
- [ ] Implement `PodDisruptionBudget` with `minAvailable` matching your SLO capacity
- [ ] Create `AnalysisTemplate` resources bound to Prometheus/Grafana metrics
- [ ] Integrate TypeScript/Node validation hooks for SLO-gated promotion in CI/CD
- [ ] Test rollback paths in staging with simulated canary failure and migration reversal
- [ ] Audit routing controllers to ensure single source of truth for traffic weights

### Decision Matrix

| Scenario | Recommended Approach | Why | Cost Impact |
|----------|---------------------|-----|-------------|
| Low-risk internal tooling | RollingUpdate + PDB | Simplicity outweighs traffic control needs; failure impact is contained | Low infra overhead, minimal CI/CD complexity |
| High-traffic customer API | Progressive Canary + ALB/Istio | Requires sub-minute rollback, dynamic weight shifting, and SLO-gated promotion | Moderate control plane cost, high reliability ROI |
| Compliance/financial workloads | Blue/Green + Immutable Audit | Binary state changes simplify compliance verification; traffic split is less critical than deterministic rollback | High infra duplication cost, low incident risk |
| AI/ML inference endpoints | Canary with latency/error thresholds | Model drift and GPU saturation require metric-driven promotion, not replica scaling | GPU cost scales with canary weight, but prevents silent accuracy degradation |

### Configuration Template
Copy this bundle to implement progressive delivery with metric analysis. Adjust thresholds to match your SLOs.

```yaml
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: error-rate-check
spec:
  metrics:
  - name: error-rate
    interval: 30s
    failureLimit: 2
    successCondition: result[0] < 0.02
    provider:
      prometheus:
        address: http://prometheus.monitoring:9090
        query: |
          sum(rate(http_requests_total{namespace="{{namespace}}", status=~"5.."}[2m]))
          /
          sum(rate(http_requests_total{namespace="{{namespace}}"}[2m]))
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: api-ingress
  annotations:
    alb.ingress.kubernetes.io/actions.canary-routing: |
      {
        "Type": "forward",
        "ForwardConfig": {
          "TargetGroups": [
            {"ServiceName": "stable-svc", "ServicePort": "80", "Weight": 90},
            {"ServiceName": "canary-svc", "ServicePort": "80", "Weight": 10}
          ]
        }
      }
spec:
  ingressClassName: alb
  rules:
  - host: api.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: canary-routing
            port:
              name: use-annotation

Quick Start Guide

1. Install Argo Rollouts: kubectl apply -f https://github.com/argoproj/argo-rollouts/releases/latest/download/install.yaml
2. Apply the Rollout and dual-service manifests to your namespace
3. Verify controller recognition: kubectl get rollout api-service -w
4. Update the image field in the Rollout spec and commit. The controller will create canary replicas, shift traffic to 10%, pause for 60s, and evaluate metrics before proceeding.
5. Monitor promotion: kubectl argo rollouts get rollout api-service or use the Argo Rollouts dashboard. Trigger manual rollback with kubectl argo rollouts abort rollout api-service if metrics breach thresholds.