CI/CD pipeline design patterns

ntation

1. Define the Pipeline Schema

We define interfaces for stages, jobs, and gates. This enforces structure and enables IDE autocomplete.

// pipeline-types.ts
export type JobStatus = 'pending' | 'running' | 'success' | 'failed' | 'skipped';

export interface QualityGate {
  name: string;
  condition: (context: PipelineContext) => Promise<boolean>;
  onFail: 'abort' | 'notify' | 'manual-approve';
}

export interface Job {
  id: string;
  name: string;
  command: string[];
  dependsOn?: string[]; // For DAG dependencies
  timeout?: number; // Minutes
  resources?: { cpu: string; memory: string };
}

export interface PipelineStage {
  name: string;
  jobs: Job[];
  gate?: QualityGate;
  parallel?: boolean; // Fan-out toggle
}

export interface PipelineConfig {
  id: string;
  stages: PipelineStage[];
  globalEnv: Record<string, string>;
  retryPolicy?: { maxAttempts: number; backoff: 'linear' | 'exponential' };
}

2. Implement the Fan-out/Fan-in Pattern

The Fan-out pattern allows parallel execution of independent jobs. The Fan-in ensures downstream stages wait for all upstream jobs to complete.

// pipeline-builder.ts
import { PipelineConfig, Job, PipelineStage } from './pipeline-types';

export class PipelineBuilder {
  private config: PipelineConfig;

  constructor(id: string) {
    this.config = {
      id,
      stages: [],
      globalEnv: {},
    };
  }

  addStage(stage: PipelineStage): this {
    // Validation: Check for duplicate job IDs within stage
    const jobIds = stage.jobs.map(j => j.id);
    if (new Set(jobIds).size !== jobIds.length) {
      throw new Error(`Duplicate job IDs detected in stage: ${stage.name}`);
    }
    this.config.stages.push(stage);
    return this;
  }

  // Fan-out implementation: Jobs run in parallel if parallel=true
  // Fan-in is implicit: Next stage waits for current stage completion
  addFanOutStage(name: string, jobs: Job[]): this {
    return this.addStage({
      name,
      jobs,
      parallel: true,
    });
  }

  // Add a Quality Gate that halts pipeline on failure
  addGate(stageName: string, gate: QualityGate): this {
    const stage = this.config.stages.find(s => s.name === stageName);
    if (!stage) throw new Error(`Stage ${stageName} not found`);
    stage.gate = gate;
    return this;
  }

  build(): PipelineConfig {
    // Final validation: Ensure DAG constraints are met
    this.validateDependencies();
    return this.config;
  }

  private validateDependencies(): void {
    // Logic to verify that dependsOn references exist and no cycles exist
    // Omitted for brevity, but critical in production
  }
}

3. Define a Concrete Pipeline

Using the builder, we construct a pipeline that applies the Gatekeeper, Fan-out, and Immutable Artifact patterns.

// my-pipeline.ts
import { PipelineBuilder } from './pipeline-builder';

const pipeline = new PipelineBuilder('app-core-delivery')
  .addStage({
    name: 'Static Analysis',
    jobs: [
      { id: 'lint', name: 'Run ESLint', command: ['npm', 'run', 'lint'] },
      { id: 'audit', name: 'Security Audit', command: ['npm', 'audit', '--json'] },
    ],
    parallel: true, // Fan-out: Lint and Audit run simultaneously
  })
  .addGate('Static Analysis', {
    name: 'Quality Gate',
    condition: async (ctx) => ctx.artifacts['audit'].exitCode === 0,
    onFail: 'abort',
  })
  .addStage({
    name: 'Build & Test',
    jobs: [
      { id: 'build', name: 'Compile', command: ['npm', 'run', 'build'] },
      { id: 'unit', name: 'Unit Tests', command: ['npm', 'test', '--coverage'] },
      { id: 'integration', name: 'Integration Tests', command: ['npm', 'run', 'test:integration'] },
    ],
    parallel: true, // Fan-out: Build, Unit, and Integration run in parallel
    // Note: In a real DAG, 'integration' might depend on 'build' output
    // This requires a more advanced dependency resolver in the builder
  })
  .addStage({
    name: 'Deploy Staging',
    jobs: [
      {
        id: 'deploy',
        name: 'Deploy to Staging',
        command: ['kubectl', 'apply', '-f', 'dist/manifest.yaml'],
        dependsOn: ['build', 'unit', 'integration'], // Fan-in: Wait for all
      },
    ],
  })
  .build();

console.log(JSON.stringify(pipeline, null, 2));

4. Execution Engine Rationale

The compiled JSON is consumed by an orchestration engine (e.g., Argo Workflows, GitHub Actions via a custom dispatcher, or a self-hosted runner). The engine parses the DAG, schedules jobs based on dependencies, and manages state. By separating definition from execution, you gain:

• Portability: Switch CI/CD providers without rewriting logic.
• Simulation: Run pipeline logic locally to debug failures.
• Optimization: The engine can dynamically allocate resources based on job resource hints.

Pitfall Guide

1. The Mutable Artifact Trap

   • Mistake: Rebuilding artifacts in deployment stages instead of passing the binary built in the CI stage.
   • Impact: "It works in CI but fails in Prod" errors due to environment drift or non-deterministic builds.
   • Fix: Hash every artifact at build time. Deployment stages must pull artifacts by hash. Never run build commands in deploy stages.

2. Secret Sprawl and Rotation Latency

   • Mistake: Storing secrets in pipeline variables or environment files without dynamic rotation.
   • Impact: Compromised secrets require manual rotation across all pipelines; long blast radius.
   • Fix: Use short-lived tokens via OIDC federation (e.g., GitHub Actions OIDC to AWS). Secrets should be injected at runtime, not stored.

3. Flaky Tests as Gatekeepers

   • Mistake: Allowing flaky tests to pass intermittently or disabling them to unblock pipelines.
   • Impact: Erosion of trust in the pipeline. Developers ignore failures. Quality degrades silently.
   • Fix: Quarantine flaky tests immediately. They must not block the main pipeline. Use retry logic only for infrastructure flakiness, never for code logic.

4. Monolithic Dependency Graphs

   • Mistake: A single failing test in a library blocks the build of an unrelated service.
   • Impact: Reduced deployment frequency; "build police" mentality.
   • Fix: Implement service-level pipelines with dependency mapping. Only rebuild services affected by library changes. Use change detection to trigger relevant pipelines.

5. Pipeline Configuration Drift

   • Mistake: Modifying pipeline settings via the CI/CD UI instead of code.
   • Impact: Configuration is not versioned, not peer-reviewed, and lost on repo deletion.
   • Fix: Enforce "Pipeline as Code" policies. Disable UI edits for production pipelines. All changes must go through PRs.

6. Over-Parallelization and Rate Limits

   • Mistake: Fan-out too aggressively without considering external rate limits (e.g., Docker Hub, npm registry).
   • Impact: Pipeline fails due to 429 errors; noisy neighbor issues in shared runners.
   • Fix: Implement concurrency limits and retry/backoff strategies. Use private artifact registries to avoid public rate limits.

7. Lack of Rollback Automation

   • Mistake: Pipeline handles deployment but lacks a defined rollback path.
   • Impact: MTTR increases significantly during incidents; manual intervention required.
   • Fix: Every deployment job must have a corresponding rollback job. Implement automated rollback triggers based on health checks (e.g., if error rate > 1% for 5 mins, rollback).

Production Bundle

Action Checklist

• Audit Pipeline Topology: Map current pipelines to identify serial bottlenecks and opportunities for Fan-out parallelization.
• Enforce Artifact Immutability: Modify build stages to output hashed artifacts and update deploy stages to consume by reference only.
• Implement Quality Gates: Add pass/fail criteria for security scans, test coverage thresholds, and lint checks before deployment stages.
• Containerize Build Environments: Ensure all pipeline jobs run in defined container images to eliminate environment drift.
• Establish Rollback Procedures: Define automated rollback jobs and configure health-check-based triggers for automatic reversion.
• Adopt OIDC for Secrets: Replace long-lived credentials with short-lived tokens using OpenID Connect federation.
• Monitor DORA Metrics: Instrument pipelines to report deployment frequency, lead time, change failure rate, and MTTR.
• Quarantine Flaky Tests: Identify tests with >2% failure variance and move them to a separate validation stream.

Decision Matrix

Scenario	Recommended Approach	Why	Cost Impact
Startup / Solo Dev	Modular Linear	Simplicity outweighs parallelization benefits; low maintenance overhead.	Low
Mid-Size / Monolith	Fan-out (Jobs)	Parallelize lint/test/build to reduce feedback time without complex DAG management.	Medium
Enterprise / Microservices	Pipeline-as-Graph	Handles complex dependencies, change detection, and isolation across 50+ services.	High (Ops)
Regulated Industry	Gate-Heavy / Manual	Compliance requires explicit approval steps and audit trails; speed is secondary.	Medium
High-Churn Feature Teams	Branch-by-Abstraction	Deploy incomplete features behind toggles to maintain pipeline velocity and reduce merge conflicts.	Low

Configuration Template

Ready-to-use TypeScript configuration for a Fan-out/Fan-in pipeline with gates.

// production-pipeline.template.ts
import { PipelineBuilder } from './pipeline-builder';

export const createProductionPipeline = (appName: string) => {
  return new PipelineBuilder(`${appName}-prod`)
    .addStage({
      name: 'Validation',
      jobs: [
        { id: 'lint', name: 'Lint', command: ['npm', 'run', 'lint'] },
        { id: 'test', name: 'Test', command: ['npm', 'run', 'test:ci'] },
        { id: 'scan', name: 'Scan', command: ['npm', 'run', 'security:scan'] },
      ],
      parallel: true,
    })
    .addGate('Validation', {
      name: 'Security & Quality',
      condition: async (ctx) => {
        const lintOk = ctx.jobs['lint'].status === 'success';
        const testOk = ctx.jobs['test'].status === 'success';
        const scanOk = ctx.jobs['scan'].exitCode === 0;
        return lintOk && testOk && scanOk;
      },
      onFail: 'abort',
    })
    .addStage({
      name: 'Build',
      jobs: [
        { id: 'build', name: 'Build App', command: ['npm', 'run', 'build'] },
      ],
    })
    .addStage({
      name: 'Deploy',
      jobs: [
        {
          id: 'deploy',
          name: 'Deploy to Prod',
          command: ['deploy-tool', 'push', '--artifact', 'build/dist'],
          dependsOn: ['build'],
        },
      ],
    })
    .build();
};

Quick Start Guide

1. Initialize Pipeline Repo: Create a ci/ directory in your repository. Install the pipeline builder library or copy the TypeScript interfaces.
2. Define Stages: Create pipeline.ts and instantiate PipelineBuilder. Add your lint, test, and build stages. Enable parallel: true for independent jobs.
3. Add Gates: Implement a quality gate checking test results and security scan exit codes. Set onFail to abort.
4. Generate Config: Run npx ts-node pipeline.ts > pipeline-config.json. This generates the execution graph.
5. Hook to Runner: Configure your CI/CD runner to consume pipeline-config.json. For GitHub Actions, use a workflow that calls a custom action to parse and execute the JSON DAG. Verify execution by pushing a commit.