uses: github/codeql-action/upload-sarif@v2
    with:
      sarif_file: 'trivy-results.sarif'

integration-tests: needs: build runs-on: ubuntu-latest services: postgres: image: postgres:15 env: POSTGRES_PASSWORD: test ports: ['5432:5432'] steps: - uses: actions/checkout@v4 - name: Run Integration Tests run: | docker run --network host
-e DATABASE_URL=postgres://postgres:test@localhost:5432/testdb
${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ github.sha }}
npm run test:integration

Stage 4: Deploy with Approval

deploy-staging: needs: [security, integration-tests] if: github.ref == 'refs/heads/main' environment: staging runs-on: ubuntu-latest steps: - name: Deploy to Staging run: | kubectl set image deployment/app
app=${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}@${{ needs.build.outputs.image-digest }}

deploy-production: needs: deploy-staging if: github.ref == 'refs/heads/main' environment: production runs-on: ubuntu-latest steps: - name: Deploy to Production run: | kubectl set image deployment/app
app=${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}@${{ needs.build.outputs.image-digest }}

### Architecture Decisions

| Decision | Recommendation | Rationale |
| :--- | :--- | :--- |
| **Pipeline Definition** | **Declarative (YAML)** | Version-controlled, auditable, and enables code review for delivery logic. |
| **Runner Architecture** | **Ephemeral & Isolated** | Prevents state pollution and ensures security compliance. Use container-based runners. |
| **Artifact Storage** | **Registry + Content Hash** | Ensures reproducibility. The SHA256 digest guarantees the binary tested is the binary deployed. |
| **Secrets Management** | **Dynamic Injection** | Never store secrets in pipeline config. Use OIDC federation for cloud providers to eliminate long-lived credentials. |
| **Feedback Loop** | **Failure Fast** | Order jobs by probability of failure. Lint and unit tests must precede expensive integration tests. |

---

## Pitfall Guide

### 1. Monolithic Pipeline Execution
**Mistake:** Running all tests in a single job or sequential job without parallelism.
**Impact:** Linear scaling of duration with test suite growth. Developers bypass pipelines due to wait times.
**Fix:** Matrix strategies and job dependency graphs.

### 2. Cache Poisoning and Staleness
**Mistake:** Caching without proper key invalidation or sharing caches across branches.
**Impact:** Builds succeed locally but fail in CI due to stale dependencies, or vice versa. Security vulnerabilities persist in cached layers.
**Fix:** Use content-hash keys (e.g., `lockfile-hash`). Scope caches to branch/PR where necessary.

### 3. Flaky Tests as Gatekeepers
**Mistake:** Allowing non-deterministic tests to block the pipeline.
**Impact:** "Boy who cried wolf" syndrome. Teams ignore pipeline failures, leading to undetected regressions.
**Fix:** Quarantine flaky tests immediately. Implement retry logic only for known infrastructure issues, not code defects.

### 4. Hardcoded Configuration
**Mistake:** Embedding environment variables, URLs, or feature flags directly in the pipeline script.
**Impact:** Pipeline fragility and inability to reuse logic across environments.
**Fix:** Externalize configuration. Use environment-specific variable sets and template rendering.

### 5. Lack of Rollback Automation
**Mistake:** Focusing solely on deployment without a defined rollback mechanism.
**Impact:** Extended MTTR when deployments fail. Manual intervention required during incidents.
**Fix:** Implement automatic rollback triggers based on health check failures or metric thresholds (e.g., error rate > 1%).

### 6. Security as an Afterthought
**Mistake:** Running SAST/DAST only on the `main` branch.
**Impact:** Vulnerabilities are discovered late, requiring expensive rework.
**Fix:** Shift left. Run lightweight SAST on PRs. Run full DAST/SBOM generation in the build stage.

### 7. Runner Resource Starvation
**Mistake:** Under-provisioning runners or sharing runners between CI and CD.
**Impact:** Queue buildup and delayed deployments. Noisy neighbor issues.
**Fix:** Separate runner pools for CI (high concurrency, short-lived) and CD (long-lived, stateful deployments).

---

## Production Bundle

### Action Checklist

- [ ] **Audit Pipeline Duration:** Identify the critical path. Measure time spent in queue vs. execution.
- [ ] **Implement Caching:** Add dependency caching using lockfile hashes. Verify cache hit rates.
- [ ] **Parallelize Independent Jobs:** Refactor sequential jobs into a matrix or parallel graph.
- [ ] **Enable OIDC:** Remove static cloud credentials. Configure Workload Identity Federation.
- [ ] **Add Concurrency Controls:** Configure `cancel-in-progress` to save resources on redundant pushes.
- [ ] **Define Rollback Strategy:** Script automated rollback based on deployment health checks.
- [ ] **Secure Secrets:** Rotate all pipeline secrets. Ensure no secrets are logged or exposed in artifacts.
- [ ] **Instrument Metrics:** Track Build Success Rate, Lead Time for Changes, and MTTR in a dashboard.

### Decision Matrix: Pipeline Tooling

| Criteria | GitHub Actions | GitLab CI | Jenkins | ArgoCD (GitOps) |
| :--- | :--- | :--- | :--- | :--- |
| **Best For** | General purpose, Ecosystem | All-in-one DevSecOps | Legacy, Complex On-prem | Kubernetes Deployments |
| **Scalability** | High (Hosted/Runners) | High (Shared/Runners) | Medium (Controller bottleneck) | High (Agentless) |
| **Learning Curve** | Low | Medium | High | Medium |
| **Cost** | Pay-per-minute | Included/Runners | Infrastructure only | Infrastructure only |
| **Security** | OIDC, Secrets, Dependabot | SAST/DAST built-in | Plugin dependent | RBAC, Policy as Code |
| **Recommendation** | **Default for new repos** | **Teams using GitLab** | **Enterprise legacy** | **K8s CD layer** |

### Configuration Template: Production-Ready Pipeline

Copy this template for a robust, secure, and optimized pipeline structure. Adapt variables to your stack.

```yaml
# .github/workflows/pipeline.yml
# Production Template: Graph-Optimized, Secure, Immutable

name: Production Pipeline

on:
  workflow_dispatch:
  push:
    branches: [main]
  pull_request:
    branches: [main]

permissions:
  contents: read
  packages: write
  id-token: write # Required for OIDC

concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true

env:
  NODE_VERSION: '20'
  REGISTRY: ghcr.io
  IMAGE_NAME: ${{ github.repository }}

jobs:
  setup:
    runs-on: ubuntu-latest
    outputs:
      cache-key: ${{ steps.cache-key.outputs.key }}
    steps:
      - uses: actions/checkout@v4
      - id: cache-key
        run: echo "key=${{ hashFiles('**/package-lock.json') }}" >> $GITHUB_OUTPUT

  build:
    needs: setup
    runs-on: ubuntu-latest
    outputs:
      digest: ${{ steps.push.outputs.digest }}
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: ${{ env.NODE_VERSION }}
          cache: 'npm'
      
      - run: npm ci
      - run: npm run build
      
      - name: Build Image
        id: push
        uses: docker/build-push-action@v5
        with:
          context: .
          push: true
          tags: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ github.sha }}
          cache-from: type=gha
          cache-to: type=gha,mode=max

  test:
    needs: [setup, build]
    runs-on: ubuntu-latest
    strategy:
      matrix:
        suite: [unit, integration]
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: ${{ env.NODE_VERSION }}
          cache: 'npm'
      - run: npm ci
      - run: npm run test:${{ matrix.suite }}

  security:
    needs: build
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Scan Image
        uses: aquasecurity/trivy-action@master
        with:
          image-ref: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ github.sha }}
          severity: CRITICAL,HIGH
          exit-code: 1

  deploy-staging:
    needs: [test, security]
    if: github.event_name == 'push'
    environment: staging
    runs-on: ubuntu-latest
    steps:
      - name: Configure AWS Credentials
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: ${{ secrets.AWS_ROLE_ARN }}
          aws-region: us-east-1
      
      - name: Update ECS Service
        run: |
          aws ecs update-service \
            --cluster production-cluster \
            --service my-service \
            --force-new-deployment \
            --desired-count 1

Quick Start Guide

1. Initialize: Create .github/workflows/ci.yml using the Configuration Template above.
2. Configure Secrets: Add AWS_ROLE_ARN (or equivalent OIDC role) and REGISTRY credentials to your repository settings.
3. Run First Build: Push a change. Verify the pipeline executes the setup, build, test, and security stages in parallel where possible.
4. Add Caching: Confirm cache hit rates in the workflow logs. Adjust cache keys if misses occur.
5. Enable Environments: Set up staging and production environments in repository settings. Add required reviewers for production if needed.
6. Monitor: Set up a dashboard tracking pipeline duration and success rate. Alert on duration regressions > 20%.

Codcompass Note: Pipeline design is iterative. Treat your pipeline code with the same rigor as application code. Review PRs for pipeline changes, refactor for performance, and continuously measure delivery metrics.