I built a self-hosted CI/CD platform with persistent queue, encrypted secrets, and rollback UI — here's what I learned

On server startup, the system queries the database for deployments in a QUEUED state that lack a corresponding active Redis job. These are re-enqueued to prevent orphaned states.

• Retry Policy: Failed deployments trigger an exponential backoff strategy (e.g., 1s, 5s, 25s) with a maximum of three attempts.
• Health Guard: A middleware layer intercepts API requests and returns a 503 Service Unavailable if the Redis connection is unhealthy, preventing silent request drops.

// interfaces/queue.interface.ts
export interface IQueueAdapter {
  enqueue(jobId: string, payload: DeploymentPayload): Promise<void>;
  recoverOrphanedJobs(): Promise<number>;
  isHealthy(): Promise<boolean>;
}

// services/deployment-queue-manager.ts
import { Queue, Job } from 'bullmq';
import { DeploymentRepository } from '../repositories/deployment.repository';

export class DeploymentQueueManager implements IQueueAdapter {
  private queue: Queue;
  private retryDelays = [1000, 5000, 25000];

  constructor(redisConfig: RedisConfig) {
    this.queue = new Queue('deployments', {
      connection: redisConfig,
      defaultJobOptions: {
        attempts: 3,
        backoff: { type: 'custom', delay: 0 }, // Custom backoff logic
      },
    });
  }

  async enqueue(jobId: string, payload: DeploymentPayload): Promise<void> {
    await this.queue.add(jobId, payload, {
      backoff: {
        type: 'custom',
        delay: this.retryDelays[0],
      },
    });
  }

  async recoverOrphanedJobs(): Promise<number> {
    const orphans = await DeploymentRepository.findQueuedWithoutActiveJob();
    let recovered = 0;
    for (const job of orphans) {
      await this.enqueue(job.id, job.payload);
      recovered++;
    }
    return recovered;
  }

  async isHealthy(): Promise<boolean> {
    try {
      await this.queue.client.ping();
      return true;
    } catch {
      return false;
    }
  }
}

2. Row-Level Secret Encryption

Secrets must never be stored in plaintext. The orchestrator employs AES-256-GCM encryption with a unique Initialization Vector (IV) for every secret row. This ensures that even if two secrets have identical values, their ciphertext differs. Decryption occurs only at deployment time, and values are redacted from all logs.

Key Implementation Details:

• Key Management: The master encryption key is a 64-character hex string stored in the environment. Rotation requires a script to re-encrypt all existing rows under the new key.
• Redaction: A logging middleware scans output for known secret names and replaces values with [REDACTED].

// services/secret-vault.ts
import crypto from 'crypto';
import { SecretRepository } from '../repositories/secret.repository';

export class SecretVault {
  private masterKey: Buffer;

  constructor(envKey: string) {
    this.masterKey = Buffer.from(envKey, 'hex');
  }

  encrypt(plaintext: string): { ciphertext: string; iv: string; authTag: string } {
    const iv = crypto.randomBytes(12);
    const cipher = crypto.createCipheriv('aes-256-gcm', this.masterKey, iv);
    
    let ciphertext = cipher.update(plaintext, 'utf8', 'hex');
    ciphertext += cipher.final('hex');
    const authTag = cipher.getAuthTag().toString('hex');

    return { ciphertext, iv: iv.toString('hex'), authTag };
  }

  decrypt(record: EncryptedSecretRecord): string {
    const decipher = crypto.createDecipheriv(
      'aes-256-gcm',
      this.masterKey,
      Buffer.from(record.iv, 'hex')
    );
    decipher.setAuthTag(Buffer.from(record.authTag, 'hex'));

    let plaintext = decipher.update(record.ciphertext, 'hex', 'utf8');
    plaintext += decipher.final('utf8');
    return plaintext;
  }
}

3. Granular RBAC with Project Membership

Access control is implemented via a two-layer model. System-wide roles define baseline permissions, while project memberships grant specific access.

• System Roles: Admin, Manager, Developer, Viewer.
• Project Roles: Owner, Member.
• Enforcement: A Developer can only trigger deployments for projects where they hold Owner or Member status. A Viewer has read-only access to logs and configurations.

4. Decoupled Notification Fan-out

The notification system avoids hard-coded integrations by using a Provider/Channel/Subscription architecture.

• Provider: Defines credentials and delivery logic (e.g., Discord, SMTP).
• Channel: Represents a specific target within a provider (e.g., a Discord channel ID).
• Subscription: An M:N mapping linking projects and event types to channels.

This design allows adding new providers without modifying the core project model. Fan-out uses Promise.allSettled to ensure a failure in one channel does not block notifications to others.

// services/alert-dispatcher.ts
import { AlertProviderRegistry } from './alert-provider-registry';

export class AlertDispatcher {
  constructor(private registry: AlertProviderRegistry) {}

  async dispatch(event: DeploymentEvent, subscriptions: Subscription[]): Promise<void> {
    const tasks = subscriptions.map(async (sub) => {
      const provider = this.registry.getProvider(sub.providerType);
      try {
        await provider.send(sub.channelId, event);
      } catch (error) {
        // Log failure with context; do not throw
        console.error(`Notification failed for channel ${sub.channelId}:`, error);
      }
    });

    await Promise.allSettled(tasks);
  }
}

Pitfall Guide

1. In-Memory Queue State Loss

   • Explanation: Using in-memory arrays or local variables for job queues results in lost deployments if the process restarts.
   • Fix: Always use an external persistence layer like Redis. Implement boot-time recovery to re-enqueue jobs found in the database but missing from the queue.

2. Shared Encryption Key Rotation Neglect

   • Explanation: Failing to rotate encryption keys or lacking a rotation script leaves secrets vulnerable if the key is compromised.
   • Fix: Implement a key rotation command that reads all secrets, decrypts them with the old key, and re-encrypts them with the new key. Store key version metadata to support gradual migration.

3. Blocking Notification Failures

   • Explanation: If a notification provider throws an error and the deployment process awaits it synchronously, a webhook failure can block the deployment pipeline.
   • Fix: Use Promise.allSettled or fire-and-forget patterns for notifications. Log failures for debugging but never let them halt the core deployment flow.

4. Log Leakage of Sensitive Data

   • Explanation: Environment variables or secrets may inadvertently appear in stdout logs during deployment execution.
   • Fix: Implement a log redaction middleware that matches secret names and replaces their values with placeholders before writing to the log store.

5. ORM Migration Drift

   • Explanation: Relying on auto-migration or manual schema changes leads to inconsistencies between development and production databases.
   • Fix: Use a strict migration tool. Version all schema changes and apply them via CI/CD pipelines. Consider lightweight alternatives like Drizzle or Kysely if Sequelize migrations become cumbersome.

6. Real-Time Complexity Overhead

   • Explanation: Implementing WebSockets for simple log streaming adds connection management overhead and complexity.
   • Fix: Evaluate Server-Sent Events (SSE) for one-way streams like logs. SSE is simpler, works over standard HTTP, and requires less infrastructure than WebSocket management.

7. Queue Backpressure Ignorance

   • Explanation: Allowing unlimited concurrent deployments can exhaust server resources.
   • Fix: Configure concurrency limits in the queue manager. Implement a backlog limit to reject new jobs when the queue exceeds a threshold, returning a clear error to the user.

Production Bundle

Action Checklist

• Configure Redis Persistence: Enable AOF or RDB persistence in Redis to prevent queue data loss during Redis restarts.
• Generate Master Key: Create a cryptographically secure 64-character hex string for the secret encryption key and store it in the environment.
• Define RBAC Policies: Map out required system roles and project permissions in the configuration file before onboarding users.
• Test Queue Recovery: Simulate a server crash mid-deployment and verify that the job is recovered and resumed upon restart.
• Verify Log Redaction: Trigger a deployment with known secrets and inspect logs to ensure no sensitive values are exposed.
• Set Up Notification Providers: Configure at least one provider (e.g., Discord or Email) and test the fan-out mechanism with multiple channels.
• Implement Health Checks: Ensure the /health endpoint reports queue and database connectivity status for load balancers.

Decision Matrix

Scenario	Recommended Approach	Why	Cost Impact
Small Team (<10 devs)	Self-Hosted Orchestrator	Low maintenance, full control, sufficient governance.	Low (Single server + Redis)
Regulated Industry	Enterprise CI/CD + Vault	Requires audit compliance, SSO, and advanced secret management.	High (Licensing + Ops)
High Scale (>50 deploys/hr)	Distributed Orchestrator	Needs horizontal scaling and load balancing.	Medium (Cluster infrastructure)
Prototype/POC	Ad-hoc Scripts	Fastest setup, no tooling overhead.	Low (Risk of technical debt)

Configuration Template

# config/production.yaml
server:
  port: 3000
  host: 0.0.0.0

database:
  dialect: mysql
  host: ${DB_HOST}
  port: 3306
  username: ${DB_USER}
  password: ${DB_PASSWORD}
  database: deploy_orchestrator

redis:
  host: ${REDIS_HOST}
  port: 6379
  password: ${REDIS_PASSWORD}
  tls: true

security:
  encryption_key: ${ENCRYPTION_KEY}
  session_secret: ${SESSION_SECRET}

queue:
  max_concurrency: 5
  retry_attempts: 3
  retry_delays: [1000, 5000, 25000]

logging:
  level: info
  redact_secrets: true
  format: json

Quick Start Guide

1. Initialize Environment: Copy .env.example to .env and populate database credentials, Redis connection details, and generate the encryption key.
2. Start Infrastructure: Run docker-compose up -d to launch MySQL, Redis, and the application container.
3. Run Migrations: Execute npm run db:migrate to create the schema and seed default roles.
4. Access UI: Navigate to http://localhost:3000 and log in with the default admin credentials.
5. Create Project: Define a new project, add environment secrets, and configure a deployment target to begin orchestrating deployments.