automation.config.yaml

ectural pillars: a reliable event bus, an adapter pattern for external APIs, and async job processing with observability. The following implementation uses TypeScript, BullMQ (Redis-backed queue), and a modular adapter interface.

Step 1: Define the Adapter Interface

External APIs change frequently. An adapter pattern isolates your core logic from provider-specific implementations.

// adapters/base.ts
export interface AutomationAdapter<TInput, TOutput> {
  readonly provider: string;
  execute(input: TInput): Promise<TOutput>;
  validate(input: TInput): boolean;
  getRateLimit(): { maxRequests: number; windowMs: number };
}

Step 2: Implement a Concrete Adapter

Example: Stripe invoice sync adapter with idempotency and rate limiting.

// adapters/stripe-invoice.ts
import Stripe from 'stripe';
import { AutomationAdapter } from './base';

export interface InvoiceInput {
  customerId: string;
  amount: number;
  currency: string;
  idempotencyKey: string;
}

export interface InvoiceOutput {
  stripeInvoiceId: string;
  status: 'pending' | 'paid' | 'void';
  hostedUrl: string | null;
}

export class StripeInvoiceAdapter implements AutomationAdapter<InvoiceInput, InvoiceOutput> {
  readonly provider = 'stripe';
  private client: Stripe;

  constructor(apiKey: string) {
    this.client = new Stripe(apiKey, { apiVersion: '2023-10-16' });
  }

  validate(input: InvoiceInput): boolean {
    return input.customerId.startsWith('cus_') && input.amount > 0;
  }

  getRateLimit() {
    return { maxRequests: 100, windowMs: 1000 }; // Stripe default
  }

  async execute(input: InvoiceInput): Promise<InvoiceOutput> {
    if (!this.validate(input)) throw new Error('Invalid invoice payload');

    const invoice = await this.client.invoices.create({
      customer: input.customerId,
      auto_advance: true,
      collection_method: 'charge_automatically',
      default_payment_method: null,
    }, { idempotencyKey: input.idempotencyKey });

    return {
      stripeInvoiceId: invoice.id,
      status: invoice.status,
      hostedUrl: invoice.hosted_invoice_url,
    };
  }
}

Step 3: Build the Event Router & Queue Worker

Webhooks arrive unpredictably. Normalize, enqueue, and process asynchronously.

// workers/automation-worker.ts
import { Queue, Worker } from 'bullmq';
import IORedis from 'ioredis';
import { StripeInvoiceAdapter, InvoiceInput, InvoiceOutput } from '../adapters/stripe-invoice';

const redisConnection = new IORedis({ maxRetriesPerRequest: null });

const automationQueue = new Queue('solopreneur-automation', {
  connection: redisConnection,
  defaultJobOptions: {
    attempts: 3,
    backoff: { type: 'exponential', delay: 2000 },
    removeOnComplete: 100,
    removeOnFail: 50,
  },
});

const stripeAdapter = new StripeInvoiceAdapter(process.env.STRIPE_SECRET_KEY!);

const worker = new Worker('solopreneur-automation', async (job) => {
  const { type, payload, correlationId } = job.data;

  console.log(`[${correlationId}] Processing ${type}`);

  switch (type) {
    case 'CREATE_STRIPE_INVOICE':
      return await stripeAdapter.execute(payload as InvoiceInput);
    default:
      throw new Error(`Unknown workflow type: ${type}`);
  }
}, { connection: redisConnection, concurrency: 5 });

export { automationQueue, worker };

Step 4: Webhook Ingestion Endpoint

Express/Fastify router to receive, validate, and enqueue.

// routes/webhooks.ts
import { Router } from 'express';
import { v4 as uuidv4 } from 'uuid';
import { automationQueue } from '../workers/automation-worker';

const router = Router();

router.post('/ingest', async (req, res) => {
  const correlationId = uuidv4();
  const { event, payload } = req.body;

  if (!event || !payload) {
    return res.status(400).json({ error: 'Missing event or payload' });
  }

  await automationQueue.add('automation-job', {
    type: event,
    payload,
    correlationId,
    timestamp: new Date().toISOString(),
  }, { jobId: `${event}-${payload.idempotencyKey || uuidv4()}` });

  res.status(202).json({ status: 'queued', correlationId });
});

export default router;

Architecture Decisions & Rationale

• BullMQ over cron/sync processing: Webhooks and API rate limits demand async handling. BullMQ provides reliable persistence, exponential backoff, concurrency control, and dead-letter queue visibility. Cron jobs poll unnecessarily and fail silently on transient errors.
• TypeScript strict mode: Cross-API payloads mutate frequently. TypeScript enforces contract stability at compile time, preventing runtime type mismatches when mapping CRM fields to billing systems.
• Adapter pattern over direct SDK calls: Swapping providers (e.g., Stripe → Paddle) requires only implementing the same interface. Core workflow logic remains untouched.
• Idempotency keys: Financial and CRM operations must not duplicate on retry. Job IDs and API idempotency keys prevent double-charging or duplicate records.
• Trade-off: Initial setup requires Redis, queue configuration, and type definitions. Long-term, this reduces vendor lock-in, eliminates polling costs, and provides audit trails unavailable in no-code platforms.

Pitfall Guide

1. Ignoring idempotency: Retries on failed webhooks or network blips will duplicate invoices, contacts, or tasks. Always pass idempotency keys to APIs and use deterministic job IDs in the queue.
2. Synchronous webhook processing: Returning 200 OK only after external API calls complete causes gateway timeouts. Acknowledge immediately, enqueue, and process asynchronously.
3. Hardcoding secrets or rate limits: API keys rotate; rate limits change. Store secrets in a vault or environment manager. Implement dynamic rate limiters that respect X-RateLimit-Remaining headers.
4. Circular workflow dependencies: Workflow A triggers B, B triggers A. This creates infinite loops that exhaust queue memory. Implement directed acyclic graph (DAG) validation or correlation ID tracking to detect cycles.
5. Skipping dead-letter queues (DLQ): Failed jobs vanish into logs. Configure DLQ routing with alerting. Inspect DLQ payloads weekly to identify schema drift or provider outages.
6. Over-nesting transformations: Chaining 5+ adapters in a single job increases blast radius. Decompose into micro-workflows with explicit state transitions. Use correlation IDs to track end-to-end flow.
7. Treating no-code as production: Visual builders lack version control, testing, and rollback capabilities. Export logic to code, add unit tests for payload mapping, and CI/CD the automation layer like any other service.

Best practices from production:

• Implement circuit breakers for external APIs (e.g., opossum or cockatiel).
• Use structured logging with correlation IDs for traceability.
• Run dry-mode validation before executing financial or destructive actions.
• Monitor queue depth, processing latency, and failure rates with Prometheus/Grafana or OpenTelemetry.
• Version your adapter contracts. Deprecate old payload shapes gracefully.

Production Bundle

Action Checklist

• Deploy Redis instance: Provision managed Redis or self-host with persistence enabled for BullMQ durability.
• Configure environment secrets: Store API keys, webhook signing secrets, and DB credentials in a vault (e.g., AWS Secrets Manager, Doppler).
• Implement adapter interfaces: Define TypeScript contracts for each external service before writing execution logic.
• Add idempotency controls: Generate deterministic job IDs and pass idempotency keys to all financial/CRM APIs.
• Wire webhook ingestion: Build Express/Fastify endpoint that validates signatures, normalizes payloads, and enqueues jobs.
• Configure dead-letter queue: Route failed jobs to a separate queue with alerting and manual retry capability.
• Enable observability: Attach correlation IDs to all logs, expose queue metrics, and set up failure rate alerts.
• Run dry-mode validation: Test workflows against staging endpoints before enabling production execution.

Decision Matrix

Scenario	Recommended Approach	Why	Cost Impact
<500 events/week, single provider	Lightweight cron + direct SDK	Low complexity, minimal infrastructure overhead	$0-$5/mo (serverless)
500-5,000 events/week, multi-provider	Event-driven queue + adapter pattern	Predictable scaling, retry safety, provider abstraction	$8-$22/mo (Redis + compute)
>5,000 events/week, compliance-heavy	Event-driven + DLQ + audit logging	Regulatory traceability, failure isolation, audit readiness	$25-$60/mo (managed queue + logging)
AI-augmented routing required	Agent orchestrator + deterministic fallback	Semantic understanding with guaranteed execution path	$35-$120/mo (LLM + queue)

Configuration Template

# automation.config.yaml
queue:
  name: solopreneur-automation
  connection:
    host: ${REDIS_HOST}
    port: ${REDIS_PORT}
    password: ${REDIS_PASSWORD}
  defaultJobOptions:
    attempts: 3
    backoff:
      type: exponential
      delay: 2000
    removeOnComplete: 100
    removeOnFail: 50

adapters:
  - name: stripe-invoice
    provider: stripe
    endpoint: /api/webhooks/stripe
    rateLimit:
      maxRequests: 100
      windowMs: 1000
    idempotency: true

observability:
  logging:
    format: json
    correlationIdHeader: X-Correlation-ID
  metrics:
    enabled: true
    endpoint: /metrics
    scrapeInterval: 15s

deadLetterQueue:
  name: solopreneur-dlq
  alertOnThreshold: 10
  retentionDays: 30

Quick Start Guide

1. Initialize project: npm init -y && npm i bullmq ioredis express uuid typescript @types/node
2. Create queue & worker: Copy the workers/automation-worker.ts and adapters/base.ts examples. Configure Redis connection string in .env.
3. Deploy webhook router: Mount routes/webhooks.ts to an Express server. Test with curl -X POST http://localhost:3000/api/webhooks/ingest -H "Content-Type: application/json" -d '{"event":"CREATE_STRIPE_INVOICE","payload":{"customerId":"cus_test","amount":1500,"currency":"usd","idempotencyKey":"test-001"}}'
4. Monitor & iterate: Check BullMQ dashboard or Redis keyspace for job states. Add adapters for your primary tools (Notion, Gmail, Paddle). Scale concurrency and retry policies based on observed load.