ing; Stripe/Square tolerate extended outages.
Core Solution
Pillar 1: Idempotency by Event ID
Every serious provider emits a unique event identifier. Persist it before executing business logic in a table with a unique constraint. If the insert fails due to a duplicate key, the event has already been processed—return 200 OK and skip execution.
Provider Event ID Mapping:
- Stripe:
id (top-level) → evt_1ABC...
- GitHub:
X-GitHub-Delivery (header) → UUID v4
- Shopify:
X-Shopify-Webhook-Id (header) → UUID v4
- Slack:
X-Slack-Request-Timestamp + body hash → composite
- Square:
event_id (in body) → UUID v4
- HubSpot:
eventId (per event in array) → numeric
- SendGrid:
sg_event_id (per event) → base64
CREATE TABLE processed_webhook_events (
event_id TEXT PRIMARY KEY,
source TEXT NOT NULL, -- 'stripe', 'github', etc.
received_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
// Express + node-postgres
import express from "express";
import { Pool } from "pg";
const pool = new Pool();
const app = express();
app.post(
"/webhooks/stripe",
express.raw({ type: "application/json" }),
async (req, res) => {
// 1. Verify signature first (always — never skip).
const event = verifyStripeSignature(req); // throws if invalid
// 2. Try to record the event ID. Unique constraint = dedupe.
try {
await pool.query(
"INSERT INTO processed_webhook_events (event_id, source) VALUES ($1, $2)",
[event.id, "stripe"],
);
} catch (err) {
if ((err as { code?: string }).code === "23505") {
// Duplicate — Stripe is retrying. We already handled this event.
return res.json({ received: true, duplicate: true });
}
throw err;
}
// 3. Now do real work. If this throws, the row stays in the table
// but the event is unprocessed. See "transactional handlers" below.
await handleStripeEvent(event);
res.json({ received: true });
},
);
Transactional Handlers (Race Condition Mitigation)
The basic pattern has a race: if handleStripeEvent throws after the ID is recorded, retries see a duplicate and skip execution, leaving the event unprocessed.
Option A — Pending/Processed Status: Add a status column. On retry, if status='pending', re-run the work. If status='processed', return immediately. Required when handlers invoke external APIs (email, Slack) that cannot be rolled back.
Option B — Single DB Transaction: Wrap the insert and business logic in one transaction. If work fails, the insert rolls back, and the next retry sees no row. Cleanest for DB-bound logic.
ALTER TABLE processed_webhook_events
ADD COLUMN status TEXT NOT NULL DEFAULT 'pending',
ADD COLUMN processed_at TIMESTAMPTZ;
await pool.query("BEGIN");
try {
await pool.query(
"INSERT INTO processed_webhook_events (event_id, source) VALUES ($1, $2)",
[event.id, "stripe"],
);
await handleStripeEvent(event); // does its own DB writes inside the txn
await pool.query("COMMIT");
} catch (err) {
await pool.query("ROLLBACK");
if ((err as { code?: string }).code === "23505") {
return res.json({ received: true, duplicate: true });
}
throw err; // sender will retry; rollback means we're idempotent
}
Pillar 2: Return 2xx Fast — Defer to Queue
Most providers timeout at 5–30 seconds. Inline processing guarantees retry storms. Acknowledge synchronously, process asynchronously.
app.post("/webhooks/stripe", async (req, res) => {
const event = verifyStripeSignature(req);
// Sync: dedupe + enqueue + return.
await enqueueForProcessing(event); // this is BullMQ / SQS / DB-backed jobs
res.status(200).json({ received: true });
});
// Worker process — handles the actual logic, retries on its own schedule.
worker.process("stripe-events", async (job) => {
await handleStripeEvent(job.data);
});
Trade-off: Introduces dual retry layers (sender + worker). The worker must also deduplicate by event ID before executing business logic.
Pillar 3: Map Sender Retry Policies
Dead-letter thresholds and replay strategies must align with upstream retry budgets.
| Provider | Retry Attempts | Backoff Schedule | Total Window |
|---|
| Stripe | Up to 17 | Exponential, ~immediately → ~3 days | 3 days |
| GitHub | Up to 50 | Exponential | ~8 hours |
| Shopify | Up to 19 | Exponential, ~hours apart | 48 hours |
| Slack | Up to 3 | 1 min, 5 min, 30 min | ~36 min |
| Twilio | Configurable (3 default) | Exponential per Webhook config | varies |
| Square | Up to 70 (!) | Exponential, ~immediately → ~72 hours | 72 hours |
| HubSpot | Up to 10 | Exponential | ~8 hours |
| SendGrid | Up to ~24 hours of retries | Exponential | 24 hours |
Architectural Implications:
- Long windows (Stripe/Square) mean handler stability must persist over days. Short outages self-resolve.
- Slack's 36-minute window requires defensive replay tooling or extended internal queuing to prevent event loss.
- Never return
4xx for expected failures (e.g., duplicates). Most senders treat 4xx as a hard rejection and permanently stop retrying.
Pitfall Guide
- Skipping Signature Verification: Never bypass cryptographic signature validation to speed up processing. Attackers can replay or forge payloads, bypassing idempotency and triggering unauthorized side effects.
- Race Conditions in Mid-Flight Handlers: Recording an event ID before business logic completes creates a "zombie" state if the process crashes. Use transactional rollbacks (Option B) or explicit
pending/processed status tracking (Option A) to guarantee exactly-once semantics.
- Returning 4xx on Expected Failures: Senders interpret
4xx as a client-side rejection and halt retries permanently. Always return 2xx for duplicates or recoverable states, and route unrecoverable failures to a dead-letter queue internally.
- Ignoring Sender-Specific Retry Windows: Applying a uniform DLQ threshold across all providers causes premature event loss for short-window senders (Slack) or unnecessary DLQ bloat for long-window senders (Square). Tune retry budgets per provider.
- Dual Retry Layer Collision: When both the sender and your async worker retry, uncoordinated backoff strategies amplify load exponentially. Align worker retry intervals with sender policies or use jittered exponential backoff to prevent thundering herds.
- Missing Dead-Letter Queue (DLQ) Implementation: Without a DLQ, events that exhaust sender retries vanish silently. Implement a DLQ with alerting, manual replay capabilities, and schema versioning to handle payload evolution over time.
Deliverables
- Webhook Receiver Architecture Blueprint: A reference architecture diagram detailing the sync acknowledgment layer, async worker pool, idempotency table schema, and DLQ routing logic. Includes provider-specific configuration matrices and backoff alignment guidelines.
- Idempotency & Retry Configuration Checklist: A step-by-step validation checklist covering signature verification, unique constraint enforcement, transactional boundary selection, queue deduplication, DLQ thresholds, and sender policy mapping. Ready for CI/CD pipeline integration and pre-deployment auditing.
