Event-Driven
Reliability
Webhook Processing
at Scale
Idempotency, retry handling, dead letter queues, and reliability patterns for processing millions of webhooks without losing data.
Webhooks seem simple until you're processing thousands per minute. Duplicate deliveries. Out-of-order events. Slow handlers causing timeouts. Failed processing losing critical data.
Here's how we reliably process webhooks from Stripe, Twilio, Slack, and a dozen other services.
Reliable Webhook Pipeline
📥 Receive
→
✓ Verify
→
🔑 Dedup
→
📤 Queue
→
⚙️ Process
Pattern 1: Immediate Acknowledgment
Never do heavy processing in the webhook handler. Acknowledge fast, process async:
webhook-handler.ts
export default {
async fetch(request: Request, env: Env, ctx: ExecutionContext) {
// 1. Verify signature (fast)
const signature = request.headers.get('X-Signature');
const body = await request.text();
if (!verifySignature(body, signature, env.WEBHOOK_SECRET)) {
return new Response('Invalid signature', { status: 401 });
}
const event = JSON.parse(body);
// 2. Check idempotency (fast)
const eventId = event.id;
const exists = await env.KV.get(`webhook:${eventId}`);
if (exists) {
// Already processed - return success
return new Response('OK', { status: 200 });
}
// 3. Queue for async processing
ctx.waitUntil(
env.WEBHOOK_QUEUE.send({
event,
received_at: Date.now()
})
);
// 4. Mark as received (not processed yet)
ctx.waitUntil(
env.KV.put(`webhook:${eventId}`, 'received', {
expirationTtl: 86400 // 24h dedup window
})
);
// 5. Return immediately
return new Response('OK', { status: 200 });
}
};
Pattern 2: Signature Verification
Always verify webhook signatures. Never trust unverified payloads:
signature-verification.ts
// Stripe-style HMAC verification
async function verifyStripeSignature(
payload: string,
header: string,
secret: string
): Promise<boolean> {
const parts = header.split(',');
const timestamp = parts.find(p => p.startsWith('t='))?.slice(2);
const signature = parts.find(p => p.startsWith('v1='))?.slice(3);
if (!timestamp || !signature) return false;
// Check timestamp freshness (prevent replay attacks)
const age = Date.now() / 1000 - parseInt(timestamp);
if (age > 300) return false; // 5 min tolerance
// Compute expected signature
const signedPayload = `${timestamp}.${payload}`;
const key = await crypto.subtle.importKey(
'raw',
new TextEncoder().encode(secret),
{ name: 'HMAC', hash: 'SHA-256' },
false,
['sign']
);
const sig = await crypto.subtle.sign(
'HMAC',
key,
new TextEncoder().encode(signedPayload)
);
const expected = Array.from(new Uint8Array(sig))
.map(b => b.toString(16).padStart(2, '0'))
.join('');
return expected === signature;
}
Pattern 3: Queue Processing with Retries
queue-consumer.ts
export default {
async queue(batch: MessageBatch, env: Env) {
for (const message of batch.messages) {
try {
await processWebhook(message.body, env);
// Mark as successfully processed
await env.KV.put(
`webhook:${message.body.event.id}`,
'processed',
{ expirationTtl: 86400 }
);
message.ack();
} catch (error) {
console.error('Processing failed', error);
// Check retry count
if (message.attempts < 3) {
// Retry with backoff
message.retry({
delaySeconds: Math.pow(2, message.attempts) * 60
});
} else {
// Send to dead letter queue
await sendToDeadLetter(message.body, error, env);
message.ack(); // Don't retry forever
}
}
}
}
};
async function sendToDeadLetter(
webhook: any,
error: Error,
env: Env
) {
await env.DLQ.send({
webhook,
error: { message: error.message, stack: error.stack },
failed_at: new Date().toISOString()
});
// Alert on failures
await sendSlackAlert(`Webhook failed: ${webhook.event.type}`, env);
}
Dead Letter Queue Strategy
Store failed webhooks with full context: the original payload, error message, stack trace, and timestamp. Build a simple admin UI to review failures, fix bugs, and replay webhooks.
Pattern 4: Event-Type Routing
event-router.ts
const handlers: Record<string, (event: any, env: Env) => Promise<void>> = {
'payment.completed': handlePaymentCompleted,
'payment.failed': handlePaymentFailed,
'subscription.created': handleSubscriptionCreated,
'subscription.cancelled': handleSubscriptionCancelled,
'lead.created': handleLeadCreated,
'lead.qualified': handleLeadQualified,
};
async function processWebhook(data: any, env: Env) {
const { event } = data;
const handler = handlers[event.type];
if (!handler) {
console.log(`Unknown event type: ${event.type}`);
return; // Don't fail on unknown events
}
await handler(event, env);
}
async function handlePaymentCompleted(event: any, env: Env) {
const { customer_id, amount, payment_id } = event.data;
// Update database
await env.DB.prepare(`
UPDATE customers SET balance = balance + ? WHERE id = ?
`).bind(amount, customer_id).run();
// Send notification
await sendSlackNotification(`💰 Payment received: $${amount/100}`, env);
// Trigger downstream workflows
await env.WORKFLOW_QUEUE.send({
type: 'payment_received',
customer_id,
payment_id
});
}
Webhook Reliability Checklist
- Verify signatures before processing any webhook
- Respond within 5 seconds (acknowledge, don't process)
- Implement idempotency with unique event IDs
- Queue webhooks for async processing
- Retry failed processing with exponential backoff
- Store failures in a dead letter queue
- Build tooling to replay failed webhooks
- Monitor webhook processing latency and failure rates
Webhooks are unreliable by nature—providers retry, networks fail, duplicates happen. Your job is to make processing reliable despite the chaos.