Reliable Webhook Delivery: Retries, Exponential Backoff, Dead-Letter and Idempotent Receivers

A webhook is not an HTTP call you make and forget. It is a delivery guarantee you have quietly promised your customers, and the moment their endpoint returns a 503 you find out whether you meant it.

Most webhook code I inherit is a bare fetch inside a request handler. It fires the POST, hopes for a 200, and logs a line if it does not. That works in the demo and fails in production the first time the subscriber’s endpoint is slow, redeploying, or rate-limiting you. The event is gone. Nobody notices until a customer asks why their downstream system is out of sync. Reliable delivery is a queue problem, and once you frame it that way the design falls out cleanly: retry with backoff, dead-letter what will not deliver, and give the receiver a contract that makes your retries safe to accept. The stack below is BullMQ on Redis 7, Node 22 and NestJS 10, but the shape is portable.

Delivery is a queue problem, not an HTTP call

The instant you accept that any given delivery attempt can fail, the synchronous request handler stops being the right place to send from. You enqueue the event, return control to the caller, and let a worker own the delivery lifecycle: attempt, back off, retry, and eventually give up in a way you can inspect. BullMQ gives you the retry machinery for free through the attempts and backoff job options. The part it does not give you — the dead-letter queue and the per-attempt log — is a few lines you write yourself on the worker’s failed event.

The sender: attempts, backoff, and a dead-letter queue

Configure the queue with a bounded number of attempts and exponential backoff. BullMQ’s exponential strategy waits 2 ^ (attemptsMade - 1) * delay milliseconds between tries, so with a 2-second base you get roughly 2s, 4s, 8s, 16s. Add jitter so a thousand simultaneously-failing events do not all retry on the same tick and hammer a recovering endpoint in lockstep. When the last attempt fails, the job lands in BullMQ’s failed set; the failed handler is where you decide it has exhausted its retries and park it in a separate dead-letter queue for triage.

// webhook-delivery.ts — outbound delivery (Node 22, Redis 7, BullMQ v5)
import { Queue, Worker } from 'bullmq';
import { createHmac } from 'node:crypto';

const connection = { host: '127.0.0.1', port: 6379 };

// Five attempts, exponential backoff with jitter to de-synchronise retries.
export const deliveryQueue = new Queue('webhook-delivery', {
  connection,
  defaultJobOptions: {
    attempts: 5,
    backoff: { type: 'exponential', delay: 2000, jitter: 0.5 },
    removeOnComplete: 1000,   // keep the last 1000 for observability
    removeOnFail: false,      // keep failures; we dead-letter them ourselves
  },
});

// Events that exhaust every attempt are parked here for human triage.
const deadLetterQueue = new Queue('webhook-dead-letter', { connection });

function sign(secret: string, body: string, ts: number): string {
  return createHmac('sha256', secret).update(`${ts}.${body}`).digest('hex');
}

const worker = new Worker(
  'webhook-delivery',
  async (job) => {
    const { url, secret, event } = job.data;
    const body = JSON.stringify(event);
    const ts = Math.floor(Date.now() / 1000);
    const res = await fetch(url, {
      method: 'POST',
      headers: {
        'content-type': 'application/json',
        'x-webhook-id': event.id,                  // stable ID for dedupe
        'x-webhook-timestamp': String(ts),
        'x-webhook-signature': sign(secret, body, ts),
      },
      body,
      signal: AbortSignal.timeout(10_000),         // do not hang the worker
    });
    // Any non-2xx is a failure, so BullMQ schedules the next attempt.
    if (!res.ok) throw new Error(`HTTP ${res.status}`);
    return { status: res.status, attempt: job.attemptsMade + 1 };
  },
  { connection, concurrency: 20 },
);

// Per-attempt logging + dead-letter on the final failure.
worker.on('failed', async (job, err) => {
  if (!job) return;
  const attempts = job.opts.attempts ?? 1;
  console.warn(
    `event=${job.data.event.id} attempt=${job.attemptsMade}/${attempts} err=${err.message}`,
  );
  if (job.attemptsMade >= attempts) {
    await deadLetterQueue.add('dead', {
      original: job.data,
      lastError: err.message,
      failedAt: new Date().toISOString(),
    });
  }
});

Two decisions in there earn their keep. The AbortSignal.timeout stops a subscriber’s hung socket from tying up a worker slot indefinitely — without it, twenty slow endpoints stall your whole delivery pipeline. And treating every non-2xx as a thrown error is what hands control back to BullMQ’s retry scheduler; if you swallow the status you have silently turned a retryable failure into a dropped event. The per-attempt log line is not decoration either: it is the raw material for the delivery-success and mean-time-to-recovery metrics you will want on a dashboard, in the same spirit as instrumenting the DORA four.

Ordering: you almost certainly do not have it

Here is the guarantee people assume and rarely have: that webhooks arrive in the order the events happened. With retries and concurrency, they do not. Event A fails and backs off for eight seconds; event B, created later, delivers on its first attempt and arrives first. A worker concurrency of twenty means twenty events are in flight with no relative ordering at all. Do not paper over this — state it in your documentation and design the receiver so it does not matter. Put a monotonic sequence or an authoritative occurred_at in the payload and let the consumer reconcile to latest-state rather than replaying a stream it assumes is ordered. If you genuinely need strict per-entity ordering, that is a different and more expensive design (a single-concurrency queue keyed per subscriber, or BullMQ’s flows), and you should reach for it deliberately, not by accident.

The receiver contract: sign, then dedupe

Retries are only safe if the receiver is idempotent, because at-least-once delivery means the same event will occasionally arrive twice — a subscriber that returns 200 after the connection drops still gets retried. The contract has two halves. First, prove the payload came from you: an HMAC over timestamp.body, compared in constant time, with a timestamp check to blunt replay. Second, make processing idempotent by recording the event ID before doing the work; the first writer wins and every duplicate becomes a no-op. This is the same discipline as idempotency keys for POST, moved to the receiving end of the wire.

// webhook.controller.ts — idempotent receiver (NestJS 10, Node 22)
import { Controller, Post, Headers, HttpCode, BadRequestException } from '@nestjs/common';
import { createHmac, timingSafeEqual } from 'node:crypto';
import Redis from 'ioredis';

const redis = new Redis();
const SECRET = process.env.WEBHOOK_SECRET!;
const MAX_SKEW = 300; // seconds

@Controller('webhooks')
export class WebhookController {
  @Post('orders')
  @HttpCode(200)
  async receive(
    @Headers('x-webhook-id') eventId: string,
    @Headers('x-webhook-timestamp') ts: string,
    @Headers('x-webhook-signature') signature: string,
    rawBody: string,   // the exact bytes, from a raw-body middleware
  ) {
    // 1. Reject stale deliveries to limit the replay window.
    if (Math.abs(Date.now() / 1000 - Number(ts)) > MAX_SKEW) {
      throw new BadRequestException('stale timestamp');
    }

    // 2. Verify the HMAC over `${ts}.${body}` in constant time.
    const expected = createHmac('sha256', SECRET).update(`${ts}.${rawBody}`).digest('hex');
    const a = Buffer.from(signature, 'hex');
    const b = Buffer.from(expected, 'hex');
    if (a.length !== b.length || !timingSafeEqual(a, b)) {
      throw new BadRequestException('bad signature');
    }

    // 3. Dedupe on the event ID: SET NX means first writer wins.
    const fresh = await redis.set(`seen:${eventId}`, '1', 'EX', 604800, 'NX');
    if (fresh === null) {
      return { status: 'duplicate' };   // already handled; acknowledge and stop
    }

    await this.process(rawBody);        // your real, side-effecting work
    return { status: 'ok' };
  }
}

Note the deliberate choices. The signature covers the raw bytes, not a re-serialised object, because JSON key ordering will differ and break verification — capture the raw body in middleware and hash that. Use timingSafeEqual rather than === so you are not leaking a byte-by-byte comparison to a timing attacker; the same care you would apply when validating a token signature. And the dedupe key carries a TTL: a seven-day window is enough to absorb any retry storm without letting the seen-set grow without bound. If your processing must be exactly-once against a database rather than best-effort, promote step three into the same transaction as the write — a unique constraint on the event ID does the same job with stronger guarantees than a Redis flag.

Give subscribers a replay button

The dead-letter queue is not an archive; it is a work queue for a human or a scheduled job. Every entry is an event that a subscriber never successfully received, which means their world is now missing a state change. So expose it. A subscriber should be able to list their failed deliveries and trigger a replay — which is nothing more than re-enqueuing the original event onto webhook-delivery. Because the receiver dedupes on event ID, a replay of something that did eventually land is harmless; the consumer no-ops on the duplicate. That safety is the whole point of building the idempotent receiver first: it is what makes replay a button you can hand to a customer rather than a risk you have to gate behind a support ticket.

Build it in this order and each piece protects the next: backoff protects the subscriber, the dead-letter queue protects the event, and the idempotent receiver protects everyone from the retries the first two generate. Skip the receiver contract and every retry you add becomes a way to corrupt your customer’s data twice as fast.