Webhook Rate Limiting and Backpressure: Surviving a Traffic Spike

Most webhook endpoints are sized for the average, because the average is what you saw when you built it. Then a customer bulk-imports 40,000 contacts, or an upstream provider comes back from an outage and flushes six hours of queued events at you in ninety seconds.

The endpoint that comfortably did four events a second is now doing four hundred.

It Doesn't Crash, It Cascades

The interesting thing about this failure is that it's rarely a clean crash. It unfolds:

Requests arrive faster than you process them, so they pile up in the web server. Each one waiting holds a connection and a database connection from the pool. The pool empties. Now every request times out — not just webhooks, but the customer trying to load their dashboard. The sender sees those timeouts, marks the deliveries failed, and queues retries. And then the retries arrive on top of the traffic that was already too much.

That last step is what makes webhook overload different from an ordinary traffic spike. A human who gets a timeout gives up and goes away. A webhook sender comes back, with reinforcements. Failure generates load, which generates more failure.

The One Rule That Matters Most

Do nothing in the request but verify, store, and acknowledge.

app.post('/webhooks', async (req, res) => {
  if (!verify(req)) return res.status(401).end();
  await queue.push(req.rawBody);
  res.status(200).end();
});

An endpoint that only enqueues absorbs an order of magnitude more traffic than one that writes to five tables and calls two APIs. More importantly it decouples your throughput from the sender's burst rate. The spike becomes a backlog that drains at whatever pace your workers manage, which is a completely different kind of problem — a queue you're watching go down, rather than an outage.

Rate Limiting Doesn't Do What You Think

This is where intuition from public APIs leads people astray. Rate limiting a webhook endpoint doesn't shed load. It defers it. Return a 429 and the sender doesn't go away — it retries, and the event comes back later, on top of whatever else is arriving then.

That's fine as a safety valve against a genuinely pathological sender. It's actively harmful as routine flow control, because if your limit sits below the sender's sustained rate you're permanently rejecting a slice of traffic, the retry backlog grows without bound, and eventually the sender exhausts its retry policy and drops those events for good.

So set the limit far above your real peak, treat it as a circuit breaker rather than a throttle, and always send Retry-After so well-behaved senders wait a sensible amount of time instead of hammering you:

HTTP/1.1 429 Too Many Requests
Retry-After: 30

Bound the Queue

An unbounded queue is a memory leak with a nicer name. Under sustained overload it grows until the process dies and takes everything in it with it — which is strictly worse than rejecting those events at the door, because the sender was told you accepted them and has already moved on.

Give it a maximum depth. When it's full, return 503 with a Retry-After. That's what backpressure actually is: handing the burden back to the party that can still hold it, instead of pretending you can absorb infinite load.

Watch Lag, Not Errors

Error rate is a lagging indicator. It looks completely healthy right up until the moment everything is on fire.

The number that tells you something useful is queue lag — the age of the oldest unprocessed event. Steady near zero means healthy. Rising steadily means your workers are underprovisioned and you have until the queue fills to do something about it, which is usually an hour of warning rather than none. Spiking and then draining means a burst arrived and the system handled it exactly as designed.

Alert on lag crossing a threshold, and on sustained growth. Both give you time.

One more thing worth doing while you're in there: give webhook ingestion its own process, or at minimum its own connection pool. If webhooks and page rendering share a pool, every spike in events is also a site outage for your customers, and there's no reason to couple those.

Moving the Absorption Upstream

Everything above is buildable. It's also a durable queue, bounded depth, a worker pool, lag metrics, and alerting — a real chunk of infrastructure for a team that mostly wants to process orders.

Putting Bluejay Relay in front changes where the burst lands. Relay accepts it at its own capacity, stores every event, and delivers to your endpoint at a rate it can survive. When your app returns 503 or times out, that delivery is retried with backoff rather than lost, and the events wait in the log until you're ready for them.

Which inverts the whole failure mode. Instead of a spike causing dropped events and a cascading outage, it causes a delivery backlog — one you can watch drain, replay selectively, and audit afterward to see exactly which events were affected. Dead letter queues and real-time monitoring cover the two halves of that in more depth.

---

Let a buffer absorb the spike instead of your app. Try Bluejay Relay free.