Building Resilient Webhook Integrations for SaaS Products

I have a mental model for webhook integration reliability that I want to share, because it took me embarrassingly long to develop it: every webhook integration has three distinct failure modes, and you need to think about each one separately.

Delivery failures: the event never reached your endpoint. The sender tried, couldn't connect, and eventually stopped retrying.

Processing failures: the event arrived, your handler tried to process it, threw an exception, and returned a 500. The event may or may not have been partially processed.

Silent failures: the event arrived, your handler returned 200, but the business logic didn't do what you intended. Wrong data written, wrong state transition, duplicate record created.

Most resilience engineering focuses on delivery failures, because those are the most visible. But processing failures and silent failures cause more production incidents, because they're harder to detect.

Handling Delivery Failures

The baseline here is making sure your endpoint is reachable from the public internet with a valid TLS certificate, and that it responds within 30 seconds (most senders have this timeout).

Beyond the basics: don't process events synchronously in the request handler. Write the raw payload to a queue or a database first, return 200 immediately, and process asynchronously. This decouples delivery reliability from processing reliability.

If you're using Bluejay Relay, delivery reliability is handled by the platform. Events are captured and persisted the moment they arrive, regardless of what your application does. If your endpoint is temporarily unreachable, Bluejay Relay retries delivery with configurable backoff. Your application doesn't need to manage any of this.

Handling Processing Failures

Processing failures need two things: idempotency and clear error signaling.

Idempotency means the same event processed twice produces the same result as processing it once. The practical implementation is an event_id column on whatever you're creating/updating, with a unique constraint. Before processing, check if you've already handled this event ID. If yes, return success without processing again.

Clear error signaling means your handler should return a 5xx response when it fails, not a 200. This tells the delivery system that something went wrong and the event may need to be retried or flagged. A handler that catches all exceptions and returns 200 regardless is hiding failures.

In Bluejay Relay, every delivery attempt is logged with the HTTP status code and the response body. If your handler returns a 500 with an error message, you can see that error in the delivery log without digging through application logs.

Handling Silent Failures

Silent failures are trickiest because the system thinks everything is fine. The best defenses are:

Schema validation before business logic: Validate that the payload has the expected shape and required fields before you do anything with it. Fail hard and return 400 if the payload doesn't match your expectations.

Audit trails: Log what business action was taken in response to each event, not just whether the event was received. "Payment processed for user X, subscription activated" is more useful than "webhook received."

Schema change monitoring: If the upstream service changes their payload format, your handler might silently misbehave without returning an error. Bluejay Relay's schema monitoring alerts you when the incoming payload structure changes, so you can review and update your handler before it causes a silent failure in production.

The Operational Reality

Here's what I've seen happen in production enough times to write it down:

1. You launch a new integration. It works great.
2. Three months later, the upstream service updates their API and the payload has a new required field. Your handler doesn't crash — it just reads null for that field and does something wrong.
3. Someone notices two weeks later when a report looks off.

The gap between "something changed" and "we noticed something was wrong" is where silent failures live. Schema monitoring shortens that gap. Comprehensive delivery logging shortens it further. Replay capability means you can fix it retroactively once you find it.

A Practical Stack for Resilient Webhooks

For a production SaaS:

• Bluejay Relay as the inbound webhook layer: capture, log, retry, schema monitoring
• A durable queue (or Bluejay Relay's delivery to an async endpoint) for processing decoupled from delivery
• Idempotency keys in your database for every event type that matters
• Schema validation in your handlers before business logic
• Structured logging of business outcomes so you can audit what happened

This isn't over-engineering. For any integration where a dropped event costs money or damages a customer relationship, this is the minimum.

---

Try Bluejay Relay free and start building webhook integrations that hold up in production.