Webhook Best Practices for 2026: Reliability, Security, and Observability

After looking at a lot of webhook integrations — debugging failures, reviewing implementations, and building infrastructure for them — the same mistakes come up over and over. Some are minor annoyances. Some will bite you in production in ways that are hard to recover from.

This guide is a comprehensive checklist organized by concern: security, reliability, observability, and operational hygiene.

Security

1. Always Verify the Signature

Every serious webhook provider signs their payloads with HMAC-SHA256 and includes the signature in a header. Stripe uses Stripe-Signature. GitHub uses X-Hub-Signature-256. Shopify uses X-Shopify-Hmac-SHA256.

If you don't verify this signature, anyone can send your webhook endpoint arbitrary payloads and your handler will process them as legitimate events. This is a real attack vector.

// Stripe example
var webhookSecret = Environment.GetEnvironmentVariable("STRIPE_WEBHOOK_SECRET");
var stripeEvent = EventUtility.ConstructEvent(
    json,
    Request.Headers["Stripe-Signature"],
    webhookSecret
);

Use the provider's official SDK for signature verification — it handles the timing-safe comparison for you. Don't roll your own.

2. Use HTTPS Only

Never accept webhooks over HTTP. TLS prevents eavesdropping on the payload in transit and helps confirm you're talking to the correct server. Most providers won't even deliver to non-HTTPS endpoints.

3. Keep Webhook Secrets Out of Source Control

Store webhook secrets in environment variables or a secrets manager. Never commit them to git. Rotate them immediately if they're exposed.

4. Validate Payload Shape, Not Just Signature

A valid signature proves the payload came from the right sender. It doesn't guarantee the payload has the structure your code expects. Validate required fields before processing and fail gracefully on unexpected shapes.

Reliability

5. Respond Quickly — Process Asynchronously

Most webhook senders have a 30-second timeout. If your handler takes longer than that to respond, the sender marks the delivery failed and may retry. Return 200 OK immediately, then process the event asynchronously via a queue or background worker.

[HttpPost("webhook")]
public async Task<IActionResult> Handle()
{
    var body = await new StreamReader(Request.Body).ReadToEndAsync();
    await _queue.EnqueueAsync(body); // fast
    return Ok();                     // return immediately
}

6. Make Your Handler Idempotent

The same event can arrive more than once. Stripe retries for 72 hours. GitHub retries on timeout. Your handler must produce the same result whether it sees an event once or ten times.

The simplest approach: store processed event IDs and skip duplicates.

if (await _db.ProcessedEvents.AnyAsync(e => e.EventId == eventId))
    return Ok();

Use a database constraint or Redis SET for the idempotency check to make it atomic.

7. Handle Retries Explicitly

Understand the retry behavior of every webhook sender you integrate with. Some retry for 72 hours (Stripe). Some retry 3 times then give up (GitHub). Some have no retry at all.

For senders with short retry windows, make sure your endpoint uptime matches or exceeds the window. For senders with no retry, consider routing through an intermediary that can retry on your behalf.

8. Don't Block on Expensive Operations

Calling third-party APIs, running ML inference, or performing heavy DB operations inside your synchronous webhook handler is a reliability risk. Any of those operations can be slow or fail, causing your handler to timeout and trigger a retry. Push expensive work to a queue.

9. Plan for Schema Changes

Upstream services update their webhook payloads. Sometimes they announce it. Sometimes they don't. If you're deserializing into a strict schema, a new field or a renamed property can break your handler silently (null reference) or loudly (5xx → retry loop).

Use lenient deserialization that ignores unknown fields. Set up monitoring to detect payload shape changes before they break your integration.

Observability

10. Log the Raw Payload

Log the raw request body before you do anything else. If something goes wrong, having the original payload is invaluable for debugging. Store it — not just the event ID.

11. Log Every Delivery Attempt

At minimum, log: event ID, event type, receipt timestamp, processing result (success/failure), and any error details. This lets you reconstruct what happened when something goes wrong three days later.

12. Set Up Alerting on Webhook Failures

If your handler starts returning 5xx at an elevated rate, you want to know before the webhook sender stops retrying. Set up alerts on your error rate for the webhook endpoint specifically.

13. Have a Replay Strategy

When your handler has a bug and you drop events, you need a way to re-process them. Some providers (Stripe) let you re-send from their dashboard. Most don't. Know your recovery options before you need them.

Options:

• Re-trigger from the provider's dashboard (if available)
• Reconstruct from the provider's REST API
• Replay from your own logs (requires you logged the raw payloads)

Operational Hygiene

14. Test With Real Payloads

Vendor documentation often has outdated or incomplete payload examples. Capture a real webhook from the provider and write your tests against the actual JSON, not the documented schema.

15. Version Your Webhook Handler

If you need to make breaking changes to your handler logic, do it behind a version flag. This lets you deploy the new handler alongside the old one while you migrate.

16. Use Separate Endpoints Per Provider

One endpoint for Stripe, a separate one for GitHub, a separate one for Shopify. This makes it easier to apply provider-specific signature verification, logging, and rate limiting. It also makes debugging much simpler.

17. Monitor Webhook Volume

Sudden drops in webhook volume (from a high-volume provider going quiet) can indicate a misconfiguration on their side or yours. Track volume per provider and alert on unexpected drops.

Checklist Summary

Security:

• HMAC signature verification for every provider
• HTTPS only
• Secrets in environment variables, not source
• Payload shape validation

Reliability:

• Respond in <5 seconds, process async
• Idempotency checking on all handlers
• Understood and documented retry behavior per provider
• No blocking operations in synchronous handler
• Lenient deserialization (ignore unknown fields)

Observability:

• Raw payload logging
• Per-attempt delivery logging
• Error rate alerting
• Replay strategy defined

Operational:

• Tests use real captured payloads
• Separate endpoints per provider
• Webhook volume monitoring

---

Bluejay Relay automates much of this checklist — raw payload logging, automatic retry, schema monitoring, and replay are all built in. Start free and skip the infrastructure work.