Webhook providers commonly retry when they do not receive an acknowledgement quickly. A delivery may therefore arrive more than once, and events may arrive out of order. Your endpoint must assume duplicate delivery rather than relying on the provider to send each event exactly once.
Establish a delivery identity
Use the provider's event or delivery ID as a deduplication key. Store it with a unique database constraint so concurrent requests cannot both claim the same event. If the provider has no stable event ID, derive a carefully scoped fingerprint from documented immutable fields; do not hash arbitrary JSON and assume semantically identical events will always serialize identically.
Separate acknowledgement from work
Verify and persist the event, then acknowledge it according to the provider's contract. Perform slow business processing in a worker when possible. The database record should track received, processing, completed and failed states, along with attempt count and timestamps. A worker should claim work safely so two workers do not execute the same event simultaneously.
Make the business operation idempotent
Deduplication at the HTTP layer is not enough if a worker crashes after making a business change but before marking the event complete. Use a transaction to commit the business change and completion state together where possible. For external side effects, use a downstream idempotency key or a durable outbox pattern so recovery does not send the same action twice.
Handle ordering explicitly
If event sequence numbers or resource versions exist, compare them before applying changes. A late “payment pending” event must not overwrite a newer “payment completed” state. When ordering cannot be established, retrieve the current resource state from the provider instead of trusting arrival order.
Operational visibility
Expose safe metrics for received, duplicate, delayed and failed events. Provide a controlled replay process that preserves idempotency and records who initiated the replay.
Further reading
- Stripe: Webhooks — https://docs.stripe.com/webhooks
- AWS: Idempotent operations — https://aws.amazon.com/builders-library/making-retries-safe-with-idempotent-APIs/