Retries & Failures

How a delivery is sent

The first attempt is normally made within about a second of the change being saved; anything it does not send is picked up by a job that runs every minute. Before every attempt the API switch, the plan, the endpoint's status, its owner and the event's module are checked again, and the receiver's host name is resolved again and must still point only at public addresses.

The body is built once, at the first attempt, as the endpoint's owner sees the record at that moment. Every retry and redelivery sends exactly the same body, signed with a fresh timestamp.

Success and failure

Any 2xx status code is a success. Everything else is a failure and is retried: other status codes, redirects, timeouts, connection errors, TLS errors and a host name that no longer resolves to a public address.

Return 2xx as soon as you have verified and stored the event, and do the real work afterwards. A receiver that takes more than 10 seconds to answer is treated as failed even if it later finishes the work.

Retry schedule

A delivery gets 8 attempts in all. After each failed attempt, the next one waits a random time up to the limit below (and never less than 30 seconds), so many failed deliveries do not all retry at the same moment:

All eight attempts happen within about 45 hours. If the last attempt fails, the delivery is marked failed.

Retries carry the same event id, so a receiver that de-duplicates on id processes each event once, even if an earlier attempt succeeded but its response was lost.

When a delivery is cancelled

A queued delivery is cancelled, not sent later, when before its next attempt:

• the API is switched off, the plan no longer includes API access or the subscription lapses;
• the endpoint is paused or turned off;
• the endpoint's owner is deactivated, offboarded or no longer an Administrator or Executive;
• the module the event belongs to (time tracking or expenses) is switched off.

A delivery is also cancelled at its first attempt if the owner cannot see the record the event is about. Nothing is sent in that case.

Cancelled deliveries are listed in the delivery log with the reason.

Automatic disabling

An endpoint is turned off automatically when every delivery to it has failed for 72 hours and at least 20 attempts in a row have failed in that time. Any successful delivery resets the count. When it is turned off:

• every queued delivery to it is cancelled;
• its owner and every Executive in the organization are emailed.

Turning the endpoint back on (Resume) is a manual step, and it does not replay what was missed. Fix the receiver, resume the endpoint, then redeliver.

Redelivering events

From the endpoint's Delivery log in Settings > API > Webhooks:

• Redeliver sends one finished delivery again: succeeded, failed or cancelled.
• Redeliver all failed queues every failed delivery, and every delivery cancelled because the endpoint was turned off for failing, up to 500 at a time.

A redelivery has the same event id and body as the original, a fresh signature and timestamp, and a fresh retry schedule. The endpoint must be active, and redelivering needs the same authority as resuming it (see Who can change an endpoint).

Through the API, POST /webhooks/{id}/deliveries/{delivery_id}/redeliver redelivers one delivery and answers 202. A delivery that is still queued cannot be redelivered (409 CONFLICT).

The delivery log

Each endpoint has a delivery log showing, for every delivery, the event, its status (Delivered, Failed, Retrying or queued or Not sent), the attempt number, the status code your server returned, the duration, the error category and when the next attempt is due. GET /webhooks/{id}/deliveries returns the same information to a token with webhooks:read.

Delivery records are kept for 30 days. A delivery that is still queued is never removed.

Concurrency

At most 4 deliveries are in flight to one endpoint at a time, and at most 20 for one organization, so a slow receiver cannot hold up the others.

Troubleshooting by status