Retries, backoff and jitter

Gatepass's SMS workers call the provider, and sometimes the call fails. Most failures are brief: a dropped connection, a provider deploy, a two-second hiccup. Trying again fixes them. But at 21:04 on fest night the provider goes down for eight seconds, two thousand workers retry as fast as they can, and when the provider comes back it is flattened by the very clients that are waiting for it.

Why retry at all

Distributed systems fail in small, temporary ways all the time. A packet is lost, a server restarts mid-request, a database fails over in three seconds. For these transient failures, the same request a moment later usually succeeds. Retrying turns a 99.5% success rate into 99.99% without anyone noticing.

Retrying only helps with transient failures, though. Before retrying, sort the error:

• Retry: timeouts, connection resets, 502, 503, 504, and 429 Too Many Requests (after waiting as long as Retry-After says).
• Don't retry: 400 Bad Request, 401, 403, 404, 422. The request itself is wrong. Sending it again gets the same answer and wastes a call.

And only retry operations that are safe to repeat. A timed-out “send SMS” may have actually sent it. That's the idempotency lesson next. For now, assume the job can run twice safely.

The retry storm

The naive retry loop is while (!ok) send(). Picture 2,000 workers running it against a provider that is down. Each worker retries every 100 ms, so the provider receives 20,000 calls a second instead of its normal few hundred. That is fine while it's down (they all fail fast). The problem is the moment it recovers. It comes back with capacity for, say, 400 calls a second, and it immediately gets 20,000. It rejects or times out almost all of them, and in many real systems that overload knocks it straight back over.

This is a retry storm: clients turn a short outage into a long one by retrying in a way that adds load exactly when the system is weakest. Retries multiply, too. If the API retries 3 times, and each call to the worker service retries 3 times, and each call to the provider retries 3 times, one user action can become 27 calls to the provider.

Exponential backoff

The first fix is to wait longer after each failure: 0.5 s, then 1 s, 2 s, 4 s, 8 s, capped at some maximum such as 16 or 30 seconds. This is exponential backoff. A short glitch is retried almost immediately. A long outage gets fewer and fewer retries, so the load on the failing system falls off quickly.

But backoff alone has a nasty side effect. All 2,000 workers failed at the same moment, so they all compute the same schedule. They all retry at 0.5 s, then all at 1.5 s, then all at 3.5 s, 7.5 s and 15.5 s. The load arrives in perfectly synchronized spikes of 2,000 calls, and the gaps between spikes are wasted. The provider comes back at 8 s, but nobody calls until 15.5 s, and then everyone calls in the same 100 ms.

Jitter breaks the synchronization

The second fix is randomness. Instead of waiting exactly the backoff time, each worker waits a random time between zero and the backoff time. This is full jitter:

sleep = random_between(0, min(cap, base * 2 ** attempt))

The workers that failed together now retry at different moments. The spikes become a smooth, lower load that the provider can serve as soon as it's back. Jitter feels like it should make things slower; in practice it finishes the backlog sooner, because the provider is never idle in a gap or swamped in a spike.

Try it: four retry strategies

Start with “Retry at once” and look at the number of calls and the height of the bars. Then try “Exponential” and find the synchronized spikes. Finish with “Exponential + jitter”. Change the outage length and capacity and compare the table.

Try it

2,000 workers, one SMS outage

The SMS provider goes down at 0 s. Every worker's call fails and each one retries by its own rule. Bars are calls per 100 ms; the dashed line is what the provider can serve once it's back.

Retry at onceEvery 1 sExponentialExponential + jitter

Outage length8 s

Provider capacity once back400/s

0 soutage ends at 8 s40 s

failed call succeeded

Total calls

690k

for 2.0k jobs

Peak load

20k/s

50.0× capacity

All delivered

not yet

1.2k still retrying

Strategy	Calls	Peak	All delivered
Retry at once	690k	20k/s	not in 40 s
Every 1 s	76k	10k/s	not in 40 s
Exponential	14k	10k/s	not in 40 s
Exponential + jitter	13k	5.8k/s	23.9 s

Look at the totals. The immediate retrier makes hundreds of thousands of calls to deliver 2,000 messages, and keeps the provider so overloaded after it recovers that most messages still aren't delivered 40 seconds later. Exponential backoff cuts the calls to a few per job, but every retry lands in the same 100 ms, so each wall overloads the provider again. Jitter delivers everything, with the fewest calls, soon after the outage ends.

Limits and budgets

Backoff and jitter shape the load. You also need to cap it:

• Max attempts. Give up after a fixed number of tries (say 6 to 8 for a background job) and send the job somewhere a human or a later process can deal with it. That's the dead-letter queue, two lessons from now.
• Retry at one layer. In a chain of services, let one layer own retries (usually the one closest to the user, or the job runner) and have the others fail fast. That stops 3 × 3 × 3 multiplication.
• A retry budget. Allow retries to add at most, say, 10% extra load. When more than 10% of recent calls are retries, stop retrying and fail. A healthy system barely touches the budget; a dying one gets protected from its clients.
• Respect Retry-After. If the server tells you when to come back, that beats any formula.

Check yourself

Gatepass's workers use exponential backoff (1, 2, 4, 8 s) without jitter. After a 5-second provider outage, what's the main problem?

AWorkers retry too slowly to ever succeedBWorkers that failed together retry together, so the provider gets synchronized spikes and idle gapsCExponential backoff increases the total number of callsDThe provider can't tell retries from first attempts

Your turn

Write the retry policy for Gatepass's SMS worker. Which errors, how many attempts, what delays, and what happens at the end?

01

02

03

04

Stuck? Separate errors that will never succeed from ones that might, and remember the provider's 429 responses.

Compare with an expertWrite at least 2 first.

Key takeaways

• Retry transient errors (timeouts, 5xx, 429). Don't retry requests that are wrong (4xx).
• Immediate retries from many clients create a retry storm that can keep a recovering system down.
• Exponential backoff reduces load during long outages; jitter stops clients that failed together from retrying together.
• Cap attempts, retry at only one layer, and set a retry budget so retries can't multiply the load.
• Every retried operation must be safe to repeat.