Retrying without making the outage worse: backoff and jitter

Picture an API that answers 503 for six hundred milliseconds. Nothing serious: a deploy, a server restarting. Except that two hundred clients get the error at the same moment, and each one does the reasonable thing: retry. From the server’s side, two hundred reasonable decisions made in the same millisecond look a lot like an attack.

I had to write this logic for real while building the SDKs for a timetable API: the client retries with exponential backoff and jitter, honours the Retry-After header up to 60 seconds, and never retries what it shouldn’t. None of these ideas are mine. Most of them come from a 2015 post on the AWS Architecture Blog, Exponential Backoff And Jitter, which is still the best reference I know on the subject.

What convinced me, though, was simulating the outage. The chart below shows why jitter matters more than backoff.

The scenario

Two hundred clients send a request within the same fraction of a second, for instance because a deploy just dropped their connections. The server is unavailable for 600 ms, then comes back, able to handle 24 requests every 100 ms. Each client gets eight attempts. Only the retry strategy changes.

Retrying right away

The first strategy is the one you write in a hurry: a loop, a fixed 100 ms pause, eight tries. During the outage the server receives two hundred requests every 100 ms, all rejected. When it comes back it accepts 24 per slot; the others fail and go straight back out.

Clients burn through their attempts during the outage, and most give up while the server is already working again. Barely a quarter get served. It is the worst of both worlds: lots of load, few results. In a real system that load lands on a service that is trying to restart, and it can be enough to knock it over again.

Doubling the delay

Exponential backoff is the textbook fix: wait 100 ms, then 200, 400, 800, and so on up to a ceiling. Clients finally let the server breathe. But look at the chart: load arrives in spikes. All two hundred clients failed at the same instant, they all waited exactly the same time, so they all come back at the same instant.

Every wave is far above capacity. The server serves 24, the rest leave together for a wait twice as long. Recovery stretches to almost ten seconds, and some clients give up anyway. Backoff reduces the number of calls. It does nothing about synchronisation.

Adding randomness

The fix is one line. Instead of waiting exactly the computed delay, each client draws a random duration between zero and that delay: the AWS post calls it full jitter. Clients that failed together no longer come back together, the spikes spread into a roughly continuous load, and that load drops under the server’s capacity.

In the simulation every client is served within a few seconds, with fewer calls than either of the other strategies. Replay it a few times: the draw changes, the result holds.

import random
import time

RETRYABLE = {429, 502, 503, 504, 529}

def send_with_retries(send, attempts=6, base=0.25, cap=8.0, deadline=30.0):
    """Retry with full jitter, honour Retry-After, and give up at the deadline."""
    start = time.monotonic()
    for attempt in range(attempts):
        response = send()
        if response.status_code not in RETRYABLE:
            return response
        delay = random.uniform(0, min(cap, base * 2**attempt))
        hint = response.headers.get("retry-after", "")
        if hint.isdigit():
            # trust the server, but never wait more than a minute
            delay = max(delay, min(int(hint), 60))
        if time.monotonic() - start + delay > deadline:
            break
        time.sleep(delay)
    return response

Listening to the server

When a server answers 429 or 503, it sometimes adds a Retry-After header, with a number of seconds or a date. It is the best information available, since it comes from the one that knows its own load. My client honours it, capped at 60 seconds: a misconfigured header announcing an hour’s wait should not freeze an application.

Language model APIs have their own codes. Anthropic’s returns 529 when it is overloaded, and its official SDKs already retry twice by default, with backoff. Before writing a loop around a client, check what it already does: otherwise you stack two retry policies without knowing it.

Knowing when to give up

A number of attempts is a practical limit, but what matters to the user is a deadline. In the same project, an Alexa skill has to play audio announcements at an exact time in each venue, and a late announcement is worse than a missed one. So the worker drops any message older than five minutes. Queue visibility is set to 30 seconds so that three attempts fit in that window, and the dead-letter queue raises an alarm but is never replayed.

Some requests should not be retried at all. A 400 or a 422 will fail the same way ten seconds later. And a POST that creates an order can only be replayed if the server can recognise the duplicate, usually thanks to an idempotency key sent by the client.

Retry in one place only

The last trap is sneakier: retries stack. If the browser retries three times, the backend retries each call to the next service three times, and that service retries its database three times, a single error at the bottom of the stack becomes 27 database requests. At the exact moment the database is struggling.

The rule I follow: one layer retries, usually the one closest to the resource that failed, and the others pass the error up. Teams running large volumes go further with a retry budget: past a small fraction of traffic, you stop retrying. A service failing that badly needs quiet, not reinforcements.

The list I keep at hand

• Capped exponential backoff, always with jitter: a delay drawn between zero and min(cap, base × 2ⁿ).
• Retry-After when present, with a sensible cap.
• A deadline rather than just a number of attempts.
• Never retry a client error, and never replay a write without an idempotency key.
• Only one layer of the stack retries.