Implement retry with exponential backoff and jitter. Why is the jitter the part that matters, and what must never be retried?

TL;DR: Exponential backoff doubles the wait after each failure, which spaces out one client's attempts and does nothing about the case that actually causes outages: a thousand clients whose requests all failed at the same moment, whose deterministic schedules bring them all back at the same moment. Jitter fixes that by randomizing each delay. In a direct measurement of a thousand clients reaching their fourth retry, deterministic backoff put all one thousand into a single ten-millisecond window, and full jitter reduced the peak to twenty, a fiftyfold reduction in the instantaneous load a recovering service sees. Two budgets bound the rest: a per-request attempt cap and total deadline, and a client-wide budget limiting retries to a small fraction of total requests, so a broad failure cannot multiply the load on an already struggling backend. And none of it is safe without classifying errors: a timeout is retryable only if the operation is idempotent, since the first attempt may have succeeded.

How to approach it

Give the schedule, then immediately make the point that jitter rather than backoff is what prevents the failure mode people care about, with the measurement. Then the two budgets, since unbounded retries turn a partial failure into a total one. Then the classification, because retrying a non-idempotent operation is a correctness bug rather than a performance one. Close with the tests.

A strong answer

A typical situation: a backend has a thirty-second outage. Every client retries with textbook exponential backoff. When the backend recovers, every client's fourth retry lands within the same few milliseconds, the backend is overwhelmed by a load spike far above its steady state, and it fails again, which synchronizes the clients even more tightly for the next round.

The schedule, and what jitter changes:

import random

def backoff_delays(attempts, base=0.1, cap=10.0, rng=None):
    """Full jitter: uniform in [0, min(cap, base * 2**i)]."""
    rng = rng or random.Random()
    return [rng.uniform(0, min(cap, base * (2 ** i))) for i in range(attempts)]

deterministic schedule, base 0.1 s, cap 10 s:
  [0.1, 0.2, 0.4, 0.8, 1.6, 3.2, 6.4, 10.0]   total 22.70 s

full jitter, same parameters, one sample:
  [0.064, 0.005, 0.110, 0.179, 1.178, 2.165, 5.710, 0.869]   total 10.28 s

the measurement that matters, 1,000 clients all reaching their fourth retry:
  peak arrivals in any 10 ms window
    no jitter:   1,000     every client waits exactly 0.8 s and arrives together
    full jitter:    20     spread uniformly over the 0 to 0.8 s window
  a 50x reduction in the instantaneous load the recovering backend sees
sanity: the expected total wait halves under full jitter, since a uniform draw averages half
        the ceiling, so jitter also makes the client faster on average. It is not a trade

Retry, Backoff and Idempotency covers the variants; full jitter is the one to default to, and the alternatives (equal jitter, decorrelated jitter) trade a little spread for a tighter lower bound on the wait, which matters when the first retry should not be immediate.

The two budgets:

import time

class RetryBudget:
    """Client-wide: retries may not exceed a fraction of total requests."""
    def __init__(self, ratio=0.1, min_per_sec=1.0):
        self.ratio, self.min_per_sec = ratio, min_per_sec
        self._requests = self._retries = 0

    def record_request(self):
        self._requests += 1

    def allow_retry(self) -> bool:
        allowed = self._requests * self.ratio + self.min_per_sec
        if self._retries < allowed:
            self._retries += 1
            return True
        return False


def call_with_retry(fn, budget, max_attempts=5, deadline_s=30.0,
                    base=0.1, cap=10.0, rng=None, idempotency_key=None):
    rng = rng or random.Random()
    started = time.monotonic()
    budget.record_request()
    last = None
    for attempt in range(max_attempts):
        try:
            return fn(idempotency_key=idempotency_key)
        except Exception as exc:
            last = exc
            if not is_retryable(exc):
                raise
            if attempt == max_attempts - 1:
                break
            if not budget.allow_retry():
                raise RetriesExhausted("client retry budget spent") from exc
            delay = rng.uniform(0, min(cap, base * (2 ** attempt)))
            if time.monotonic() - started + delay > deadline_s:
                break
            time.sleep(delay)
    raise last

per-request budget    max_attempts and a total deadline
                      bounds one request's cost and stops a slow failure from consuming a
                      caller's whole timeout budget
client-wide budget    retries as a fraction of total requests, typically 10%
                      this is the one that prevents amplification: during a total outage,
                      every request fails, and without this budget every request also retries
                      max_attempts times, so the load on the recovering backend is 5x normal
                      at exactly the moment it can least absorb it
                      with a 10% budget, retries are capped at a tenth of traffic no matter
                      how bad the failure is
the amplification arithmetic, which is what the client-wide budget is for:

  during a total outage every request fails, so with max_attempts = 5 and no budget
    attempts per request  = 1 initial + 4 retries = 5
    load on the recovering backend = 5 x normal traffic
  with a 10% client-wide budget
    retries allowed       = 0.1 x requests
    total attempts        = requests x (1 + 0.1) = 1.1 x normal
    reduction             = 5 / 1.1 = 4.5x less load

  and the expected wait, deterministic against full jitter over 8 attempts
    deterministic sum     = 0.1 + 0.2 + 0.4 + 0.8 + 1.6 + 3.2 + 6.4 + 10.0 = 22.70 s
    full jitter expected  = half of each ceiling = 22.70 / 2 = 11.35 s
    one observed sample   = 10.28 s
sanity: the budget turns a 5x load spike into a 1.1x one at exactly the moment the backend can
        least absorb it, which is the difference between a backend recovering and a backend
        held down by its own clients. It is the piece most implementations lack

The classification, which is a correctness question:

the timeout row is the important one
  a timeout means you do not know whether it succeeded
  retrying a non-idempotent operation after a timeout can duplicate it: a second charge, a
    second job submission, a second message
  the fix is idempotency keys: the client generates one per logical operation and sends it
    with every attempt, and the server deduplicates
  with a key, a timeout is safely retryable. Without one, it is not, and no amount of backoff
    tuning changes that

The tests:

def test_delays_within_ceiling():
    d = backoff_delays(8, base=0.1, cap=10.0, rng=random.Random(0))
    for i, x in enumerate(d):
        assert 0 <= x <= min(10.0, 0.1 * 2 ** i)

def test_non_retryable_raises_immediately(fake_clock):
    calls = []
    def fn(**kw):
        calls.append(1); raise BadRequest()
    with pytest.raises(BadRequest):
        call_with_retry(fn, RetryBudget())
    assert len(calls) == 1                      # no retries at all

def test_budget_caps_retries():
    b = RetryBudget(ratio=0.1, min_per_sec=0)
    for _ in range(100):
        b.record_request()
    allowed = sum(b.allow_retry() for _ in range(50))
    assert allowed == 10                        # 10% of 100

def test_deadline_respected(fake_clock):
    # a call with a 1 s deadline must not sleep past it
    ...

def test_idempotency_key_is_stable_across_attempts():
    seen = []
    def fn(idempotency_key=None):
        seen.append(idempotency_key); raise Timeout()
    with pytest.raises(Timeout):
        call_with_retry(fn, RetryBudget(), max_attempts=3, idempotency_key="abc")
    assert seen == ["abc", "abc", "abc"]        # same key, so the server can deduplicate

The last test is the one that catches the real bug: an implementation that generates a fresh key per attempt has an idempotency mechanism that does nothing, and every retry after a timeout creates a duplicate. The key identifies the logical operation, not the attempt.

The reversal condition: retry is the wrong tool when the failure is not transient. Retrying a request that fails because the backend is overloaded adds load to an overloaded system, which is why the client-wide budget exists and why a server should shed load explicitly with a retry-after rather than timing out silently. At the extreme, a circuit breaker replaces retry entirely: after enough consecutive failures, stop calling for a period, which protects both sides better than any backoff schedule. Backoff handles transient failures and a breaker handles sustained ones, and a client facing a sustained failure with only backoff will keep hammering at a reduced rate for as long as the failure lasts.

What interviewers probe next

• "Why full jitter rather than adding a small random amount?" A small perturbation on a deterministic schedule still clusters. Full jitter spreads uniformly across the whole window, which is what breaks the synchronization.
• "Where does the retry-after header fit?" It overrides the computed delay, since the server knows more than the client. Respect it, and cap it so a hostile or buggy value cannot stall the client indefinitely.
• "How do you pick the budget ratio?" From what the backend can absorb above its steady state. Ten percent is a common default; the arithmetic is that a full outage then costs the backend 1.1 times its normal load rather than 5 times.
• "Should the server also do something?" Yes: shed load with an explicit rejection and a retry-after rather than timing out, since a rejection is cheap and a timeout costs a held connection on both sides. Capacity and Backpressure covers that server-side half.

Common mistakes

• Exponential backoff without jitter, which synchronizes every client that failed together.
• No client-wide budget, so a total outage multiplies the load on the recovering backend by the attempt count.
• Retrying timeouts on non-idempotent operations, which duplicates work.
• Generating a fresh idempotency key per attempt, which makes the mechanism decorative.

Key takeaways

• Jitter, not backoff, prevents the thundering herd: 1,000 clients on the fourth retry put 1,000 arrivals in one 10 ms window deterministically and 20 with full jitter.
• Full jitter also halves the expected wait, so it is not a trade against latency.
• Two budgets: per-request attempts and deadline, plus a client-wide cap of roughly 10% of requests that bounds amplification during a full outage.
• Retry timeouts only with a stable idempotency key reused across every attempt of the same logical operation.