Retry and Backoff: Exponential, Jittered, and Token Bucket Strategies

Retries are one of those things that seem obvious until you watch a retry storm take down a service. "If a call fails, try again" — fine in isolation, catastrophic at scale. The discipline of retries with backoff is what separates resilient systems from systems that turn brief failures into prolonged outages.

This post is about the choices that actually matter when implementing retries: what to retry, when to give up, how to space the attempts, and how to avoid amplifying a problem the retried service is already suffering from.

The Three Failures of Naive Retries

Retries that look reasonable in code often cause real outages in production:

The thundering herd. A backend becomes briefly unavailable. Every client retries immediately. The backend, just coming back up, gets hit with more traffic than it had before, and dies again. Repeat.

The infinite loop. A 500 error is returned because of a malformed request. Retrying does not fix malformed requests. The client retries forever.

The waste of resources. A 5-second timeout with 5 retries means a slow dependency can pin a worker for 30 seconds before failing. With enough concurrent failures, every worker is stuck retrying and the application is effectively down.

Every retry strategy in this post exists to solve at least one of these.

Exponential Backoff

The first discipline is to wait progressively longer between retries. Linear backoff (wait 1s, 2s, 3s) is too gentle. Exponential backoff (wait 1s, 2s, 4s, 8s, 16s) gives the failing service real time to recover.

$attempt = 0;
$baseDelay = 100; // ms

while (true) {
    try {
        return $client->call();
    } catch (TransientException $e) {
        if ($attempt >= 5) {
            throw $e;
        }
        $delay = $baseDelay * (2 ** $attempt);
        usleep($delay * 1000);
        $attempt++;
    }
}

Without jitter (covered below), exponential backoff is still vulnerable to thundering herds — if 1,000 clients fail at the same time, they all retry at the same time, just on a logarithmic schedule.

Jitter

Adding randomness to the backoff interval breaks up the synchronized retries. The simplest and best version is "full jitter" — pick a uniformly random delay between 0 and the calculated exponential delay.

$delay = $baseDelay * (2 ** $attempt);
$jitteredDelay = random_int(0, $delay);
usleep($jitteredDelay * 1000);

AWS's classic blog post on this showed that full jitter outperforms the more intuitive "exponential delay plus or minus 50%" because it spreads the retry traffic more evenly. The math works out.

For systems with strict service-level objectives on retry latency, "equal jitter" is a softer alternative — half the calculated delay plus a random half. The total range is smaller, but the spread is still broken up.

Which Errors to Retry

The single most important retry rule: only retry transient errors. Permanent errors are not fixed by retrying.

Timeouts are the tricky case. If the call timed out, did the server start processing it? Retrying a timed-out call to a non-idempotent endpoint can cause duplicate effects. The defensive answer is to only retry idempotent operations on timeout, or to use an idempotency key so the server can detect duplicates.

Idempotency Keys

A retry-safe call carries an idempotency key — a unique identifier for this attempt. The server records the result against the key. If the same key arrives twice, the server returns the original result instead of doing the work again.

POST /payments/charge HTTP/1.1
Idempotency-Key: charge-attempt-7f8b3c91-e842-4d
Content-Type: application/json

{"amount": 1000, "currency": "USD", "customer": "cus_abc"}

Stripe, GitHub's API, and most well-designed payment APIs require idempotency keys for non-idempotent operations. The pattern is the only safe way to retry POSTs.

Token Bucket and Retry Budgets

Even with good backoff, an aggressive retry policy can multiply load on a struggling service. A retry budget caps how much of a service's traffic can be retries.

Allowed retries per second = (current request rate) × 0.10

If a service is doing 1,000 RPS, you allow at most 100 retries/sec. When the bucket is empty, retries are rejected client-side instead of sent.

The Envoy proxy implements this explicitly; you can also implement it yourself with a token bucket rate limiter. The pattern protects the downstream service from being overrun by retries when it is the thing that needs the most protection.

Circuit Breakers as Retry Discipline

A circuit breaker is retry policy at the system level. If the last N calls to a dependency have failed, stop sending new calls — fail fast instead. After a cooldown, let one test call through; if it succeeds, resume normal operation.

class CircuitBreaker
{
    public function call(callable $fn)
    {
        if ($this->state === 'open') {
            if (now()->diffInSeconds($this->openedAt) < $this->cooldown) {
                throw new CircuitOpenException();
            }
            $this->state = 'half-open';
        }

        try {
            $result = $fn();
            $this->recordSuccess();
            return $result;
        } catch (Throwable $e) {
            $this->recordFailure();
            throw $e;
        }
    }
}

The retry policy and the circuit breaker work together. Within a single request, you might retry 3 times with backoff. If the circuit is open from prior failures, the first retry never happens — the call fails immediately.

Deadline Propagation

A retry policy must respect the overall deadline of the request. If the user is waiting for a response that has a 2-second timeout at the API gateway, you cannot retry for 30 seconds. The deadline propagates: each layer passes it down so the next layer knows how much time is left.

// HTTP client respecting parent deadline
$remaining = $deadline - microtime(true);
if ($remaining <= 0) {
    throw new DeadlineExceeded();
}

$response = Http::timeout(min($defaultTimeout, $remaining))
    ->get($url);

Without deadline propagation, retries at lower layers cause upper layers to time out anyway — but with the lower layers still chewing through retries that no one cares about. Wasted work, worse incident behavior.

Defaults Worth Stealing

A reasonable default retry policy for non-critical HTTP calls:

• Max 3 attempts (the original plus 2 retries)
• Exponential backoff: 100 ms, 200 ms, 400 ms base
• Full jitter on each delay
• Retry only on 5xx, 429, connection errors, idempotent timeouts
• Respect the request deadline
• Behind a circuit breaker with a 30-second cooldown
• Behind a retry budget of 10% of normal traffic

For critical calls, add idempotency keys. For internal calls between services you own, consider tighter retry budgets and longer cooldowns.

The Real Lesson

Retries are not free, and the failure modes of bad retry policies are worse than the failures they are trying to mitigate. A service with no retries that returns errors to its callers is annoying; a service with bad retries that piles on a dying dependency is the source of a major incident.

Write the retry policy as deliberately as you write the business logic. Most of the production retry stories that end well had someone thinking carefully about backoff, jitter, idempotency, and budgets — not just calling the API in a loop.

Designing the retry policy for a service that talks to flaky dependencies? We help teams build resilience into client code without amplifying upstream problems. scopeforged.com