Retry, Timeout, and Backoff: When Should We Stop Retrying?

Retries are among the first mechanisms we consider when designing resilient distributed systems.

A connection fails, so we retry. A request times out, so we try again. A service becomes temporarily unavailable, so we wait and reconnect.

These mechanisms are useful, but they can also become part of the failure.

In systems where multiple services depend on one another, uncontrolled retries may transform a temporary slowdown into a much larger availability problem.

A timeout is not proof of failure

A timeout tells us that the expected response was not received within the configured time.

It does not necessarily tell us whether the remote operation was executed.

The request may never have reached its destination. It may still be running. Or it may have completed successfully while the response was lost.

These outcomes can look identical to the caller, but they have different implications for retry safety.

Repeating a read-only operation may be acceptable. Repeating a payment or a physical device command may introduce unintended side effects.

This brings us back to idempotency.

A timeout is an observation, not proof of the operation’s outcome.

How retries amplify failures

Suppose a downstream service becomes overloaded.

Client requests begin to time out. Each client immediately retries several times.

The overloaded service receives even more requests, increasing latency and triggering further retries.

A feedback loop develops: slow responses cause retries, and retries make responses slower.

The effect can multiply across service boundaries. If five layers independently make up to three total attempts, a failure at the deepest layer can theoretically result in 3⁵, or 243 attempts.

Real systems do not always reach this worst case, but the example demonstrates why retry placement and limits matter.

Exponential backoff and jitter

Exponential backoff introduces progressively longer delays between attempts.

For example:

Initial request    immediately
Retry 1            after 200 ms
Retry 2            after 400 ms
Retry 3            after 800 ms

This can give the downstream service time to recover.

However, many clients using identical delays may still retry simultaneously.

Jitter introduces controlled randomness into those delays, spreading retry attempts over time and reducing synchronized bursts.

Schematic — How jitter spreads synchronized retries

A retry policy also needs limits on attempt count, elapsed time, and acceptable load. Jitter alone is not a complete resilience strategy.

A small .NET example

The following code applies bounded retries with jitter to a read-only HTTP GET request:

using System.Net;

static bool IsTransient(HttpStatusCode status) =>
    status is HttpStatusCode.RequestTimeout
        or HttpStatusCode.TooManyRequests
        or HttpStatusCode.BadGateway
        or HttpStatusCode.ServiceUnavailable
        or HttpStatusCode.GatewayTimeout;

static async Task<HttpResponseMessage> GetWithRetryAsync(
    HttpClient client,
    string url,
    CancellationToken cancellationToken)
{
    const int maxAttempts = 4;

    for (var attempt = 1; ; attempt++)
    {
        using var timeout = CancellationTokenSource
            .CreateLinkedTokenSource(cancellationToken);

        timeout.CancelAfter(TimeSpan.FromSeconds(2));

        HttpResponseMessage? response = null;

        try
        {
            response = await client.GetAsync(
                url,
                timeout.Token);

            if (!IsTransient(response.StatusCode) ||
                attempt == maxAttempts)
            {
                return response;
            }
        }
        catch (OperationCanceledException)
            when (!cancellationToken.IsCancellationRequested &&
                  attempt < maxAttempts)
        {
            // This attempt timed out.
        }
        catch (HttpRequestException)
            when (attempt < maxAttempts)
        {
            // The failure may be transient.
        }

        response?.Dispose();

        var capMs = Math.Min(
            200 * (1 << (attempt - 1)),
            2000);

        var delay = TimeSpan.FromMilliseconds(
            Random.Shared.Next(0, capMs + 1));

        await Task.Delay(delay, cancellationToken);
    }
}

This example intentionally focuses on the basic retry mechanism rather than providing a complete production policy.

A real system must also consider total execution budgets, server-provided Retry-After guidance, telemetry, load limits, and the semantics of the operation.

The example is restricted to GET requests. Applying the same mechanism to operations with non-idempotent side effects requires additional safeguards.

Testing the behavior: What do retries actually improve?

Rather than evaluating retries, backoff, and jitter through theory alone, I wanted to compare their behavior under controlled failure conditions.

As part of the Distributed Reliability Lab, I developed two experiments. E03-A examines general retry behavior during transient failures, while E03-B focuses on recovery after a large number of clients experience a synchronized disruption.

The initial results revealed an important limitation: different retry strategies could generate very similar total request volumes, and jitter did not consistently improve success rates.

This led me to focus on when requests arrive, rather than only how many requests are sent.

Experimental setup

E03-B simulates 200 clients.

The downstream dependency freezes between 8.5 and 10.0 seconds. When it resumes at 10.0 seconds, it responds to the accumulated requests with HTTP 503.

I compared four strategies:

• A — Immediate retry: Failed attempts are retried without delay.
• B — Exponential backoff: Increasing delays are introduced between attempts.
• C — Backoff with jitter: Randomness is added to the backoff delays.
• D — Backoff with jitter and a retry budget: Additional retries are constrained to approximately 10% of the initial request volume.

All strategies share the same basic limits: a maximum of four attempts, a two-second timeout per attempt, and a five-second total operation budget.

The dependency is a synthetic model: 10 workers at 40 ms per request give a capacity of 250 requests per second, with a queue of 100 and an immediate HTTP 503 once capacity is full. For five seconds after the restart (10–15 s), capacity drops to 125 requests per second. I ran the experiments on a single laptop, inside Docker. The results belong to this model and these conditions; I do not claim they represent any real service one to one.

Total request volume does not tell the whole story

In the E03-B run with seed 42, strategies B and C generated nearly identical request volumes: 2,956 and 2,959 requests respectively.

Their peaks within a one-second window were also nearly identical: 229 requests for B and 225 for C.

At that resolution, jitter appeared to make little difference.

But when I measured the traffic in 100-millisecond intervals, the difference became clear.

Strategy B peaked at 143 requests, while C peaked at 76.

In this scenario, jitter reduced short-interval peak concentration by approximately 47%, without materially reducing total request volume.

The reason is synchronization. Clients that fail simultaneously and use identical backoff schedules tend to retry together.

Jitter spreads those attempts across a wider interval.

Figure 1 — Temporal distribution of retry attempts

• A · Immediate retry
• B · Exponential backoff
• C · Backoff + jitter

Show the data as a table

Time	A · Immediate retry	B · Exponential backoff	C · Backoff + jitter
8 s	7	10	9
8.1 s	10	9	9
8.2 s	12	11	12
8.3 s	11	12	11
8.4 s	9	9	8
8.5 s	9	9	9
8.6 s	6	6	7
8.7 s	10	8	8
8.8 s	8	10	10
8.9 s	7	7	7
9 s	10	10	10
9.1 s	9	8	8
9.2 s	12	11	10
9.3 s	12	13	14
9.4 s	10	10	11
9.5 s	6	7	5
9.6 s	7	7	8
9.7 s	8	8	8
9.8 s	8	9	9
9.9 s	11	11	10
10 s	232	11	76
10.1 s	4	3	73
10.2 s	5	143	19
10.3 s	6	6	13
10.4 s	7	11	11
10.5 s	8	8	15
10.6 s	4	36	8
10.7 s	3	4	4
10.8 s	5	6	4
10.9 s	1	1	2
11 s	1	0	0
11.1 s	8	4	4
11.2 s	6	5	5
11.3 s	3	2	3
11.4 s	8	8	1
11.5 s	6	3	4
11.6 s	4	5	5
11.7 s	4	2	7
11.8 s	8	5	4
11.9 s	7	3	6
12 s	3	5	3
12.1 s	11	11	7
12.2 s	8	5	5
12.3 s	8	8	9
12.4 s	13	7	12
12.5 s	14	9	12
12.6 s	6	10	9
12.7 s	11	9	13
12.8 s	14	10	8
12.9 s	14	5	9
13 s	12	8	7

This highlights an important measurement issue:

The observed effect of a resilience mechanism can depend on the time resolution of the metrics.

A one-second aggregate may conceal a substantial burst occurring within a much shorter interval.

Retry budgets introduce a different trade-off

Strategy D introduces a stricter limit on retry volume.

In the seed 42 run, it reduced total requests to 2,858, the one-second peak to 120, and the 100 ms peak to 19.

However, the success rate among affected operations fell to 78%, compared with 100% for B and C.

That does not mean the retry budget failed.

It means the policy prioritized limiting downstream work, even when doing so prevented some requests from recovering.

Protecting a dependency and maximizing the success rate of individual requests are not always the same objective.

Whether this trade-off is acceptable depends on the application’s requirements and available fallback behavior.

Figure 2 — Strategy comparison

Strategy	Successaffected operations (8.5–15 s)	Total requestsfirst attempts + retries	Retriessent	Retriesdenied by budget	Peakper 1 s	Peakper 100 ms	p95all operations	p99all operations	Recovery
A · Immediate retry	92.9%	3,040	220	0	275	232	628 ms	1,600 ms	1 s
B · Exponential backoff	100.0%	2,956	182	0	229	143	844 ms	2,132 ms	0 s
C · Backoff + jitter	100.0%	2,959	171	0	225	76	870 ms	2,043 ms	0 s
D · C + retry budget	78.0%	2,858	10	126	120	19	131 ms	1,192 ms	1 s

We should also avoid interpreting lower p99 latency as automatically better performance.

Strategy D had the lowest reported p99 at 1,192 ms, but also the lowest success rate.

The treatment of failed requests in latency calculations matters. Success rate and tail latency must be interpreted together.

Repeating the experiment

I repeated E03-B using five seeds. The main pattern remained consistent: jitter reduced short-interval request concentration, while overall request volumes for B and C remained similar.

However, changing the workload produced different outcomes.

With a smaller queue, jitter improved the success rate. With longer disruptions or more clients, C sometimes produced more total requests and a higher one-second peak than B.

Each alternative configuration was tested in a single run, so these observations should not be treated as evidence of general superiority.

In E03-A, variation between random seeds sometimes exceeded the differences between strategies.

The conclusion is important:

A retry policy cannot be evaluated reliably using one scenario, one random seed, or one metric.

What the experiment taught me

Backoff, jitter, and retry budgets solve related but distinct problems.

Backoff delays retries. Jitter can reduce synchronization between them. Retry budgets constrain the additional work that retries are allowed to generate.

Their effects depend on workload characteristics, downstream capacity, and recovery conditions.

Instead of asking which retry algorithm is universally better, we should ask:

What behavior is acceptable when this particular system fails and recovers?

Answering that requires evaluating success rates, short-term traffic bursts, total request costs, latency, and recovery behavior together.

Run the experiment yourself

The code, tests, and raw results of both experiments are open source in the distributed-reliability-lab repository. Docker is the only requirement:

git clone https://github.com/hemreg/distributed-reliability-lab.git
cd distributed-reliability-lab/experiments/retry-resilience
docker compose up --build

The command takes about four to five minutes. It runs E03-A, then E03-B, and finally prints the counterparts of the tables in this article. Afterwards, docker compose down cleans up, and ./test-e03.sh runs the unit tests. They verify the delay schedules, the retry loop’s attempt and time limits, the retry budget, and the failure timeline; they do not verify the experiment’s results or that any strategy is better. On a laptop, sleep pauses the experiment; on macOS, caffeinate -di docker compose up --build prevents it.

Parameters can be changed with environment variables, for example E03_SEED=7 docker compose up --build for a different seed. The repository README explains each parameter and why it was chosen.

Your numbers may differ slightly, because timing varies from machine to machine. In an independent run from a fresh clone with the same seed, the 100 ms peaks of B and C were 144 and 77, and D’s success rate was 77.9%. The differences between strategies and their order should hold. If they do not, that is a result worth sharing; you can report it by opening an issue in the repository.

When should we stop retrying?

Retrying is not always the right response.

We should reconsider retries when:

• Repeating the same invalid request cannot change the outcome.
• The previous operation may already have produced a non-idempotent side effect.
• The total execution budget has been exhausted.
• A dependency requests that clients stop or delay further attempts.
• The downstream system cannot safely absorb additional traffic.

The appropriate alternative may be returning an error, delaying work, querying the operation state, queueing the request, or requiring operator intervention.

In industrial environments, retry decisions may also depend on command semantics and the observed state of a physical device.

Retry is only one resilience mechanism

Retries may need to work alongside circuit breakers, backpressure, rate limiting, and load shedding.

Each mechanism introduces its own trade-offs. An overly aggressive circuit breaker, for instance, may continue rejecting requests even after a dependency has recovered.

Choosing thresholds and observing recovery behavior are therefore as important as selecting the mechanism itself.

Conclusion: Retry is a policy, not a reflex

The purpose of a retry mechanism is not simply to resend a failed request.

The E03 experiments showed that different retry policies can affect success rates, total request volume, and short-term traffic concentration in different ways.

Under certain conditions, jitter reduced synchronized retry bursts. Retry budgets constrained downstream traffic more aggressively, but at the cost of lower recovery success for some operations.

Choosing an algorithm is therefore not enough. Operation semantics, time budgets, downstream capacity, and business consequences must all be considered.

Reliability is not about retrying after every failure. It is about knowing when retrying is the right decision.

Further reading

• Timeouts, retries, and backoff with jitter — Amazon Builders’ Library
• Build resilient HTTP apps — Microsoft Learn (Microsoft.Extensions.Http.Resilience)