Retries and circuit breakers

Why retries fix blips and worsen outages, and how a breaker helps recovery.

On the roadmap:Intermediate · step 6: Protecting services

Intermediateabout 10 to 15 min6 experiments

Experiments

New here? A few short steps show what happens, one at a time.

Settings

When a call fails

A failed request is just an error. Nothing extra is sent.

Requests from users per second70

What the service can answer per second100

Outage length (seconds)10

Warm-up after it returns (seconds)10

Random failures0%

Requests that failed

1,630

Peak load on the service

1.0x normal

Back to normal

29 s after

Breaker open

no breaker

The service was down for 10 seconds and users saw errors until it recovered, 29 seconds after it came back. Try retrying, and then a circuit breaker.

One minute, second by second

requests per second

Users answeredAttempts reaching the serviceWhat it can answer

The service goes down at second 10 for 10 seconds. Where the load line rises far above the traffic, retries are adding to it.

In plain words

When a call to another service fails, the easy fix is to try again. That fixes the odd blip, but when the other service is really down or overloaded, every retry is extra load on something already struggling. A circuit breaker stops calling for a while so it can recover.

Why it matters

Many large outages are made worse by well-meant retries. Here users send steady traffic to a service that goes down for a while and comes back cold, answering a little more each second. You can see what each policy does to the load, to the users, and to how soon it all returns to normal.

How real systems do it

The simulation is a simplified model. Here is what production systems add or do.

Backoff with jitter, which spreads retries over time with some randomness, is the standard advice from AWS for clients that retry against a shared service.
Netflix popularised the circuit breaker with Hystrix, which is now in maintenance mode. Resilience4j is a common successor on the JVM, and service meshes such as Envoy can break circuits too.
A breaker has three states: closed (normal), open (calls fail at once), and half open (a few probes test whether the service is back).
Some libraries cap retries with a retry budget, a limit on how large a share of calls may be retries, so that retrying cannot multiply the load. Only retry operations that are safe to repeat, such as those with an idempotency key.