Cookie preferences

Essential cookies keep this site working. Optional Google Analytics cookies help us understand page visits and improve the site. You can change your choice anytime. Cookie policy

Skip to content
ForCSGuy System Design Atlas home

Library / Blogs

Using load shedding to avoid overload

How to reject excess work early and fairly so a service stays healthy under a surge.

AdvancedResilienceRate Limiting

By Amazon Builders' Library · Amazon Web Services · Added 2026-10-12

Open original source

All content belongs to Amazon Builders' Library / Amazon Web Services. We only link to it and give credit. Report a problem

Learn this article

About 37 min left. Your work is saved as you go.

What you are about to study

Amazon Web Services wrote this to explain how they thought about a problem. It assumes you are comfortable with distributed systems.

Ideas you will meet

Resilience
Timeouts, retries, circuit breakers, bulkheads and load shedding.
Rate Limiting
Token bucket, leaky bucket and sliding-window limiters.

What we already know about this

Our own lessons on these ideas. Read them here before you start, so it makes sense.

  • Advanced Distributed Systems · 13 minResilience: timeouts, retries, bulkheads, backpressure and circuit breakersHow to stop one slow or failing dependency from taking everything down: the patterns, how they fit together and where each one goes wrong.Read here

    The core problem

    In a system of many services, slowness is more dangerous than failure. A dependency that fails fast is easy to handle. One that responds in 30 seconds makes callers hold threads, connections and memory until they too stop responding: a cascading failure. Resilience patterns limit how far a problem can spread.

    The toolbox

    Figure SDA-ADS-06-1 · Resilience layers around a call · © forcsguy.com

    Timeouts

    Every network call needs a timeout, including connect, read and overall. Pick from measured latency (for example slightly above the 99th percentile), not a guess, and make the caller's budget smaller than its own caller's so work is abandoned in an orderly way. Propagate deadlines across calls.

    Retries

    Retry only transient failures and only idempotent operations (or use idempotency keys). Use exponential backoff with jitter (random spread) so clients do not retry in lockstep, cap the attempts, and apply a retry budget (for example no more than 10 percent extra traffic). Retry at one layer, not at every layer, or attempts multiply.

    Circuit breaker

    A breaker watches failures or slow calls. Closed: normal. After too many failures it opens: calls fail immediately (or use the fallback) so the dependency can recover and callers stop wasting resources. After a cool-down it goes half-open: a few trial calls decide whether to close again.

    Figure SDA-ADS-06-2 · Circuit breaker states · © forcsguy.com

    Bulkheads

    Partition resources so one dependency cannot consume them all: separate thread pools or connection pools per dependency, separate queues per tenant or per priority, or separate instances for critical paths. A slow recommendation service then cannot starve checkout.

    Backpressure and load shedding

    If producers outpace consumers, work piles up. Without limits the buffer grows until memory is exhausted or latency becomes huge. Options:

    • Bounded queues that reject or block when full.
    • Signalling upstream to slow down (credit-based flow in streaming systems, TCP windows, reactive streams).
    • Load shedding: deliberately drop low-priority work early (return 429 or 503 quickly) to protect the rest.
    • Rate limiting at the edge (see the rate-limiter lesson).

    Failing fast and visibly under overload is better than accepting everything and failing slowly.

    Fallbacks and graceful degradation

    Return something useful when a dependency fails: cached data, a default, a simpler algorithm, or hide a non-essential section. Decide which features are essential before an incident, and test the degraded mode.

    How they interact

    • A timeout is the minimum; without it nothing else works.
    • Retries inside a breaker inside a bulkhead is the usual nesting.
    • Everything needs metrics: breaker state, rejected calls, queue depth, timeout counts. Alert on them.
    • Test it: inject latency and failures (chaos experiments) to confirm the patterns work before production does it for you.

    Common mistakes

    Retries with no backoff; timeouts longer than the caller's; unbounded queues; a shared thread pool for everything; breakers that never close; fallbacks that call the same failing service; and testing only the happy path.

    Real-world reading

    Related reading

    Show 6 more

    Open this lesson with its quiz and exercise

  • System Design Foundations · 12 minDesigning for failure: reliabilityTimeouts, retries, circuit breakers, rate limiting and redundancy: how systems stay up when parts of them break.Read here

    Failure is normal

    At scale something is always broken: a disk, a network link, a deployment, a dependency having a bad day. Reliable systems do not avoid failure, they expect it and contain it.

    Two words to keep apart:

    • Availability: the fraction of time the system works (99.9% allows about 8.8 hours of downtime a year; 99.99% about 53 minutes).
    • Durability: the chance stored data is not lost.

    Remove single points of failure

    A single point of failure (SPOF) is any component whose failure takes everything down. Hunt for them:

    • Run several application servers behind a load balancer.
    • Replicate databases and plan failover (lesson 5).
    • Spread across availability zones, and for the most critical services, across regions.
    • Do not forget the "invisible" ones: the load balancer, DNS, a certificate that expires, a single queue.

    Figure SDA-FND-10-1 · Removing single points of failure tier by tier · © forcsguy.com

    Protect calls between services

    Most outages spread through calls between components. Four basic defences:

    1. Timeouts

    Every network call needs a timeout. Without it a slow dependency holds your threads hostage. Set timeouts based on what a healthy call looks like (for example a little above the 99th percentile), and make inner timeouts shorter than outer ones.

    2. Retries, carefully

    Transient errors often succeed on a second try. But retries multiply load on a struggling service, so:

    • Retry only idempotent operations or those with an idempotency key (lesson 8).
    • Use exponential backoff with jitter.
    • Cap the number of attempts, and ideally cap the share of traffic that can be retries.

    3. Circuit breakers

    If a dependency keeps failing, stop calling it for a short time and fail fast.

    A vintage electrical circuit breaker, the physical idea behind the software circuit breakerImage: Alf van Beem · CC0 · Source · Wikimedia Commons: Circuit breaker, Single pole, Non-tripfree, 20Amp-30Volts, AN 3160 pic1

    Figure SDA-FND-10-2 · Circuit breaker states · © forcsguy.com

    • Closed: normal operation, failures are counted.
    • Open: calls fail immediately, saving resources.
    • Half-open: a few trial calls check whether the dependency has recovered.

    4. Bulkheads

    Give each dependency its own pool of threads or connections, like watertight compartments in a ship, so one slow dependency cannot use up everything.

    Before: a slow dependency drags everyone down

    After: a breaker cuts the slow dependency off

    What changed

    • Callers stop waiting on a failing service
    • Threads and connections are not used up
    • A fallback answer is returned while the service recovers

    Protect yourself from too much traffic

    • Rate limiting caps how many requests a client may make (common algorithms: token bucket, sliding window). It protects against abuse and accidental floods.
    • Load shedding: when overloaded, deliberately reject low-priority work early so important requests still succeed. It is better to serve 80% of requests well than 100% badly.
    • Back pressure: tell producers to slow down instead of letting queues grow without bound.

    Degrade gracefully

    When a non-critical part fails, keep the rest working: show the page without recommendations, serve cached prices, queue the email. Decide in advance which features are essential.

    Practise failure

    • Health checks so traffic leaves broken instances.
    • Chaos testing: deliberately kill instances in a controlled way to prove the system copes.
    • Runbooks and game days so people have rehearsed the response.

    A worked example: the arithmetic of availability

    Downtime in a month. 99% allows about 7.3 hours; 99.9% about 43 minutes; 99.99% about 4.4 minutes; 99.999% about 26 seconds. Each extra nine is ten times harder and usually much more expensive, so choose the target your users need, not the biggest number.

    Dependencies in series multiply. If a request needs three services, each 99.9% available, the request is available only 0.999 x 0.999 x 0.999 = 99.7%, about 2 hours 11 minutes of downtime a month. The more things in the critical path, the worse it gets, which is why teams shorten the critical path and make non-essential calls optional.

    Redundancy in parallel helps. Two independent replicas, each 99% available, are both down only 1% x 1% = 0.01% of the time: 99.99% together. The word "independent" carries all the weight: replicas that share a rack, a deploy, a bug or a dependency fail together, and the formula no longer applies.

    Retries multiply load. If every one of 1,000 requests per second fails and each caller retries 3 times, the failing service sees up to 4,000 requests per second, making recovery harder. And if three layers each retry 3 times, one user request can become 64 attempts at the bottom. Retries need limits, backoff with jitter, and a retry budget (for example, retries may be at most 10% of normal traffic).

    Figure SDA-FND-10-3 · How availability combines · © forcsguy.com

    Anatomy of a cascading failure

    The classic outage is rarely one failure; it is one failure that spreads.

    1. A database gets slow (a bad query, a noisy neighbour). Responses that took 10 ms now take 5 seconds.
    2. App servers wait on it. Each request holds a thread for 5 seconds, so threads and connections fill up.
    3. The app can no longer accept new requests, even ones that do not need the database. The load balancer sees timeouts and marks servers unhealthy, so the same traffic lands on fewer servers.
    4. Callers time out and retry, which adds load to the already struggling system.
    5. Everything is down, and fixing the original slow query no longer brings it back, because the retry storm keeps it overloaded.

    The defences are cheap compared with the outage:

    • Timeouts on every call, shorter than the caller's own timeout.
    • Circuit breakers that stop calling a failing dependency for a while and fail fast.
    • Bulkheads: separate thread or connection pools per dependency, so one slow one cannot use all of them.
    • Load shedding: when overloaded, reject some requests early to keep serving the rest.
    • Backoff with jitter and retry budgets.

    Figure SDA-FND-10-4 · A circuit breaker protects both sides · © forcsguy.com

    Mistakes people make

    • No timeouts, or timeouts longer than the caller's. The default in many libraries is "wait forever" or minutes.
    • Retrying everything everywhere. Retry only safe, idempotent operations, at one layer, with backoff, jitter and a budget.
    • Health checks that lie. A server that answers /health but cannot reach its database should fail the check.
    • Untested failover. Failover that has never run is a guess. Practise it on purpose, in business hours.
    • Shared fate. Two "redundant" systems in one zone, on one deploy pipeline, or on one dependency are one system.
    • No graceful degradation. Decide in advance which features can be switched off (recommendations, images, analytics) to protect the critical path.

    Try it yourself

    In the System Builder, lose three servers at peak and find how much spare capacity you need. Read the resilience patterns lesson for circuit breakers and bulkheads in more depth, and build the lease leader election lab to see failover without split brain.

    More on this site

    Related reading

    Show 6 more

    Open this lesson with its quiz and exercise

  • Classic System Designs · 12 minDesign a rate limiterCap how often a client can call an API: the common algorithms, where to put the limiter and how to make it work across many servers.Read here

    The problem

    A rate limiter restricts how many requests a client may make in a period. It protects a service from abuse, accidental floods and runaway scripts, keeps costs predictable and shares capacity fairly.

    Requirements

    • Limit by user, API key or IP, with different limits per rule or plan.
    • Add very little latency to each request.
    • Work across many servers, not just one.
    • Tell rejected clients clearly what happened.
    • If the limiter itself fails, the API should not go down with it.

    Heavy traffic on a road: without limits, demand beyond capacity makes everyone slowerImage: Digitura · CC0 · Source · Wikimedia Commons: Gulf, traffic jam 1970s

    Where to put it

    Figure SDA-CSD-01-1 · Rate limiter in front of the service · © forcsguy.com

    The usual home is the API gateway or a middleware layer, because it sees every request early and rejects cheaply. Building it into each service duplicates effort, and a client-side limiter cannot be trusted.

    Here is the same service without a limiter, and with one in front. Switch on Simulate and raise the traffic to see what each version does when a script floods it.

    Before: Nothing in front

    After: Limiter at the gateway

    What changed

    • The abusive client is stopped at the gateway, not at the service
    • Counters sit in one shared store, so every server sees the same count
    • Blocked requests cost almost nothing to answer

    Algorithms

    AlgorithmIdeaGoodBad
    Token bucketA bucket refills at a steady rate; each request takes a tokenAllows bursts, tiny memoryTwo numbers to tune
    Leaky bucketRequests join a queue processed at a fixed rateSmooth output rateBursts fill the queue, old requests delayed
    Fixed window counterCount per minute, reset at the boundarySimplestBurst across a boundary doubles the limit
    Sliding window logKeep a timestamp per request, count those within the windowExactMemory per request
    Sliding window counterWeight the previous window's count by overlapCheap and close to exactApproximate

    Token bucket is a strong default: it is cheap, and bursts are usually fine for real clients. The sliding window counter is a good choice when you need smoother accuracy.

    Token bucket in a few lines

    For each key store (tokens, last_refill_time). On a request:

    1. Add elapsed x refill_rate tokens, up to the bucket size.
    2. If at least one token is available, take it and allow the request.
    3. Otherwise reject.

    The algorithms in pictures

    Token bucket. Tokens drip into a bucket of fixed size; every request must take one.

    Figure SDA-CSD-01-2 · Token bucket decision · © forcsguy.com

    A walk through time (bucket of 4, refilled to full each minute):

    Figure SDA-CSD-01-3 · Token bucket over a minute · © forcsguy.com

    Leaky bucket. Requests wait in a bounded queue and leave at a fixed rate.

    Figure SDA-CSD-01-4 · Leaky bucket · © forcsguy.com

    Fixed window counter and its boundary problem. One counter per clock window is simple, but two bursts on either side of the boundary can pass double the limit.

    Figure SDA-CSD-01-5 · Fixed window boundary burst · © forcsguy.com

    Sliding window log. Keep a timestamp for every accepted request and count those inside the last window.

    Figure SDA-CSD-01-6 · Sliding window log (2 per minute) · © forcsguy.com

    Sliding window counter. Blend the previous window's count with the current one, weighted by how much of the previous window still overlaps the last minute.

    Figure SDA-CSD-01-7 · Sliding window counter estimate · © forcsguy.com

    Making it work across servers

    Counters must be shared, otherwise each server enforces its own limit and a client gets N times the allowance. A fast in-memory store such as Redis works well.

    • Use an atomic operation (INCR with expiry, or a small Lua script) so the check and the update happen together. A separate read then write lets two servers both slip through.
    • Put the key in a predictable shape: rate:{user}:{rule}:{window}.
    • Partition keys across several Redis nodes for scale (consistent hashing, covered in lesson 2).
    • To save round trips, some systems keep local counters and sync periodically, accepting small overshoot.

    Why counters must be shared when there are several limiter servers:

    Figure SDA-CSD-01-8 · Why limiter servers need shared state · © forcsguy.com

    The system around the algorithm

    Figure SDA-CSD-01-9 · Detailed rate limiter design · © forcsguy.com

    The race condition when two servers read the same counter at once, and the fix:

    Figure SDA-CSD-01-10 · Read-modify-write race on a counter · © forcsguy.com

    Responses and visibility

    Return HTTP 429. Include headers so clients can behave: how many requests are allowed, how many remain, and when they reset, plus Retry-After. Log rejections and watch them, since a sudden rise can mean an attack, a bug in a client or a limit set too low.

    Going deeper

    Two races hide in "read, check, write". If two requests read the counter at 3 at the same moment, both write 4 and one request goes uncounted. A lock fixes it but slows every request; the usual answers are an atomic script that does the read, check and increment in one step inside Redis, or a sorted set that stores one timestamp per request so each request is a single atomic add.

    Many limiter servers need shared state. If each server counted only the traffic it sees, a client could send half its requests to each and double its quota. Sticky routing (always the same limiter for a client) helps but breaks when servers change and uneven load builds up. A central store such as Redis, or periodic sync with eventual consistency, is the common fix. Across regions, accept a little over-admission in exchange for not paying a cross-ocean round trip on every request.

    Where the rules live. Keep rules as data on disk or in config (for example "login: 5 per minute", "marketing messages: 5 per day"), have workers refresh them into a cache, and let the limiter read them from there, so tightening a rule never needs a deploy.

    Measure it. Watch two things: rules that are too strict (many valid users rejected) and an algorithm that cannot absorb a flash-sale burst. In the second case a token bucket, which allows short bursts, is the usual swap.

    Other layers and the client's side. The same idea works lower down, for example per IP address in the network layer rather than per user in HTTP. Well-behaved clients cache responses, respect the limit headers, handle the rejection gracefully, and back off before retrying.

    An approximate sliding-window counter blends the previous window's count (weighted by how much of it still overlaps) with the current one. It assumes traffic was even in the previous window; published tests by one large CDN found only a tiny fraction of requests were misjudged, so for most uses the saving in memory is worth it.

    Failure and edge cases

    • Limiter store down: usually fail open (let requests through, alert loudly) so you do not turn a limiter outage into an API outage. For security-sensitive endpoints (login attempts) you may prefer failing closed.
    • Clock differences between servers: base windows on the store's clock, not each server's.
    • Hard versus soft limits: some rules throttle (delay), others reject.
    • Many identities: also limit by IP for anonymous traffic, and watch for one client rotating keys.

    A worked example: setting the limits

    Say your public API has a free plan and a paid plan. You decide a free client may make 60 requests per minute with bursts up to 10, and a paid client 600 per minute.

    • Token bucket for the free plan: capacity 10, refill 1 token per second. A client idle for 10 seconds can fire 10 requests at once (the burst) and then settles to 1 per second. After a 429, it must wait about a second for the next token.
    • How much traffic is that? If 50,000 free clients all use their full allowance, that is 3,000,000 requests per minute, or 50,000 per second. You almost never see that, because most clients use a fraction. Sizing from the allowance gives the worst case; sizing from observed use gives the normal case. Plan the limiter and the servers behind it for the first, and alert on the second.
    • Counter memory: a token bucket needs two numbers per client (tokens, last refill time), about 16 bytes plus key overhead, say 100 bytes. For 10 million active clients that is about 1 GB, comfortably within one Redis node. A sliding window log would store a timestamp per request, up to 60 per client, which is 5 to 10 times more, one reason counters and buckets are the usual choice.

    Figure SDA-CSD-01-11 · Choosing limits by plan · © forcsguy.com

    Walking through a rejected request

    1. A client sends its 61st request in a minute.
    2. The edge gateway extracts the key (API key, falling back to IP address) and asks the limiter for that key.
    3. The limiter reads the bucket from the shared store, sees no tokens, and answers "no" without touching the application servers.
    4. The gateway returns HTTP 429 with a Retry-After header and, if you publish them, headers such as X-RateLimit-Limit and X-RateLimit-Remaining, so well-behaved clients can slow themselves down.
    5. The rejection is counted in metrics. A rapid rise from one key suggests abuse; a rise from many keys suggests your limit is too tight or a new client release is misbehaving.

    The cost of refusing is tiny compared with the cost of doing the work, which is the whole point: the limiter's decision must be much cheaper than the request it protects.

    When the limiter itself fails

    A limiter is in the path of every request, so decide in advance what happens when its shared store is slow or down:

    • Fail open (let requests through): users keep working, but you lose protection exactly when something odd is happening. Good for most public APIs, usually with a local fallback limit per server.
    • Fail closed (reject everything): safe for anything where overload is worse than refusal, such as an expensive or sensitive operation.
    • Keep a small local, in-memory limit on each server as a backstop, so even with the shared store down a single server cannot be flooded.

    Figure SDA-CSD-01-12 · Fail open with a local backstop · © forcsguy.com

    Mistakes people make

    • Limiting only by IP address. Many users share one address (offices, mobile carriers), and attackers rotate addresses. Prefer an API key or user id, with IP as a fallback.
    • Using the wrong clock. Limiter servers whose clocks drift apart count windows differently. Take time from the shared store, not each server.
    • Race conditions on counters. Reading a counter, adding one and writing it back lets two requests slip through together. Use an atomic increment or a small server-side script.
    • No visibility. Without rejection metrics you cannot tell abuse from a limit set too low.
    • Rejecting without guidance. Send Retry-After so clients can back off properly instead of retrying immediately and making things worse.

    Try it yourself

    Open the System Builder challenge A noisy client. One abusive client sends 97% of the traffic. Try to survive it by adding servers (you cannot, within 12), then switch on the rate limiter and watch the real users' errors disappear. Then build the sliding window rate limiter lab and see the boundary burst for yourself. You can also open the noisy-client design on the free canvas and place the rate limiter where you think it belongs.

    More on this site

    Related reading

    Show 6 more

    Open this lesson with its quiz and exercise

Continue to Amazon Web Services

Go deeper on this

A path through videos, articles, other case studies and code on the same ideas as Using load shedding to avoid overload.

  1. Warm upGet the vocabulary first.

  2. WatchHow others solve the same problem, on video.

    Videos belong to their creators and play from YouTube. More on YouTube (opens in a new tab)

  3. ReadEngineers explain it in their own words.

  4. See it in another companyThe same ideas, a different system.

  5. Read the codeSee how it looks when it is real.

Keep learning

Loading community…