Avoiding insurmountable queue backlogs

Ideas you will meet

Event Driven Architecture

Loosely coupled services reacting to events.

Resilience

Timeouts, retries, circuit breakers, bulkheads and load shedding.

What we already know about this

Our own lessons on these ideas. Read them here before you start, so it makes sense.

• Advanced Distributed Systems · 13 minResilience: timeouts, retries, bulkheads, backpressure and circuit breakersHow to stop one slow or failing dependency from taking everything down: the patterns, how they fit together and where each one goes wrong.Read hereClose

  The core problem

  In a system of many services, slowness is more dangerous than failure. A dependency that fails fast is easy to handle. One that responds in 30 seconds makes callers hold threads, connections and memory until they too stop responding: a cascading failure. Resilience patterns limit how far a problem can spread.

  The toolbox

  100%

  Figure SDA-ADS-06-1 · Resilience layers around a call · © forcsguy.com

  Timeouts

  Every network call needs a timeout, including connect, read and overall. Pick from measured latency (for example slightly above the 99th percentile), not a guess, and make the caller's budget smaller than its own caller's so work is abandoned in an orderly way. Propagate deadlines across calls.

  Retries

  Retry only transient failures and only idempotent operations (or use idempotency keys). Use exponential backoff with jitter (random spread) so clients do not retry in lockstep, cap the attempts, and apply a retry budget (for example no more than 10 percent extra traffic). Retry at one layer, not at every layer, or attempts multiply.

  Circuit breaker

  A breaker watches failures or slow calls. Closed: normal. After too many failures it opens: calls fail immediately (or use the fallback) so the dependency can recover and callers stop wasting resources. After a cool-down it goes half-open: a few trial calls decide whether to close again.

  100%

  Figure SDA-ADS-06-2 · Circuit breaker states · © forcsguy.com

  Bulkheads

  Partition resources so one dependency cannot consume them all: separate thread pools or connection pools per dependency, separate queues per tenant or per priority, or separate instances for critical paths. A slow recommendation service then cannot starve checkout.

  Backpressure and load shedding

  If producers outpace consumers, work piles up. Without limits the buffer grows until memory is exhausted or latency becomes huge. Options:

  • Bounded queues that reject or block when full.
  • Signalling upstream to slow down (credit-based flow in streaming systems, TCP windows, reactive streams).
  • Load shedding: deliberately drop low-priority work early (return 429 or 503 quickly) to protect the rest.
  • Rate limiting at the edge (see the rate-limiter lesson).

  Failing fast and visibly under overload is better than accepting everything and failing slowly.

  Fallbacks and graceful degradation

  Return something useful when a dependency fails: cached data, a default, a simpler algorithm, or hide a non-essential section. Decide which features are essential before an incident, and test the degraded mode.

  How they interact

  • A timeout is the minimum; without it nothing else works.
  • Retries inside a breaker inside a bulkhead is the usual nesting.
  • Everything needs metrics: breaker state, rejected calls, queue depth, timeout counts. Alert on them.
  • Test it: inject latency and failures (chaos experiments) to confirm the patterns work before production does it for you.

  Common mistakes

  Retries with no backoff; timeouts longer than the caller's; unbounded queues; a shared thread pool for everything; breakers that never close; fallbacks that call the same failing service; and testing only the happy path.

  Real-world reading

  Related reading

  • Timeouts, retries, and backoff with jitterAmazon Builders' Library · Amazon Web ServicesBlogs
  • Using load shedding to avoid overloadAmazon Builders' Library · Amazon Web ServicesBlogs
  • Circuit BreakerMartin FowlerBlogs
  • Circuit Breaker patternMicrosoft · Microsoft AzureDocumentation

  Show 6 moreShow fewer

  • Google SRE Book: Addressing Cascading FailuresGoogle SRE team · GoogleBlogs
  • Chaos Monkey (source)Netflix · GitHubGitHub
  • Bulkheads: Partition and Tolerate Failure in One PartSkife engineers · SkifeCase Studies
  • Protector: Circuit Breaker for Time Series Databases at TrivagoTrivago engineers · TrivagoCase Studies
  • Autoscaling Jenkins at TrivagoTrivago engineers · TrivagoCase Studies
  • Troubleshooting a Connection Timeout Issue with tcp_tw_recycle Enabled at eBayeBay engineers · eBayCase Studies

  Open this lesson with its quiz and exercise

• Classic System Designs · 10 minDesign a notification systemSend push, SMS and email notifications reliably at scale: queues per channel, user preferences, retries and avoiding spam.Read hereClose

  The problem

  Applications tell users things: "your order shipped", "someone replied", "your code is 482913". A notification system delivers such messages over several channels:

  • Mobile push through Apple (APNs) and Google (FCM) services.
  • SMS through a telecom provider.
  • Email through an email provider.
  • Sometimes in-app messages or webhooks.

  Requirements: soft real time (seconds, not hours), high volume with spikes, no lost notifications, no duplicate floods, user control over what they receive.

  High-level design

  100%

  Figure SDA-CSD-06-1 · Notification pipeline with a queue per channel · © forcsguy.com

  1. Other services call the notification service (an API or an event) with who and what.
  2. It validates, applies preferences, renders the message from a template and picks the channel.
  3. It places a job on a per-channel queue.
  4. Workers call the third-party provider and record the result.

  Putting a queue between each stage absorbs spikes and lets you scale the channels independently.

  100%

  Figure SDA-CSD-06-2 · Should this event become a notification? · © forcsguy.com

  Image: Océanos y dados · CC0 · Source · Wikimedia Commons: Smartphone Use

  Channels and their providers

  100%

  Figure SDA-CSD-06-3 · One provider per channel · © forcsguy.com

  Collecting contact details

  100%

  Figure SDA-CSD-06-4 · Contact information flow and tables · © forcsguy.com

  Before: one shared queue

  100%

  After: separate lanes

  100%

  What changed

  • A marketing blast no longer delays login codes
  • Each channel scales and fails on its own
  • Priorities can be set per lane

  Supporting data

  Data	Purpose
  Contact info and device tokens	where to send
  User preferences	channels allowed, topics, quiet hours, language
  Templates	consistent text, translations
  Notification log	state per notification (queued, sent, failed), for debugging and dedupe

  Reliability

  • Persist first: write the notification to the log before queueing, so a crash does not lose it.
  • At-least-once delivery from the queue, plus a dedupe check (notification_id already sent?) so retries do not double send.
  • Retries with exponential backoff and jitter; after repeated failures move to a dead-letter queue and alert.
  • Provider failures: use a second provider as a fallback for critical channels such as SMS verification codes.
  • Invalid tokens: providers tell you when a device token is dead; remove it.

  Protecting users and the system

  • Preferences and opt-out are mandatory for marketing messages (and legally required in many places).
  • Rate limit per user and per topic, and collapse bursts into a digest ("12 new comments").
  • Priority: security codes and payment alerts jump the queue ahead of marketing.
  • Scheduling: delay non-urgent notifications until a sensible local time.

  Why the design looks this way

  A first version with one notification server has three weaknesses: it is a single point of failure, everything scales together (database, cache and sending), and slow third-party calls drag down the whole thing at peak. The fixes are the ones in the design: move state out of the server, run several servers, and put a queue per channel between the API and the workers so an outage at one provider (say the SMS vendor) only backs up its own queue.

  Other points worth knowing: providers differ by region (a push service available in one market may not be in another), so keep each provider behind a plug-in interface. Only internal services or verified clients may call the send API, using an app key and secret, or the system becomes a spam relay. Notifications may be delayed or reordered but must never be silently lost, so persist each one before sending and retry from the stored record. And the single most useful health signal is the number of queued notifications: if it keeps growing, add workers.

  Life of a notification

  100%

  Figure SDA-CSD-06-5 · Notification states for tracking · © forcsguy.com

  100%

  Figure SDA-CSD-06-6 · The complete notification path · © forcsguy.com

  Observability

  Track sent, delivered, opened and failed counts per channel and per provider, plus queue depth and age. A growing queue age is the earliest sign something is wrong. Keep a trail so support can answer "did this user get the message?".

  Worked numbers: a steady stream and a blast

  Take 10 million users who each get about 5 notifications a day across push, email and in-app: 50 million a day, about 580 per second on average. That is easy. The hard case is a campaign blast: a push to all 10 million users within 10 minutes means 16,700 sends per second, and the push providers rate-limit each connection, so you need many parallel workers and connections, and you must prevent the blast from delaying a login code.

  Log volume: at about 200 bytes per delivery record, 50 million a day is 10 GB a day. Keeping 30 days of detail is about 300 GB, which is manageable; keep aggregates much longer.

  Provider failure: a push provider that fails for 10 minutes during a blast would otherwise cause 10 million retries at once. The queue absorbs it, workers back off, and delivery continues when the provider recovers.

  100%

  Figure SDA-CSD-06-7 · Separate lanes for urgent and bulk messages · © forcsguy.com

  One notification, start to finish

  1. Event: the order service publishes "order 4711 shipped" with the user id.
  2. Decide: the notification service loads the user's preferences (allowed channels, quiet hours, language), checks the per-user limit (not more than a few promotional messages a day) and the dedupe key (order-4711-shipped), and renders the template.
  3. Enqueue: a message is placed on the right queue with an idempotency key.
  4. Send: a worker calls the provider. On success it records "sent". On a temporary error it retries with backoff. On a permanent error (for example, an invalid device token) it removes the token and records the failure.
  5. Track: delivery and open receipts update the record; the data feeds dashboards and "did not receive" support tools.

  100%

  Figure SDA-CSD-06-8 · Deduplicate before sending · © forcsguy.com

  Mistakes people make

  • Sharing one queue between login codes and marketing. A blast delays the code people are waiting for. Use separate lanes.
  • No per-user rate limit. A bug that emits an event in a loop can send a user a hundred messages.
  • Ignoring preferences and legal rules. Unsubscribe, quiet hours and consent are requirements, not nice-to-haves.
  • Keeping dead device tokens. Providers tell you when a token is invalid. Remove it, or your sender reputation and efficiency suffer.
  • Retry storms. Back off with jitter and cap retries, or you will hammer a provider that is already failing.
  • Time zones. "9 am local time" needs the user's zone, and the send time must be scheduled per zone.

  Try it yourself

  Use the System Builder to see a burst absorbed by a queue, and read the rule engine lesson for how triggers, cooldowns and idempotent actions fit the "decide" step.

  More on this site

  Related reading

  • Avoiding insurmountable queue backlogsAmazon Builders' Library · Amazon Web ServicesBlogs
  • Timeouts, retries, and backoff with jitterAmazon Builders' Library · Amazon Web ServicesBlogs
  • Event SourcingMartin FowlerBlogs
  • Uber's Real-Time Push PlatformUber Engineering · UberCase Studies

  Show 6 moreShow fewer

  • NATS ServerNATS · CNCF / NATSGitHub
  • CQRSMartin FowlerBlogs
  • CQRS patternMicrosoft · Microsoft AzureDocumentation
  • Scaling Push Messaging for Millions of Devices at NetflixNetflix engineers · NetflixCase Studies
  • Reference Architectures for Big Data SystemsCmu engineers · CmuCase Studies
  • Messaging System Model at DropboxDropbox engineers · DropboxCase Studies

  Open this lesson with its quiz and exercise

• Core Concepts: How Things Actually Work · 9 minHow AWS Lambda and serverless platforms workWhat happens between an event and your function running: micro-VMs, cold starts, scaling and the limits that follow.Read hereClose

  What "serverless" means

  You provide a function; the platform decides where and when it runs, scales it up and down (even to zero), and bills for the time used. Servers still exist, but you do not manage them. AWS Lambda popularised the model, and Google Cloud Functions and Azure Functions work similarly.

  The path of an invocation

  100%

  Figure SDA-CORE-19-1 · Serverless invocation path · © forcsguy.com

  1. An event arrives from an API gateway, a queue, a file upload, a timer and so on.
  2. The platform's front-end service authenticates it and finds your function.
  3. A placement service looks for a free, warm execution environment for your function. If none exists it picks a worker host and creates one.
  4. The environment runs your code in an isolated micro-VM (AWS uses Firecracker, a lightweight virtual machine that starts in a fraction of a second) so that different customers' code is strongly isolated on shared hardware.
  5. The response flows back (synchronous calls), or the result is recorded (asynchronous calls).

  Cold and warm starts

  An execution environment goes through phases:

  1. Create the micro-VM and download your code or container image.
  2. Initialise the runtime and run your initialisation code (imports, creating clients).
  3. Invoke the handler.
  4. After finishing, the environment is frozen and kept for a while, ready to be reused (a warm start) for the next request.

  The first three steps together are a cold start. It adds tens to hundreds of milliseconds, or more for large packages and heavy runtimes. Warm requests skip straight to the handler.

  100%

  Figure SDA-CORE-19-2 · Synchronous and asynchronous invocation · © forcsguy.com

  Scaling

  The platform runs one request per environment at a time. More simultaneous requests mean more environments, created on demand, up to concurrency limits (account and per function). Scaling is fast but not instant, and a burst can hit cold starts. You can reserve capacity with provisioned concurrency, and cap a function so it cannot overwhelm a downstream database.

  Invocation types

  Type	Behaviour	Failure handling
  Synchronous (API calls)	The caller waits for the result	The caller sees the error and decides to retry
  Asynchronous (events)	Queued internally, run later	Automatic retries, then a dead-letter queue or failure destination
  Stream or queue polling	The platform reads batches from a queue or stream and invokes your function	Failed batches retry, so use partial-failure reporting and idempotent handlers

  Limits to design around

  • A maximum run time (for Lambda, 15 minutes at the time of writing), and limits on memory, package size and payload size. Memory settings also determine CPU share.
  • Ephemeral local storage: do not rely on it for state. Keep state in databases, caches or object storage.
  • No inbound connections to the function: it is invoked, it does not listen.
  • Concurrency pressure on dependencies: thousands of parallel functions can open thousands of database connections. Use a connection pooler or proxy, or limit concurrency.

  When serverless fits

  Good for spiky, event-driven or glue workloads, APIs with uneven traffic, scheduled jobs, file processing, and anything where idle time dominates. Less suited to very long-running jobs, steady high load (where always-on servers can be cheaper), latency-critical paths that cannot tolerate cold starts, or workloads needing special hardware.

  Practical tips

  • Keep packages small and imports lazy.
  • Do expensive setup outside the handler.
  • Make handlers idempotent and stateless.
  • Set timeouts and memory deliberately, and measure cost, since more memory can make a function faster and sometimes cheaper.
  • Add observability: structured logs, metrics for duration, errors and cold starts, and tracing across functions.

  Real-world reading

  Related reading

  • Using load shedding to avoid overloadAmazon Builders' Library · Amazon Web ServicesBlogs
  • GitHub Availability Report: June 2023GitHub Engineering · GitHubCase Studies
  • Avoiding insurmountable queue backlogsAmazon Builders' Library · Amazon Web ServicesBlogs
  • Timeouts, retries, and backoff with jitterAmazon Builders' Library · Amazon Web ServicesBlogs

  Show 6 moreShow fewer

  • Uber's Real-Time Push PlatformUber Engineering · UberCase Studies
  • NATS ServerNATS · CNCF / NATSGitHub
  • GitHub EngineeringGitHub engineers · GitHubBlogs
  • Bulkheads: Partition and Tolerate Failure in One PartSkife engineers · SkifeCase Studies
  • Data Platform at DoorDashDoorDash engineers · DoorDashCase Studies
  • API Platform at Riot GamesRiot Games engineers · Riot GamesCase Studies

  Open this lesson with its quiz and exercise