Queues, streams and asynchronous work

Not everything has to happen now

A user clicks "Sign up". Creating the account must happen before you reply. Sending the welcome email, resizing the avatar and updating dashboards do not. Doing them inside the request makes it slow and fragile: if the email service hiccups, signup fails.

Image: W.carter · CC0 · Source · Wikimedia Commons: People waiting in line at Stena Line ferry terminal, Frederikshavn

Asynchronous processing moves such work out of the request. The request records what must happen and returns; other workers do it shortly after.

100%

Figure SDA-FND-08-1 · Asynchronous work via a queue · © forcsguy.com

What a queue gives you

• Decoupling: the producer does not know or care who consumes.
• Load levelling: a spike of 10,000 jobs is stored and worked off at a steady rate instead of crashing the consumer.
• Resilience: if a consumer is down, messages wait.
• Independent scaling: add more workers when the queue grows.

The price is eventual completion: the work is done soon, not now, and the system has more moving parts.

Queues versus streams

• A message queue (RabbitMQ, SQS) delivers each message to one consumer, which then removes it. Great for task distribution.
• A log or stream (Kafka, Kinesis) keeps messages in order for a retention period. Many independent consumers can read it, and each can replay from any point. Great for event pipelines and for feeding several systems from one source of truth.

Delivery guarantees

Networks and consumers fail, so a broker cannot give every guarantee for free.

Guarantee	Meaning	Risk
At-most-once	Delivered zero or one time	May lose messages
At-least-once	Delivered one or more times	May duplicate messages
Exactly-once	Effect happens exactly once	Costly, and usually achieved by at-least-once plus idempotency

The practical approach: use at-least-once delivery and make handlers idempotent.

100%

Figure SDA-FND-08-2 · Why the moment of acknowledgement decides the guarantee · © forcsguy.com

Idempotency in practice

Make repeating an operation harmless:

• Prefer "set the order status to paid" over "add 10 to the balance".
• Attach a unique idempotency key to a request and remember the keys already processed.
• Use database constraints (a unique key) so a duplicate insert fails safely.

Failure handling

• Retries with backoff: wait longer between attempts (1 s, 2 s, 4 s...) with a little random jitter so many consumers do not retry in lockstep.
• Dead-letter queue: after N failures, move the message aside so it cannot block the rest, and alert a human.
• Ordering: strict global order is expensive. Ordering is usually guaranteed only within a partition, so pick a key (like user id) so related events land in the same one.
• Back pressure: if consumers fall behind, the queue grows. Monitor its depth and age, and scale workers or slow producers.

A worked example: sizing workers and watching the backlog

A photo app creates thumbnails in the background. Each job takes 2 seconds of one worker. Normal traffic is 50 uploads per second.

• Workers needed at the average: 50 jobs/s x 2 s = 100 workers busy all the time. Plan for about 70% utilisation, so roughly 143 workers, which can do about 71 jobs a second.
• A spike: a campaign sends 500 uploads per second for 60 seconds: 30,000 jobs. The queue accepts them instantly; the workers keep draining at about 71 per second. The backlog grows by 500 - 71 = 429 per second, to about 25,700 jobs by the end of the spike.
• Recovery: after the spike, arrivals fall back to 50 per second and the workers drain a net 71 - 50 = 21 per second. Clearing 25,700 jobs takes about 1,200 seconds, 20 minutes. That is the real cost of a spike: not an outage, but a delay.

This is why the metric to watch is queue age (how long the oldest message has waited), not just depth. Depth of 25,000 can be fine for a thumbnail and disastrous for a payment confirmation. Decide how stale each job may become, and alert on that.

100%

Figure SDA-FND-08-3 · A queue turns a spike into a delay · © forcsguy.com

What happens to a message that keeps failing

Workers fail: a bug, a missing record, a dependency down. Without a plan, one bad message can block everything or loop forever.

1. A worker receives a message. The queue hides it for a visibility timeout (say 60 seconds) instead of deleting it.
2. If the worker finishes, it acknowledges and the message is deleted.
3. If the worker crashes or fails, no acknowledgement arrives, the timeout expires and the message becomes visible again for another try.
4. Retries use exponential backoff with jitter so a struggling dependency is not hammered.
5. After a fixed number of attempts (often 3 to 10) the message moves to a dead-letter queue so it stops blocking others. A person or a tool inspects it, fixes the cause, and replays it.

100%

Figure SDA-FND-08-4 · Retries and the dead-letter queue · © forcsguy.com

Because a message can be delivered more than once, every consumer must be idempotent: processing it twice must have the same effect as once. Typical tools are a unique message id recorded in a table (insert first, skip on conflict), or operations that are naturally repeatable ("set status to shipped" rather than "add one").

Mistakes people make

• Non-idempotent consumers. "Charge the card" run twice is a double charge. Use an idempotency key.
• Unbounded retries. A permanently failing message that retries forever wastes capacity. Cap attempts and dead-letter.
• Ignoring queue age. Alert on how old the oldest message is, per queue.
• One queue for everything. A flood of low-priority jobs starves urgent ones. Separate queues, or priorities, by importance.
• Assuming order. Most queues do not guarantee order across consumers. If order matters per user or per order, partition by that key.
• Huge messages. Put big payloads in object storage and send a reference.
• Using a queue when the caller needs the answer now. Queues make work asynchronous; if the user is waiting for the result, they only add delay.

Try it yourself

In the System Builder, open the challenge "A burst of writes": run it without a queue and then with one, and watch the "Writes waiting in queue" chart. For a database-backed version with retries and a dead-letter state, read the SKIP LOCKED queue lesson.

More on this site