Queues and latency

Why latency climbs steeply as servers fill up, and what headroom buys.

On the roadmap:Intermediate · step 4: Messaging

Intermediateabout 10 to 15 min6 experiments

Experiments

New here? A few short steps show what happens, one at a time.

Settings

Requests per second90

Work per request (ms)10

Servers sharing one line1

How uneven the jobs aretypical

These servers can do about 100 requests a second between them. Past that, the line never empties.

Busy

90%

Average response

100 ms

Slowest 1% (about)

461 ms

Average wait

90 ms

In the system

9.0

At 90% busy a request waits 90 ms on average, 9.0 times its own work. A small rise in traffic would make this much worse. Add capacity or lower the load.

Average response as the servers fill up

average time for a request

Your 1 server

Your servers are 90% busy now. The curve is flat, then climbs steeply. Add servers to see how sharing one line changes the shape.

In plain words

A request that finds every server busy has to wait in line. The wait is small while servers are mostly idle and grows steeply as they fill up. That is why a system that is fine at 70% busy can fall over at 95%.

Why it matters

Teams are often surprised that a modest rise in traffic can double the response time. The curve is not a line: it is flat for a long way and then climbs almost straight up. Here you can move along it, see how many servers and how even the work is change its shape, and see why capacity plans leave headroom.

How real systems do it

The simulation is a simplified model. Here is what production systems add or do.

Little's law says the average number of requests in the system equals the arrival rate times the average time each spends there. It holds for any stable system, which makes it handy for sizing queues and thread pools.
For one server with random arrivals and service times, average time in the system is the service time divided by one minus the busy fraction, which is where the steep climb near 100% comes from.
Autoscalers are usually set to add capacity well before servers are full, often targeting 50 to 70% utilisation, to leave room for surges and for the time new servers take to start.
Latency percentiles climb even faster than the average near saturation, so the slowest 1% of requests suffer first.