Latency, throughput and bandwidth
Three numbers that are often confused, and why a fast network can still feel slow.
On the roadmap:Beginner · step 1: Think like a designer
Lesson 1 of 23 · 6 min read · quiz · exercise
Three different questions
- Latency: how long does one request take from start to finish? Measured in milliseconds.
- Bandwidth: how much could the pipe carry per second? A capacity, measured in bits per second.
- Throughput: how much actually flows per second, measured in requests or bytes per second. It is at most the bandwidth, and often much less.
A motorway analogy: bandwidth is the number of lanes, latency is the time one car needs to drive the length, and throughput is how many cars arrive per hour. Widening the road (more lanes) lets more cars through but does not make any single car arrive sooner.
Why it matters in design
Many speed problems are really latency problems, and more bandwidth will not fix them:
| Symptom | Likely cause | Lever |
|---|---|---|
| Page makes 80 small requests and feels slow | Round-trip latency adds up | Fewer requests, bundling, HTTP/2, caching |
| Users far away complain | Distance | CDN, regional servers |
| Big downloads are slow | Throughput or bandwidth | Compression, parallel transfers, bigger pipe |
| System is slow only at busy times | Throughput limit | Scale out, queue, shed load |
Image: Syced · CC0 · Source · Wikimedia Commons: Peabird Ethernet switch in 2003
Where the time goes
Figure SDA-CORE-01-1 · A latency budget for one page · © forcsguy.com
Tail latency
Averages hide the worst experiences. If 1% of requests take 3 seconds, a page that makes 50 calls will hit one of those slow calls most of the time. Track percentiles (p95, p99), and design to keep the slowest requests under control with timeouts and hedged requests.
Latency budget
Break a target (say 300 ms for a page) into a budget: network 80 ms, application 60 ms, database 50 ms, rendering 80 ms, slack 30 ms. When the total exceeds the target you know where to cut.
Real-world reading
Related reading
- Optimizing Web Servers for High Throughput and Low Latency at DropboxDropbox engineers · DropboxCase Studies
- Natural Language Processing and Content Analysis (2 parts) at Condé NastCondé Nast engineers · Condé NastCase Studies
- Finding Kafka throughput limit in infrastructure at DropboxDropbox engineers · DropboxCase Studies
- Scaling Gradient Boosted Trees for Click-Through-Rate Prediction at YelpYelp engineers · YelpCase Studies
Show 6 moreShow fewer
- Machine Learning (2 parts) at Condé NastCondé Nast engineers · Condé NastCase Studies
- DataSift Architecture: Realtime Datamining at 120,000 Tweets Per Second - High Scalability -High Scalability engineers · High ScalabilityCase Studies
- Deploying Brotli for Static Content at DropboxDropbox engineers · DropboxCase Studies
- How Twitter Handles 3,000 Images Per Second - High Scalability -High Scalability engineers · High ScalabilityCase Studies
- ESPN's Architecture at Scale - Operating at 100,000 Duh Nuh Nuhs Per Second - High Scalability -High Scalability engineers · High ScalabilityCase Studies
- Architecture NotesArchitecture Notes · architecturenotes.coPodcasts & Newsletters
Check your understanding
Question 1 of 3
0 correct so far
A road has many lanes (high bandwidth) but a long length. A single car still takes a long time. Which term describes the time for one car?
Practise it: Trade latency for throughput
Everything below is already set up for this lesson: change the design, run the traffic, and see whether you meet the goal.
- 1The task
- 2Build and test
Reads take the slow path to the database every time. You need fast responses at 2,000 requests per second.
Goal: Slowest 5% under 80 ms, under 1% errors.
A memory lookup is much faster than a database read. What serves most reads from memory?
Try it yourself
A file download over a 1 Gbps link with 200 ms round-trip latency feels slow for small files but fine for large ones. Explain why.
Click what a strong design here would need:
More in Building blocks
© 2026 forcsguy.com. Original lesson text and diagrams. Books and sites linked above belong to their authors.