Library / Blogs

Scalable Web Architecture and Distributed Systems

A full chapter on the building blocks of large websites: availability, performance, reliability, scalability, caches, proxies, indexes, load balancers and queues. Free to read here under CC BY 3.0.

Beginner40 minDistributed Systems Caching Load Balancer

By Kate Matsudaira · The Architecture of Open Source Applications · Added 2026-10-22

Open original source

All content belongs to Kate Matsudaira / The Architecture of Open Source Applications. We only link to it and give credit. License: CC BY 3.0. Report a problem

Learn this article

About 67 min left. Your work is saved as you go.

What you are about to study

The Architecture of Open Source Applications wrote this to explain how they thought about a problem. You can follow it without prior background.

Ideas you will meet

Distributed Systems: Consensus, time, failure and the fundamentals of scale-out.
Caching: Cache-aside, write-through, eviction and invalidation strategies.
Load Balancer: Spread traffic across servers with L4/L7 balancing, health checks and algorithms.

What we already know about this

Our own lessons on these ideas. Read them here before you start, so it makes sense.

System Design Foundations · 11 minCachingHow keeping popular data in fast memory makes systems quicker, and the invalidation problems that come with it.Read here

Why a cache helps

Reading from memory is thousands of times faster than a database round trip (lesson 3). A cache keeps copies of frequently needed data in fast storage so most requests never reach the slower source. The results: lower latency, less load on the database, and lower cost.

Figure SDA-FND-06-1 · Cache-aside read path · © forcsguy.com

Before: every read hits the database

After: a cache answers most reads

What changed

Most reads never reach the database
Responses are faster because memory is quicker than disk
The cache can be stale, so you decide how long is acceptable

Where caches live

In the browser: controlled with HTTP headers such as Cache-Control.
At the edge (CDN): copies of pages and files near the user (lesson 9).
In the application tier: a shared cache like Redis or Memcached.
Inside the database: buffer pools and query caches.

You can use several at once, each answering whatever it can.

Common strategies

Strategy	How it works	Good for
Cache-aside	App checks cache, on miss reads the database and fills the cache	the default for most read-heavy data
Read-through	The cache itself loads from the database on a miss	simpler app code
Write-through	Writes go to the cache and the database together	data you read right after writing
Write-behind	Writes go to the cache first and reach the database later	very high write rates, if you can risk losing a little

Write-behind is fast but dangerous: if the cache fails before it flushes, writes are lost.

Writes can use the cache in three different ways. The sequence shows when the database learns about the change:

Figure SDA-FND-06-2 · Write-through: update the cache and the database together · © forcsguy.com

Figure SDA-FND-06-3 · Write-back: acknowledge first, save later · © forcsguy.com

Figure SDA-FND-06-4 · Write-around: skip the cache on writes · © forcsguy.com

Watch one read with and without a warm cache. Press Play both together, and use the Simulate tab to slow the network and see the gap grow.

Before: no cache

After: cache hit

What changed

The database is not asked at all
Two network hops disappear from the request
The saving grows as the network gets slower

The hard part: invalidation

There are only two hard things in computer science: cache invalidation and naming things.

A cache holds a copy, and copies go stale. Your options:

Time to live (TTL): entries expire after a set time. Simple, but data can be stale until then.
Explicit invalidation: delete or update the cache entry when the source changes. Fresher, but you must never forget a code path.
Versioned keys: put a version in the key (product:42:v7) so a change simply starts using a new key.

Most real systems combine a TTL (a safety net) with explicit invalidation on writes.

Figure SDA-FND-06-5 · Defending the database from lookups of keys that do not exist · © forcsguy.com

Eviction: what to drop when full

Memory is limited, so a cache removes items when it fills. Common policies: LRU (least recently used), LFU (least frequently used) and FIFO. LRU is a good default.

Failure modes to know

Cache stampede: a hot key expires and many requests rebuild it at once. Fix with a lock, request coalescing, or adding random jitter to expiry times.
Cache penetration: requests for keys that do not exist always miss and reach the database. Fix by caching "not found" briefly, or with a Bloom filter.
Cold start: after a restart the cache is empty and the database takes the full load. Warm the cache or ramp traffic up.
Inconsistency: two writers update the cache and database in different orders. Prefer deleting the cache entry on write over updating it.

A worked example: what a cache is really worth

Suppose a product page reads from a database in 20 ms and from a cache in 1 ms, and the site serves 10,000 reads per second.

Cache hit rate	Average latency	Reads reaching the database
0% (no cache)	20 ms	10,000 per second
50%	10.5 ms	5,000 per second
90%	2.9 ms	1,000 per second
99%	1.2 ms	100 per second

The average latency is hit x 1 + (1 - hit) x 20. Notice two things. First, going from 90% to 99% barely changes the average (2.9 to 1.2 ms) but cuts database load ten times. That is usually the real prize: a cache is a way to protect the database. Second, the first 50% is cheap and the last 9% is expensive: it needs a much bigger cache, because the remaining misses are the long tail of rarely requested items.

How big should it be? Traffic is usually skewed: a small share of items gets most requests (often described as "80% of requests go to 20% of items", and for popular content it is more extreme). Estimate your working set: the items requested within your cache lifetime. If 2 million product pages exist but 100,000 get 90% of the traffic, a cache that holds 150,000 of them at about 5 KB each needs only 750 MB. Measure the hit rate while changing the size, and stop growing when the curve flattens.

Figure SDA-FND-06-6 · Hit rate rises quickly, then flattens · © forcsguy.com

Stampedes: when a popular key expires

Take a homepage fragment read 5,000 times a second and cached for 60 seconds. At the moment it expires, thousands of requests miss at once, and every one of them rebuilds it from the database. The database, which was lightly loaded because of the cache, suddenly takes a spike. If the rebuild is slow, requests pile up, the spike lasts longer, and the system can fall over exactly when it is busiest.

Ways to defend:

Request coalescing (single flight). Let one request rebuild the value while the others wait for its result.
A short lock on rebuild. The first request takes a lock and rebuilds; the rest either wait briefly or serve the old value.
Stale-while-revalidate. Keep serving the expired value for a few more seconds while one background request refreshes it. Users see slightly old data and never see a spike.
Jittered expiry. Add a random amount to each TTL (for example 60 seconds plus up to 10) so thousands of keys written together do not all expire together.
Refresh ahead. For a handful of known-hot keys, refresh them on a schedule before they expire.

Figure SDA-FND-06-7 · One request rebuilds, the others wait or use the old value · © forcsguy.com

Mistakes people make

Caching without a plan for invalidation. "Cache it for an hour" is a promise that users may see hour-old data. Decide what staleness each kind of data tolerates (a like count: minutes; a price or permission: seconds or never).
Caching per-user data under a shared key. If a personalised page is cached under a key that does not include the user, the next visitor sees someone else's data. Include everything that changes the response in the key, or do not cache it.
Treating the cache as the source of truth. Caches lose data on restart or eviction. The database must be able to rebuild everything.
Ignoring the hot key. One enormously popular key can overload a single cache node. Replicate that key across nodes or add a small in-process cache in front.
Not measuring. Track hit rate, evictions, memory use and the latency of misses. A cache you do not measure is a guess.

Try it yourself

In the System Builder, open "The database becomes the bottleneck". Run it without a cache and watch the database turn red, then add a cache and try different sizes. In the sandbox, raise the "popular-item skew" slider and notice how much a small cache helps when a few items dominate. You can also open the cached design on the free canvas and try wiring the cache in front of different parts.

More on this site

Two ways to grow

When one server is not enough you have two options.

A technician working at a server rack in a data center Image: Derrick Coetzee · CC0 · Source · Wikimedia Commons: Technician with laptop working on server rack at NERSC

Vertical scaling ("scale up"): use a bigger machine with more CPU, memory or disk. It is simple because nothing in the design changes.
Horizontal scaling ("scale out"): add more machines and share the work between them. It needs more design effort but has no hard ceiling.

	Vertical	Horizontal
Effort	Low	Higher
Ceiling	Hardware limit	Practically none
Failure	One machine is a single point of failure	Survives losing a machine
Cost curve	Gets steep for the largest machines	Roughly linear with commodity machines

Real systems use both: scale up until it gets expensive or risky, then scale out.

Figure SDA-FND-02-1 · Scale up versus scale out · © forcsguy.com

What a load balancer does

Once you have several servers, something must decide which one handles each request. That is the load balancer.

Figure SDA-FND-02-2 · Load balancer in front of a server pool · © forcsguy.com

It does three jobs:

Distributes requests so no server is overloaded.
Checks health. It probes each server regularly and stops sending traffic to ones that fail.
Hides the pool. Clients talk to one address, and servers can be added or removed without clients noticing.

Common strategies

Round robin: take turns. Simple and fine when requests are similar.
Least connections: send to the server with the fewest active requests. Better when request cost varies.
Hashing (for example on user id or IP): the same key goes to the same server. Useful when a server keeps a local cache for that key.
Weighted: bigger machines get a bigger share.

Load balancers work at different layers. A layer 4 balancer routes by IP and port and is very fast. A layer 7 balancer reads the HTTP request, so it can route by URL path or header and terminate TLS.

A security benefit comes with this layout: only the balancer needs a public address. The servers behind it use private addresses, reachable from inside the network but not from the internet, so clients cannot bypass the balancer or attack a server directly. If one server goes offline, the balancer sends its traffic to the others and a replacement can join the pool.

Before: one server

After: load balancer and several servers

What changed

Traffic is shared across servers
One server failing no longer takes the site down
You can add capacity without downtime

Stateless servers make scaling easy

A server is stateless when it keeps nothing about a user between requests. Everything it needs comes from the request or from a shared store.

Figure SDA-FND-02-3 · Stateful versus stateless servers · © forcsguy.com

With stateless servers any request can go to any server, so you can add servers at will, replace a failed one without losing data, and run auto-scaling. The state does not disappear. It moves to a database, a cache or a session store, which you then have to scale and protect, which is what the next lessons are about.

Putting it together

A typical first scalable layout:

Users reach a load balancer.
It spreads requests over a pool of stateless application servers.
Servers read and write a shared database (and later a cache).
Static files such as images and scripts are served from a CDN so the servers do not carry that traffic (lesson 9).

This design handles a surprisingly large amount of traffic. The first thing to give way is usually the database, which is where we go next.

A worked example: how many servers do you need?

Capacity planning is arithmetic you can do on a napkin. Say your product has 5 million daily users and each makes about 20 requests a day.

Requests per day: 5,000,000 x 20 = 100 million.
Per second on average: 100,000,000 / 86,400 = about 1,160 requests per second.
Traffic is not flat. A common rule of thumb is that the busiest second is 2 to 3 times the average, so plan for about 3,500 per second.
One server handles about 400 requests per second flat out, but you never want servers at 100%: queues build and latency explodes near full load. Plan for roughly 70% utilisation, so 280 requests per second each.
3,500 / 280 = 12.5, so 13 servers carry the peak.
Add spare capacity so you survive failures: with 13 servers, losing 2 leaves 11, which carry only 3,080 requests per second. Plan for N+2: 15 servers.

Figure SDA-FND-02-4 · From users to a server count · © forcsguy.com

The same method works for any tier. Only the "requests per server" number changes, and you should measure it with a load test rather than guess.

Choosing a balancing method with real numbers

Round robin is fair in the number of requests, not in the work. Imagine three servers and a stream where most requests take 10 ms but every fifth takes 1 second (a report export, say). Round robin gives each server the same count, but by chance one server collects several slow requests at once, its queue grows, and its users wait while the other two are nearly idle.

Least connections notices: the server with the most open requests gets fewer new ones, so the slow requests spread out as they arrive. It costs the balancer a little bookkeeping and helps most when request costs vary a lot. Use plain round robin when requests are similar and servers identical; use least connections, or "power of two choices" (pick two servers at random and send to the less busy one), when they are not. For caches and stateful workloads, consistent hashing keeps the same key on the same server so its memory stays useful.

Health checks and draining

A balancer is only as good as its knowledge of which servers are alive.

Active health checks call a lightweight endpoint (for example /health) every few seconds. After two or three failures in a row, the server leaves rotation; after a few successes it returns. Make the check reflect real readiness: a server that is up but cannot reach its database should fail the check.
Passive checks watch real traffic: too many errors or timeouts from one server and it is ejected.
Connection draining matters at deploys. Before stopping a server, tell the balancer to send it no new requests, let in-flight requests finish (typically 30 to 60 seconds), then stop it. Without draining, every deploy drops a few user requests.

Figure SDA-FND-02-5 · Rolling out a new version without dropping requests · © forcsguy.com

Mistakes people make

Treating the balancer as magic. It is a server too and can be a single point of failure. Run at least two, or use a managed one.
Adding servers behind a single database. App servers scale out easily; if the database is the bottleneck, more app servers only make it fail faster.
Keeping state on the server. Sessions, uploads and caches held in one server's memory break the moment a request lands elsewhere. Move them to a shared store.
Sizing for the average. The system fails at peak, which is when it matters.
Never testing failure. Kill a server during a load test. If users notice, you have found a gap before they did.

Try it yourself

Open the System Builder challenge "One server is not enough". Run the test on one server and watch it fail, add a load balancer without adding servers, then add servers and see where the numbers settle. Or skip the challenge and open a load-balanced system on the free canvas, then change the wiring yourself.

More on this site

1. Clarify requirements

Functional

Given a long URL, create a short one (sho.rt/Ab3dE9).
Opening the short URL redirects to the long one.
Optional custom alias and expiry (we will add these later).

Non-functional

Redirects are fast (under about 100 ms) and highly available. If redirects fail, every link on the internet that points to us is broken.
Short keys must be unique. Links must not disappear unexpectedly.
Keys should not be guessable in sequence (so people cannot enumerate every link).

Out of scope for now: user accounts, detailed analytics.

2. Estimate

Using the numbers from lesson 3: 100 million new links a month is about 40 writes per second. With reads at 10 times writes, about 400 reads per second, peaking maybe at 2,000. Five years of data is about 6 billion records at roughly 500 bytes each, so around 3 TB.

Conclusion: traffic is modest and data is small enough for a single replicated relational database for a long time. The design should be simple, with the focus on the read path.

3. High-level design

Create a link (write path)

Client sends POST /links with the long URL.
The server obtains a unique ID and encodes it into a short key.
It stores (key, long_url, created_at, expires_at) in the database.
It returns the short URL.

Open a link (read path)

Client requests GET /Ab3dE9.
The server looks in the cache. On a hit it returns the redirect.
On a miss it reads the database, fills the cache, then redirects.

4. The interesting parts

Generating short keys

Use base62 (a-z, A-Z, 0-9). Seven characters give 62^7, about 3.5 trillion keys, far more than the 6 billion we need.

Options for producing a unique key:

Approach	Good	Bad
Hash the URL and take 7 characters	stateless	collisions, same URL gives same key
Random key, check the database	simple	needs a check and retry, extra write load
Unique ID, then base62	no collisions by construction	needs an ID generator, and sequential IDs are guessable

A good choice: obtain IDs from a generator (a counter handed out in ranges to each server, or a Snowflake-style 64-bit ID), then base62-encode. To stop people guessing the next link, scramble the ID with a reversible function before encoding.

How base-62 conversion works: divide the number by 62 repeatedly and read the remainders from the last to the first. The number 11157 gives remainders 59, 55 and 2, and those map to the symbols X, T and 2, so the key is 2TX. With seven positions there are 62^7, about 3.5 trillion, possible keys.

The database

One table is enough:

Column	Notes
`key` (primary key)	the short code
`long_url`	the target
`created_at`, `expires_at`	for expiry

Lookup is always by key, so the primary key index serves every read. A relational database works; a key-value store would too. Start with the one you can operate.

The read path and the cache

The same few links are opened over and over. A cache such as Redis in front of the database absorbs most reads. With cache-aside and a TTL of a day, plus LRU eviction, the hit rate is typically very high. Redirect responses are tiny, so the cache is small.

A 301 redirect (permanent) can be cached by browsers, reducing traffic but hiding repeat clicks from analytics. A 302 (temporary) keeps every click visible. Choose based on whether you need accurate click counts.

The two flows end to end

5. Scale and failure

Application servers are stateless, so add more behind the balancer (lesson 2).
Database reads: add read replicas if the cache is not enough. Writes are only about 40 per second, so sharding is unnecessary. If it ever becomes necessary, shard by key hash (lesson 5).
Cache failure: the database must survive a cold cache. Warm it, or ramp traffic back gradually.
Database failure: replicas with automatic failover, backups tested by restoring.
Abuse: rate limit link creation per client, and check submitted URLs for malware, since shorteners are popular with spammers.
Unknown keys: return 404 and cache the miss briefly, so repeated lookups of missing keys do not reach the database (lesson 6).

6. What would you change at 100 times the size?

Trace the bottleneck honestly: first the cache and replica count, then the ID generator becoming a coordination point, then the database size. At that point you would shard by key hash, use a multi-region deployment with a global key space, and move click counting fully onto an event stream (lesson 8).

Worked numbers, end to end

Assume 100 million new short links a month and a read to write ratio of 100 to 1.

Writes: 100,000,000 / (30 x 86,400) = about 40 per second on average, perhaps 120 at peak.
Reads: 40 x 100 = 4,000 per second on average, 12,000 at peak.
Storage: about 500 bytes per record (long URL, short key, dates, owner). Over 5 years: 100M x 12 x 5 = 6 billion links x 500 bytes = 3 TB. That fits on a handful of database nodes, so the data size is not the hard part.
Key length: with base 62 (a to z, A to Z, 0 to 9), 7 characters give 62^7 = about 3.5 trillion keys, comfortably more than the 6 billion we need, with room to spare against guessing.
Cache: if 20% of links get 80% of reads, caching the hottest 20% of one day's reads needs roughly 4,000 x 86,400 x 0.2 x 500 bytes = about 35 GB, which one or two cache nodes hold.

So the real engineering problems are not size. They are generating unique keys without collisions at scale, keeping redirects fast, and not going down when one link goes viral.

Redirect: 301 or 302?

A short link answers with a redirect. 301 (permanent) lets browsers and intermediaries cache the redirect, which cuts load on your servers but means you never see the later clicks, so click analytics become inaccurate, and you cannot change the destination for people who cached it. 302 (temporary) is not cached, so every click reaches you: more traffic, but accurate analytics and the ability to change or disable a link immediately. Products that sell analytics, or must be able to remove abusive links, use 302 (or 307). Mention the trade-off in an interview; it shows you think about product consequences.

Handling a link that goes viral

One link suddenly gets 50,000 requests per second. Everything funnels onto one key: one cache node, one database row. Defences, in order of cost:

Cache the redirect in a CDN with a short TTL (say 60 seconds). Most of those requests never reach you.
Replicate hot keys across several cache nodes, or keep a tiny in-memory cache on each app server for the hottest few hundred keys.
Rate limit per client so a script cannot add to the storm.
Degrade gracefully: if the database is struggling, serve from cache even past TTL.

Mistakes people make

Guessable keys. Sequential ids let anyone enumerate every link. Use random or encoded ids if links may be private.
Collisions from hashing the URL. Truncating a hash of the long URL will eventually collide. Detect and retry, or generate keys from a counter or a pre-generated pool.
Forgetting abuse. Short links hide destinations, so they are used for phishing. Check destinations against blocklists and let people report links.
A single counter as a single point of failure. If one service hands out every id, it needs replicas or ranges handed to each app server in blocks.
Ignoring expiry and cleanup. Decide what happens to unused or expired links; otherwise storage and the key space fill with dead entries.

Try it yourself

Use the System Builder challenges "The database becomes the bottleneck" and "A viral link", which follow this shortener's growth. Then build the Snowflake ID lab and think about whether a time-based id is suitable for short keys. To build your own, open the URL shortener on the free canvas.

More on this site

Go deeper on this

A path through videos, articles, other case studies and code on the same ideas as Scalable Web Architecture and Distributed Systems.

1
Warm upGet the vocabulary first.
- Courses
  Google SRE Book
  Free online book on running reliable production systems at Google scale.
  GoogleAdvanced
  Study this Original (opens in a new tab)
- Documentation
  Google Cloud Architecture Framework
  Design principles for reliability, performance and operational excellence on GCP.
  Google CloudIntermediate
  Study this Original (opens in a new tab)
WatchHow others solve the same problem, on video.
Videos belong to their creators and play from YouTube. More on YouTube (opens in a new tab)
3
ReadEngineers explain it in their own words.
- Blogs
  High Scalability
  Long-running blog dissecting the architectures of large internet companies.
  High ScalabilityIntermediate
  Study this Original (opens in a new tab)
- Papers · Read here
  Consistent Hashing and Random Trees (paper)
  The original consistent-hashing paper, written for distributing web caches.
  MITAdvanced
  Study this Original (opens in a new tab)
4
See it in another companyThe same ideas, a different system.
- Case Studies · Read here
  CDN Services at Spotify
  Spotify built its business on flawless content delivery. Our streaming platform serves up more than 50 million tracks (plus an array of images and other assets) to more than 230 million monthly active users around the...
  SpotifyIntermediate
  Study this Original (opens in a new tab)
- Case Studies · Read here
  Moving persistent data out of Redis at Github
  Historically, we have used Redis in two ways at GitHub: We used it as an LRU cache to conveniently store the results of expensive computations over data originally persisted in…
  GithubIntermediate
  Study this Original (opens in a new tab)
5
Read the codeSee how it looks when it is real.
- GitHub
  Awesome System Design Resources
  Concepts, patterns and case-study links organised for interview preparation.
  ashishps1Beginner
  Study this Original (opens in a new tab)
- GitHub
  The System Design Primer
  The most-starred system design study guide: scalability concepts, trade-offs and Anki flashcards.
  donnemartinBeginner
  Study this Original (opens in a new tab)

Keep learning

Loading community…

Cookie preferences

Scalable Web Architecture and Distributed Systems

Learn this article

What you are about to study

Why a cache helps

Where caches live

Common strategies

The hard part: invalidation

Eviction: what to drop when full

Failure modes to know

A worked example: what a cache is really worth

Stampedes: when a popular key expires

Mistakes people make

Try it yourself

More on this site

Two ways to grow

What a load balancer does

Common strategies

Stateless servers make scaling easy

Putting it together

A worked example: how many servers do you need?

Choosing a balancing method with real numbers

Health checks and draining

Mistakes people make

Try it yourself

More on this site

1. Clarify requirements

2. Estimate

3. High-level design

4. The interesting parts

Generating short keys

The database

The read path and the cache

The two flows end to end

5. Scale and failure

6. What would you change at 100 times the size?

Worked numbers, end to end

Redirect: 301 or 302?

Handling a link that goes viral

Mistakes people make

Try it yourself

More on this site

Go deeper on this

Warm upGet the vocabulary first.

WatchHow others solve the same problem, on video.

ReadEngineers explain it in their own words.

See it in another companyThe same ideas, a different system.

Read the codeSee how it looks when it is real.

Keep learning