Caching

Why a cache helps

Reading from memory is thousands of times faster than a database round trip (lesson 3). A cache keeps copies of frequently needed data in fast storage so most requests never reach the slower source. The results: lower latency, less load on the database, and lower cost.

100%

Figure SDA-FND-06-1 · Cache-aside read path · © forcsguy.com

Where caches live

• In the browser: controlled with HTTP headers such as Cache-Control.
• At the edge (CDN): copies of pages and files near the user (lesson 9).
• In the application tier: a shared cache like Redis or Memcached.
• Inside the database: buffer pools and query caches.

You can use several at once, each answering whatever it can.

Common strategies

Strategy	How it works	Good for
Cache-aside	App checks cache, on miss reads the database and fills the cache	the default for most read-heavy data
Read-through	The cache itself loads from the database on a miss	simpler app code
Write-through	Writes go to the cache and the database together	data you read right after writing
Write-behind	Writes go to the cache first and reach the database later	very high write rates, if you can risk losing a little

Write-behind is fast but dangerous: if the cache fails before it flushes, writes are lost.

Writes can use the cache in three different ways. The sequence shows when the database learns about the change:

100%

Figure SDA-FND-06-2 · Write-through: update the cache and the database together · © forcsguy.com

100%

Figure SDA-FND-06-3 · Write-back: acknowledge first, save later · © forcsguy.com

100%

Figure SDA-FND-06-4 · Write-around: skip the cache on writes · © forcsguy.com

Watch one read with and without a warm cache. Press Play both together, and use the Simulate tab to slow the network and see the gap grow.

The hard part: invalidation

There are only two hard things in computer science: cache invalidation and naming things.

A cache holds a copy, and copies go stale. Your options:

1. Time to live (TTL): entries expire after a set time. Simple, but data can be stale until then.
2. Explicit invalidation: delete or update the cache entry when the source changes. Fresher, but you must never forget a code path.
3. Versioned keys: put a version in the key (product:42:v7) so a change simply starts using a new key.

Most real systems combine a TTL (a safety net) with explicit invalidation on writes.

100%

Figure SDA-FND-06-5 · Defending the database from lookups of keys that do not exist · © forcsguy.com

Eviction: what to drop when full

Memory is limited, so a cache removes items when it fills. Common policies: LRU (least recently used), LFU (least frequently used) and FIFO. LRU is a good default.

Failure modes to know

• Cache stampede: a hot key expires and many requests rebuild it at once. Fix with a lock, request coalescing, or adding random jitter to expiry times.
• Cache penetration: requests for keys that do not exist always miss and reach the database. Fix by caching "not found" briefly, or with a Bloom filter.
• Cold start: after a restart the cache is empty and the database takes the full load. Warm the cache or ramp traffic up.
• Inconsistency: two writers update the cache and database in different orders. Prefer deleting the cache entry on write over updating it.

A worked example: what a cache is really worth

Suppose a product page reads from a database in 20 ms and from a cache in 1 ms, and the site serves 10,000 reads per second.

Cache hit rate	Average latency	Reads reaching the database
0% (no cache)	20 ms	10,000 per second
50%	10.5 ms	5,000 per second
90%	2.9 ms	1,000 per second
99%	1.2 ms	100 per second

The average latency is hit x 1 + (1 - hit) x 20. Notice two things. First, going from 90% to 99% barely changes the average (2.9 to 1.2 ms) but cuts database load ten times. That is usually the real prize: a cache is a way to protect the database. Second, the first 50% is cheap and the last 9% is expensive: it needs a much bigger cache, because the remaining misses are the long tail of rarely requested items.

How big should it be? Traffic is usually skewed: a small share of items gets most requests (often described as "80% of requests go to 20% of items", and for popular content it is more extreme). Estimate your working set: the items requested within your cache lifetime. If 2 million product pages exist but 100,000 get 90% of the traffic, a cache that holds 150,000 of them at about 5 KB each needs only 750 MB. Measure the hit rate while changing the size, and stop growing when the curve flattens.

100%

Figure SDA-FND-06-6 · Hit rate rises quickly, then flattens · © forcsguy.com

Stampedes: when a popular key expires

Take a homepage fragment read 5,000 times a second and cached for 60 seconds. At the moment it expires, thousands of requests miss at once, and every one of them rebuilds it from the database. The database, which was lightly loaded because of the cache, suddenly takes a spike. If the rebuild is slow, requests pile up, the spike lasts longer, and the system can fall over exactly when it is busiest.

Ways to defend:

• Request coalescing (single flight). Let one request rebuild the value while the others wait for its result.
• A short lock on rebuild. The first request takes a lock and rebuilds; the rest either wait briefly or serve the old value.
• Stale-while-revalidate. Keep serving the expired value for a few more seconds while one background request refreshes it. Users see slightly old data and never see a spike.
• Jittered expiry. Add a random amount to each TTL (for example 60 seconds plus up to 10) so thousands of keys written together do not all expire together.
• Refresh ahead. For a handful of known-hot keys, refresh them on a schedule before they expire.

100%

Figure SDA-FND-06-7 · One request rebuilds, the others wait or use the old value · © forcsguy.com

Mistakes people make

• Caching without a plan for invalidation. "Cache it for an hour" is a promise that users may see hour-old data. Decide what staleness each kind of data tolerates (a like count: minutes; a price or permission: seconds or never).
• Caching per-user data under a shared key. If a personalised page is cached under a key that does not include the user, the next visitor sees someone else's data. Include everything that changes the response in the key, or do not cache it.
• Treating the cache as the source of truth. Caches lose data on restart or eviction. The database must be able to rebuild everything.
• Ignoring the hot key. One enormously popular key can overload a single cache node. Replicate that key across nodes or add a small in-process cache in front.
• Not measuring. Track hit rate, evictions, memory use and the latency of misses. A cache you do not measure is a guess.

Try it yourself

In the System Builder, open "The database becomes the bottleneck". Run it without a cache and watch the database turn red, then add a cache and try different sizes. In the sandbox, raise the "popular-item skew" slider and notice how much a small cache helps when a few items dominate. You can also open the cached design on the free canvas and try wiring the cache in front of different parts.

More on this site