Caching

LRU vs LFU vs FIFO. See why a small cache can absorb most traffic.

On the roadmap:Beginner · step 7: Caching basics

Starterabout 10 to 15 min7 experiments

Experiments

New here? A few short steps show what happens, one at a time.

Settings

Which entry to throw out

What the traffic looks like

Cache size (items)50

Distinct keys1000

Traffic skew (0 = uniform)1.0

Requests simulated20000

What it costs you

Requests per second10,000

Cache answers in (ms)1

Database answers in (ms)20

Hit rate

45.9%

Average answer time

11.3 ms

Reach the database

5,411/s

Database load saved

46%

Without a cache every answer takes 20 ms and all 10,000 requests a second hit the database. 9,178 of 20,000 simulated requests were hits.

A cache holding just 5% of the keys still serves 45.9% of requests, because a few keys are asked for again and again. That is why small caches are so effective.

What is in the cache

The 100 most popular keys, hottest first. A lit square is a key the cache holds. 11 of the 20 hottest keys are cached.

Hit rate as the cache warms up

hit rate so far

Hit rate so far

An empty cache misses at first. The line shows the hit rate over all requests so far, and it settles once the popular keys have been loaded.

In plain words

A cache is a small, fast copy of data you already have somewhere slower. When a request arrives you look in the cache first. If the item is there (a hit) you are done. If not (a miss) you fetch it the slow way and keep a copy, throwing something else out if the cache is full.

Why it matters

Real traffic is lopsided: a few things are asked for again and again. A cache that holds just those can answer most requests and shield the database. Here you can see how much that depends on how lopsided the traffic is, how big the cache is, and which entry you throw away.

How real systems do it

The simulation is a simplified model. Here is what production systems add or do.

Redis evicts keys by an approximated LRU (it samples a few keys instead of tracking every one) or, since version 4.0, by LFU. You choose with the maxmemory-policy setting.
Caffeine, a popular Java cache, uses Window-TinyLFU: a new entry only gets in if it looks more valuable than the one it would replace. That is the idea behind the admission policy here.
Caches come in layers: the browser, a CDN, the application and the database's own memory. A request reaches the database only when every layer misses.
Eviction is only half the story. Expiry and invalidation decide how stale the data you serve can be, and stale data is the main price of caching.