Distributed Cache for S3 at ClickHouse

Ideas you will meet

Caching

Cache-aside, write-through, eviction and invalidation strategies.

Distributed Systems

Consensus, time, failure and the fundamentals of scale-out.

What we already know about this

Our own lessons on these ideas. Read them here before you start, so it makes sense.

• System Design Foundations · 11 minCachingHow keeping popular data in fast memory makes systems quicker, and the invalidation problems that come with it.Read hereClose

  Why a cache helps

  Reading from memory is thousands of times faster than a database round trip (lesson 3). A cache keeps copies of frequently needed data in fast storage so most requests never reach the slower source. The results: lower latency, less load on the database, and lower cost.

  100%

  Figure SDA-FND-06-1 · Cache-aside read path · © forcsguy.com

  Before: every read hits the database

  100%

  After: a cache answers most reads

  100%

  What changed

  • Most reads never reach the database
  • Responses are faster because memory is quicker than disk
  • The cache can be stale, so you decide how long is acceptable

  Where caches live

  • In the browser: controlled with HTTP headers such as Cache-Control.
  • At the edge (CDN): copies of pages and files near the user (lesson 9).
  • In the application tier: a shared cache like Redis or Memcached.
  • Inside the database: buffer pools and query caches.

  You can use several at once, each answering whatever it can.

  Common strategies

  Strategy	How it works	Good for
  Cache-aside	App checks cache, on miss reads the database and fills the cache	the default for most read-heavy data
  Read-through	The cache itself loads from the database on a miss	simpler app code
  Write-through	Writes go to the cache and the database together	data you read right after writing
  Write-behind	Writes go to the cache first and reach the database later	very high write rates, if you can risk losing a little

  Write-behind is fast but dangerous: if the cache fails before it flushes, writes are lost.

  Writes can use the cache in three different ways. The sequence shows when the database learns about the change:

  100%

  Figure SDA-FND-06-2 · Write-through: update the cache and the database together · © forcsguy.com

  100%

  Figure SDA-FND-06-3 · Write-back: acknowledge first, save later · © forcsguy.com

  100%

  Figure SDA-FND-06-4 · Write-around: skip the cache on writes · © forcsguy.com

  Watch one read with and without a warm cache. Press Play both together, and use the Simulate tab to slow the network and see the gap grow.

  Before: no cache

  100%

  After: cache hit

  100%

  What changed

  • The database is not asked at all
  • Two network hops disappear from the request
  • The saving grows as the network gets slower

  The hard part: invalidation

  There are only two hard things in computer science: cache invalidation and naming things.

  A cache holds a copy, and copies go stale. Your options:

  1. Time to live (TTL): entries expire after a set time. Simple, but data can be stale until then.
  2. Explicit invalidation: delete or update the cache entry when the source changes. Fresher, but you must never forget a code path.
  3. Versioned keys: put a version in the key (product:42:v7) so a change simply starts using a new key.

  Most real systems combine a TTL (a safety net) with explicit invalidation on writes.

  100%

  Figure SDA-FND-06-5 · Defending the database from lookups of keys that do not exist · © forcsguy.com

  Eviction: what to drop when full

  Memory is limited, so a cache removes items when it fills. Common policies: LRU (least recently used), LFU (least frequently used) and FIFO. LRU is a good default.

  Failure modes to know

  • Cache stampede: a hot key expires and many requests rebuild it at once. Fix with a lock, request coalescing, or adding random jitter to expiry times.
  • Cache penetration: requests for keys that do not exist always miss and reach the database. Fix by caching "not found" briefly, or with a Bloom filter.
  • Cold start: after a restart the cache is empty and the database takes the full load. Warm the cache or ramp traffic up.
  • Inconsistency: two writers update the cache and database in different orders. Prefer deleting the cache entry on write over updating it.

  A worked example: what a cache is really worth

  Suppose a product page reads from a database in 20 ms and from a cache in 1 ms, and the site serves 10,000 reads per second.

  Cache hit rate	Average latency	Reads reaching the database
  0% (no cache)	20 ms	10,000 per second
  50%	10.5 ms	5,000 per second
  90%	2.9 ms	1,000 per second
  99%	1.2 ms	100 per second

  The average latency is hit x 1 + (1 - hit) x 20. Notice two things. First, going from 90% to 99% barely changes the average (2.9 to 1.2 ms) but cuts database load ten times. That is usually the real prize: a cache is a way to protect the database. Second, the first 50% is cheap and the last 9% is expensive: it needs a much bigger cache, because the remaining misses are the long tail of rarely requested items.

  How big should it be? Traffic is usually skewed: a small share of items gets most requests (often described as "80% of requests go to 20% of items", and for popular content it is more extreme). Estimate your working set: the items requested within your cache lifetime. If 2 million product pages exist but 100,000 get 90% of the traffic, a cache that holds 150,000 of them at about 5 KB each needs only 750 MB. Measure the hit rate while changing the size, and stop growing when the curve flattens.

  100%

  Figure SDA-FND-06-6 · Hit rate rises quickly, then flattens · © forcsguy.com

  Stampedes: when a popular key expires

  Take a homepage fragment read 5,000 times a second and cached for 60 seconds. At the moment it expires, thousands of requests miss at once, and every one of them rebuilds it from the database. The database, which was lightly loaded because of the cache, suddenly takes a spike. If the rebuild is slow, requests pile up, the spike lasts longer, and the system can fall over exactly when it is busiest.

  Ways to defend:

  • Request coalescing (single flight). Let one request rebuild the value while the others wait for its result.
  • A short lock on rebuild. The first request takes a lock and rebuilds; the rest either wait briefly or serve the old value.
  • Stale-while-revalidate. Keep serving the expired value for a few more seconds while one background request refreshes it. Users see slightly old data and never see a spike.
  • Jittered expiry. Add a random amount to each TTL (for example 60 seconds plus up to 10) so thousands of keys written together do not all expire together.
  • Refresh ahead. For a handful of known-hot keys, refresh them on a schedule before they expire.

  100%

  Figure SDA-FND-06-7 · One request rebuilds, the others wait or use the old value · © forcsguy.com

  Mistakes people make

  • Caching without a plan for invalidation. "Cache it for an hour" is a promise that users may see hour-old data. Decide what staleness each kind of data tolerates (a like count: minutes; a price or permission: seconds or never).
  • Caching per-user data under a shared key. If a personalised page is cached under a key that does not include the user, the next visitor sees someone else's data. Include everything that changes the response in the key, or do not cache it.
  • Treating the cache as the source of truth. Caches lose data on restart or eviction. The database must be able to rebuild everything.
  • Ignoring the hot key. One enormously popular key can overload a single cache node. Replicate that key across nodes or add a small in-process cache in front.
  • Not measuring. Track hit rate, evictions, memory use and the latency of misses. A cache you do not measure is a guess.

  Try it yourself

  In the System Builder, open "The database becomes the bottleneck". Run it without a cache and watch the database turn red, then add a cache and try different sizes. In the sandbox, raise the "popular-item skew" slider and notice how much a small cache helps when a few items dominate. You can also open the cached design on the free canvas and try wiring the cache in front of different parts.

  More on this site

  Related reading

  • Flannel: An Application-Level Edge Cache to Make Slack ScaleSlack Engineering · SlackCase Studies
  • Redis UniversityRedis · Redis Ltd.Courses
  • Redis BlogRedis engineers · Redis Ltd.Blogs
  • RedisRedis · Redis Ltd.GitHub

  Show 6 moreShow fewer

  • Redis DocumentationRedis · Redis Ltd.Documentation
  • TAO: Facebook's Distributed Data Store for the Social Graph (paper)Bronson et al. · Meta (Facebook)Papers
  • Build Your Own XCodeCrafters · GitHubGitHub
  • Memory Optimization in Redis at WattpadWattpad engineers · WattpadCase Studies
  • Redis Fleet at HerokuHeroku engineers · HerokuCase Studies
  • Redis at TrivagoTrivago engineers · TrivagoCase Studies

  Open this lesson with its quiz and exercise

• System Design Foundations · 14 minPutting it together: design a URL shortenerA complete worked design using everything so far: requirements, estimates, data model, key generation, caching, scaling and failure handling.Read hereClose

  This lesson uses every earlier idea on one concrete problem. Read it as a model of how to talk through a design.

  1. Clarify requirements

  Functional

  • Given a long URL, create a short one (sho.rt/Ab3dE9).
  • Opening the short URL redirects to the long one.
  • Optional custom alias and expiry (we will add these later).

  Non-functional

  • Redirects are fast (under about 100 ms) and highly available. If redirects fail, every link on the internet that points to us is broken.
  • Short keys must be unique. Links must not disappear unexpectedly.
  • Keys should not be guessable in sequence (so people cannot enumerate every link).

  Out of scope for now: user accounts, detailed analytics.

  2. Estimate

  Using the numbers from lesson 3: 100 million new links a month is about 40 writes per second. With reads at 10 times writes, about 400 reads per second, peaking maybe at 2,000. Five years of data is about 6 billion records at roughly 500 bytes each, so around 3 TB.

  Conclusion: traffic is modest and data is small enough for a single replicated relational database for a long time. The design should be simple, with the focus on the read path.

  3. High-level design

  100%

  Figure SDA-FND-12-1 · URL shortener architecture · © forcsguy.com

  Create a link (write path)

  1. Client sends POST /links with the long URL.
  2. The server obtains a unique ID and encodes it into a short key.
  3. It stores (key, long_url, created_at, expires_at) in the database.
  4. It returns the short URL.

  Open a link (read path)

  1. Client requests GET /Ab3dE9.
  2. The server looks in the cache. On a hit it returns the redirect.
  3. On a miss it reads the database, fills the cache, then redirects.

  4. The interesting parts

  Generating short keys

  Use base62 (a-z, A-Z, 0-9). Seven characters give 62^7, about 3.5 trillion keys, far more than the 6 billion we need.

  Options for producing a unique key:

  Approach	Good	Bad
  Hash the URL and take 7 characters	stateless	collisions, same URL gives same key
  Random key, check the database	simple	needs a check and retry, extra write load
  Unique ID, then base62	no collisions by construction	needs an ID generator, and sequential IDs are guessable

  A good choice: obtain IDs from a generator (a counter handed out in ranges to each server, or a Snowflake-style 64-bit ID), then base62-encode. To stop people guessing the next link, scramble the ID with a reversible function before encoding.

  100%

  Figure SDA-FND-12-2 · Two ways to make a short key · © forcsguy.com

  How base-62 conversion works: divide the number by 62 repeatedly and read the remainders from the last to the first. The number 11157 gives remainders 59, 55 and 2, and those map to the symbols X, T and 2, so the key is 2TX. With seven positions there are 62^7, about 3.5 trillion, possible keys.

  The database

  One table is enough:

  Column	Notes
  key (primary key)	the short code
  long_url	the target
  created_at, expires_at	for expiry

  Lookup is always by key, so the primary key index serves every read. A relational database works; a key-value store would too. Start with the one you can operate.

  The read path and the cache

  The same few links are opened over and over. A cache such as Redis in front of the database absorbs most reads. With cache-aside and a TTL of a day, plus LRU eviction, the hit rate is typically very high. Redirect responses are tiny, so the cache is small.

  A 301 redirect (permanent) can be cached by browsers, reducing traffic but hiding repeat clicks from analytics. A 302 (temporary) keeps every click visible. Choose based on whether you need accurate click counts.

  The two flows end to end

  100%

  Figure SDA-FND-12-3 · Creating a short link · © forcsguy.com

  100%

  Figure SDA-FND-12-4 · Following a short link · © forcsguy.com

  5. Scale and failure

  • Application servers are stateless, so add more behind the balancer (lesson 2).
  • Database reads: add read replicas if the cache is not enough. Writes are only about 40 per second, so sharding is unnecessary. If it ever becomes necessary, shard by key hash (lesson 5).
  • Cache failure: the database must survive a cold cache. Warm it, or ramp traffic back gradually.
  • Database failure: replicas with automatic failover, backups tested by restoring.
  • Abuse: rate limit link creation per client, and check submitted URLs for malware, since shorteners are popular with spammers.
  • Unknown keys: return 404 and cache the miss briefly, so repeated lookups of missing keys do not reach the database (lesson 6).

  6. What would you change at 100 times the size?

  Trace the bottleneck honestly: first the cache and replica count, then the ID generator becoming a coordination point, then the database size. At that point you would shard by key hash, use a multi-region deployment with a global key space, and move click counting fully onto an event stream (lesson 8).

  Worked numbers, end to end

  Assume 100 million new short links a month and a read to write ratio of 100 to 1.

  • Writes: 100,000,000 / (30 x 86,400) = about 40 per second on average, perhaps 120 at peak.
  • Reads: 40 x 100 = 4,000 per second on average, 12,000 at peak.
  • Storage: about 500 bytes per record (long URL, short key, dates, owner). Over 5 years: 100M x 12 x 5 = 6 billion links x 500 bytes = 3 TB. That fits on a handful of database nodes, so the data size is not the hard part.
  • Key length: with base 62 (a to z, A to Z, 0 to 9), 7 characters give 62^7 = about 3.5 trillion keys, comfortably more than the 6 billion we need, with room to spare against guessing.
  • Cache: if 20% of links get 80% of reads, caching the hottest 20% of one day's reads needs roughly 4,000 x 86,400 x 0.2 x 500 bytes = about 35 GB, which one or two cache nodes hold.

  So the real engineering problems are not size. They are generating unique keys without collisions at scale, keeping redirects fast, and not going down when one link goes viral.

  100%

  Figure SDA-FND-12-5 · Capacity numbers for the shortener · © forcsguy.com

  Redirect: 301 or 302?

  A short link answers with a redirect. 301 (permanent) lets browsers and intermediaries cache the redirect, which cuts load on your servers but means you never see the later clicks, so click analytics become inaccurate, and you cannot change the destination for people who cached it. 302 (temporary) is not cached, so every click reaches you: more traffic, but accurate analytics and the ability to change or disable a link immediately. Products that sell analytics, or must be able to remove abusive links, use 302 (or 307). Mention the trade-off in an interview; it shows you think about product consequences.

  Handling a link that goes viral

  One link suddenly gets 50,000 requests per second. Everything funnels onto one key: one cache node, one database row. Defences, in order of cost:

  1. Cache the redirect in a CDN with a short TTL (say 60 seconds). Most of those requests never reach you.
  2. Replicate hot keys across several cache nodes, or keep a tiny in-memory cache on each app server for the hottest few hundred keys.
  3. Rate limit per client so a script cannot add to the storm.
  4. Degrade gracefully: if the database is struggling, serve from cache even past TTL.

  100%

  Figure SDA-FND-12-6 · Layers of defence for a viral link · © forcsguy.com

  Mistakes people make

  • Guessable keys. Sequential ids let anyone enumerate every link. Use random or encoded ids if links may be private.
  • Collisions from hashing the URL. Truncating a hash of the long URL will eventually collide. Detect and retry, or generate keys from a counter or a pre-generated pool.
  • Forgetting abuse. Short links hide destinations, so they are used for phishing. Check destinations against blocklists and let people report links.
  • A single counter as a single point of failure. If one service hands out every id, it needs replicas or ranges handed to each app server in blocks.
  • Ignoring expiry and cleanup. Decide what happens to unused or expired links; otherwise storage and the key space fill with dead entries.

  Try it yourself

  Use the System Builder challenges "The database becomes the bottleneck" and "A viral link", which follow this shortener's growth. Then build the Snowflake ID lab and think about whether a time-based id is suitable for short keys. To build your own, open the URL shortener on the free canvas.

  More on this site

  Related reading

  • MongoDB: Shard KeysMongoDB · MongoDB Inc.Documentation
  • Consistent Hashing and Random Trees (paper)Karger et al. · MITPapers
  • TAO: Facebook's Distributed Data Store for the Social Graph (paper)Bronson et al. · Meta (Facebook)Papers
  • Hello Interview: System Design in a HurryHello InterviewCourses

  Show 6 moreShow fewer

  • Hello Interview: System Design WalkthroughsEvan King & Stefan Mai · Hello InterviewYouTube
  • The System Design PrimerDonne Martin · donnemartinGitHub
  • TAO: The power of the graphMeta Engineering · MetaCase Studies
  • Megastore: Providing Scalable, Highly Available Storage (paper)Baker et al. · GooglePapers
  • Sharding MySQL at TwilioTwilio engineers · TwilioCase Studies
  • Sharding out Elasticsearch at VintedVinted engineers · VintedCase Studies

  Open this lesson with its quiz and exercise

• Classic System Designs · 10 minDesign search autocompleteSuggest completions as the user types, within a tenth of a second: tries, top-k per prefix and how suggestions are updated.Read hereClose

  The problem

  As a user types "syst", show a handful of likely completions: "system design", "systemd", "system of a down". The suggestions must appear almost instantly (under about 100 ms) or they are useless, and they should reflect what people actually search.

  Requirements

  • Fast: respond within ~100 ms.
  • Return the top 5 to 10 completions for the typed prefix, ranked by popularity.
  • Handle high volume (a request per keystroke).
  • Updated regularly with new trends.

  The core data structure: a trie

  A trie (prefix tree) stores words along paths of letters, so all words that share a prefix share a path.

  100%

  Figure SDA-CSD-09-1 · Trie with word frequencies · © forcsguy.com

  To find completions for a prefix, walk to its node, then look at everything beneath it. Doing that on every keystroke is too slow for a big trie, so we precompute: at each node store the top k most popular completions for that prefix.

  100%

  Figure SDA-CSD-09-2 · Top-k completions cached per prefix · © forcsguy.com

  Now a query is: walk the prefix (a few steps), read the stored list. That is fast and independent of how large the subtree is.

  From raw queries to counts

  100%

  Figure SDA-CSD-09-3 · Counting queries into a frequency table · © forcsguy.com

  Two services

  100%

  Figure SDA-CSD-09-4 · Online query path and offline trie build · © forcsguy.com

  Query service (online)

  1. Receive the prefix.
  2. Look it up in the in-memory trie or a prefix cache.
  3. Return the top suggestions.

  Data pipeline (offline)

  1. Search queries are logged.
  2. A periodic job (hourly or daily) aggregates counts per query, filtering out spam and abusive terms.
  3. A builder creates the new trie with top-k lists.
  4. The new version is published to the serving machines and swapped in.

  Real-time updates for every search would be costly and unstable. For breaking trends you can add a small fast path for recent spikes.

  Looking up a prefix in the trie

  100%

  Figure SDA-CSD-09-5 · Autocomplete lookup on a trie with cached top lists · © forcsguy.com

  When one query's count changes, every node on the path above it must refresh its stored list, which is why the trie is rebuilt in a batch rather than edited per search:

  100%

  Figure SDA-CSD-09-6 · Updating a count touches all ancestors · © forcsguy.com

  100%

  Figure SDA-CSD-09-7 · A trie stored as key-value pairs · © forcsguy.com

  100%

  Figure SDA-CSD-09-8 · Filtering unwanted suggestions · © forcsguy.com

  100%

  Figure SDA-CSD-09-9 · Sharding the trie by prefix · © forcsguy.com

  Making it faster and bigger

  • Client side: wait a short pause after typing (debounce), skip very short prefixes, and cache previous responses in the browser.
  • Server side: keep the trie in memory; cache hot prefixes; set a CDN or edge cache for the most common ones.
  • Limit prefix length (for example 50 characters) to bound work.
  • Shard the trie across machines, for example by the first letters, balancing by traffic not by alphabet (some letters are far more common).
  • Replicate the serving tier for availability.

  The numbers and the trie's cost

  Each keystroke is a request, so 10 million users doing 10 searches of about 20 keystrokes is on the order of 24,000 requests per second (about double at peak), while only roughly 0.4 GB of new query text arrives per day. The trie's walk to a prefix is proportional to the prefix length, then reading its children and sorting them would be slow; two changes make lookup effectively constant time: cap the prefix length, and store the top few suggestions on every node (trading space for time). The cost moves to updates: changing one count means refreshing the cached lists on every ancestor up to the root, which is why the trie is normally rebuilt in a batch (say weekly) from aggregated logs and swapped in, rather than edited per query.

  Storing the trie. Either serialise each weekly snapshot into a document store, or flatten it into a key-value store where each prefix is a key and the node's data is the value. Serve from a distributed cache; on a miss, fill the cache from the store.

  Log less. Sample the query log (record one in N) because logging every keystroke-level request is costly and top results are statistically stable.

  Take-down. Put a filter layer in front of the cache to hide unsafe suggestions immediately, and delete them from the data before the next rebuild.

  Variants. Different countries can have their own tries (even served from a CDN); other languages need Unicode in the nodes; trending queries need a faster path, such as stream processing and a ranking that weights recent searches more.

  Quality details

  • Ranking can combine popularity, recency and personalisation.
  • Safety: filter offensive or sensitive suggestions.
  • Typos: optionally tolerate small spelling errors.
  • Languages: tokenisation and ordering differ per language, so plan for it early if you need it.

  Worked numbers: does the whole thing fit in memory?

  Say the system serves the 10 million most common prefixes, storing the top 5 suggestions for each at roughly 100 bytes (five short strings plus scores). That is about 1 GB, comfortably inside one machine's memory, and a few GB with overhead. Even 100 million prefixes is only about 10 GB. So the "answer from memory, rebuilt in the background" design is realistic; no search over a huge dataset happens per keystroke.

  Latency budget for a 100 ms target as seen by the user: the browser debounce is an intentional pause (not part of the budget); network round trip 30 to 50 ms; edge or load balancer 5 ms; service 5 to 10 ms to hash the prefix and read the cached list; response and render 10 ms. There is little room for anything slower, which is why the lookup must be a single key read.

  Request volume: 10 million users x 10 searches x about 20 keystrokes is 2 billion prefix lookups a day, about 24,000 per second on average. Debouncing (waiting about 150 ms after the last keystroke) and not querying for very short prefixes can cut that by half or more.

  100%

  Figure SDA-CSD-09-10 · Debounce and ignore late answers · © forcsguy.com

  Ranking: popular, recent and personal

  Raw popularity is a good start but has two problems. It favours old favourites over trending queries, and it is the same for everyone. A common improvement blends signals:

  score = popularity x decay + trending boost + personal boost

  • Decay: weight recent searches more, for example by halving a query's weight every day or week, so a sudden event rises quickly and fades.
  • Trending: a fast path watches a stream of recent queries and lifts those whose rate jumps compared with their normal level.
  • Personal: a user's own recent searches, kept separately (and cached per user, not in the shared trie).
  • Safety and quality: filter offensive or illegal suggestions and anything with too few searches to be useful or private.

  100%

  Figure SDA-CSD-09-11 · Blending slow and fast ranking signals · © forcsguy.com

  Mistakes people make

  • Searching the data per keystroke. Precompute the top suggestions per prefix.
  • Ignoring out-of-order responses. A slow response for "sy" arriving after "syst" overwrites the list with stale results.
  • Putting personal data in the shared cache. Keep personal suggestions out of anything shared across users.
  • Forgetting normalisation. Case, accents, extra spaces and different Unicode forms must map to the same prefix.
  • No filtering. Autocomplete amplifies whatever people type, including offensive or private terms.
  • Rebuilding on every search. Rebuild in batches and swap versions atomically.

  Try it yourself

  Build the tiny search engine lab for the index side, and read the query understanding lesson for how typed text is cleaned before it is matched.

  Real-world reading

  Related reading

  • YouTube ArchitectureHigh ScalabilityCase Studies
  • Hello Interview: System Design in a HurryHello InterviewCourses
  • Hello Interview: System Design WalkthroughsEvan King & Stefan Mai · Hello InterviewYouTube
  • Meta Engineering BlogMeta Engineering · MetaBlogs

  Show 6 moreShow fewer

  • Cloudflare Learning Center: What is a CDN?CloudflareDocumentation
  • Solving Remote Build Cache Misses (2 parts) at SoundCloudSoundCloud engineers · SoundCloudCase Studies
  • Scaling Memcache at Facebook (paper)Nishtala et al. · Meta (Facebook)Papers
  • Reduce Memcached Memory Usage by 50% at TrivagoTrivago engineers · TrivagoCase Studies
  • Amazon Architecture - High Scalability -High Scalability engineers · High ScalabilityCase Studies
  • LINE Engineering BlogLINE engineers · LINEBlogs

  Open this lesson with its quiz and exercise