Putting it together: design a URL shortener

This lesson uses every earlier idea on one concrete problem. Read it as a model of how to talk through a design.

1. Clarify requirements

Functional

• Given a long URL, create a short one (sho.rt/Ab3dE9).
• Opening the short URL redirects to the long one.
• Optional custom alias and expiry (we will add these later).

Non-functional

• Redirects are fast (under about 100 ms) and highly available. If redirects fail, every link on the internet that points to us is broken.
• Short keys must be unique. Links must not disappear unexpectedly.
• Keys should not be guessable in sequence (so people cannot enumerate every link).

Out of scope for now: user accounts, detailed analytics.

2. Estimate

Using the numbers from lesson 3: 100 million new links a month is about 40 writes per second. With reads at 10 times writes, about 400 reads per second, peaking maybe at 2,000. Five years of data is about 6 billion records at roughly 500 bytes each, so around 3 TB.

Conclusion: traffic is modest and data is small enough for a single replicated relational database for a long time. The design should be simple, with the focus on the read path.

3. High-level design

100%

Figure SDA-FND-12-1 · URL shortener architecture · © forcsguy.com

Create a link (write path)

1. Client sends POST /links with the long URL.
2. The server obtains a unique ID and encodes it into a short key.
3. It stores (key, long_url, created_at, expires_at) in the database.
4. It returns the short URL.

Open a link (read path)

1. Client requests GET /Ab3dE9.
2. The server looks in the cache. On a hit it returns the redirect.
3. On a miss it reads the database, fills the cache, then redirects.

4. The interesting parts

Generating short keys

Use base62 (a-z, A-Z, 0-9). Seven characters give 62^7, about 3.5 trillion keys, far more than the 6 billion we need.

Options for producing a unique key:

Approach	Good	Bad
Hash the URL and take 7 characters	stateless	collisions, same URL gives same key
Random key, check the database	simple	needs a check and retry, extra write load
Unique ID, then base62	no collisions by construction	needs an ID generator, and sequential IDs are guessable

A good choice: obtain IDs from a generator (a counter handed out in ranges to each server, or a Snowflake-style 64-bit ID), then base62-encode. To stop people guessing the next link, scramble the ID with a reversible function before encoding.

100%

Figure SDA-FND-12-2 · Two ways to make a short key · © forcsguy.com

How base-62 conversion works: divide the number by 62 repeatedly and read the remainders from the last to the first. The number 11157 gives remainders 59, 55 and 2, and those map to the symbols X, T and 2, so the key is 2TX. With seven positions there are 62^7, about 3.5 trillion, possible keys.

The database

One table is enough:

Column	Notes
key (primary key)	the short code
long_url	the target
created_at, expires_at	for expiry

Lookup is always by key, so the primary key index serves every read. A relational database works; a key-value store would too. Start with the one you can operate.

The read path and the cache

The same few links are opened over and over. A cache such as Redis in front of the database absorbs most reads. With cache-aside and a TTL of a day, plus LRU eviction, the hit rate is typically very high. Redirect responses are tiny, so the cache is small.

A 301 redirect (permanent) can be cached by browsers, reducing traffic but hiding repeat clicks from analytics. A 302 (temporary) keeps every click visible. Choose based on whether you need accurate click counts.

The two flows end to end

100%

Figure SDA-FND-12-3 · Creating a short link · © forcsguy.com

100%

Figure SDA-FND-12-4 · Following a short link · © forcsguy.com

5. Scale and failure

• Application servers are stateless, so add more behind the balancer (lesson 2).
• Database reads: add read replicas if the cache is not enough. Writes are only about 40 per second, so sharding is unnecessary. If it ever becomes necessary, shard by key hash (lesson 5).
• Cache failure: the database must survive a cold cache. Warm it, or ramp traffic back gradually.
• Database failure: replicas with automatic failover, backups tested by restoring.
• Abuse: rate limit link creation per client, and check submitted URLs for malware, since shorteners are popular with spammers.
• Unknown keys: return 404 and cache the miss briefly, so repeated lookups of missing keys do not reach the database (lesson 6).

6. What would you change at 100 times the size?

Trace the bottleneck honestly: first the cache and replica count, then the ID generator becoming a coordination point, then the database size. At that point you would shard by key hash, use a multi-region deployment with a global key space, and move click counting fully onto an event stream (lesson 8).

Worked numbers, end to end

Assume 100 million new short links a month and a read to write ratio of 100 to 1.

• Writes: 100,000,000 / (30 x 86,400) = about 40 per second on average, perhaps 120 at peak.
• Reads: 40 x 100 = 4,000 per second on average, 12,000 at peak.
• Storage: about 500 bytes per record (long URL, short key, dates, owner). Over 5 years: 100M x 12 x 5 = 6 billion links x 500 bytes = 3 TB. That fits on a handful of database nodes, so the data size is not the hard part.
• Key length: with base 62 (a to z, A to Z, 0 to 9), 7 characters give 62^7 = about 3.5 trillion keys, comfortably more than the 6 billion we need, with room to spare against guessing.
• Cache: if 20% of links get 80% of reads, caching the hottest 20% of one day's reads needs roughly 4,000 x 86,400 x 0.2 x 500 bytes = about 35 GB, which one or two cache nodes hold.

So the real engineering problems are not size. They are generating unique keys without collisions at scale, keeping redirects fast, and not going down when one link goes viral.

100%

Figure SDA-FND-12-5 · Capacity numbers for the shortener · © forcsguy.com

Redirect: 301 or 302?

A short link answers with a redirect. 301 (permanent) lets browsers and intermediaries cache the redirect, which cuts load on your servers but means you never see the later clicks, so click analytics become inaccurate, and you cannot change the destination for people who cached it. 302 (temporary) is not cached, so every click reaches you: more traffic, but accurate analytics and the ability to change or disable a link immediately. Products that sell analytics, or must be able to remove abusive links, use 302 (or 307). Mention the trade-off in an interview; it shows you think about product consequences.

Handling a link that goes viral

One link suddenly gets 50,000 requests per second. Everything funnels onto one key: one cache node, one database row. Defences, in order of cost:

1. Cache the redirect in a CDN with a short TTL (say 60 seconds). Most of those requests never reach you.
2. Replicate hot keys across several cache nodes, or keep a tiny in-memory cache on each app server for the hottest few hundred keys.
3. Rate limit per client so a script cannot add to the storm.
4. Degrade gracefully: if the database is struggling, serve from cache even past TTL.

100%

Figure SDA-FND-12-6 · Layers of defence for a viral link · © forcsguy.com

Mistakes people make

• Guessable keys. Sequential ids let anyone enumerate every link. Use random or encoded ids if links may be private.
• Collisions from hashing the URL. Truncating a hash of the long URL will eventually collide. Detect and retry, or generate keys from a counter or a pre-generated pool.
• Forgetting abuse. Short links hide destinations, so they are used for phishing. Check destinations against blocklists and let people report links.
• A single counter as a single point of failure. If one service hands out every id, it needs replicas or ranges handed to each app server in blocks.
• Ignoring expiry and cleanup. Decide what happens to unused or expired links; otherwise storage and the key space fill with dead entries.

Try it yourself

Use the System Builder challenges "The database becomes the bottleneck" and "A viral link", which follow this shortener's growth. Then build the Snowflake ID lab and think about whether a time-based id is suitable for short keys. To build your own, open the URL shortener on the free canvas.

More on this site