Scaling from one user to millions

Start simple, then react to evidence

Most systems that serve millions began on one server. They were not designed for millions on day one. Each step below is a response to a measured bottleneck, and the order matters: start with the cheapest change that removes the problem.

100%

Figure SDA-FND-13-1 · Scaling path: one server to sharded database · © forcsguy.com

1. One server

The web app, the database and the files all run on one machine. It is simple and cheap, and fine for early users. Its weakness is also clear: one machine is a single point of failure and everything competes for the same resources.

100%

Figure SDA-FND-13-2 · One server: the whole request path · © forcsguy.com

2. Separate the database

Move the database to its own server. Web and data tiers can now be sized separately, and you can choose hardware suited to each. Choose the database type by your data and queries: usually relational first, with a NoSQL store only where it genuinely fits (see the databases lesson).

3. Load balancer and several web servers

Put a load balancer in front of two or more stateless web servers. Move sessions out of server memory into a shared store so any server can handle any request. Now web capacity scales by adding servers, and losing one is no longer an outage.

4. Read replicas and a cache

The database becomes the next bottleneck.

• Read replicas copy data from a primary to followers that serve reads. Reads scale; writes still go to the primary.
• A cache (Redis, Memcached) holds hot data in memory so most reads never reach the database.

Do the cache first when the same data is read repeatedly, and replicas when reads are varied.

What happens when a database server fails:

100%

Figure SDA-FND-13-3 · Database failure cases with one primary and replicas · © forcsguy.com

5. CDN

Serve images, scripts, styles and video from a content delivery network: copies near users. This removes a large share of traffic from your servers and cuts latency.

6. Several data centres

For lower latency worldwide and survival of a regional failure, run in several data centres or regions. Geo-routing (geo DNS) sends users to the nearest healthy one. Hard parts: keeping data in sync across regions and handling failover. Test failure on purpose.

Several data centres, and what happens when one is lost:

100%

Figure SDA-FND-13-4 · Two data centres and a failover · © forcsguy.com

7. Queues and background workers

Move slow work out of the request: emails, image processing, reports, notifications. A message queue decouples producers and consumers, absorbs bursts, and lets workers scale independently.

Decoupling slow work with a queue, and scaling the workers:

100%

Figure SDA-FND-13-5 · Queue between producers and workers · © forcsguy.com

8. Shard the database

When a single primary can no longer handle the writes or the data size, split data across several databases (sharding) by a good key. This is powerful and permanently complicates queries and transactions, so do it last, after indexing, caching, replicas and bigger machines.

100%

Figure SDA-FND-13-6 · Sharding by user id, and its costs · © forcsguy.com

Throughout: observe and automate

• Logging, metrics and tracing so you see problems before users do.
• Automation: continuous integration and delivery, infrastructure as code, automated tests and rollbacks.
• Capacity planning and load tests so you scale ahead of demand.
• Backups you have actually restored.

What this teaches

• Keep web servers stateless.
• Cache what is read often.
• Use a CDN for static content.
• Replicate data for availability and read capacity.
• Decouple with queues.
• Shard only when forced.
• Measure, then change one thing at a time.

Worked example: growing a product by the numbers

Follow one product as it grows. The numbers are round and illustrative; the pattern is what matters.

Stage	Users per day	Peak requests per second	What breaks	The fix
Launch	1,000	5	Nothing	One server with the app and the database
Early traction	50,000	60	The database competes with the app for memory and CPU	Move the database to its own machine
Growing	500,000	600	One app server is saturated, and a deploy takes the site down	Load balancer and 3 to 4 app servers
Popular	5,000,000	3,500	Database reads are too heavy; pages are slow	Cache the hot reads, add read replicas, put static files on a CDN
Large	50,000,000	30,000	Writes overload the primary; slow work blocks requests	Queues and workers for slow jobs, then shard the database
Global	200,000,000	100,000+	Users far away see high latency, and one region is a single point of failure	Several regions with geo-routing and replicated data

Two habits matter more than any individual technique. Measure before you scale: at each row, someone looked at graphs and found the actual bottleneck, rather than adding components because large companies have them. Add one thing at a time: each step above introduces new failure modes (cache invalidation, replication lag, cross-shard queries), so you want to learn them one by one.

100%

Figure SDA-FND-13-7 · The scaling loop · © forcsguy.com

What each step costs

Every piece you add solves one problem and creates another. Know the price before you pay it.

• Load balancer and more servers: servers must be stateless, and you now need health checks, deployments across many machines, and a balancer that is not itself a single point of failure.
• Cache: faster reads, but stale data, invalidation bugs and a new component that can fail. Decide what happens when the cache is down.
• Read replicas: more read capacity, but replication lag means users can read old data.
• CDN: big savings for static and cacheable content, but changes take time to propagate and you must version your files.
• Queues and workers: smooth spikes and move slow work out of the request, but results are asynchronous, so the product must tolerate delay, retries and duplicates.
• Sharding: the only way to scale writes past one machine, but cross-shard queries, transactions and rebalancing become your problem. It is the last step for a reason.
• Multiple regions: lower latency and survival of a regional outage, but you must choose how to keep data consistent across regions.

100%

Figure SDA-FND-13-8 · Each addition solves one problem and adds another · © forcsguy.com

Try the whole journey

The System Builder lets you do this yourself. It starts with one server and gives you six challenges that follow the table above: spreading load, protecting the database with a cache, absorbing a viral spike with a CDN, smoothing a write burst with a queue, surviving lost servers, and surviving a database outage. After each pass it tells you why the change worked. Prefer to wire it yourself? Open the finished design on the free canvas and break parts of it.

Real-world reading