Load balancer
On the roadmap:Beginner · step 6: Scaling one server
Pick a backend for each request with round-robin or least-connections, and stop sending traffic to unhealthy servers.
In plain words
A switchboard in front of several servers. For each request it picks one server, skipping any that are unhealthy.
Why it matters
This is how one address can serve more traffic than one machine can handle, and how a dead server stops hurting users. The choice of rule (round-robin or least busy) changes how evenly the load spreads.
Where you meet it
- Nginx, HAProxy and cloud load balancers
- Kubernetes services and API gateways
- Every large web site
You will be able to
- Implement round-robin and least-connections
- Skip unhealthy backends
- Say when each rule is the better pick
In interviews
Appears in nearly every design. Be ready to compare rules and explain health checks.
Practise on the Interview Ready roadmap1Learn the idea first
2See it work before you read the code
3Build it and run the check
What you build
- Backend records with a health flag and an active-request count
- A round-robin picker that walks the healthy list
- A least-connections picker that chooses the least busy healthy backend
- start() and finish() calls that keep the counts honest
export type Backend = { id: string; healthy: boolean; active: number };
export type Strategy = "round-robin" | "least-connections";
export class LoadBalancer {
private turn = 0;
constructor(public backends: Backend[], private strategy: Strategy = "round-robin") {}
/** Choose a backend without changing any counters. */
pick(): Backend | null {
const up = this.backends.filter((b) => b.healthy);
if (up.length === 0) return null;
if (this.strategy === "least-connections") return up.reduce((best, b) => (b.active < best.active ? b : best));
// ponytail: the turn counter indexes the healthy list, so it shifts when health changes; fine for a lab
return up[this.turn++ % up.length];
}
/** Begin a request: pick a backend and count it as busy. */
start(): Backend | null {
const b = this.pick();
if (b) b.active++;
return b;
}
finish(b: Backend): void {
b.active = Math.max(0, b.active - 1);
}
}
Rather run it on your own computer?
Save the code as index.ts and the check as check.ts in one folder. The check exits with an error if anything is wrong and prints load-balancer ok when everything passes. You need Node.js installed.
mkdir load-balancer && cd load-balancer
# save index.ts and check.ts in this folder
npx tsx check.tsHow real systems do it
This lab is the core idea in a small form. Here is what production versions add or do differently.
- Production balancers add weights, slow start for new servers, connection draining before a server is removed, and both active health checks (probes) and passive ones (watching real failures).
- 'Power of two choices', picking two random servers and sending to the less busy one, gets most of least-connections' benefit without a global view.
- Layer 4 balancers route connections; layer 7 balancers read the request and can route by path, header or cookie.
Ways to pick a server
| How it picks | Handles a slow server? | Needs | Pick it when | |
|---|---|---|---|---|
| Round-robin | Takes turns | No: it keeps giving a slow server its share | Nothing | Requests and servers are similar |
| Least connections | The server with the fewest active requests | Yes | Live counts of active requests | Requests vary in length |
| Weighted | A share proportional to a weight | If you set the weights | Weights set by hand | Servers differ in size |
| Hash on a key (sticky) | The same key always goes to the same server | No | A key such as a user or session | Caches or sessions live on the server |
| Power of two choices | Pick two at random, send to the less busy one | Mostly | Active counts for two servers | Large fleets without a global view |
What it cannot do
The balancer can be a single point of failure
Production runs several, behind one address.
Health checks lag
A server that just failed may still get a few requests before it is marked down.
It does not know a request's cost
Ten cheap requests and one expensive one look the same to round-robin.
Sticky routing fights balance
Sending a user to the same server keeps caches warm but can leave one server busier.
4Now extend it
Each one makes your version closer to what real systems do. Pick one and make the check cover it.
- Add weights so bigger machines get more traffic
- Add a passive health check: mark a backend down after 3 failures in a row
- Add consistent hashing for sticky sessions