Load balancing

Six strategies against slow servers, big requests, busy clients and failures.

On the roadmap:Beginner · step 6: Scaling one server

Starterabout 10 to 15 min6 experiments

Experiments

New here? A few short steps show what happens, one at a time.

Settings

Strategy

Takes the servers in turn. Simple and fair when servers and requests are alike.

Servers4

How busy the servers are70%

Share of very big requests0%

Server 1 is 2.5 times slowerThe last server dies halfwayHealth check (notices a dead server)

Typical response

1 ms

Slowest 5%

1 ms

Requests lost

Busiest server's share

1.00x even

Servers and requests are alike, so most strategies cope. Change something: slow a server, make some requests bigger, or take a server down.

Share of the requests each server received

An even share would be 25% each. Bars turn red when the busiest server has more than one and a half times that.

How busy each server was, and its longest line

Server 1
Working 70% of the time. Longest line: 0 requests.
Server 2
Working 70% of the time. Longest line: 0 requests.
Server 3
Working 70% of the time. Longest line: 0 requests.
Server 4
Working 70% of the time. Longest line: 0 requests.

In plain words

A load balancer stands in front of several servers and decides which one gets each request. The rule it uses is the strategy. A good one keeps every server about equally busy, and a poor one leaves some servers swamped while others sit idle.

Why it matters

Adding servers only helps if the work is shared well. Here the same traffic meets six strategies, so you can see which one copes when a server is slow, when some requests are far bigger than others, when a few clients send most of the traffic, and when a server dies. Each tick is a millisecond.

How real systems do it

The simulation is a simplified model. Here is what production systems add or do.

NGINX offers round robin, least connections, IP hash and a random choice between two servers. HAProxy offers round robin, least connections and source hashing, among others.
Envoy's least-request balancer picks two servers at random and sends the request to the one with fewer open requests. That gets most of the benefit of least connections without looking at every server.
AWS Application Load Balancers can route by round robin or by least outstanding requests, which suits requests of uneven size.
Load balancers check health in two ways: active checks that call each server regularly, and passive checks that watch real traffic for errors. A dead server that still looks idle is one reason a health check matters.