Autoscaling

A spike against policies that react, count the queue, or look ahead.

On the roadmap:Beginner · step 6: Scaling one server

Intermediateabout 10 to 15 min7 experiments

Experiments

New here? A few short steps show what happens, one at a time.

Settings

What the traffic does

Scaling policy

The number of servers never changes. Simple, and either wasteful or too small.

Time for a new server to start (seconds)60

Target busy level60%

Normal traffic (requests per second)400

Requests one server handles per second100

Longest wait

2.0 min

Seconds users waited over 1 s

239

Most servers

Cost

35 server-min

With a fixed fleet the capacity never moves, so once traffic passes it the line of waiting requests only grows: 2.0 min at worst.

Traffic and the capacity that is ready

requests per second

Requests per secondWhat the servers can handle

Capacity trails the traffic by the time new servers take to start. Where demand is above capacity, a line of waiting requests forms.

Requests waiting in line

requests waiting

Waiting requests

Every request in this line is a user waiting. It only shrinks once capacity passes the traffic, and it takes time to drain.

In plain words

An autoscaler adds servers when traffic rises and removes them when it falls. The difficulty is time: a new server takes a while to start, so a rule that reacts to trouble is always behind it. How early it acts, and how big a step it takes, decide whether users wait.

Why it matters

Spikes are what autoscaling is for, and also where it most often fails. Here you can send a sudden spike at a fixed fleet and at three policies, change how long servers take to start, and see why the usual fix is headroom and faster starts, not just a smarter rule.

How real systems do it

The simulation is a simplified model. Here is what production systems add or do.

AWS Auto Scaling offers target tracking (keep a metric at a value), step scaling (add a set number for a set breach) and scheduled and predictive scaling, which forecasts load from past days.
Kubernetes' Horizontal Pod Autoscaler scales on CPU, memory or custom metrics. KEDA adds scaling on the length of a queue or a stream's lag, which sees a backlog before CPU does.
A new server is not useful the moment it launches: it boots, pulls an image, warms caches and passes health checks. Cooldown and warm-up settings stop the scaler from ordering more while earlier orders are still starting.
Teams usually keep spare capacity, with targets commonly between 50 and 70% utilisation, and scale in slowly. Scaling out is urgent, and scaling in rarely is.