Leader election (Raft)

Crash the leader or split the network and watch five servers keep agreeing.

On the roadmap:Advanced · step 4: Distributed systems

Intermediateabout 15 to 20 min6 experiments

Experiments

New here? A few short steps show what happens, one at a time.

Settings and controls

The network

Shortest election timeout (ticks)15

Longest election timeout (ticks)30

Heartbeat every (ticks)5

Each server waits a random time between the two timeouts before starting an election. Keep heartbeats well below the shortest timeout, or followers will call elections while the leader is fine.

Time

0 ticks

Highest term

Leader

none

Committed entries

There is no leader right now. Servers wait out their random timeouts, and the first to expire starts an election for term 1. Elections can fail when two start at once, so timeouts are random.

Node 1

follower

Term 0 · timeout in 18

empty log

Node 2

follower

Term 0 · timeout in 19

empty log

Node 3

follower

Term 0 · timeout in 20

empty log

Node 4

follower

Term 0 · timeout in 16

empty log

Node 5

follower

Term 0 · timeout in 26

empty log

What happened

Press Step or Run. The first election starts after the shortest timeout.

A solid entry is committed (a majority has it). A dashed entry is on this server only so far.

In plain words

Raft lets several servers agree on one sequence of events even when some of them crash or the network splits. The servers elect a leader, the leader takes every write and copies it to the others, and an entry counts as committed only when a majority has stored it.

Why it matters

Anything that must be correct, such as locks, configuration or the metadata of a database, cannot live on one machine, and copies disagree unless something orders them. Consensus is that something. Here you can crash the leader, cut the network in two, and watch the cluster keep its promise: nothing committed is ever lost.

How real systems do it

The simulation is a simplified model. Here is what production systems add or do.

Raft was published in 2014 by Diego Ongaro and John Ousterhout as an easier-to-understand alternative to Paxos. The paper suggests election timeouts of about 150 to 300 milliseconds.
etcd, which holds Kubernetes' state, and HashiCorp's Consul are built on Raft. CockroachDB and TiKV run a separate Raft group for each range of data.
A cluster of 2f+1 servers tolerates f failures, because a majority must survive. Three servers tolerate one failure and five tolerate two. Even sizes add cost without adding fault tolerance.
Every leader has a term number. A message from an older term is ignored, so a server that was cut off cannot disturb the cluster when it comes back.