A cluster, end to end

Add and kill nodes, split the network, and watch keys, leadership and traffic react together.

On the roadmap:Advanced · step 4: Distributed systems

Intermediateabout 10 to 15 min5 experiments

Experiments

Cluster

N1 · leader
N2
N3
N4

Traffic

Network

Nodes up

4 of 4

Leader

Requests

Errors

0 (0%)

Everything is healthy. Kill a node, or split the network, to see how the cluster responds.

Where the keys live (consistent hashing)

133

135

116

4 nodes sharing 500 keys. Imbalance: 1.08x the average — close to even.

Request latency

Send a request or a burst to see latency.

Event log

Cluster started with 4 nodes. N1 is leader for term 1.

In plain words

A real cluster is not one idea in isolation. It is a set of nodes that hold keys (consistent hashing), agree on a leader for writes (election), and answer requests spread across whoever is up (load balancing) — and a network partition hits all three at once. This simulation keeps one small cluster and lets you poke at it from every angle instead of one sim per concept.

Why it matters

Most lessons isolate one idea to make it learnable, which is right for a first pass. But in production, a node going down is never just a hashing problem or just an election problem: it is both, at the same time, and the two interact. Seeing them on one cluster is closer to the real debugging experience of 'why did this fail' than four separate tools.

How real systems do it

The simulation is a simplified model. Here is what production systems add or do.

etcd, the coordination store behind Kubernetes, uses the Raft algorithm for exactly this: a majority of nodes must agree before a write is considered committed, and a minority partition stops accepting writes rather than risk disagreement.
Cassandra separates these concerns differently: it has no single leader for writes (any node can accept one), trading the 'only a majority can write' guarantee for always being able to write somewhere, then reconciling later.
Google's Chubby and Spanner both pay the latency cost of cross-region consensus deliberately, because for a lock service or a globally consistent database, a split-brain (two nodes both believing they're the leader) is worse than being briefly slower.
Production clusters are usually deployed with an odd number of nodes (3, 5, 7) specifically so a partition can never produce two equal halves with no majority anywhere — the 'even split' experiment above is the scenario odd sizing exists to avoid.