The Raft Consensus Algorithm (site and visualisation)

Ideas you will meet

Consensus

Raft, Paxos and how replicas agree on one order.

Distributed Systems

Consensus, time, failure and the fundamentals of scale-out.

What we already know about this

Our own lessons on these ideas. Read them here before you start, so it makes sense.

• Advanced Distributed Systems · 12 minConsensus: Raft and PaxosHow a group of machines agrees on one order of operations even when some crash or messages are delayed, and why a majority is the magic number.Read hereClose

  The problem consensus solves

  Several machines must agree on one sequence of operations (who is the leader, what is the next log entry, who owns a lock) even though machines crash and messages are delayed or lost. Without a protocol, replicas drift apart and two nodes may both believe they are the leader.

  Systems that rely on consensus include coordination services (etcd, ZooKeeper), many databases for their metadata or their data, and Kubernetes (through etcd).

  Why a majority

  A decision is valid when a majority (quorum) of nodes accepts it. Any two majorities share at least one node, so a later decision always meets someone who remembers the earlier one. A cluster of n nodes tolerates floor((n-1)/2) failures: 3 nodes tolerate 1, 5 nodes tolerate 2. That is why clusters have an odd number of nodes; a 4th node adds cost but no extra fault tolerance over 3.

  Raft in three parts

  Raft splits the problem so it can be understood:

  100%

  Figure SDA-ADS-01-1 · Raft roles and elections · © forcsguy.com

  1. Leader election. Time is divided into numbered terms. A node that hears nothing becomes a candidate and requests votes. A node votes at most once per term, and only for a candidate whose log is at least as up to date as its own. Randomised timeouts make split votes rare.
  2. Log replication. The leader appends a client command to its log and sends it to followers. When a majority has stored the entry, it is committed; the leader applies it and replies to the client. Followers apply committed entries in order, so every replica runs the same sequence.
  3. Safety. The vote rule guarantees that any new leader already has every committed entry, so committed entries are never lost or reordered.

  100%

  Figure SDA-ADS-01-2 · Replicating one write · © forcsguy.com

  Paxos, briefly

  Paxos reaches the same guarantee with two phases (prepare/promise, then accept/accepted) over a majority. It is famous for being hard to implement and explain, and for being the basis of many systems. Multi-Paxos repeats it for a log of decisions. Raft is essentially a clearer arrangement of the same ideas with a strong leader, which is why most new systems choose it.

  What consensus costs and what it does not give

  • Latency: every committed write waits for a majority round trip, so cross-region clusters are slow. Keep consensus groups within a region or use few, small groups for metadata.
  • Throughput: one leader handles all writes in a group. Large databases run many small groups (one per data range) to spread load.
  • Availability under partition: the minority side cannot make progress. This is the CAP trade-off made concrete: consensus systems are consistent first.
  • It does not remove all problems: slow nodes, clock assumptions, and reconfiguration (adding or removing nodes safely) need care.

  Where you will meet it

  Leader election for databases and queues, distributed locks and configuration stores, metadata for storage systems, and cluster state in Kubernetes. When a design needs "exactly one leader" or "all replicas agree on this order", you are describing consensus. See also the distributed-locks lesson in the core-concepts course.

  Real-world reading

  Related reading

  • In Search of an Understandable Consensus Algorithm (Raft paper)Ongaro and Ousterhout · Stanford UniversityPapers
  • The Raft Consensus Algorithm (site and visualisation)Diego Ongaro and community · raft.github.ioDocumentation
  • ZippyDB: Facebook's key-value storeMeta Engineering · MetaCase Studies
  • How Ringpop from Uber Engineering Helps Distribute Your ApplicationUber Engineering · UberCase Studies

  Show 6 moreShow fewer

  • etcd Raft libraryetcd · CNCF / etcdGitHub
  • Google SRE Book: Managing Critical State (Distributed Consensus)Google SRE team · GoogleBlogs
  • Paxos Made Simple (paper)Leslie Lamport · Microsoft ResearchPapers
  • HashiCorp RaftHashiCorp · GitHubGitHub
  • Distributed Systems (4th edition)Maarten van Steen and Andrew Tanenbaum · distributed-systems.netBooks
  • Scaling Hadoop YARN cluster beyond 10,000 nodes at LinkedInLinkedIn engineers · LinkedInCase Studies

  Open this lesson with its quiz and exercise

• Core Concepts: How Things Actually Work · 8 minTime, clocks and ordering of eventsWhy you cannot trust timestamps across machines, and how logical clocks give a safe ordering.Read hereClose

  Why time is hard

  On one machine, order is easy: the clock only goes forward (mostly). Across machines, there is no shared clock.

  • Each machine's clock drifts, and synchronisation (NTP) is only accurate to some milliseconds, sometimes much worse.
  • Clocks can jump (corrections, leap seconds, virtual machine pauses).
  • Messages take unpredictable time, so "A happened before B" cannot be read from two clocks.

  So never use wall-clock timestamps from different machines to order important events (or to decide who wins a conflict, or to measure the timeout of a lock).

  Image: National Institute of Standards and Technology (NIST) · Public domain · Source · Wikimedia Commons: NIST-F2 cesium fountain atomic clock

  What we can know: happened-before

  Event A happened before B if A could have caused B: same process in order, or a message sent at A and received at B (and transitivity). Events with no such chain are concurrent.

  Logical clocks

  Lamport clock

  Each process keeps a counter. Increase it on each event; attach it to messages; on receiving, set it to max(local, received) + 1. Guarantee: if A happened before B, then clock(A) < clock(B). The reverse does not hold, so equal-looking order does not prove causality.

  Vector clock

  Each process keeps a vector of counters, one per process. Comparing two vectors tells you whether one event happened before the other or whether they are concurrent. Used by Dynamo-style stores to detect conflicting writes.

  100%

  Figure SDA-CORE-17-1 · Vector clock · © forcsguy.com

  Practical choices

  Need	Tool
  A total order of events	A single leader, or consensus (Raft, Paxos), or a log such as Kafka partitions
  Detect concurrent updates	Version vectors, or version numbers with conditional writes
  Merge concurrent edits	CRDTs, operational transformation, application-level merge
  Unique, roughly time-ordered ids	Snowflake-style ids (accepting small skew)
  Real-time bounds	Databases that use atomic clocks and wait out uncertainty (such as Spanner)

  Rules of thumb

  • Use monotonic clocks (not wall time) to measure durations on one machine.
  • Treat timestamps as approximate and for humans; use versions for correctness.
  • Make operations idempotent and commutative where possible, so arrival order matters less.

  More on this site

  Related reading

  • Time, Clocks, and the Ordering of Events in a Distributed System (paper)Leslie Lamport · Microsoft ResearchPapers
  • Spanner: Google's Globally Distributed Database (paper)Corbett et al. · GooglePapers
  • The Raft Consensus Algorithm (site and visualisation)Diego Ongaro and community · raft.github.ioDocumentation
  • In Search of an Understandable Consensus Algorithm (Raft paper)Ongaro and Ousterhout · Stanford UniversityPapers

  Show 6 moreShow fewer

  • How Ringpop from Uber Engineering Helps Distribute Your ApplicationUber Engineering · UberCase Studies
  • Google SRE Book: Managing Critical State (Distributed Consensus)Google SRE team · GoogleBlogs
  • The Chubby Lock Service (paper)Mike Burrows · GooglePapers
  • Clock Skew when Syncing Password Payloads at DrobboxDrobbox engineers · DrobboxCase Studies
  • Airflow at GrabGrab engineers · GrabCase Studies
  • OpenTable BlogOpenTable engineers · OpenTableBlogs

  Open this lesson with its quiz and exercise

• Advanced Distributed Systems · 15 minA map of distributed-system patternsThirty recurring building blocks (logs, quorums, leases, clocks, partitions, heartbeats and more) grouped by the problem each solves, so you can name them in a design.Read hereClose

  Why a pattern map

  Real distributed systems (Kafka, etcd, Cassandra, Kubernetes and others) are assembled from a small set of recurring solutions. Knowing their names lets you read documentation faster, discuss designs precisely and spot what a design is missing. This lesson groups thirty well-known patterns by the problem they solve. Each is explained in a sentence or two; the catalogue linked below gives full treatment. Many have their own lessons on this site (consensus, storage engines, Kafka, Redis).

  100%

  Figure SDA-ADS-09-1 · Problems and the pattern families that solve them · © forcsguy.com

  1. Durability and ordering

  • Write-ahead log: record every change durably before applying it, so a crash can be recovered by replaying (see the storage engines lesson).
  • Segmented log: split the log into segment files so old data can be deleted or compacted cheaply and lookups stay fast (Kafka does this).
  • Low-water mark: the oldest log position still needed; everything before it can be discarded.
  • High-water mark: the newest position known to be safely replicated; readers must not go beyond it.
  • Replicated log: a log kept identical on several nodes, usually driven by a leader; the basis of consensus-backed systems.

  2. Replication and agreement

  • Leader and followers: one node orders writes; others copy and can serve reads.
  • Majority quorum: act only when more than half of the nodes agree, so any two decisions overlap.
  • Paxos: a protocol that makes a majority agree on a value despite failures (see the consensus lesson; Raft is its more understandable cousin).
  • Consistent core: a small strongly consistent cluster (the "core") makes decisions for a much larger, weakly coupled cluster, such as metadata and leadership.
  • Two-phase commit: a coordinator asks all participants to prepare, then to commit; gives atomicity across nodes but blocks if the coordinator fails, so use sparingly.
  • Follower reads: allow reads from followers to spread load, accepting some staleness (or bounding it).
  • Emergent leader: nodes agree on a leader from membership data (for example the oldest member) without a separate vote.

  3. Failure handling and authority

  • Heartbeat: periodic "I am alive" messages; absence signals failure.
  • Gossip dissemination: nodes pass information to random peers, so news (membership, failure) spreads exponentially without a central point (used by Cassandra and Dynamo-style stores).
  • Lease: a time-bound right (to be leader or hold a lock) that expires automatically unless renewed.
  • Generation clock (also called epoch or term): a number increased at each leadership change and sent with requests; receivers ignore lower numbers, stopping a stale leader.
  • Idempotent receiver: handle repeated requests safely by remembering what was already processed.
  • State watch: clients subscribe to changes of a key or range and are notified, instead of polling (etcd and ZooKeeper watches).

  4. Time and versions

  • Lamport clock: a logical counter that orders events consistently with causality without synchronised wall clocks.
  • Version vector: per-node counters that detect whether two copies are ordered or concurrent (conflict); see the clocks lesson.
  • Hybrid clock: combines physical time with a logical counter so timestamps are close to real time yet still ordered.
  • Clock-bound wait: wait out the clock uncertainty before committing, so timestamps reflect real order (used by globally consistent databases with bounded-error clocks).
  • Versioned value: store each update as a new version so reads can be consistent snapshots and conflicts can be detected.

  5. Partitioning

  • Fixed partitions: split data into a fixed number of partitions up front and move whole partitions between nodes when scaling (like Redis hash slots or Kafka partitions).
  • Key-range partitions: split by sorted ranges of keys, good for range scans; ranges split as they grow (like Bigtable and HBase), at the risk of hot ranges.

  6. Performance and concurrency

  • Request batch: group several requests into one network call or one disk write to amortise cost.
  • Request pipeline: send further requests without waiting for earlier responses, keeping the link busy.
  • Request waiting list: park a request until the cluster has replied or an event happens, then answer the client.
  • Singular update queue: funnel all updates through one thread and queue to avoid locks and keep order (the idea behind single-threaded cores in Redis and exchanges).
  • Single-socket channel: one connection per peer, with ordered messages, to preserve order and reduce overhead.

  Seeing the patterns in a system

  System	Patterns visible
  Kafka	Segmented log, replicated log, leader and followers, high-water mark, fixed partitions, request batch, heartbeats
  etcd or ZooKeeper	Replicated log with majority quorum, leader election, state watch, leases, generation clock
  Cassandra	Gossip, fixed or token partitions, version-based conflict resolution, write-ahead log, tunable quorum reads
  A global SQL database	Consensus per partition, key-range partitions, hybrid or bounded clocks, two-phase commit across partitions
  Redis Cluster	Fixed partitions (16,384 slots), leader and followers, gossip, singular update queue

  How to use this in an interview

  When you propose "a leader", immediately address: how it is elected, how a stale leader is fenced (generation clock or lease), how failure is detected (heartbeat) and how followers catch up (replicated log, high-water mark). Naming these shows you know what a one-word box hides.

  Real-world reading

  Related reading

  • In Search of an Understandable Consensus Algorithm (Raft paper)Ongaro and Ousterhout · Stanford UniversityPapers
  • Spanner: Google's Globally Distributed Database (paper)Corbett et al. · GooglePapers
  • ZippyDB: Facebook's key-value storeMeta Engineering · MetaCase Studies
  • The Raft Consensus Algorithm (site and visualisation)Diego Ongaro and community · raft.github.ioDocumentation

  Show 6 moreShow fewer

  • Time, Clocks, and the Ordering of Events in a Distributed System (paper)Leslie Lamport · Microsoft ResearchPapers
  • Patterns of Distributed SystemsUnmesh Joshi · Martin FowlerBlogs
  • ZooKeeper: Wait-free Coordination for Internet-scale Systems (paper)Hunt et al. · Yahoo! ResearchPapers
  • How Ringpop from Uber Engineering Helps Distribute Your ApplicationUber Engineering · UberCase Studies
  • Architecture to Handle 80K RPS Celebrity Sales at Shopify - Simon Eskildsen, Engineering Lead at ShopifyShopifyYouTube
  • Building a scalable ELK stack at EnvatoEnvato engineers · EnvatoCase Studies

  Open this lesson with its quiz and exercise