Google SRE Book: Managing Critical State (Distributed Consensus)

Ideas you will meet

Consensus

Raft, Paxos and how replicas agree on one order.

CAP Theorem

Consistency, availability and partition tolerance trade-offs.

What we already know about this

Our own lessons on these ideas. Read them here before you start, so it makes sense.

• Advanced Distributed Systems · 12 minConsensus: Raft and PaxosHow a group of machines agrees on one order of operations even when some crash or messages are delayed, and why a majority is the magic number.Read hereClose

  The problem consensus solves

  Several machines must agree on one sequence of operations (who is the leader, what is the next log entry, who owns a lock) even though machines crash and messages are delayed or lost. Without a protocol, replicas drift apart and two nodes may both believe they are the leader.

  Systems that rely on consensus include coordination services (etcd, ZooKeeper), many databases for their metadata or their data, and Kubernetes (through etcd).

  Why a majority

  A decision is valid when a majority (quorum) of nodes accepts it. Any two majorities share at least one node, so a later decision always meets someone who remembers the earlier one. A cluster of n nodes tolerates floor((n-1)/2) failures: 3 nodes tolerate 1, 5 nodes tolerate 2. That is why clusters have an odd number of nodes; a 4th node adds cost but no extra fault tolerance over 3.

  Raft in three parts

  Raft splits the problem so it can be understood:

  100%

  Figure SDA-ADS-01-1 · Raft roles and elections · © forcsguy.com

  1. Leader election. Time is divided into numbered terms. A node that hears nothing becomes a candidate and requests votes. A node votes at most once per term, and only for a candidate whose log is at least as up to date as its own. Randomised timeouts make split votes rare.
  2. Log replication. The leader appends a client command to its log and sends it to followers. When a majority has stored the entry, it is committed; the leader applies it and replies to the client. Followers apply committed entries in order, so every replica runs the same sequence.
  3. Safety. The vote rule guarantees that any new leader already has every committed entry, so committed entries are never lost or reordered.

  100%

  Figure SDA-ADS-01-2 · Replicating one write · © forcsguy.com

  Paxos, briefly

  Paxos reaches the same guarantee with two phases (prepare/promise, then accept/accepted) over a majority. It is famous for being hard to implement and explain, and for being the basis of many systems. Multi-Paxos repeats it for a log of decisions. Raft is essentially a clearer arrangement of the same ideas with a strong leader, which is why most new systems choose it.

  What consensus costs and what it does not give

  • Latency: every committed write waits for a majority round trip, so cross-region clusters are slow. Keep consensus groups within a region or use few, small groups for metadata.
  • Throughput: one leader handles all writes in a group. Large databases run many small groups (one per data range) to spread load.
  • Availability under partition: the minority side cannot make progress. This is the CAP trade-off made concrete: consensus systems are consistent first.
  • It does not remove all problems: slow nodes, clock assumptions, and reconfiguration (adding or removing nodes safely) need care.

  Where you will meet it

  Leader election for databases and queues, distributed locks and configuration stores, metadata for storage systems, and cluster state in Kubernetes. When a design needs "exactly one leader" or "all replicas agree on this order", you are describing consensus. See also the distributed-locks lesson in the core-concepts course.

  Real-world reading

  Related reading

  • In Search of an Understandable Consensus Algorithm (Raft paper)Ongaro and Ousterhout · Stanford UniversityPapers
  • The Raft Consensus Algorithm (site and visualisation)Diego Ongaro and community · raft.github.ioDocumentation
  • ZippyDB: Facebook's key-value storeMeta Engineering · MetaCase Studies
  • How Ringpop from Uber Engineering Helps Distribute Your ApplicationUber Engineering · UberCase Studies

  Show 6 moreShow fewer

  • etcd Raft libraryetcd · CNCF / etcdGitHub
  • Google SRE Book: Managing Critical State (Distributed Consensus)Google SRE team · GoogleBlogs
  • Paxos Made Simple (paper)Leslie Lamport · Microsoft ResearchPapers
  • HashiCorp RaftHashiCorp · GitHubGitHub
  • Distributed Systems (4th edition)Maarten van Steen and Andrew Tanenbaum · distributed-systems.netBooks
  • Scaling Hadoop YARN cluster beyond 10,000 nodes at LinkedInLinkedIn engineers · LinkedInCase Studies

  Open this lesson with its quiz and exercise

• System Design Foundations · 12 minConsistency, availability and the CAP theoremWhat goes wrong when data lives on several machines, and the vocabulary for choosing what to guarantee.Read hereClose

  The problem

  On one machine, data has one value. Spread it over several machines and new questions appear:

  • If I write to machine A, when can machine B see it?
  • If A and B cannot talk to each other, should they keep serving requests?
  • If two people change the same item at the same moment, who wins?

  Consistency models are the answers to the first question. The CAP theorem frames the second.

  Consistency: what a read may return

  From strongest to weakest, in plain words:

  • Strong (linearizable): once a write completes, every later read anywhere sees it. It behaves like one copy. Easiest to reason about, slowest and least available.
  • Read-your-writes: you always see your own updates, though others may lag.
  • Eventual: if writes stop, all copies will eventually agree. Reads may be stale in between. Fast and highly available.

  100%

  Figure SDA-FND-07-1 · Replica lag and eventual consistency · © forcsguy.com

  Stronger guarantees cost latency (replicas must coordinate) and availability (some must be reachable). Weaker ones are cheaper but push complexity to the application.

  The CAP theorem

  For a distributed data store, three properties are in tension:

  • Consistency (C): every read sees the latest write (in the strong sense above).
  • Availability (A): every request to a working node gets a response.
  • Partition tolerance (P): the system keeps working even when messages between nodes are lost or delayed.

  Networks do fail, so P is not optional. The real choice is what to do when a partition happens:

  100%

  Figure SDA-FND-07-2 · Choosing C or A during a partition · © forcsguy.com

  • CP systems (for example many coordination stores and strongly consistent databases) may reject requests rather than risk a wrong answer.
  • AP systems (for example many Dynamo-style stores) stay up and let replicas diverge briefly, then merge.

  A more useful lens: latency too

  Even without partitions, strong consistency costs time because replicas must agree. The PACELC idea adds this: if there is a Partition choose A or C, Else (normal operation) choose between Latency and Consistency. It explains why a store may feel slower when configured to be strongly consistent.

  100%

  Figure SDA-FND-07-3 · How many copies must confirm a write · © forcsguy.com

  Resolving conflicts

  If two replicas accept different writes, the system needs a rule:

  • Last write wins: simple, but silently drops one update.
  • Version vectors or merge logic: detect that the writes were concurrent, then merge or ask the user.
  • CRDTs: data types designed so concurrent updates always merge cleanly (used for counters and collaborative editing).

  Choosing per feature, not per system

  You rarely need one consistency level for the whole product. Use strong consistency where wrong data is costly (payments, inventory, seat booking) and eventual consistency where speed and uptime matter more (counts, feeds, recommendations). Many databases let you pick per query.

  A worked example: quorums with numbers

  A key is stored on N = 3 replicas. A write must be acknowledged by W of them, and a read asks R of them and takes the newest version.

  Setting	W	R	R + W > N?	What you get
  Fast reads	3	1	yes (4 > 3)	Strong reads, but one dead replica blocks writes
  Balanced	2	2	yes (4 > 3)	Strong reads; survives one replica down for both reads and writes
  Fast writes	1	3	yes (4 > 3)	Quick writes, slow reads, reads fail if any replica is down
  Very fast	1	1	no (2 < 3)	Fastest, but a read can miss the latest write: eventual consistency

  The rule R + W > N guarantees the set of replicas read overlaps the set that acknowledged the last write, so at least one has the newest value. Concretely, with W = 2, R = 2: a write of version 5 reaches replicas A and B; C is still at version 4. A read that asks B and C sees versions 5 and 4 and returns 5. Whichever two replicas you ask, at least one has version 5.

  100%

  Figure SDA-FND-07-4 · Why read and write quorums overlap · © forcsguy.com

  A story: the comment that disappeared

  A user posts a comment. The write goes to the primary and the page reloads, but the read is served by a replica that is 300 ms behind. The comment is missing, the user posts it again, and now there are duplicates.

  This is a read-your-writes violation, the most common user-visible consistency bug. Fixes, from cheapest to strongest:

  1. For a short time after a user writes (say 5 seconds), route that user's reads to the primary.
  2. Return the new item in the write response and show it on the page without re-reading.
  3. Track a version or log position from the write, and have the replica serve the read only when it has caught up to it.
  4. Make that data strongly consistent end to end (higher latency, fewer failure modes tolerated).

  Not every read deserves strong consistency. A like counter that is a few seconds old is fine; a balance after a transfer is not.

  What a network partition actually does

  Imagine a three-node store split into two nodes on one side and one node on the other.

  • A CP system (consistency over availability) lets only the side with a majority accept writes. The lone node refuses requests, which is unavailable but correct.
  • An AP system (availability over consistency) lets both sides accept writes. Everyone gets an answer, but the two sides now hold different versions, and after the partition heals the system must merge: last-write-wins (simple, can lose data and trusts clocks), version vectors that keep both and ask the application, or data types designed to merge (CRDTs such as counters and sets).

  100%

  Figure SDA-FND-07-5 · Behaviour during a partition · © forcsguy.com

  Mistakes people make

  • Reading CAP as "pick any two". Partitions happen whether you like it or not, so the real choice is what to give up during a partition. Without partitions you can have both.
  • Assuming "eventual" means "soon enough". It means "if writes stop, replicas converge", with no bound. Measure the real lag.
  • Last-write-wins on unsynchronised clocks. A node with a clock a second ahead silently overwrites newer data. Prefer logical versions.
  • One setting for the whole system. Choose per feature: strong for payments and inventory, eventual for counters, feeds and recommendations.
  • Forgetting the latency side. Even without partitions, stronger consistency costs round trips between replicas, a trade-off often called PACELC.

  Try it yourself

  Use the CAP theorem playground to partition a cluster and watch CP and AP behave differently, and the replication playground to see stale reads and what a failover loses. Then try the System Builder with replicas.

  More on this site

  Related reading

  • Amazon Dynamo: Highly Available Key-Value StoreWerner Vogels · AmazonCase Studies
  • Please stop calling databases CP or APMartin Kleppmann · martin.kleppmann.comBlogs
  • CAP Twelve Years Later: How the Rules Have ChangedEric Brewer · InfoQBlogs
  • Martin Kleppmann's BlogMartin Kleppmann · martin.kleppmann.comBlogs

  Show 6 moreShow fewer

  • Martin Kleppmann: Distributed Systems LecturesMartin Kleppmann · University of CambridgeYouTube
  • Amazon S3 strong consistencyAmazon Web ServicesDocumentation
  • Designing Data-Intensive ApplicationsMartin Kleppmann · O'ReillyBooks
  • Read Consistency with Database Replicas at ShopifyShopify engineers · ShopifyCase Studies
  • All Things DistributedWerner Vogels · AmazonBlogs
  • Aurora BlogAurora engineers · AuroraBlogs

  Open this lesson with its quiz and exercise

• Core Concepts: How Things Actually Work · 8 minTime, clocks and ordering of eventsWhy you cannot trust timestamps across machines, and how logical clocks give a safe ordering.Read hereClose

  Why time is hard

  On one machine, order is easy: the clock only goes forward (mostly). Across machines, there is no shared clock.

  • Each machine's clock drifts, and synchronisation (NTP) is only accurate to some milliseconds, sometimes much worse.
  • Clocks can jump (corrections, leap seconds, virtual machine pauses).
  • Messages take unpredictable time, so "A happened before B" cannot be read from two clocks.

  So never use wall-clock timestamps from different machines to order important events (or to decide who wins a conflict, or to measure the timeout of a lock).

  Image: National Institute of Standards and Technology (NIST) · Public domain · Source · Wikimedia Commons: NIST-F2 cesium fountain atomic clock

  What we can know: happened-before

  Event A happened before B if A could have caused B: same process in order, or a message sent at A and received at B (and transitivity). Events with no such chain are concurrent.

  Logical clocks

  Lamport clock

  Each process keeps a counter. Increase it on each event; attach it to messages; on receiving, set it to max(local, received) + 1. Guarantee: if A happened before B, then clock(A) < clock(B). The reverse does not hold, so equal-looking order does not prove causality.

  Vector clock

  Each process keeps a vector of counters, one per process. Comparing two vectors tells you whether one event happened before the other or whether they are concurrent. Used by Dynamo-style stores to detect conflicting writes.

  100%

  Figure SDA-CORE-17-1 · Vector clock · © forcsguy.com

  Practical choices

  Need	Tool
  A total order of events	A single leader, or consensus (Raft, Paxos), or a log such as Kafka partitions
  Detect concurrent updates	Version vectors, or version numbers with conditional writes
  Merge concurrent edits	CRDTs, operational transformation, application-level merge
  Unique, roughly time-ordered ids	Snowflake-style ids (accepting small skew)
  Real-time bounds	Databases that use atomic clocks and wait out uncertainty (such as Spanner)

  Rules of thumb

  • Use monotonic clocks (not wall time) to measure durations on one machine.
  • Treat timestamps as approximate and for humans; use versions for correctness.
  • Make operations idempotent and commutative where possible, so arrival order matters less.

  More on this site

  Related reading

  • Time, Clocks, and the Ordering of Events in a Distributed System (paper)Leslie Lamport · Microsoft ResearchPapers
  • Spanner: Google's Globally Distributed Database (paper)Corbett et al. · GooglePapers
  • The Raft Consensus Algorithm (site and visualisation)Diego Ongaro and community · raft.github.ioDocumentation
  • In Search of an Understandable Consensus Algorithm (Raft paper)Ongaro and Ousterhout · Stanford UniversityPapers

  Show 6 moreShow fewer

  • How Ringpop from Uber Engineering Helps Distribute Your ApplicationUber Engineering · UberCase Studies
  • Google SRE Book: Managing Critical State (Distributed Consensus)Google SRE team · GoogleBlogs
  • The Chubby Lock Service (paper)Mike Burrows · GooglePapers
  • Clock Skew when Syncing Password Payloads at DrobboxDrobbox engineers · DrobboxCase Studies
  • Airflow at GrabGrab engineers · GrabCase Studies
  • OpenTable BlogOpenTable engineers · OpenTableBlogs

  Open this lesson with its quiz and exercise