How to do distributed locking

Ideas you will meet

Distributed Systems

Consensus, time, failure and the fundamentals of scale-out.

Consensus

Raft, Paxos and how replicas agree on one order.

What we already know about this

Our own lessons on these ideas. Read them here before you start, so it makes sense.

• Advanced Distributed Systems · 12 minConsensus: Raft and PaxosHow a group of machines agrees on one order of operations even when some crash or messages are delayed, and why a majority is the magic number.Read hereClose

  The problem consensus solves

  Several machines must agree on one sequence of operations (who is the leader, what is the next log entry, who owns a lock) even though machines crash and messages are delayed or lost. Without a protocol, replicas drift apart and two nodes may both believe they are the leader.

  Systems that rely on consensus include coordination services (etcd, ZooKeeper), many databases for their metadata or their data, and Kubernetes (through etcd).

  Why a majority

  A decision is valid when a majority (quorum) of nodes accepts it. Any two majorities share at least one node, so a later decision always meets someone who remembers the earlier one. A cluster of n nodes tolerates floor((n-1)/2) failures: 3 nodes tolerate 1, 5 nodes tolerate 2. That is why clusters have an odd number of nodes; a 4th node adds cost but no extra fault tolerance over 3.

  Raft in three parts

  Raft splits the problem so it can be understood:

  100%

  Figure SDA-ADS-01-1 · Raft roles and elections · © forcsguy.com

  1. Leader election. Time is divided into numbered terms. A node that hears nothing becomes a candidate and requests votes. A node votes at most once per term, and only for a candidate whose log is at least as up to date as its own. Randomised timeouts make split votes rare.
  2. Log replication. The leader appends a client command to its log and sends it to followers. When a majority has stored the entry, it is committed; the leader applies it and replies to the client. Followers apply committed entries in order, so every replica runs the same sequence.
  3. Safety. The vote rule guarantees that any new leader already has every committed entry, so committed entries are never lost or reordered.

  100%

  Figure SDA-ADS-01-2 · Replicating one write · © forcsguy.com

  Paxos, briefly

  Paxos reaches the same guarantee with two phases (prepare/promise, then accept/accepted) over a majority. It is famous for being hard to implement and explain, and for being the basis of many systems. Multi-Paxos repeats it for a log of decisions. Raft is essentially a clearer arrangement of the same ideas with a strong leader, which is why most new systems choose it.

  What consensus costs and what it does not give

  • Latency: every committed write waits for a majority round trip, so cross-region clusters are slow. Keep consensus groups within a region or use few, small groups for metadata.
  • Throughput: one leader handles all writes in a group. Large databases run many small groups (one per data range) to spread load.
  • Availability under partition: the minority side cannot make progress. This is the CAP trade-off made concrete: consensus systems are consistent first.
  • It does not remove all problems: slow nodes, clock assumptions, and reconfiguration (adding or removing nodes safely) need care.

  Where you will meet it

  Leader election for databases and queues, distributed locks and configuration stores, metadata for storage systems, and cluster state in Kubernetes. When a design needs "exactly one leader" or "all replicas agree on this order", you are describing consensus. See also the distributed-locks lesson in the core-concepts course.

  Real-world reading

  Related reading

  • In Search of an Understandable Consensus Algorithm (Raft paper)Ongaro and Ousterhout · Stanford UniversityPapers
  • The Raft Consensus Algorithm (site and visualisation)Diego Ongaro and community · raft.github.ioDocumentation
  • ZippyDB: Facebook's key-value storeMeta Engineering · MetaCase Studies
  • How Ringpop from Uber Engineering Helps Distribute Your ApplicationUber Engineering · UberCase Studies

  Show 6 moreShow fewer

  • etcd Raft libraryetcd · CNCF / etcdGitHub
  • Google SRE Book: Managing Critical State (Distributed Consensus)Google SRE team · GoogleBlogs
  • Paxos Made Simple (paper)Leslie Lamport · Microsoft ResearchPapers
  • HashiCorp RaftHashiCorp · GitHubGitHub
  • Distributed Systems (4th edition)Maarten van Steen and Andrew Tanenbaum · distributed-systems.netBooks
  • Scaling Hadoop YARN cluster beyond 10,000 nodes at LinkedInLinkedIn engineers · LinkedInCase Studies

  Open this lesson with its quiz and exercise

• Core Concepts: How Things Actually Work · 6 minOptimistic and pessimistic lockingTwo strategies for concurrent updates: lock first, or detect conflicts at the end and retry.Read hereClose

  The problem

  Two users load the same record, both change it, both save. Without protection the later save silently overwrites the earlier one (a lost update). Two families of solutions exist.

  Image: Svetlov Artem · CC0 · Source · Wikimedia Commons: RZD Selizarovo 2016-08 searchlight signal 2

  Pessimistic locking: assume conflict

  Lock the data before changing it, so others must wait.

  BEGIN;
  SELECT * FROM seats WHERE id = 7 FOR UPDATE;  -- others block here
  UPDATE seats SET taken = true WHERE id = 7;
  COMMIT;
  

  • Safe and simple to reason about.
  • Cost: waiting, reduced throughput, risk of deadlock, and trouble if the lock is held while waiting for a user.

  Optimistic locking: assume no conflict, check at the end

  Read freely, and when saving check nothing changed.

  UPDATE pages SET body = :new, version = version + 1
  WHERE id = 5 AND version = :versionIRead;
  -- 0 rows updated means someone else saved first
  

  • No locks held while the user thinks.
  • Cost: on conflict the work is rejected and must be retried or merged. Under heavy contention many attempts fail.

  Before: pessimistic: lock first

  100%

  After: optimistic: check at the end

  100%

  What changed

  • Pessimistic makes others wait, optimistic lets them all try
  • Optimistic rejects a stale save instead of overwriting it
  • Optimistic wastes work only when a conflict really happens

  Optimistic locking in action

  100%

  Figure SDA-CORE-03-1 · Two writers with a version check · © forcsguy.com

  Choosing

  	Optimistic	Pessimistic
  Conflicts	rare	frequent
  Lock duration	none	whole transaction
  Failure mode	retry or merge	waiting, deadlocks
  Good for	editing, profiles, most web apps	inventory, seat booking, money moves

  Beyond one database

  For resources not in a database (a file, an external job), use a distributed lock or a lease, covered in a later lesson. They need extra care because the lock holder can crash or stall.

  Real-world reading

  Related reading

  • ZooKeeper: Wait-free Coordination for Internet-scale Systems (paper)Hunt et al. · Yahoo! ResearchPapers
  • Chaos Monkey (source)Netflix · GitHubGitHub
  • Google SRE Book: Addressing Cascading FailuresGoogle SRE team · GoogleBlogs
  • Chaos MonkeyNetflixDocumentation

  Show 6 moreShow fewer

  • Static stability using Availability ZonesAmazon Builders' Library · Amazon Web ServicesBlogs
  • The Network is ReliableKyle Kingsbury · aphyr.comBlogs
  • Avoiding fallback in distributed systemsAmazon Builders' Library · Amazon Web ServicesBlogs
  • How to do distributed lockingMartin Kleppmann · martin.kleppmann.comBlogs
  • Distributed Locking at UberUberYouTube
  • Resilience Engineering with Project Waterbear at LinkedInLinkedIn engineers · LinkedInCase Studies

  Open this lesson with its quiz and exercise