Distributed systems theory for the distributed systems engineer

Ideas you will meet

Distributed Systems

Consensus, time, failure and the fundamentals of scale-out.

Consensus

Raft, Paxos and how replicas agree on one order.

What we already know about this

Our own lessons on these ideas. Read them here before you start, so it makes sense.

• Advanced Distributed Systems · 12 minConsensus: Raft and PaxosHow a group of machines agrees on one order of operations even when some crash or messages are delayed, and why a majority is the magic number.Read hereClose

  The problem consensus solves

  Several machines must agree on one sequence of operations (who is the leader, what is the next log entry, who owns a lock) even though machines crash and messages are delayed or lost. Without a protocol, replicas drift apart and two nodes may both believe they are the leader.

  Systems that rely on consensus include coordination services (etcd, ZooKeeper), many databases for their metadata or their data, and Kubernetes (through etcd).

  Why a majority

  A decision is valid when a majority (quorum) of nodes accepts it. Any two majorities share at least one node, so a later decision always meets someone who remembers the earlier one. A cluster of n nodes tolerates floor((n-1)/2) failures: 3 nodes tolerate 1, 5 nodes tolerate 2. That is why clusters have an odd number of nodes; a 4th node adds cost but no extra fault tolerance over 3.

  Raft in three parts

  Raft splits the problem so it can be understood:

  100%

  Figure SDA-ADS-01-1 · Raft roles and elections · © forcsguy.com

  1. Leader election. Time is divided into numbered terms. A node that hears nothing becomes a candidate and requests votes. A node votes at most once per term, and only for a candidate whose log is at least as up to date as its own. Randomised timeouts make split votes rare.
  2. Log replication. The leader appends a client command to its log and sends it to followers. When a majority has stored the entry, it is committed; the leader applies it and replies to the client. Followers apply committed entries in order, so every replica runs the same sequence.
  3. Safety. The vote rule guarantees that any new leader already has every committed entry, so committed entries are never lost or reordered.

  100%

  Figure SDA-ADS-01-2 · Replicating one write · © forcsguy.com

  Paxos, briefly

  Paxos reaches the same guarantee with two phases (prepare/promise, then accept/accepted) over a majority. It is famous for being hard to implement and explain, and for being the basis of many systems. Multi-Paxos repeats it for a log of decisions. Raft is essentially a clearer arrangement of the same ideas with a strong leader, which is why most new systems choose it.

  What consensus costs and what it does not give

  • Latency: every committed write waits for a majority round trip, so cross-region clusters are slow. Keep consensus groups within a region or use few, small groups for metadata.
  • Throughput: one leader handles all writes in a group. Large databases run many small groups (one per data range) to spread load.
  • Availability under partition: the minority side cannot make progress. This is the CAP trade-off made concrete: consensus systems are consistent first.
  • It does not remove all problems: slow nodes, clock assumptions, and reconfiguration (adding or removing nodes safely) need care.

  Where you will meet it

  Leader election for databases and queues, distributed locks and configuration stores, metadata for storage systems, and cluster state in Kubernetes. When a design needs "exactly one leader" or "all replicas agree on this order", you are describing consensus. See also the distributed-locks lesson in the core-concepts course.

  Real-world reading

  Related reading

  • In Search of an Understandable Consensus Algorithm (Raft paper)Ongaro and Ousterhout · Stanford UniversityPapers
  • The Raft Consensus Algorithm (site and visualisation)Diego Ongaro and community · raft.github.ioDocumentation
  • ZippyDB: Facebook's key-value storeMeta Engineering · MetaCase Studies
  • How Ringpop from Uber Engineering Helps Distribute Your ApplicationUber Engineering · UberCase Studies

  Show 6 moreShow fewer

  • etcd Raft libraryetcd · CNCF / etcdGitHub
  • Google SRE Book: Managing Critical State (Distributed Consensus)Google SRE team · GoogleBlogs
  • Paxos Made Simple (paper)Leslie Lamport · Microsoft ResearchPapers
  • HashiCorp RaftHashiCorp · GitHubGitHub
  • Distributed Systems (4th edition)Maarten van Steen and Andrew Tanenbaum · distributed-systems.netBooks
  • Scaling Hadoop YARN cluster beyond 10,000 nodes at LinkedInLinkedIn engineers · LinkedInCase Studies

  Open this lesson with its quiz and exercise

• Low-Level Design Practice · 11 minMovie ticket bookingSeats, shows and temporary holds: how to let many people pick seats at once without selling the same seat twice.Read hereClose

  The problem

  A theatre chain runs screens that show movies at set times. Customers browse shows, pick seats, pay and receive a ticket. Many customers pick seats for the same popular show at once. A seat must never be sold twice.

  Entities

  100%

  Figure SDA-LLD-05-1 · Movie booking classes · © forcsguy.com

  The key idea: ShowSeat is a seat in one show. A physical seat exists for all shows; it is available or taken per show. Status values: AVAILABLE, HELD, BOOKED.

  The seat lifecycle

  100%

  Figure SDA-LLD-05-2 · Show seat status · © forcsguy.com

  class ShowSeat {
    status: "AVAILABLE" | "HELD" | "BOOKED" = "AVAILABLE";
    heldBy?: string; heldUntil = 0;
  
    // must be atomic: a lock per seat in memory, a conditional UPDATE in a database
    tryHold(user: string, ms = 10 * 60_000): boolean {
      if (this.status === "HELD" && Date.now() > this.heldUntil) this.status = "AVAILABLE";
      if (this.status !== "AVAILABLE") return false;
      this.status = "HELD"; this.heldBy = user; this.heldUntil = Date.now() + ms;
      return true;
    }
    confirm(user: string): boolean {
      if (this.status !== "HELD" || this.heldBy !== user) return false;
      this.status = "BOOKED"; return true;
    }
  }
  

  In a database this is one statement: UPDATE show_seat SET status='HELD', held_by=?, held_until=? WHERE id=? AND (status='AVAILABLE' OR held_until < now()) and the affected-row count tells you whether you won.

  Booking several seats

  100%

  Figure SDA-LLD-05-3 · Booking flow · © forcsguy.com

  If any seat in the group cannot be held, release the ones you did hold and tell the customer. This avoids a half-booked group.

  Design notes

  • Pricing by seat category, time and day is a strategy or a price table on ShowSeat.
  • Expiry can be a background job or a check at read time (as above). A job keeps the displayed seat map accurate.
  • Hot shows: at huge scale put the seat state in a fast store with atomic operations, and queue requests; this overlaps with the flash-sale design in the real-world systems course.
  • Idempotent payment: a retried payment callback must not issue a second ticket.
  • Extensions: seat recommendations, food add-ons, cancellations with refund policy, and notifications on confirm.

  Real-world reading

  Related reading

  • Rewriting the heart of our sync engineDropboxCase Studies
  • Megastore: Providing Scalable, Highly Available Storage (paper)Baker et al. · GooglePapers
  • Scaling Dropbox - Kevin Modzelewski, Back-end Engineer at DropboxDropboxYouTube
  • Hussein Nasser: Backend Engineering ShowHussein Nasser · Hussein Nasser (YouTube)YouTube

  Show 6 moreShow fewer

  • Uber Engineering — Where the Digital World Meets the Physical OneUber engineers · UberBlogs
  • Handles Millions of Digital Transactions Safely Everyday at PaytmPaytm engineers · PaytmCase Studies
  • Distributed systems theory for the distributed systems engineerHenry Robinson · The Paper TrailBlogs
  • The Pragmatic EngineerGergely Orosz · SubstackPodcasts & Newsletters
  • Soundcloud Engineering BlogSoundcloud engineers · SoundcloudBlogs
  • Pluralsight Engineering BlogPluralsight engineers · PluralsightBlogs

  Open this lesson with its quiz and exercise