The Byzantine Generals Problem (paper)

Ideas you will meet

Consensus

Raft, Paxos and how replicas agree on one order.

Distributed Systems

Consensus, time, failure and the fundamentals of scale-out.

What we already know about this

Our own lessons on these ideas. Read them here before you start, so it makes sense.

• Advanced Distributed Systems · 12 minConsensus: Raft and PaxosHow a group of machines agrees on one order of operations even when some crash or messages are delayed, and why a majority is the magic number.Read hereClose

  The problem consensus solves

  Several machines must agree on one sequence of operations (who is the leader, what is the next log entry, who owns a lock) even though machines crash and messages are delayed or lost. Without a protocol, replicas drift apart and two nodes may both believe they are the leader.

  Systems that rely on consensus include coordination services (etcd, ZooKeeper), many databases for their metadata or their data, and Kubernetes (through etcd).

  Why a majority

  A decision is valid when a majority (quorum) of nodes accepts it. Any two majorities share at least one node, so a later decision always meets someone who remembers the earlier one. A cluster of n nodes tolerates floor((n-1)/2) failures: 3 nodes tolerate 1, 5 nodes tolerate 2. That is why clusters have an odd number of nodes; a 4th node adds cost but no extra fault tolerance over 3.

  Raft in three parts

  Raft splits the problem so it can be understood:

  100%

  Figure SDA-ADS-01-1 · Raft roles and elections · © forcsguy.com

  1. Leader election. Time is divided into numbered terms. A node that hears nothing becomes a candidate and requests votes. A node votes at most once per term, and only for a candidate whose log is at least as up to date as its own. Randomised timeouts make split votes rare.
  2. Log replication. The leader appends a client command to its log and sends it to followers. When a majority has stored the entry, it is committed; the leader applies it and replies to the client. Followers apply committed entries in order, so every replica runs the same sequence.
  3. Safety. The vote rule guarantees that any new leader already has every committed entry, so committed entries are never lost or reordered.

  100%

  Figure SDA-ADS-01-2 · Replicating one write · © forcsguy.com

  Paxos, briefly

  Paxos reaches the same guarantee with two phases (prepare/promise, then accept/accepted) over a majority. It is famous for being hard to implement and explain, and for being the basis of many systems. Multi-Paxos repeats it for a log of decisions. Raft is essentially a clearer arrangement of the same ideas with a strong leader, which is why most new systems choose it.

  What consensus costs and what it does not give

  • Latency: every committed write waits for a majority round trip, so cross-region clusters are slow. Keep consensus groups within a region or use few, small groups for metadata.
  • Throughput: one leader handles all writes in a group. Large databases run many small groups (one per data range) to spread load.
  • Availability under partition: the minority side cannot make progress. This is the CAP trade-off made concrete: consensus systems are consistent first.
  • It does not remove all problems: slow nodes, clock assumptions, and reconfiguration (adding or removing nodes safely) need care.

  Where you will meet it

  Leader election for databases and queues, distributed locks and configuration stores, metadata for storage systems, and cluster state in Kubernetes. When a design needs "exactly one leader" or "all replicas agree on this order", you are describing consensus. See also the distributed-locks lesson in the core-concepts course.

  Real-world reading

  Related reading

  • In Search of an Understandable Consensus Algorithm (Raft paper)Ongaro and Ousterhout · Stanford UniversityPapers
  • The Raft Consensus Algorithm (site and visualisation)Diego Ongaro and community · raft.github.ioDocumentation
  • ZippyDB: Facebook's key-value storeMeta Engineering · MetaCase Studies
  • How Ringpop from Uber Engineering Helps Distribute Your ApplicationUber Engineering · UberCase Studies

  Show 6 moreShow fewer

  • etcd Raft libraryetcd · CNCF / etcdGitHub
  • Google SRE Book: Managing Critical State (Distributed Consensus)Google SRE team · GoogleBlogs
  • Paxos Made Simple (paper)Leslie Lamport · Microsoft ResearchPapers
  • HashiCorp RaftHashiCorp · GitHubGitHub
  • Distributed Systems (4th edition)Maarten van Steen and Andrew Tanenbaum · distributed-systems.netBooks
  • Scaling Hadoop YARN cluster beyond 10,000 nodes at LinkedInLinkedIn engineers · LinkedInCase Studies

  Open this lesson with its quiz and exercise

• Core Concepts: How Things Actually Work · 9 minDistributed locks and leader electionMaking sure only one worker does a job at a time, and why a lock needs a lease and a fencing token to be safe.Read hereClose

  The need

  Sometimes only one worker may do something at a time: run a nightly job, update a shared resource, be the primary of a group. On one machine you use a mutex. Across machines you need a distributed lock or leader election, and failure makes it tricky.

  A naive lock and why it breaks

  SET lock:job 1 NX   (set only if not exists)
  ... do the work ...
  DEL lock:job
  

  Problems:

  • The holder crashes before deleting: the lock is stuck forever.
  • Add an expiry (a lease) to fix that, and a new problem appears: the holder is merely slow (a long pause, a network stall), the lease expires, a second worker takes the lock, then the first wakes up and keeps working. Now two hold it.

  Leases and fencing tokens

  A lease is a lock that expires unless renewed. It lets the system recover from failures, but it cannot, by itself, stop a stalled holder from acting after it has lost the lock.

  The fix is a fencing token: every time the lock is granted, the lock service hands out a number that always increases. The holder sends the token with every write to the protected resource, and the resource rejects any request with a token older than one it has already seen.

  100%

  Figure SDA-CORE-18-1 · Leases and fencing tokens · © forcsguy.com

  This makes safety depend on the resource, not on timing.

  Where to keep locks

  Use a service built for coordination, with consensus underneath: etcd, ZooKeeper, Consul. A single Redis instance gives a convenient but weaker lock (it can lose the lock on failover). If correctness matters, prefer a consensus-based service or a database transaction.

  Often you can avoid a lock: make the operation idempotent, use a conditional write (compare-and-set on a version), or let a queue hand each job to just one consumer.

  Image: Senior Airman John Wright · Public domain · Source · Wikimedia Commons: Lajes Field Air Traffic Control

  Leader election

  The same tools elect a leader: nodes compete to create an ephemeral key or hold a lease; the winner leads and renews it. If it fails, the lease expires and another is chosen. Followers watch for changes.

  • Expect brief periods with no leader during failover.
  • Guard against a former leader acting too late (fencing tokens again, or stepping down when it cannot renew).
  • Avoid split brain (two leaders) by requiring a majority (a quorum) to agree.

  Rules of thumb

  • Keep critical sections short, and renew the lease while working.
  • Prefer designs that need no lock: idempotency, conditional writes, partitioned ownership.
  • Always handle "I lost the lock" by stopping work.
  • Test with pauses and network partitions, not only crashes.

  More on this site

  Related reading

  • ZooKeeper: Wait-free Coordination for Internet-scale Systems (paper)Hunt et al. · Yahoo! ResearchPapers
  • How Ringpop from Uber Engineering Helps Distribute Your ApplicationUber Engineering · UberCase Studies
  • The Chubby Lock Service (paper)Mike Burrows · GooglePapers
  • Google SRE Book: Managing Critical State (Distributed Consensus)Google SRE team · GoogleBlogs

  Show 6 moreShow fewer

  • The Byzantine Generals Problem (paper)Lamport, Shostak and Pease · Microsoft ResearchPapers
  • Paxos Made Simple (paper)Leslie Lamport · Microsoft ResearchPapers
  • Apache ZooKeeperApache Software Foundation · GitHubGitHub
  • Google SRE Book: Distributed Periodic Scheduling with CronGoogle SRE team · GoogleBlogs
  • Nextdoor Engineering BlogNextdoor engineers · NextdoorBlogs
  • Localytics Engineering BlogLocalytics engineers · LocalyticsBlogs

  Open this lesson with its quiz and exercise

• Software Design Essentials · 9 minHow to learn design patternsWhat patterns are, the three families, a map of the classic catalogue, and a practical way to learn them without over-engineering.Read hereClose

  What a pattern is

  A design pattern is a named solution to a recurring design problem, together with when it applies and what it costs. The famous catalogue, Design Patterns (1994) by Gamma, Helm, Johnson and Vlissides ("the Gang of Four"), described 23 patterns in object-oriented software.

  Patterns give you three things:

  1. A shared vocabulary: "use a factory here" communicates a whole idea in a few words.
  2. Proven structures you do not have to invent under pressure.
  3. Awareness of trade-offs, since each pattern comes with consequences.

  They are not copy-paste code, and not a goal in themselves.

  The three families

  100%

  Figure SDA-SDE-01-1 · The three families · © forcsguy.com

  The catalogue at a glance

  Pattern	One-line idea	Typical use
  Factory Method	A method decides which concrete class to create	Create objects without hard-coding the class
  Abstract Factory	Create families of related objects	Themes, platform-specific widget sets
  Builder	Build a complex object step by step	Objects with many optional parts
  Prototype	Create objects by copying an existing one	Costly-to-create objects
  Singleton	Exactly one instance, globally reachable	Rare; often better as dependency injection
  Adapter	Make one interface look like another	Wrap a third-party library
  Bridge	Separate an abstraction from its implementation	Several dimensions of variation
  Composite	Treat a tree of objects like a single object	File systems, UI trees
  Decorator	Add behaviour by wrapping an object	Logging, caching, compression layers
  Facade	A simple front for a complex subsystem	A clean API over messy internals
  Flyweight	Share common state to save memory	Many similar small objects
  Proxy	A stand-in that controls access	Lazy loading, access control, remote calls
  Strategy	Swap algorithms behind an interface	Pricing rules, sorting choices
  Observer	Notify dependants when something changes	Events, UI updates
  Command	Turn a request into an object	Undo, queues, macros
  State	Behaviour changes with internal state	Workflows, connections
  Template Method	A skeleton algorithm with customisable steps	Frameworks, hooks
  Iterator	Walk a collection without exposing its structure	Loops over any collection
  Chain of Responsibility	Pass a request along handlers until one handles it	Middleware, validation chains
  Mediator	One object coordinates others	Chat rooms, form logic
  Memento	Save and restore an object's state	Undo, snapshots
  Visitor	Add operations to a structure without changing it	Compilers, reports over trees
  Interpreter	Evaluate sentences of a small language	Rules, expressions

  The next lessons explain the most useful ones with examples.

  A practical way to learn

  1. Learn the problems first. Each pattern exists because of a pain: rigid code, duplicated code, tangled dependencies. Understand the pain, then the cure.
  2. Start with the ones you already use without naming them: Iterator (every for...of), Observer (event listeners), Decorator and Chain of Responsibility (middleware), Strategy (a function you pass in).
  3. Implement each once, small, in your language, and write what you gave up.
  4. Spot them in real code: frameworks and standard libraries are full of patterns.
  5. Refactor towards a pattern when the code hurts, rather than starting with one.

  Patterns look different in different languages

  In languages with first-class functions, several patterns shrink to a function: Strategy is a function argument, Command is a closure, Observer is a callback list, and Template Method is passing the varying step. In dynamic languages, Factory and Adapter are often trivial. Learn the intent, then use the lightest form your language allows.

  Anti-patterns to avoid

  • Pattern overload: five layers of abstraction for a feature that needs ten lines.
  • Singleton as global state: hides dependencies and makes testing hard.
  • Naming something by the pattern, not by what it does (ManagerFactoryProvider).
  • Applying a pattern "in case": wait until the second or third time you need the flexibility.

  Real-world reading

  Related reading

  • Google SRE Book: Managing Critical State (Distributed Consensus)Google SRE team · GoogleBlogs
  • Paxos Made Simple (paper)Leslie Lamport · Microsoft ResearchPapers
  • Design Patterns CatalogueRefactoring GuruDocumentation
  • The Byzantine Generals Problem (paper)Lamport, Shostak and Pease · Microsoft ResearchPapers

  Show 6 moreShow fewer

  • etcd Raft libraryetcd · CNCF / etcdGitHub
  • Domain-Driven Design ReferenceEric Evans · Domain LanguageBooks
  • Scaling Gradient Boosted Trees for Click-Through-Rate Prediction at YelpYelp engineers · YelpCase Studies
  • Natural Language Processing and Content Analysis (2 parts) at Condé NastCondé Nast engineers · Condé NastCase Studies
  • Building and Deploying MySQL Raft at MetaMeta engineers · MetaCase Studies
  • Lessons of Scale at Facebook - Bobby Johnson, Director of Engineering at FacebookFacebookYouTube

  Open this lesson with its quiz and exercise