Guardian Engineering Blog

Ideas you will meet

Distributed Systems

Consensus, time, failure and the fundamentals of scale-out.

What we already know about this

Our own lessons on these ideas. Read them here before you start, so it makes sense.

• Interview Preparation · 9 minKeep learning: papers, blogs and a practice habitHow to keep growing after the courses: a short reading path of classic papers, where to find real engineering write-ups, and a weekly routine that turns reading into skill.Read hereClose

  Why this lesson exists

  The courses give you a toolbox. Interviews and real work reward something further: the habit of studying real systems and reasoning about their trade-offs. This lesson is a short guide to doing that on your own.

  A reading path

  Read in this order, a little at a time:

  1. Foundations first. Make sure the basics are solid: scaling, caching, replication, queues, consistency. (The Foundations and Core Concepts courses.)
  2. The classic papers. A handful of papers explain most of what modern systems copy:
     • The Google File System and MapReduce: how large-scale storage and batch processing began.
     • Bigtable: wide-column storage on a log-structured design.
     • Dynamo: availability-first replication with quorums, hinted handoff and conflict handling (see the key-value store lesson).
     • Raft: consensus you can actually implement (see the consensus lesson).
  3. Pattern catalogues. Named building blocks such as the replicated log, lease and generation clock (see the distributed-system patterns lesson).
  4. Operations and reliability. The SRE book and cloud provider "builders' libraries" cover timeouts, retries, deployments and incident response.

  You do not need to read papers cover to cover. Read the abstract and introduction, the architecture section and the evaluation, and skim the rest.

  How to read an engineering blog post

  Companies publish write-ups of how they solved scaling problems. To get value:

  • Find the trigger: what broke or what grew that forced a change?
  • Note the numbers: users, requests per second, data size, latency targets.
  • List the options they considered and why they rejected them.
  • Look for the failure: what went wrong, and how did they recover?
  • Separate their constraints from yours. A design for a billion users may be wrong for ten thousand.

  Reading the engineering blog of a company you are interviewing with also tells you what technology they use and which problems they care about.

  A weekly routine that works

  100%

  Figure SDA-INT-04-1 · A learning loop that builds skill · © forcsguy.com

  Other ways to practise

  • Read the source or docs of one open-source system (a database, a queue, a proxy) and trace one request through it.
  • Build small versions: a rate limiter, a key-value store with a log, a URL shortener with a cache. Small builds teach what diagrams hide.
  • Do mock interviews with a friend and swap roles; explaining is the skill being tested.
  • Keep a design journal: one page per system with requirements, estimate, diagram, trade-offs and what you would change.
  • Follow incidents: public post-mortems show how real systems fail.

  A closing thought

  No single book or course covers everything, and every large system is a set of trade-offs made under constraints you may not see. The goal is not to memorise designs but to understand why they look the way they do, so that you can design for a new problem. Keep the habit of asking: what is the requirement, what are the numbers, what could fail, and what did we give up?

  Real-world reading

  Related reading

  • In Search of an Understandable Consensus Algorithm (Raft paper)Ongaro and Ousterhout · Stanford UniversityPapers
  • The Raft Consensus Algorithm (site and visualisation)Diego Ongaro and community · raft.github.ioDocumentation
  • ZippyDB: Facebook's key-value storeMeta Engineering · MetaCase Studies
  • The Byzantine Generals Problem (paper)Lamport, Shostak and Pease · Microsoft ResearchPapers

  Show 6 moreShow fewer

  • MDN: HTTP cachingMDN contributors · MozillaDocumentation
  • Amazon Dynamo: Highly Available Key-Value StoreWerner Vogels · AmazonCase Studies
  • Patterns of Distributed SystemsUnmesh Joshi · Martin FowlerBlogs
  • Android Developers BlogGoogle engineers · GoogleBlogs
  • Postmark Engineering BlogPostmark engineers · PostmarkBlogs
  • Anatomy of a System Design InterviewHackernoon engineers · HackernoonBlogs

  Open this lesson with its quiz and exercise