Seagull: Distributed System that Helps Running > 20 Million Tests Per Day at Yelp

Ideas you will meet

Distributed Systems

Consensus, time, failure and the fundamentals of scale-out.

What we already know about this

Our own lessons on these ideas. Read them here before you start, so it makes sense.

• Interview Preparation · 9 minTrade-off and numbers cheat sheetA one-page reference: latency numbers, availability maths, capacity formulas and the standard trade-offs, for quick revision.Read hereClose

  Round numbers worth remembering

  These are approximate orders of magnitude, enough for estimates (exact figures vary by hardware and year).

  Operation	Rough time
  CPU cache or memory read	nanoseconds to about 100 ns
  Read 1 MB sequentially from memory	tens of microseconds
  SSD random read	around 100 microseconds
  Round trip within a data centre	around 0.5 ms
  Read 1 MB sequentially from SSD	around 1 ms
  Disk seek (hard disk)	around 10 ms
  Round trip, same continent	tens of ms
  Round trip, across oceans	100 ms and above

  Take-aways: memory is far faster than disk, disk faster than network across regions; avoid unneeded round trips; sequential access beats random.

  Powers and sizes

  Unit	Approximate size
  1 KB	a thousand bytes (a short text)
  1 MB	a million bytes (a photo, a small document)
  1 GB	a billion bytes
  1 TB	a thousand GB
  1 PB	a thousand TB
  Seconds per day	about 86,400 (use about 100,000 for quick maths)
  Seconds per month	about 2.6 million
  Seconds per year	about 31.5 million

  Quick capacity formulas

  • Requests per second (average) = requests per day / 86,400. Peak = average times a peak factor (2 to 5).
  • Storage = records per day times bytes per record times days retained (times replication factor).
  • Bandwidth = requests per second times average response size.
  • Servers = peak requests per second / requests per second one server handles, plus headroom.
  • Cache size = hot working set (often 20 percent of data serves 80 percent of reads).
  • Connections (for chat or streaming) = concurrent users; memory per connection times that gives a first estimate.

  Example: 10 million users, each posting 2 items a day of 1 KB: 20 million items a day, about 230 writes per second, 20 GB a day, about 7 TB a year before replication.

  Availability maths

  Availability	Downtime per year	Per month (30 days)
  99%	about 3.65 days	about 7.2 hours
  99.9%	about 8.8 hours	about 43 minutes
  99.99%	about 53 minutes	about 4.3 minutes
  99.999%	about 5.3 minutes	about 26 seconds

  Components in series multiply (two 99.9% parts in a chain give about 99.8%); components in parallel (redundant) improve availability: failure needs both to fail. This is why redundancy across zones helps and long dependency chains hurt.

  The standard trade-offs

  100%

  Figure SDA-INT-03-1 · Trade-offs to name in every design · © forcsguy.com

  Choice	Option A	Option B
  Data model	Relational: constraints, transactions	NoSQL: flexible, easier horizontal scale
  Scaling	Vertical: simple, limited	Horizontal: scalable, more complex
  Replication	Synchronous: safer, slower	Asynchronous: faster, may lose data on failover
  Feeds	Fan-out on write: fast reads, heavy writes	Fan-out on read: cheap writes, slow reads
  Delivery	At-least-once plus idempotency	At-most-once for loss-tolerant data
  Caching	Cache-aside: simple, risk of staleness	Write-through: fresher, slower writes
  Consistency	Strong: easy to reason about	Eventual: available and fast
  Communication	Synchronous calls: simple	Asynchronous messages: resilient, harder to trace
  Architecture	Monolith: simple to start	Microservices: independent scaling and teams, more operations
  ID generation	Sequential: sortable, coordination	Random (UUID): no coordination, poor locality

  Patterns to reach for

  Problem	Pattern
  Reads overwhelm the database	Cache, read replicas, CDN
  Writes overwhelm one database	Sharding, queues to smooth bursts, LSM-based stores
  Hot key	Replicate, split the key, local caching
  Slow external dependency	Timeout, bulkhead, circuit breaker, queue
  Duplicate deliveries	Idempotency keys, deduplication
  Cross-service transaction	Saga with compensation
  Need history or audit	Event log, event sourcing
  Unique "does this exist" checks at scale	Bloom filter in front of storage
  Even spread when nodes change	Consistent hashing
  Overload	Rate limit, load shed, backpressure

  Final checks before you finish an answer

  • Requirements and scale stated; numbers used.
  • Main flow and data model clear.
  • Bottlenecks and single points of failure named, with mitigations.
  • Failures, consistency and security discussed.
  • Monitoring and next steps mentioned.
  • Trade-offs spoken aloud.

  Real-world reading

  Related reading

  • MySQL High Availability at GitHubGitHub Engineering · GitHubCase Studies
  • Mitigating MySQL Replication Lag and Reducing Read Load at GithubGithub engineers · GithubCase Studies
  • Read Consistency with Database Replicas at ShopifyShopify engineers · ShopifyCase Studies
  • Uber Schemaless: A Scalable MySQL DatastoreUber Engineering · UberCase Studies

  Show 6 moreShow fewer

  • Apache CassandraApache Software Foundation · GitHubGitHub
  • Jordan Has No Life: Systems Design Concepts SeriesJordan · Jordan Has No LifeYouTube
  • Black-Box Auditing: Verifying End-to-End Replication Integrity between MySQL and Redshift at YelpYelp engineers · YelpCase Studies
  • Cassandra architecture overviewApache Cassandra · Apache Software FoundationDocumentation
  • Eliminate the Database for Higher Availability at American ExpressAmerican Express engineers · American ExpressCase Studies
  • High-availability SaaS Infrastructure at FreeAgentFreeAgent engineers · FreeAgentCase Studies

  Open this lesson with its quiz and exercise