How Facebook Live Streams to 800,000 Simultaneous Viewers - High Scalability -

Ideas you will meet

Distributed Systems

Consensus, time, failure and the fundamentals of scale-out.

What we already know about this

Our own lessons on these ideas. Read them here before you start, so it makes sense.

• Advanced Distributed Systems · 13 minDistributed counters, leaderboards and live reactionsThree high-write designs: counting at huge scale without a hot row, ranking millions of players, and broadcasting reactions to millions of viewers.Read hereClose

  The common difficulty

  All three problems involve huge numbers of small writes to shared state that many people read at once. The pattern behind the solutions is the same: do not contend on one place. Spread the writes, combine the results, and accept that some answers can be approximate or slightly late.

  1. Distributed counters

  Counting likes, views or API calls looks trivial until millions of updates per second hit one row.

  100%

  Figure SDA-ADS-10-1 · A sharded counter · © forcsguy.com

  Options, from simple to scalable:

  • One atomic increment in a fast store (Redis INCR): thousands per second per key, fine for many uses.
  • Sharded counter: split one logical counter into N sub-counters; write to a random one; sum on read.
  • Local aggregation then merge: each server counts in memory and sends deltas periodically; a collector merges (the approach of metrics systems).
  • Event log then aggregate: write events to a log, count with a stream job; exact, replayable, slower to show.
  • Mergeable counters (CRDTs): counters that can be updated on several replicas and merged without conflict (see collaborative editing).
  • Approximate counting: for unique counts (distinct viewers) use HyperLogLog, which counts millions of items in a few kilobytes with about one percent error (see the probabilistic structures lesson).

  Decide the accuracy you need: a like count can lag and be approximate; a bill must be exact (use an append-only log and idempotent processing).

  Before: one hot counter

  100%

  After: sharded counters

  100%

  What changed

  • Writes spread over many counters instead of queuing on one
  • A background job adds the pieces together
  • The total can lag slightly behind the truth

  2. Leaderboards

  A leaderboard needs: update a player's score, get their rank, show the top N, and show neighbours around a player.

  100%

  Figure SDA-ADS-10-2 · Leaderboard with a sorted set · © forcsguy.com

  • Redis sorted sets (ZADD, ZRANGE, ZRANK) do this in logarithmic time and handle millions of members on one node.
  • Persistence: keep scores in a database too (the sorted set is a fast view that can be rebuilt).
  • Scale: split by season or time window (daily, weekly boards), by region or game mode, or by score range; compute a global top-N by merging each shard's top results.
  • Approximate ranks for players far from the top are often enough ("you are in the top 12 percent"); exact rank is needed only near the top.
  • Ties: break by earliest time or id so ordering is deterministic.
  • Cheating and fairness: validate scores server-side.

  3. Live comments and reactions

  A live event with millions of viewers sends a flood of reactions that everyone expects to see "live".

  100%

  Figure SDA-ADS-10-3 · Reactions at massive scale · © forcsguy.com

  For live comments (text), you cannot aggregate, so:

  • Moderate and rank: filter spam, show a sampled or top subset to big audiences.
  • Partition by chat room and cap rates per user.
  • Fan-out via a pub/sub layer to connection gateways, which hold the WebSocket connections (see the chat lesson).
  • Store comments in a log or wide-column store ordered by time for replay and history.

  Key tricks across all of these: aggregate before you fan out, sample, throttle clients, batch writes, and accept approximate, eventually consistent displays while keeping money-related counts exact elsewhere.

  Questions to ask in an interview

  How exact must the number be? How fast must it be visible? What is the write rate per key (hot key)? What happens on failure (can we lose a second of counts)? Can we pre-aggregate at the edge? These questions steer you from "a table with a counter column" to a design that survives a viral moment.

  Real-world reading

  Related reading

  • Scale with Redis ClusterRedis · Redis Ltd.Documentation
  • Redis persistenceRedis · Redis Ltd.Documentation
  • Arpit Bhayani: Asli EngineeringArpit Bhayani · Asli EngineeringYouTube
  • Moving Food Feed from Redis to Cassandra at ZomatoZomato engineers · ZomatoCase Studies

  Show 6 moreShow fewer

  • Improving key expiration in Redis at TwitterTwitter engineers · TwitterCase Studies
  • Scaling Redis at TwitterTwitter engineers · TwitterCase Studies
  • Ratequeue: Core Queueing-And-Rate-Limiting System at TwilioTwilio engineers · TwilioCase Studies
  • Scaling (a NSFW site) to 200 Million Views A Day And Beyond - Eric Pickup, Lead Platform Developer at MindGeekMindGeekYouTube
  • How GIPHY Delivers a GIF to 300 Millions Users - Alex Hoang and Nima Khoshini, Services Engineers at GIPHYGIPHY engineers · GIPHYCase Studies
  • UC Berkeley CS 162: Operating SystemsBerkeley EECS · UC BerkeleyCourses

  Open this lesson with its quiz and exercise