Dropbox Tech Blog

Ideas you will meet

Storage Engines

Write-ahead logs, B-trees and LSM-trees.

Distributed Systems

Consensus, time, failure and the fundamentals of scale-out.

What we already know about this

Our own lessons on these ideas. Read them here before you start, so it makes sense.

• Real-World Systems · 13 minHow a Twitter-style service worksPosting, timelines, search, trends and counters at huge scale: the architecture of a short-message social network.Read hereClose

  The problem

  A service where people post short messages, follow others, and read a timeline of recent posts. The numbers are what make it hard: hundreds of millions of users, huge read traffic, and a few accounts followed by tens of millions.

  Core operations

  • Post a message (with media, mentions, hashtags).
  • Follow and unfollow.
  • Read the home timeline (people I follow), and a user timeline (one person's posts).
  • Search posts and see trends.
  • Likes, replies, reposts and notifications.

  Estimates

  Say 300 million monthly users, 100 million daily active, each reading their timeline 10 times a day and posting a few times a week.

  • Timeline reads: 1 billion a day, about 12,000 a second on average and several times that at peaks.
  • Posts: about 50 million a day, a few hundred per second on average.
  • Reads outnumber writes by roughly 20 to 1, so we optimise for reading.

  High-level architecture

  100%

  Figure SDA-RWS-16-1 · Short-message service architecture · © forcsguy.com

  Posting

  1. The tweet service validates the post and obtains a unique, time-sortable id (Snowflake-style, see the ID generator lesson).
  2. It stores the tweet in a sharded tweet store, partitioned by tweet or user id. Media goes to object storage and is served through a CDN.
  3. It publishes a "tweet created" event to a stream, so the rest of the work happens asynchronously: fan-out, search indexing, notifications, trends and analytics.

  Home timeline: fan-out

  The home timeline is the hot path. Two strategies (from the news feed lesson):

  • Fan-out on write: when a user posts, workers look up their followers and push the tweet id into each follower's cached timeline list (for example Redis, keeping only the latest few hundred ids). Reading a timeline is then one quick lookup.
  • Fan-out on read: build the timeline on demand by merging recent tweets of everyone you follow. Cheap to post, expensive to read.

  The hybrid used at scale: fan out on write for most accounts, but skip accounts with huge follower counts. Their tweets are fetched at read time and merged into the cached list. This avoids one post creating tens of millions of writes.

  Reading the timeline

  1. Fetch the user's cached list of tweet ids.
  2. Merge in recent tweets from followed popular accounts, plus promoted content.
  3. Hydrate ids into tweets, authors and counts using caches (tweet cache, user cache, counters).
  4. Rank (recency, engagement, relationship, relevance) and return a page with a cursor.
  5. Filter muted, blocked and deleted items at read time.

  User timeline

  A user's own posts are a simple range query: tweets for one author in time order. They are stored by author id and cached, and paginated with a cursor.

  The social graph

  Who follows whom is a huge graph with fast lookups in both directions (followers of A, accounts followed by B). It is stored in a specialised, sharded store, with caches, and updated asynchronously when users follow or unfollow.

  Search

  Posts are indexed in an inverted index (term to list of tweet ids), usually built on a Lucene-style engine. The indexer consumes the event stream, so new tweets become searchable within seconds. The index is sharded by time (recent tweets are the most searched and kept in memory) and replicated. Queries combine text relevance, recency and engagement.

  Trends

  Trending topics are not the most common words, but the ones that are unusually frequent right now. A stream processor counts hashtags and phrases in sliding windows (say 5 minutes), compares them to their normal rate, filters spam and sensitive terms, and publishes the result per region.

  100%

  Figure SDA-RWS-16-2 · Search indexing and push notifications from the same event · © forcsguy.com

  Counters and notifications

  Likes, reposts and view counts are updated very frequently. Use sharded counters and approximate, eventually consistent values, batching updates, since an exact real-time number is rarely required. Notifications are produced from the event stream and delivered through the notification system (see that lesson).

  Scale and reliability

  • Cache everything hot: timelines, tweets, users, counters. The tweet store is the source of truth, not the read path.
  • Shard stores by id, and replicate across zones and regions.
  • Rate limit posting, following and API calls to fight spam and abuse.
  • Degrade gracefully: if ranking is slow, return a chronological timeline; if search lags, show fewer results.
  • Moderation and abuse systems run on the same event stream.
  • Real systems evolve, and details in public talks differ from year to year; the principles above (async fan-out, hybrid push and pull, caching, sharding) are the durable part.

  Real-world reading

  Related reading

  • Hello Interview: System Design in a HurryHello InterviewCourses
  • Hello Interview: System Design WalkthroughsEvan King & Stefan Mai · Hello InterviewYouTube
  • Consistent Hashing and Random Trees (paper)Karger et al. · MITPapers
  • The System Design PrimerDonne Martin · donnemartinGitHub

  Show 6 moreShow fewer

  • How Discord Stores Trillions of MessagesBo Ingram · DiscordBlogs
  • How Discord Stores Billions of MessagesDiscordCase Studies
  • YouTube ArchitectureHigh ScalabilityCase Studies
  • TAO: The power of the graphMeta Engineering · MetaCase Studies
  • Dropbox Tech BlogDropbox engineers · DropboxBlogs
  • Productionizing ML with Workflows at TwitterTwitter engineers · TwitterCase Studies

  Open this lesson with its quiz and exercise

• Interview Preparation · 12 minA method for system design interviewsA 45-minute plan: clarify, estimate, sketch, deep-dive and wrap up, and the habits and mistakes that separate strong answers from weak ones.Read hereClose

  What the interview tests

  A system design interview checks how you think with an open, ambiguous problem: do you clarify, structure, make reasoned trade-offs, communicate and adapt when the interviewer pushes back. There is no single right answer; there are well-reasoned and poorly-reasoned ones.

  The 45-minute plan

  100%

  Figure SDA-INT-01-1 · A time plan for the interview · © forcsguy.com

  1. Clarify (functional and non-functional)

  • Functional: what must the system do? Pick the top three to five features and say what you leave out.
  • Non-functional: scale (users, requests), latency, availability, consistency needs, durability, security, cost, regions.
  • Constraints: platforms, existing systems, deadlines.
  • Write the agreed list where both of you can see it.

  2. Estimate

  Turn the scale into numbers: daily active users, actions per user, requests per second (average and peak, often 2 to 3 times average), read to write ratio, storage per day and over years, bandwidth. Use round numbers and show your arithmetic (see the estimation lesson). Say what the numbers imply: "about 5,000 reads per second means one database with a cache is plausible; 500,000 means we shard and use a CDN."

  3. High-level design

  Draw the main components and data flow for the core use cases: clients, load balancer, services, databases, caches, queues, storage. Define the APIs and the main data entities. Keep it simple and correct first; do not add Kafka because it sounds good.

  4. Deep dive

  Go deeper where it matters or where the interviewer points:

  • Data model and storage choice and why (see the databases lessons).
  • The hot path: how a request is served fast (caching, indexing, precomputation).
  • Scaling a bottleneck: sharding key, replication, fan-out.
  • Consistency and failure handling: what happens when a node, region or dependency fails; retries and idempotency.
  • Specific algorithms if central (rate limiting, ranking, geospatial search, unique ids).

  5. Wrap up

  Name the bottlenecks and single points of failure, how you would monitor the system, security and privacy, cost, and what you would do next with more time. Summarise the trade-offs you chose.

  Habits of strong answers

  • Think aloud and check in: "Is this the direction you want?"
  • Start simple, then scale: a working design first, evolve it as numbers demand.
  • Justify each choice with a reason tied to a requirement; offer alternatives and why you did not choose them.
  • Quantify: use your estimates to decide.
  • Be explicit about trade-offs: consistency versus availability, latency versus cost, simplicity versus flexibility.
  • Handle failure: every box can break; say what happens.
  • Use the interviewer's hints; they are guiding you.

  Common mistakes

  Mistake	Better
  Jumping to technology names	Start from requirements and numbers
  Designing for imaginary scale	Design for stated scale, and say how you would grow
  No estimates	Compute the key numbers and use them
  One giant box labelled "backend"	Break into components with clear responsibilities
  Ignoring failures and consistency	Discuss what breaks and what guarantees you need
  Going shallow everywhere	Go deep on two or three important parts
  Memorised answer for a different problem	Adapt to this problem's requirements
  Silence	Talk through your reasoning

  Low-level design interviews

  The same spirit applies but at class level: clarify, find entities, assign responsibilities, relate them, apply patterns where useful, code the core flow and discuss extensions (see the Low-Level Design Practice course).

  Preparing

  1. Learn the building blocks (the foundations and core courses).
  2. Study classic designs end to end (classic and real-world courses).
  3. Practise out loud with a timer and a friend, or record yourself.
  4. Read engineering blogs of the company you are interviewing with to learn their technology and concerns.
  5. After each practice, write what you missed and revisit the lesson.

  Real-world reading

  Related reading

  • Netflix Technology BlogNetflix Engineering · NetflixBlogs
  • PlanetScale BlogPlanetScale engineers · PlanetScaleBlogs
  • Tech Dummies: System Design Case StudiesNarendra L · Tech DummiesYouTube
  • Dropbox Tech BlogDropbox engineers · DropboxBlogs

  Show 6 moreShow fewer

  • Tech Interview HandbookYangshun Tay · techinterviewhandbook.orgCourses
  • Minimizing Read-Write MySQL Downtime at YelpYelp engineers · YelpCase Studies
  • Cockroach Labs BlogCockroach Labs engineers · Cockroach LabsBlogs
  • Fivetran BlogFivetran engineers · FivetranBlogs
  • How NOT to design Netflix in your 45-minute System Design Interview?Hackernoon engineers · HackernoonBlogs
  • jOOQ Engineering BlogjOOQ engineers · jOOQBlogs

  Open this lesson with its quiz and exercise