Design a news feed

The problem

A news feed shows a user a stream of recent posts from the accounts they follow, newest or most relevant first. It must load fast, even for users who follow thousands of accounts, and new posts should appear within seconds.

Two operations

• Publish a post: store it and make it visible to followers.
• Get feed: return the first page of the user's feed quickly.

Estimates

Say 300 million daily users who open the feed several times a day. Reads vastly outnumber posts, so read speed is the priority. Feeds are hot data, which suits caching.

Two basic strategies

Fan-out on write (push)

When a post is published, immediately push its id into each follower's feed list (kept in a cache such as Redis).

• Reading a feed is a single quick lookup.
• Publishing is expensive for users with many followers, and wasted for followers who never come back.

Fan-out on read (pull)

Build the feed when it is requested: fetch recent posts of everyone the user follows and merge.

• Publishing is cheap.
• Reading is slow and heavy, especially for users who follow many accounts.

The hybrid that works

Use push for most accounts and pull for very popular ones.

100%

Figure SDA-CSD-07-1 · Hybrid feed: push for most, pull for popular accounts · © forcsguy.com

Publish flow

1. Save the post.
2. Put a job on a queue so the request returns quickly.
3. Workers look up the author's followers (from a graph or follower store) and push the post id to each follower's feed list, keeping only the latest few hundred.
4. For accounts above a follower threshold, skip step 3.

Read flow

1. Fetch the user's feed list (post ids) from the feed cache.
2. Fetch recent posts from followed popular accounts and merge.
3. Hydrate the ids by loading post content, author profile and like counts from caches, falling back to the database.
4. Rank and return a page with a cursor for the next one.

Storing only ids in feeds keeps them small. Content lives once, in the post store and its cache.

The two flows

100%

Figure SDA-CSD-07-2 · Publishing a post · © forcsguy.com

100%

Figure SDA-CSD-07-3 · Reading the feed · © forcsguy.com

Components

Component	Role
Post service and store	Create and store posts
Social graph	Who follows whom
Fan-out workers	Build feeds asynchronously
Feed cache	Per-user list of recent post ids
Content and counter caches	Fast hydration
Ranking	Order by recency, relevance and relationships

The cache layers

A feed system leans on separate caches for separate jobs: the feed lists (post ids per user, capped to a few hundred or thousand since few users scroll further), the content of posts (with a hot tier for popular ones), the social graph (who follows whom, often held in a graph-friendly store), actions (did this user like or reply to this post) and counters (likes, replies, followers). Fan-out also respects settings: a muted or hidden account is filtered out when feeds are built, even though the follow still exists. Media files go through a CDN rather than the feed service.

Other concerns

• Pagination: use cursors (the id or time of the last item), not page numbers, so new posts do not shift pages.
• Deletes and privacy changes: mark posts as deleted and filter at read time, since removing ids from millions of feeds is expensive.
• Cold start: a new or returning user has an empty feed, so build it by pull the first time.
• Consistency: feeds are eventually consistent. A post appearing a few seconds late is acceptable.
• Hot keys: viral posts concentrate reads, so cache and replicate them widely.

Worked numbers: why a hybrid is needed

Take 300 million daily users, each opening the feed 5 times a day, and an average user with 300 followers.

• Reads: 300M x 5 = 1.5 billion feed reads a day, about 17,000 per second, 50,000 at peak. Feeds are read far more than they are written, which favours doing work at write time.
• Fan-out on write for an ordinary user: one post becomes 300 feed insertions. If 10 million posts a day are made, that is 3 billion insertions a day, about 35,000 per second. Heavy, but steady and it runs in the background.
• Fan-out on write for a celebrity with 50 million followers: one post becomes 50 million insertions. At 100,000 insertions per second that takes 500 seconds, over eight minutes, during which the post appears to some followers and not others, and the write workers are busy with nothing else. Posting twice an hour would keep the system saturated.
• Fan-out on read for the same celebrity: nothing happens at write time. Each follower's feed request pulls the celebrity's latest posts and merges them: cheap per request, because one celebrity's recent posts are tiny and sit in cache.

Hence the hybrid: push for ordinary accounts (fast reads, bounded write cost), pull for very popular ones (no write explosion), merged when the feed is built.

100%

Figure SDA-CSD-07-4 · The cost of one post, push versus pull · © forcsguy.com

Ranking: chronological versus relevance

A feed can be ordered by time (simple, predictable) or by predicted relevance (better engagement, much more machinery). A relevance feed is usually built in stages: candidate generation gathers a few hundred recent posts from the people you follow; ranking scores each with a model that uses features such as who posted, how you interacted before, post age and content type; filtering removes seen, blocked or policy-violating posts; and the top results are stored as the feed page. Mention that ranking changes the cache design: ranked pages are personal and short-lived, whereas the lists of post ids are shared and reusable.

100%

Figure SDA-CSD-07-5 · A ranked feed in stages · © forcsguy.com

Mistakes people make

• Offset paging on a live feed. New posts shift the list, so users see duplicates or miss items. Use a cursor ("posts older than this id"); see the keyset pagination lab.
• Storing full posts in every follower's feed. Store post ids and fetch the content from a shared cache; otherwise an edit or delete must be repeated millions of times.
• Forgetting deletes and blocks. A deleted or blocked post must disappear from feeds that already contain it; filter at read time.
• Treating all users alike. Inactive users do not need their feeds precomputed. Build lazily on their return.
• No limit on feed length. Keep only the latest few hundred ids per user and fall back to a slower path for older history.

Try it yourself

The write-heavy System Builder challenge "A burst of writes" shows the same pressure as a celebrity post: many writes that the database cannot apply at once. Try it with and without a queue, and compare how it feels to push work into the background. To model the feed's write path yourself, open the social feed on the free canvas.

More on this site