Clustering Similar Stories Using LDA at Flipboard

Ideas you will meet

Distributed Systems

Consensus, time, failure and the fundamentals of scale-out.

Machine Learning Systems

Recommendations, ranking, feature stores and serving models at scale.

What we already know about this

Our own lessons on these ideas. Read them here before you start, so it makes sense.

• Low-Level Design Practice · 10 minSocial feed (Instagram-style)Users, follows, posts and likes: a feed model with the Observer pattern for notifications and a strategy for ordering.Read hereClose

  The problem

  Users follow each other, publish photo posts, like and comment, and see a home feed of posts from the accounts they follow. This lesson models the classes and rules. Scaling the feed (fan-out strategies) is covered in the news-feed lesson of the classic designs course.

  Entities

  100%

  Figure SDA-LLD-15-1 · Social feed classes · © forcsguy.com

  Publishing and the feed

  100%

  Figure SDA-LLD-15-2 · Publishing a post · © forcsguy.com

  interface FeedStrategy { build(userId: string, limit: number): Post[] }
  
  const newestFirst: FeedStrategy = {
    build(userId, limit) {
      const followees = follows.acceptedFolloweesOf(userId);
      return posts.byAuthors(followees).filter(p => canView(userId, p))
        .sort((a, b) => b.createdAt - a.createdAt).slice(0, limit);
    },
  };
  
  function canView(viewer: string, p: Post): boolean {          // one place for visibility rules
    const author = users.get(p.authorId);
    return !author.isPrivate || author.id === viewer || follows.isAccepted(viewer, author.id);
  }
  

  Putting visibility in one function means the feed, profile and search all agree about who may see what.

  Likes and comments

  • A like is unique per (user, post); enforce it with a uniqueness rule, not just a check, so double taps do not create two likes.
  • Keep a like count on the post for fast display and update it with each like or unlike, accepting that the count may briefly lag.
  • Comments are ordered by time and can have a parent id for replies.

  Design notes

  • Pagination uses a cursor (the time or id of the last item) so new posts do not shift pages.
  • Deleting a post must remove it from feeds and counts; a soft delete flag filtered at read time is the usual first step.
  • Blocking and muting are visibility rules too.
  • Extensions: stories with expiry, hashtags, tagging, search, and a ranking strategy that mixes recency with interest.

  Real-world reading

  Related reading

  • Self-Ranking Search with Elasticsearch at WattpadWattpad engineers · WattpadCase Studies
  • Monitoring Powers High Availability for LinkedIn FeedUSENIX engineers · USENIXCase Studies
  • Search Federation Architecture at LinkedIn (2018)LinkedIn engineers · LinkedInCase Studies
  • Scaling the Hadoop Distributed File System at LinkedInLinkedIn engineers · LinkedInCase Studies

  Show 6 moreShow fewer

  • Galene: Search Architecture of LinkedInLinkedIn engineers · LinkedInCase Studies
  • Infrastructure to Run User Forecasts at SpotifySpotify engineers · SpotifyCase Studies
  • Scalable User Privacy at SpotifySpotify engineers · SpotifyCase Studies
  • Moving Food Feed from Redis to Cassandra at ZomatoZomato engineers · ZomatoCase Studies
  • Clustering Similar Stories Using LDA at FlipboardFlipboard engineers · FlipboardCase Studies
  • Data Orchestration Service at SpotifySpotify engineers · SpotifyCase Studies

  Open this lesson with its quiz and exercise