Scaling the Hadoop Distributed File System at LinkedIn

Ideas you will meet

Distributed Systems

Consensus, time, failure and the fundamentals of scale-out.

What we already know about this

Our own lessons on these ideas. Read them here before you start, so it makes sense.

• Low-Level Design Practice · 10 minSocial feed (Instagram-style)Users, follows, posts and likes: a feed model with the Observer pattern for notifications and a strategy for ordering.Read hereClose

  The problem

  Users follow each other, publish photo posts, like and comment, and see a home feed of posts from the accounts they follow. This lesson models the classes and rules. Scaling the feed (fan-out strategies) is covered in the news-feed lesson of the classic designs course.

  Entities

  100%

  Figure SDA-LLD-15-1 · Social feed classes · © forcsguy.com

  Publishing and the feed

  100%

  Figure SDA-LLD-15-2 · Publishing a post · © forcsguy.com

  interface FeedStrategy { build(userId: string, limit: number): Post[] }
  
  const newestFirst: FeedStrategy = {
    build(userId, limit) {
      const followees = follows.acceptedFolloweesOf(userId);
      return posts.byAuthors(followees).filter(p => canView(userId, p))
        .sort((a, b) => b.createdAt - a.createdAt).slice(0, limit);
    },
  };
  
  function canView(viewer: string, p: Post): boolean {          // one place for visibility rules
    const author = users.get(p.authorId);
    return !author.isPrivate || author.id === viewer || follows.isAccepted(viewer, author.id);
  }
  

  Putting visibility in one function means the feed, profile and search all agree about who may see what.

  Likes and comments

  • A like is unique per (user, post); enforce it with a uniqueness rule, not just a check, so double taps do not create two likes.
  • Keep a like count on the post for fast display and update it with each like or unlike, accepting that the count may briefly lag.
  • Comments are ordered by time and can have a parent id for replies.

  Design notes

  • Pagination uses a cursor (the time or id of the last item) so new posts do not shift pages.
  • Deleting a post must remove it from feeds and counts; a soft delete flag filtered at read time is the usual first step.
  • Blocking and muting are visibility rules too.
  • Extensions: stories with expiry, hashtags, tagging, search, and a ranking strategy that mixes recency with interest.

  Real-world reading

  Related reading

  • Self-Ranking Search with Elasticsearch at WattpadWattpad engineers · WattpadCase Studies
  • Monitoring Powers High Availability for LinkedIn FeedUSENIX engineers · USENIXCase Studies
  • Search Federation Architecture at LinkedIn (2018)LinkedIn engineers · LinkedInCase Studies
  • Scaling the Hadoop Distributed File System at LinkedInLinkedIn engineers · LinkedInCase Studies

  Show 6 moreShow fewer

  • Galene: Search Architecture of LinkedInLinkedIn engineers · LinkedInCase Studies
  • Infrastructure to Run User Forecasts at SpotifySpotify engineers · SpotifyCase Studies
  • Scalable User Privacy at SpotifySpotify engineers · SpotifyCase Studies
  • Moving Food Feed from Redis to Cassandra at ZomatoZomato engineers · ZomatoCase Studies
  • Clustering Similar Stories Using LDA at FlipboardFlipboard engineers · FlipboardCase Studies
  • Data Orchestration Service at SpotifySpotify engineers · SpotifyCase Studies

  Open this lesson with its quiz and exercise