Cloud Object Store at Yahoo

Ideas you will meet

Distributed Systems

Consensus, time, failure and the fundamentals of scale-out.

What we already know about this

Our own lessons on these ideas. Read them here before you start, so it makes sense.

• Software Design Essentials · 9 minCreational patterns: Factory, Builder, SingletonWays to create objects without hard-wiring the exact class: factories, builders, prototypes and why Singleton is usually a last resort.Read hereClose

  The goal

  Creational patterns manage how objects are created, so the rest of the code does not depend on exact classes or messy construction steps.

  Factory Method (and simple factories)

  Callers ask for what they need, and a factory decides which concrete class to create.

  interface Sender { send(to: string, text: string): Promise<void> }
  
  class EmailSender implements Sender { async send(to: string, text: string) { /* ... */ } }
  class SmsSender   implements Sender { async send(to: string, text: string) { /* ... */ } }
  
  function createSender(channel: "email" | "sms"): Sender {
    switch (channel) {
      case "email": return new EmailSender();
      case "sms":   return new SmsSender();
    }
  }
  
  await createSender(user.preferredChannel).send(user.contact, "Your code is 482913");
  

  Benefits: one place to change when classes change, callers depend only on the interface, and tests can supply fakes. A registry (a map from name to constructor) makes adding a type a one-line change.

  Factory Method as a picture

  100%

  Figure SDA-SDE-03-1 · Calling code, factory and products · © forcsguy.com

  Abstract Factory

  Create families of related objects that must be used together, such as a UI toolkit's button, checkbox and menu for one platform or theme.

  interface UiFactory { button(): Button; checkbox(): Checkbox }
  class DarkUi implements UiFactory { /* returns dark variants */ }
  class LightUi implements UiFactory { /* returns light variants */ }
  

  Use it when several products must stay consistent. Skip it when you only vary one object.

  Builder

  Construct a complex object step by step, especially with many optional parts. In languages with named parameters or option objects this is often just that.

  const request = new RequestBuilder("https://api.example.com/orders")
    .method("POST")
    .header("Authorization", token)
    .json({ items })
    .timeout(5000)
    .build();   // validates and returns an immutable Request
  

  Benefits: readable construction, validation in one place, and immutable results. The alternative, a constructor with ten positional parameters, is easy to misuse.

  Prototype

  Create new objects by copying an existing one, useful when creation is expensive or when you want variations of a configured template.

  const base = { retries: 3, timeout: 2000, headers: { accept: "json" } };
  const slow = { ...base, timeout: 10000 };            // shallow copy with a change
  const deep = structuredClone(base);                  // deep copy
  

  Beware shallow versus deep copies: sharing nested objects causes surprising bugs.

  Singleton

  Ensures only one instance exists and gives global access to it.

  class Config {
    private static instance?: Config;
    static get(): Config { return (Config.instance ??= new Config()); }
  }
  

  It looks convenient but has real costs: hidden dependencies, global mutable state, difficulty in testing (state leaks between tests), and trouble with concurrency. Often what you really need is one instance, not global access.

  Better: create the single instance once, in the application's start-up code (the composition root), and inject it into what needs it. You keep "only one" without making it global.

  Choosing

  Situation	Pattern
  Which class to create depends on input or configuration	Factory Method or a registry
  Several related objects must match	Abstract Factory
  Many optional parts, validation needed	Builder or an options object
  Cloning a configured template	Prototype or a copy
  "There should be only one"	Create one in the composition root and inject it

  Prefer the lightest form your language offers: a function that returns an object is often all the "factory" you need.

  Real-world reading

  Related reading

  • Ambry: Distributed Immutable Object Store at LinkedInLinkedIn engineers · LinkedInCase Studies
  • XCMetrics: All-in-One Tool for Tracking Xcode Build Metrics at SpotifySpotify engineers · SpotifyCase Studies
  • Hugging Face BlogHugging Face engineers · Hugging FaceBlogs
  • Design Patterns CatalogueRefactoring GuruDocumentation

  Show 6 moreShow fewer

  • The Twelve-Factor AppAdam Wiggins · 12factor.netDocumentation
  • Scaling Global CDN at Netflix - Dave Temkin, Director of Global Networks at NetflixNetflixYouTube
  • 8 Things You Need to Know Before a System Design InterviewGainlo engineers · GainloBlogs
  • Cloud Object Store at YahooYahoo engineers · YahooCase Studies
  • Dynamometer: Scale Testing HDFS on Minimal Hardware with Maximum Fidelity at LinkedInLinkedIn engineers · LinkedInCase Studies
  • Kakao Blog (Korean)Kakao engineers · KakaoBlogs

  Open this lesson with its quiz and exercise

• Low-Level Design Practice · 11 minChess gameBoard, pieces and rules: polymorphic pieces that generate moves, validating turns and check, and keeping the board model separate from the UI.Read hereClose

  The problem

  Two players take turns on an 8 by 8 board. The system validates moves, detects check, checkmate and stalemate, and records the game.

  Entities

  100%

  Figure SDA-LLD-12-1 · Chess classes · © forcsguy.com

  Inheritance is justified here: each piece type is a real kind of piece with its own movement rule.

  Pieces know their own movement

  abstract class Piece {
    constructor(readonly color: "W" | "B") {}
    abstract legalMoves(board: Board, from: Pos): Pos[];     // ignoring check
  }
  
  class Rook extends Piece {
    legalMoves(board: Board, from: Pos): Pos[] {
      return slide(board, from, [[1,0],[-1,0],[0,1],[0,-1]], this.color);   // along rank and file until blocked
    }
  }
  class Knight extends Piece {
    legalMoves(board: Board, from: Pos): Pos[] {
      return [[1,2],[2,1],[-1,2],[-2,1],[1,-2],[2,-1],[-1,-2],[-2,-1]]
        .map(([dx, dy]) => ({ x: from.x + dx, y: from.y + dy }))
        .filter((p) => board.inside(p) && board.pieceAt(p)?.color !== this.color);
    }
  }
  

  A helper slide is shared by rook, bishop and queen so the logic is not repeated.

  Validating a move

  100%

  Figure SDA-LLD-12-2 · Validating a move · © forcsguy.com

  Game status: check if the opponent's king is attacked; checkmate if in check with no legal move; stalemate if not in check but no legal move.

  Moves as objects

  class Move {
    constructor(readonly from: Pos, readonly to: Pos, readonly captured?: Piece, readonly kind: "normal" | "castle" | "enPassant" | "promotion" = "normal") {}
  }
  class History { done: Move[] = []; undone: Move[] = [];
    play(m: Move, b: Board) { b.apply(m); this.done.push(m); this.undone = []; }
    undo(b: Board) { const m = this.done.pop(); if (m) { b.revert(m); this.undone.push(m); } }
  }
  

  A move object makes undo, redo, history and notation straightforward: this is the Command pattern.

  Design notes

  • Special rules: castling (neither piece has moved, path clear and not attacked), en passant (needs the previous move), promotion (choose a piece). Each needs a little extra state: hasMoved on pieces, last move on the game.
  • Model and view stay separate; an AI or network player implements the same Player interface.
  • Clocks are a separate component observing the turn.
  • Extensions: a move-suggestion engine, saved games, variants with different boards, online play with a server validating every move (never trust the client).

  Real-world reading

  Related reading

  • Operating Systems: Three Easy PiecesRemzi and Andrea Arpaci-Dusseau · University of WisconsinBooks
  • Ambry: Distributed Immutable Object Store at LinkedInLinkedIn engineers · LinkedInCase Studies
  • Cloud Object Store at YahooYahoo engineers · YahooCase Studies
  • Netflix: What Happens When You Press Play?High Scalability engineers · High ScalabilityBlogs

  Show 6 moreShow fewer

  • 8 Things You Need to Know Before a System Design InterviewGainlo engineers · GainloBlogs
  • The Dataflow Model (paper)Akidau et al. · GooglePapers
  • Atomic Object Engineering BlogAtomic Object engineers · Atomic ObjectBlogs
  • Cloud Bouncer: Distributed Rate Limiting at YahooYahoo engineers · YahooCase Studies
  • Object StorageDatacenterknowledge engineers · DatacenterknowledgeCase Studies
  • Remind Engineering BlogRemind engineers · RemindBlogs

  Open this lesson with its quiz and exercise