Benchmarking Streaming Computation Engines at Yahoo

Ideas you will meet

Event Driven Architecture

Loosely coupled services reacting to events.

Distributed Systems

Consensus, time, failure and the fundamentals of scale-out.

What we already know about this

Our own lessons on these ideas. Read them here before you start, so it makes sense.

• Software Design Essentials · 11 minBehavioural patterns: Strategy, Observer, Command, StatePatterns for how objects cooperate: swapping algorithms, reacting to events, turning actions into objects and changing behaviour with state.Read hereClose

  The goal

  Behavioural patterns distribute responsibility between objects and define how they communicate.

  Strategy

  Define a family of interchangeable algorithms behind an interface and pick one at runtime.

  type Pricing = (subtotalCents: number) => number;
  
  const regular: Pricing = (s) => s;
  const member:  Pricing = (s) => Math.round(s * 0.9);
  const blackFriday: Pricing = (s) => Math.round(s * 0.7);
  
  function checkout(subtotal: number, pricing: Pricing) { return pricing(subtotal); }
  checkout(10_000, member);   // the strategy is passed in
  

  Replaces long if/else or switch chains and supports the open/closed principle. In languages with first-class functions the "strategy" is simply a function you pass in.

  Observer (publish and subscribe)

  Objects subscribe to a subject and are notified when it changes, without the subject knowing who listens.

  class Emitter<T> {
    private handlers = new Set<(v: T) => void>();
    on(h: (v: T) => void) { this.handlers.add(h); return () => this.handlers.delete(h); } // returns "unsubscribe"
    emit(v: T) { this.handlers.forEach((h) => h(v)); }
  }
  const cart = new Emitter<number>();
  const stop = cart.on((count) => updateBadge(count));
  cart.emit(3);
  stop();                                              // always clean up
  

  Used for UI events, domain events and message buses. Watch for leaks (forgotten unsubscribes), ordering surprises and cascades of updates that are hard to follow.

  Observer as a picture

  100%

  Figure SDA-SDE-05-1 · One subject, many observers · © forcsguy.com

  Command

  Turn a request into an object containing everything needed to run it. That lets you queue, log, retry, schedule and undo it.

  interface Command { run(): void; undo(): void }
  
  class InsertText implements Command {
    constructor(private doc: Doc, private at: number, private text: string) {}
    run()  { this.doc.insert(this.at, this.text); }
    undo() { this.doc.remove(this.at, this.text.length); }
  }
  
  const history: Command[] = [];
  function perform(c: Command) { c.run(); history.push(c); }
  function undo() { history.pop()?.undo(); }
  

  Editors, job queues, macros and transaction logs all use it.

  State

  Let an object change behaviour when its internal state changes, by delegating to a state object (or a transition table).

  type State = "new" | "paid" | "shipped" | "cancelled";
  const next: Record<State, Partial<Record<"pay" | "ship" | "cancel", State>>> = {
    new:       { pay: "paid", cancel: "cancelled" },
    paid:      { ship: "shipped", cancel: "cancelled" },
    shipped:   {},
    cancelled: {},
  };
  function apply(state: State, action: "pay" | "ship" | "cancel"): State {
    const to = next[state][action];
    if (!to) throw new Error(`Cannot ${action} an order that is ${state}`);
    return to;
  }
  

  Makes illegal transitions explicit and impossible, replacing scattered flags and nested conditions.

  Template Method

  Define the skeleton of an algorithm in one place and let steps vary.

  abstract class Importer {
    run() { const raw = this.read(); const rows = this.parse(raw); this.save(rows); } // fixed order
    protected abstract read(): string;
    protected abstract parse(raw: string): Row[];
    protected save(rows: Row[]) { /* default */ }
  }
  

  Frameworks use it widely ("override these hooks"). With functions, you can pass the varying steps in instead of subclassing.

  Chain of Responsibility

  Pass a request along a chain of handlers until one handles it (or each adds something). Middleware pipelines in web servers are this pattern.

  type Handler = (req: Req, next: () => Promise<Res>) => Promise<Res>;
  // authenticate -> rateLimit -> validate -> handle
  

  Iterator and others

  • Iterator: walk a collection without exposing its structure. In most languages this is built in (for...of, generators).
  • Mediator: one object coordinates many (a chat room, a form controller) so participants do not reference each other.
  • Memento: capture and restore an object's state (undo snapshots).
  • Visitor: add new operations over a structure of object types without modifying them; common in compilers.
  • Interpreter: evaluate expressions of a small language.

  Choosing

  Problem	Pattern
  Many alternative algorithms or rules	Strategy
  React to changes without coupling	Observer
  Undo, queue, log or retry actions	Command
  Behaviour depends on lifecycle state	State
  Same steps, different details	Template Method
  A pipeline of optional processing steps	Chain of Responsibility

  Remember: with first-class functions many of these become callbacks, closures and tables. Use the lightest form that keeps the code clear.

  Real-world reading

  Related reading

  • Cape: Event Stream Processing Framework at DropboxDropbox engineers · DropboxCase Studies
  • NATS ServerNATS · CNCF / NATSGitHub
  • Samza: Stream Processing System for Latency Insighs at LinkedInLinkedIn engineers · LinkedInCase Studies
  • Autoscaling AWS Step Functions Activities at YelpYelp engineers · YelpCase Studies

  Show 6 moreShow fewer

  • Distributed Messaging, Queuing, and Event StreamingArxivPapers
  • Steady State: Always Put Logs on Separate DiskMicrosoft engineers · MicrosoftCase Studies
  • Self-Serve Reporting Platform on Hadoop at LinkedInLinkedInPapers
  • Ambry: Distributed Immutable Object Store at LinkedInLinkedIn engineers · LinkedInCase Studies
  • Benchmarking Streaming Computation Engines at YahooYahoo engineers · YahooCase Studies
  • NodeJS High Availability at YahooYahoo engineers · YahooCase Studies

  Open this lesson with its quiz and exercise