Autoscaling AWS Step Functions Activities at Yelp

Ideas you will meet

Resilience

Timeouts, retries, circuit breakers, bulkheads and load shedding.

What we already know about this

Our own lessons on these ideas. Read them here before you start, so it makes sense.

• Advanced Distributed Systems · 13 minResilience: timeouts, retries, bulkheads, backpressure and circuit breakersHow to stop one slow or failing dependency from taking everything down: the patterns, how they fit together and where each one goes wrong.Read hereClose

  The core problem

  In a system of many services, slowness is more dangerous than failure. A dependency that fails fast is easy to handle. One that responds in 30 seconds makes callers hold threads, connections and memory until they too stop responding: a cascading failure. Resilience patterns limit how far a problem can spread.

  The toolbox

  100%

  Figure SDA-ADS-06-1 · Resilience layers around a call · © forcsguy.com

  Timeouts

  Every network call needs a timeout, including connect, read and overall. Pick from measured latency (for example slightly above the 99th percentile), not a guess, and make the caller's budget smaller than its own caller's so work is abandoned in an orderly way. Propagate deadlines across calls.

  Retries

  Retry only transient failures and only idempotent operations (or use idempotency keys). Use exponential backoff with jitter (random spread) so clients do not retry in lockstep, cap the attempts, and apply a retry budget (for example no more than 10 percent extra traffic). Retry at one layer, not at every layer, or attempts multiply.

  Circuit breaker

  A breaker watches failures or slow calls. Closed: normal. After too many failures it opens: calls fail immediately (or use the fallback) so the dependency can recover and callers stop wasting resources. After a cool-down it goes half-open: a few trial calls decide whether to close again.

  100%

  Figure SDA-ADS-06-2 · Circuit breaker states · © forcsguy.com

  Bulkheads

  Partition resources so one dependency cannot consume them all: separate thread pools or connection pools per dependency, separate queues per tenant or per priority, or separate instances for critical paths. A slow recommendation service then cannot starve checkout.

  Backpressure and load shedding

  If producers outpace consumers, work piles up. Without limits the buffer grows until memory is exhausted or latency becomes huge. Options:

  • Bounded queues that reject or block when full.
  • Signalling upstream to slow down (credit-based flow in streaming systems, TCP windows, reactive streams).
  • Load shedding: deliberately drop low-priority work early (return 429 or 503 quickly) to protect the rest.
  • Rate limiting at the edge (see the rate-limiter lesson).

  Failing fast and visibly under overload is better than accepting everything and failing slowly.

  Fallbacks and graceful degradation

  Return something useful when a dependency fails: cached data, a default, a simpler algorithm, or hide a non-essential section. Decide which features are essential before an incident, and test the degraded mode.

  How they interact

  • A timeout is the minimum; without it nothing else works.
  • Retries inside a breaker inside a bulkhead is the usual nesting.
  • Everything needs metrics: breaker state, rejected calls, queue depth, timeout counts. Alert on them.
  • Test it: inject latency and failures (chaos experiments) to confirm the patterns work before production does it for you.

  Common mistakes

  Retries with no backoff; timeouts longer than the caller's; unbounded queues; a shared thread pool for everything; breakers that never close; fallbacks that call the same failing service; and testing only the happy path.

  Real-world reading

  Related reading

  • Timeouts, retries, and backoff with jitterAmazon Builders' Library · Amazon Web ServicesBlogs
  • Using load shedding to avoid overloadAmazon Builders' Library · Amazon Web ServicesBlogs
  • Circuit BreakerMartin FowlerBlogs
  • Circuit Breaker patternMicrosoft · Microsoft AzureDocumentation

  Show 6 moreShow fewer

  • Google SRE Book: Addressing Cascading FailuresGoogle SRE team · GoogleBlogs
  • Chaos Monkey (source)Netflix · GitHubGitHub
  • Bulkheads: Partition and Tolerate Failure in One PartSkife engineers · SkifeCase Studies
  • Protector: Circuit Breaker for Time Series Databases at TrivagoTrivago engineers · TrivagoCase Studies
  • Autoscaling Jenkins at TrivagoTrivago engineers · TrivagoCase Studies
  • Troubleshooting a Connection Timeout Issue with tcp_tw_recycle Enabled at eBayeBay engineers · eBayCase Studies

  Open this lesson with its quiz and exercise

• Software Design Essentials · 11 minBehavioural patterns: Strategy, Observer, Command, StatePatterns for how objects cooperate: swapping algorithms, reacting to events, turning actions into objects and changing behaviour with state.Read hereClose

  The goal

  Behavioural patterns distribute responsibility between objects and define how they communicate.

  Strategy

  Define a family of interchangeable algorithms behind an interface and pick one at runtime.

  type Pricing = (subtotalCents: number) => number;
  
  const regular: Pricing = (s) => s;
  const member:  Pricing = (s) => Math.round(s * 0.9);
  const blackFriday: Pricing = (s) => Math.round(s * 0.7);
  
  function checkout(subtotal: number, pricing: Pricing) { return pricing(subtotal); }
  checkout(10_000, member);   // the strategy is passed in
  

  Replaces long if/else or switch chains and supports the open/closed principle. In languages with first-class functions the "strategy" is simply a function you pass in.

  Observer (publish and subscribe)

  Objects subscribe to a subject and are notified when it changes, without the subject knowing who listens.

  class Emitter<T> {
    private handlers = new Set<(v: T) => void>();
    on(h: (v: T) => void) { this.handlers.add(h); return () => this.handlers.delete(h); } // returns "unsubscribe"
    emit(v: T) { this.handlers.forEach((h) => h(v)); }
  }
  const cart = new Emitter<number>();
  const stop = cart.on((count) => updateBadge(count));
  cart.emit(3);
  stop();                                              // always clean up
  

  Used for UI events, domain events and message buses. Watch for leaks (forgotten unsubscribes), ordering surprises and cascades of updates that are hard to follow.

  Observer as a picture

  100%

  Figure SDA-SDE-05-1 · One subject, many observers · © forcsguy.com

  Command

  Turn a request into an object containing everything needed to run it. That lets you queue, log, retry, schedule and undo it.

  interface Command { run(): void; undo(): void }
  
  class InsertText implements Command {
    constructor(private doc: Doc, private at: number, private text: string) {}
    run()  { this.doc.insert(this.at, this.text); }
    undo() { this.doc.remove(this.at, this.text.length); }
  }
  
  const history: Command[] = [];
  function perform(c: Command) { c.run(); history.push(c); }
  function undo() { history.pop()?.undo(); }
  

  Editors, job queues, macros and transaction logs all use it.

  State

  Let an object change behaviour when its internal state changes, by delegating to a state object (or a transition table).

  type State = "new" | "paid" | "shipped" | "cancelled";
  const next: Record<State, Partial<Record<"pay" | "ship" | "cancel", State>>> = {
    new:       { pay: "paid", cancel: "cancelled" },
    paid:      { ship: "shipped", cancel: "cancelled" },
    shipped:   {},
    cancelled: {},
  };
  function apply(state: State, action: "pay" | "ship" | "cancel"): State {
    const to = next[state][action];
    if (!to) throw new Error(`Cannot ${action} an order that is ${state}`);
    return to;
  }
  

  Makes illegal transitions explicit and impossible, replacing scattered flags and nested conditions.

  Template Method

  Define the skeleton of an algorithm in one place and let steps vary.

  abstract class Importer {
    run() { const raw = this.read(); const rows = this.parse(raw); this.save(rows); } // fixed order
    protected abstract read(): string;
    protected abstract parse(raw: string): Row[];
    protected save(rows: Row[]) { /* default */ }
  }
  

  Frameworks use it widely ("override these hooks"). With functions, you can pass the varying steps in instead of subclassing.

  Chain of Responsibility

  Pass a request along a chain of handlers until one handles it (or each adds something). Middleware pipelines in web servers are this pattern.

  type Handler = (req: Req, next: () => Promise<Res>) => Promise<Res>;
  // authenticate -> rateLimit -> validate -> handle
  

  Iterator and others

  • Iterator: walk a collection without exposing its structure. In most languages this is built in (for...of, generators).
  • Mediator: one object coordinates many (a chat room, a form controller) so participants do not reference each other.
  • Memento: capture and restore an object's state (undo snapshots).
  • Visitor: add new operations over a structure of object types without modifying them; common in compilers.
  • Interpreter: evaluate expressions of a small language.

  Choosing

  Problem	Pattern
  Many alternative algorithms or rules	Strategy
  React to changes without coupling	Observer
  Undo, queue, log or retry actions	Command
  Behaviour depends on lifecycle state	State
  Same steps, different details	Template Method
  A pipeline of optional processing steps	Chain of Responsibility

  Remember: with first-class functions many of these become callbacks, closures and tables. Use the lightest form that keeps the code clear.

  Real-world reading

  Related reading

  • Cape: Event Stream Processing Framework at DropboxDropbox engineers · DropboxCase Studies
  • NATS ServerNATS · CNCF / NATSGitHub
  • Samza: Stream Processing System for Latency Insighs at LinkedInLinkedIn engineers · LinkedInCase Studies
  • Autoscaling AWS Step Functions Activities at YelpYelp engineers · YelpCase Studies

  Show 6 moreShow fewer

  • Distributed Messaging, Queuing, and Event StreamingArxivPapers
  • Steady State: Always Put Logs on Separate DiskMicrosoft engineers · MicrosoftCase Studies
  • Self-Serve Reporting Platform on Hadoop at LinkedInLinkedInPapers
  • Ambry: Distributed Immutable Object Store at LinkedInLinkedIn engineers · LinkedInCase Studies
  • Benchmarking Streaming Computation Engines at YahooYahoo engineers · YahooCase Studies
  • NodeJS High Availability at YahooYahoo engineers · YahooCase Studies

  Open this lesson with its quiz and exercise

• Low-Level Design Practice · 9 minSnake gameA small real-time model: a grid, a moving snake, food and collisions, with a clean game loop that is easy to test.Read hereClose

  The problem

  A snake moves on a grid. The player steers it. Eating food makes it longer and scores a point. Hitting a wall or itself ends the game.

  Entities

  100%

  Figure SDA-LLD-13-1 · Snake classes · © forcsguy.com

  One step of the game

  100%

  Figure SDA-LLD-13-2 · One game step · © forcsguy.com

  type P = { x: number; y: number };
  const delta = { UP: { x: 0, y: -1 }, DOWN: { x: 0, y: 1 }, LEFT: { x: -1, y: 0 }, RIGHT: { x: 1, y: 0 } };
  const opposite = { UP: "DOWN", DOWN: "UP", LEFT: "RIGHT", RIGHT: "LEFT" } as const;
  
  class Game {
    body: P[] = [{ x: 5, y: 5 }];      // index 0 is the head
    dir: keyof typeof delta = "RIGHT"; food: P = { x: 8, y: 5 }; score = 0; over = false;
    constructor(readonly w = 20, readonly h = 20) {}
  
    turn(d: keyof typeof delta) { if (d !== opposite[this.dir]) this.dir = d; }
  
    step() {
      if (this.over) return;
      const head = { x: this.body[0].x + delta[this.dir].x, y: this.body[0].y + delta[this.dir].y };
      const outside = head.x < 0 || head.y < 0 || head.x >= this.w || head.y >= this.h;
      const ate = head.x === this.food.x && head.y === this.food.y;
      const hitsSelf = this.body.slice(0, ate ? undefined : -1).some(c => c.x === head.x && c.y === head.y);
      if (outside || hitsSelf) { this.over = true; return; }
      this.body.unshift(head);
      if (ate) { this.score++; this.food = this.freeCell(); } else this.body.pop();
    }
    freeCell(): P { /* pick a random cell not in body */ return { x: 0, y: 0 }; }
  }
  

  The subtle detail: when not eating, the tail moves away this step, so the head may legally enter the cell the tail is leaving. That is why the self-collision check excludes the tail unless the snake grows.

  Design notes

  • Separate model from loop and view: step() is pure game logic. An outer timer calls it; a renderer draws the state. Tests call step() directly.
  • Input buffering: store the next direction and apply it at the next step, so quick double taps do not skip.
  • Free-cell search: for a nearly full board, keep a set of free cells rather than retrying random picks.
  • Extensions: levels and speed, walls, two players, power-ups, saved high scores.

  Real-world reading

  Related reading

  • Cross-Lingual End-to-End Product Search with Deep Learning at ZalandoZalando engineers · ZalandoCase Studies
  • Moving Core Business Search to Elasticsearch at YelpYelp engineers · YelpCase Studies
  • Self-Ranking Search with Elasticsearch at WattpadWattpad engineers · WattpadCase Studies
  • Autoscaling AWS Step Functions Activities at YelpYelp engineers · YelpCase Studies

  Show 6 moreShow fewer

  • Discovering Popular Dishes with Deep Learning at YelpYelp engineers · YelpCase Studies
  • Boosting Site Speed Using Brotli Compression at LinkedInLinkedIn engineers · LinkedInCase Studies
  • The Tail at Scale (paper)Dean and Barroso · GooglePapers
  • Nautilus: Search Engine at DropboxDropbox engineers · DropboxCase Studies
  • CaffeOnSpark: Distributed Deep Learning on Big Data Clusters at YahooYahoo engineers · YahooCase Studies
  • Kafka at YelpYelp engineers · YelpCase Studies

  Open this lesson with its quiz and exercise