The Dataflow Model (paper)

Ideas you will meet

Event Driven Architecture

Loosely coupled services reacting to events.

Distributed Systems

Consensus, time, failure and the fundamentals of scale-out.

What we already know about this

Our own lessons on these ideas. Read them here before you start, so it makes sense.

• Software Design Essentials · 12 minDomain-driven design essentialsA shared language, bounded contexts, entities, value objects, aggregates and domain events: modelling complex business software.Read hereClose

  What DDD is for

  Domain-driven design (DDD), introduced by Eric Evans, is an approach for software where the business domain is complex. Its core claim: the hardest part is not technology but understanding and modelling the business, so code should reflect the domain, in the domain's own language, developed with domain experts.

  DDD has a strategic side (how to divide a big system) and a tactical side (how to model inside a part).

  Strategic design

  Ubiquitous language

  Developers and domain experts agree on one shared vocabulary, and use it everywhere: conversations, documents and code. If the business says "policy lapse", the code has Policy.lapse(), not status = 3. Misunderstandings shrink because there is no translation layer in people's heads.

  Bounded contexts

  One model cannot serve a whole large organisation: words mean different things in different places. A bounded context is a boundary inside which a model and its language are consistent.

  100%

  Figure SDA-SDE-07-1 · Bounded contexts · © forcsguy.com

  Each context has its own model, and often its own team, code and database. Contexts relate through explicit patterns: shared kernel, customer-supplier, conformist, and the anti-corruption layer, a translator that protects your model from a messy external one. Contexts are natural candidates for microservices boundaries.

  Subdomains

  Break the business into subdomains and treat them differently:

  • Core: what makes the business different. Invest your best people and custom design.
  • Supporting: needed but not distinctive. Build simply.
  • Generic: solved problems (authentication, payments). Buy or reuse.

  Tactical design: building blocks

  Entities

  Objects defined by identity, which persists as attributes change: a Customer, an Order. Two customers with the same name are still different.

  Value objects

  Objects defined only by their values, immutable, and interchangeable: Money(10, "EUR"), Address, DateRange. They carry their own rules (money cannot add different currencies) and remove "primitive obsession".

  class Money {
    private constructor(readonly cents: number, readonly currency: string) {}
    static of(cents: number, currency: string) { if (cents < 0) throw new Error("negative"); return new Money(cents, currency); }
    add(o: Money) { if (o.currency !== this.currency) throw new Error("currency mismatch"); return new Money(this.cents + o.cents, this.currency); }
  }
  

  Aggregates

  A cluster of entities and value objects treated as one unit for consistency, with a single aggregate root through which everything is accessed. The root enforces invariants ("an order total equals the sum of its lines").

  class Order {                                     // aggregate root
    private lines: OrderLine[] = [];
    addLine(product: ProductId, qty: number, price: Money) {
      if (this.status !== "draft") throw new Error("Order is locked");   // invariant
      this.lines.push(new OrderLine(product, qty, price));
    }
  }
  

  Rules of thumb: keep aggregates small, change one aggregate per transaction, and refer to other aggregates by id, not by object reference.

  Domain events

  Facts that happened in the domain, named in the past tense: OrderPlaced, PaymentReceived. They let other parts react without tight coupling and carry changes between aggregates and contexts, usually with the outbox pattern for reliable delivery.

  Repositories and services

  • A repository loads and saves aggregates, hiding persistence (a port in hexagonal architecture).
  • A domain service holds business logic that does not belong to one entity (for example a pricing calculation across several aggregates).
  • An application service (use case) orchestrates: load, call domain methods, save, publish events.

  Factories

  Create complex aggregates in a valid state.

  How DDD fits with the rest

  • The domain model sits at the centre of clean or hexagonal architecture, free of framework code.
  • Event-driven designs and CQRS often pair with DDD.
  • Bounded contexts guide microservice boundaries.

  When to use it

  DDD pays off where the domain is complicated and changes: insurance, logistics, finance, healthcare. For simple CRUD applications it is overkill; a plain data model is fine.

  Starting small

  1. Talk to domain experts and write down their words as a glossary.
  2. Draw the contexts and how they exchange information (a context map).
  3. Model one core subdomain deeply, with aggregates and invariants as tests.
  4. Use event storming (a workshop mapping events on sticky notes) to discover the domain together.
  5. Keep refining the model as understanding improves.

  Real-world reading

  Related reading

  • Domain-Driven Design ReferenceEric Evans · Domain LanguageBooks
  • Moving Core Business Search to Elasticsearch at YelpYelp engineers · YelpCase Studies
  • Tech Dummies: System Design Case StudiesNarendra L · Tech DummiesYouTube
  • Fast Order Search using Data Pipeline and Elasticsearch at YelpYelp engineers · YelpCase Studies

  Show 6 moreShow fewer

  • Event SourcingMartin FowlerBlogs
  • The Dataflow Model (paper)Akidau et al. · GooglePapers
  • Service Architecture (3 parts: Domain Gateways, Value-Added Services, BFF) at SoundCloudSoundCloud engineers · SoundCloudCase Studies
  • Pattern: Transactional outboxChris Richardson · microservices.ioDocumentation
  • Tech Stack at ShopifyShopify engineers · ShopifyCase Studies
  • Introduction to gRPCgRPC · CNCF / gRPCDocumentation

  Open this lesson with its quiz and exercise

• Low-Level Design Practice · 11 minChess gameBoard, pieces and rules: polymorphic pieces that generate moves, validating turns and check, and keeping the board model separate from the UI.Read hereClose

  The problem

  Two players take turns on an 8 by 8 board. The system validates moves, detects check, checkmate and stalemate, and records the game.

  Entities

  100%

  Figure SDA-LLD-12-1 · Chess classes · © forcsguy.com

  Inheritance is justified here: each piece type is a real kind of piece with its own movement rule.

  Pieces know their own movement

  abstract class Piece {
    constructor(readonly color: "W" | "B") {}
    abstract legalMoves(board: Board, from: Pos): Pos[];     // ignoring check
  }
  
  class Rook extends Piece {
    legalMoves(board: Board, from: Pos): Pos[] {
      return slide(board, from, [[1,0],[-1,0],[0,1],[0,-1]], this.color);   // along rank and file until blocked
    }
  }
  class Knight extends Piece {
    legalMoves(board: Board, from: Pos): Pos[] {
      return [[1,2],[2,1],[-1,2],[-2,1],[1,-2],[2,-1],[-1,-2],[-2,-1]]
        .map(([dx, dy]) => ({ x: from.x + dx, y: from.y + dy }))
        .filter((p) => board.inside(p) && board.pieceAt(p)?.color !== this.color);
    }
  }
  

  A helper slide is shared by rook, bishop and queen so the logic is not repeated.

  Validating a move

  100%

  Figure SDA-LLD-12-2 · Validating a move · © forcsguy.com

  Game status: check if the opponent's king is attacked; checkmate if in check with no legal move; stalemate if not in check but no legal move.

  Moves as objects

  class Move {
    constructor(readonly from: Pos, readonly to: Pos, readonly captured?: Piece, readonly kind: "normal" | "castle" | "enPassant" | "promotion" = "normal") {}
  }
  class History { done: Move[] = []; undone: Move[] = [];
    play(m: Move, b: Board) { b.apply(m); this.done.push(m); this.undone = []; }
    undo(b: Board) { const m = this.done.pop(); if (m) { b.revert(m); this.undone.push(m); } }
  }
  

  A move object makes undo, redo, history and notation straightforward: this is the Command pattern.

  Design notes

  • Special rules: castling (neither piece has moved, path clear and not attacked), en passant (needs the previous move), promotion (choose a piece). Each needs a little extra state: hasMoved on pieces, last move on the game.
  • Model and view stay separate; an AI or network player implements the same Player interface.
  • Clocks are a separate component observing the turn.
  • Extensions: a move-suggestion engine, saved games, variants with different boards, online play with a server validating every move (never trust the client).

  Real-world reading

  Related reading

  • Operating Systems: Three Easy PiecesRemzi and Andrea Arpaci-Dusseau · University of WisconsinBooks
  • Ambry: Distributed Immutable Object Store at LinkedInLinkedIn engineers · LinkedInCase Studies
  • Cloud Object Store at YahooYahoo engineers · YahooCase Studies
  • Netflix: What Happens When You Press Play?High Scalability engineers · High ScalabilityBlogs

  Show 6 moreShow fewer

  • 8 Things You Need to Know Before a System Design InterviewGainlo engineers · GainloBlogs
  • The Dataflow Model (paper)Akidau et al. · GooglePapers
  • Atomic Object Engineering BlogAtomic Object engineers · Atomic ObjectBlogs
  • Cloud Bouncer: Distributed Rate Limiting at YahooYahoo engineers · YahooCase Studies
  • Object StorageDatacenterknowledge engineers · DatacenterknowledgeCase Studies
  • Remind Engineering BlogRemind engineers · RemindBlogs

  Open this lesson with its quiz and exercise