Cape: Event Stream Processing Framework at Dropbox

Ideas you will meet

Event Driven Architecture

Loosely coupled services reacting to events.

Distributed Systems

Consensus, time, failure and the fundamentals of scale-out.

What we already know about this

Our own lessons on these ideas. Read them here before you start, so it makes sense.

• Real-World Systems · 13 minDesign a stock exchangeAn order book, a matching engine and a sequencer: how an exchange matches buyers and sellers fairly in microseconds.Read hereClose

  The problem

  An exchange lets buyers and sellers trade. Participants submit orders; the exchange matches them and reports the trades.

  Key properties:

  • Fairness: orders are handled in a strict, auditable order.
  • Very low latency: microseconds from order to acknowledgment.
  • High throughput: hundreds of thousands to millions of orders per second at peaks.
  • Correctness and durability: money is involved. No lost or duplicated trades.
  • Availability: an exchange outage halts a market.

  The order book

  For each symbol the order book holds resting orders:

  • Bids (buy orders), highest price first.
  • Asks (sell orders), lowest price first.
  • Within a price level, orders queue by arrival time.

  Common order types: limit (a worst price you accept; unmatched part rests in the book) and market (take the best available now).

  Matching: price-time priority

  An incoming buy matches the lowest-priced asks it can afford; among equal prices the earliest order matches first. Matching repeats until the order is filled or no longer matches, and any remainder rests in the book.

  Image: Thomas J. O'Halloran · Public domain · Source · Wikimedia Commons: NY stock exchange traders floor LC-U9-10548-6

  Architecture

  100%

  Figure SDA-RWS-05-1 · Exchange: gateway, sequencer, matching engine · © forcsguy.com

  1. Gateways authenticate participants, validate orders, and run pre-trade risk checks (limits, available funds).
  2. The sequencer gives every event a sequence number: one agreed total order, the basis of fairness and recovery.
  3. A matching engine applies events in order to the in-memory order book and emits trades.
  4. Everything is written to a durable event log. Executions go to participants, and a market data stream tells everyone about prices and depth.
  5. Clearing and settlement later move money and securities between accounts.

  100%

  Figure SDA-RWS-05-2 · An order book with a buy order sweeping the sell side · © forcsguy.com

  How it gets so fast

  • In-memory order books and no database in the critical path.
  • Single-threaded matching per symbol group: no locks, deterministic behaviour. Parallelism comes from partitioning symbols across engines.
  • Event sourcing: state is the result of applying an ordered log. Persist the event first (to a memory-mapped file or replicated log), then process.
  • Pre-allocated memory, ring buffers, no garbage-collection pauses, and careful cache-friendly data structures.
  • Hardware and network: servers co-located in the exchange's data centre, fast network cards and kernel-bypass networking, precise time sources.

  100%

  Figure SDA-RWS-05-3 · A fast engine on one machine: shared log and busy loops · © forcsguy.com

  Reliability

  • A hot standby applies the same event log and can take over within moments. Use sequence numbers to detect gaps and avoid duplicate processing.
  • Replicate the log across machines or sites before acknowledging, trading a little latency for durability.
  • Idempotent order identifiers prevent a resubmitted order executing twice.
  • Circuit breakers halt trading when prices move too fast.

  Market data fairness

  Everyone should see the same information at about the same time. Exchanges publish through multicast feeds, give all participants equal-length cables inside the data centre, and timestamp events at the matching engine. A separate, slower snapshot and retransmission service helps receivers that miss packets.

  What is out of scope here

  Clearing houses, settlement cycles, regulation and surveillance are large systems of their own. The core design is the sequencer plus deterministic matching engine plus durable log.

  Real-world reading

  Related reading

  • Event SourcingMartin FowlerBlogs
  • Monolith FirstMartin FowlerBlogs
  • Rewriting the heart of our sync engineDropboxCase Studies
  • Cape: Event Stream Processing Framework at DropboxDropbox engineers · DropboxCase Studies

  Show 6 moreShow fewer

  • Improving Search Engine Efficiency by over 25% at eBayeBay engineers · eBayCase Studies
  • Finding Kafka throughput limit in infrastructure at DropboxDropbox engineers · DropboxCase Studies
  • Bullet: Forward-Looking Query Engine for Streaming Data at YahooYahoo engineers · YahooCase Studies
  • MicroservicesJames Lewis and Martin Fowler · Martin FowlerBlogs
  • The LMAX ArchitectureMartin FowlerCase Studies
  • Microservices and OrchestrationMartinfowler engineers · MartinfowlerCase Studies

  Open this lesson with its quiz and exercise

• Big Data and Cloud Data Platforms · 12 minBatch and stream processingMapReduce and Spark for large batch jobs, and streaming concepts: event time, windows, watermarks and exactly-once results.Read hereClose

  Two ways to process data

  • Batch: process a bounded dataset (yesterday's logs) as a job that starts, runs and finishes. High throughput, results later.
  • Stream: process an unbounded flow of events continuously as they arrive. Low latency, harder to get right.

  Most platforms need both: stream for fresh insight, batch for heavy recomputation and history.

  Before: batch: a big job on a schedule

  100%

  After: stream: process each event

  100%

  What changed

  • Results arrive in seconds instead of hours
  • Late and duplicate events must be handled
  • Batch stays useful for reprocessing history

  Batch: the MapReduce idea

  Word count is the classic example.

  100%

  Figure SDA-BDP-02-1 · MapReduce word count flow · © forcsguy.com

  1. Map: each worker reads a slice and emits key-value pairs (("system", 1)).
  2. Shuffle: the framework groups all pairs with the same key and sends them to the same reducer. This is the expensive step (network and sorting).
  3. Reduce: each reducer combines the values for its keys (("system", 4812)).

  Failures are handled by re-running tasks, which is possible because inputs are immutable and tasks have no side effects.

  Spark improved this model: it keeps intermediate data in memory, expresses jobs as chains of transformations on DataFrames (with a query optimiser), and offers SQL, machine learning and streaming APIs on one engine. Today you usually write Spark or SQL, not raw MapReduce.

  Image: Unknown authorUnknown author · Public domain · Source · Wikimedia Commons: Ford assembly line - 1913

  Stream processing

  A stream processor reads events from a log (such as Kafka), transforms them, and writes results to another log, a database or a dashboard.

  Common operations: filter and enrich, aggregate, join streams, and detect patterns.

  Time is the hard part

  • Processing time: when the system sees the event.
  • Event time: when it actually happened.

  Phones go offline, networks lag, and events arrive late and out of order. Computing "clicks per minute" by arrival time gives wrong answers. Use event time.

  Windows

  Aggregations need boundaries:

  Window	Meaning
  Tumbling	Fixed, non-overlapping (every minute)
  Sliding	Overlapping (last 5 minutes, updated each minute)
  Session	Grouped by activity, closed by a gap of inactivity

  Watermarks and late data

  A watermark is the system's estimate of "we have seen everything up to event time T". When it passes the end of a window, the window can close and emit a result. Choosing the delay trades latency against completeness. Events arriving after the watermark are late: drop them, send them to a side output, or update the earlier result.

  State and failures

  Windows and joins need state. Engines such as Flink keep state locally and checkpoint it periodically to durable storage, together with the position in the input log. After a failure they restore the checkpoint and replay from that position.

  Delivery guarantees

  • At-most-once: may lose events.
  • At-least-once: may duplicate on replay.
  • Exactly-once effect: achieved by replayable sources, checkpointed state and idempotent or transactional sinks, so a replay does not change results.

  Architectures that combine both

  • Lambda architecture: a batch layer computes accurate views from all data, a speed layer gives fast approximations; the two are merged. Powerful but means writing logic twice.
  • Kappa architecture: treat everything as a stream. Keep the log long enough to replay history through the same code when logic changes. Simpler, if your log and engine can handle it.
  • Modern engines and the dataflow model increasingly unify batch and streaming under one API.

  100%

  Figure SDA-BDP-02-2 · Lambda and Kappa architectures · © forcsguy.com

  Choosing

  Need	Choice
  Nightly reports, big joins, model training on history	Batch (Spark, SQL warehouse)
  Fraud detection, live dashboards, alerting, real-time features	Streaming (Flink, Kafka Streams, Spark Structured Streaming)
  Both with minimal duplication	Streaming core with replayable log, or a unified engine

  Real-world reading

  Related reading

  • MapReduce: Simplified Data Processing on Large Clusters (paper)Dean and Ghemawat · GooglePapers
  • Cape: Event Stream Processing Framework at DropboxDropbox engineers · DropboxCase Studies
  • Pagelets Parallelize Server-side Processing at YelpYelp engineers · YelpCase Studies
  • Optimizing CAL Report Hadoop MapReduce Jobs at eBayeBay engineers · eBayCase Studies

  Show 6 moreShow fewer

  • Bullet: Forward-Looking Query Engine for Streaming Data at YahooYahoo engineers · YahooCase Studies
  • Samza: Stream Processing System for Latency Insighs at LinkedInLinkedIn engineers · LinkedInCase Studies
  • Netflix: What Happens When You Press Play?High Scalability engineers · High ScalabilityBlogs
  • CMU Database GroupCarnegie Mellon UniversityYouTube
  • Omid: Transaction Processing Platform at YahooYahoo engineers · YahooCase Studies
  • Big Data Processing at UberUberPapers

  Open this lesson with its quiz and exercise

• Software Design Essentials · 11 minBehavioural patterns: Strategy, Observer, Command, StatePatterns for how objects cooperate: swapping algorithms, reacting to events, turning actions into objects and changing behaviour with state.Read hereClose

  The goal

  Behavioural patterns distribute responsibility between objects and define how they communicate.

  Strategy

  Define a family of interchangeable algorithms behind an interface and pick one at runtime.

  type Pricing = (subtotalCents: number) => number;
  
  const regular: Pricing = (s) => s;
  const member:  Pricing = (s) => Math.round(s * 0.9);
  const blackFriday: Pricing = (s) => Math.round(s * 0.7);
  
  function checkout(subtotal: number, pricing: Pricing) { return pricing(subtotal); }
  checkout(10_000, member);   // the strategy is passed in
  

  Replaces long if/else or switch chains and supports the open/closed principle. In languages with first-class functions the "strategy" is simply a function you pass in.

  Observer (publish and subscribe)

  Objects subscribe to a subject and are notified when it changes, without the subject knowing who listens.

  class Emitter<T> {
    private handlers = new Set<(v: T) => void>();
    on(h: (v: T) => void) { this.handlers.add(h); return () => this.handlers.delete(h); } // returns "unsubscribe"
    emit(v: T) { this.handlers.forEach((h) => h(v)); }
  }
  const cart = new Emitter<number>();
  const stop = cart.on((count) => updateBadge(count));
  cart.emit(3);
  stop();                                              // always clean up
  

  Used for UI events, domain events and message buses. Watch for leaks (forgotten unsubscribes), ordering surprises and cascades of updates that are hard to follow.

  Observer as a picture

  100%

  Figure SDA-SDE-05-1 · One subject, many observers · © forcsguy.com

  Command

  Turn a request into an object containing everything needed to run it. That lets you queue, log, retry, schedule and undo it.

  interface Command { run(): void; undo(): void }
  
  class InsertText implements Command {
    constructor(private doc: Doc, private at: number, private text: string) {}
    run()  { this.doc.insert(this.at, this.text); }
    undo() { this.doc.remove(this.at, this.text.length); }
  }
  
  const history: Command[] = [];
  function perform(c: Command) { c.run(); history.push(c); }
  function undo() { history.pop()?.undo(); }
  

  Editors, job queues, macros and transaction logs all use it.

  State

  Let an object change behaviour when its internal state changes, by delegating to a state object (or a transition table).

  type State = "new" | "paid" | "shipped" | "cancelled";
  const next: Record<State, Partial<Record<"pay" | "ship" | "cancel", State>>> = {
    new:       { pay: "paid", cancel: "cancelled" },
    paid:      { ship: "shipped", cancel: "cancelled" },
    shipped:   {},
    cancelled: {},
  };
  function apply(state: State, action: "pay" | "ship" | "cancel"): State {
    const to = next[state][action];
    if (!to) throw new Error(`Cannot ${action} an order that is ${state}`);
    return to;
  }
  

  Makes illegal transitions explicit and impossible, replacing scattered flags and nested conditions.

  Template Method

  Define the skeleton of an algorithm in one place and let steps vary.

  abstract class Importer {
    run() { const raw = this.read(); const rows = this.parse(raw); this.save(rows); } // fixed order
    protected abstract read(): string;
    protected abstract parse(raw: string): Row[];
    protected save(rows: Row[]) { /* default */ }
  }
  

  Frameworks use it widely ("override these hooks"). With functions, you can pass the varying steps in instead of subclassing.

  Chain of Responsibility

  Pass a request along a chain of handlers until one handles it (or each adds something). Middleware pipelines in web servers are this pattern.

  type Handler = (req: Req, next: () => Promise<Res>) => Promise<Res>;
  // authenticate -> rateLimit -> validate -> handle
  

  Iterator and others

  • Iterator: walk a collection without exposing its structure. In most languages this is built in (for...of, generators).
  • Mediator: one object coordinates many (a chat room, a form controller) so participants do not reference each other.
  • Memento: capture and restore an object's state (undo snapshots).
  • Visitor: add new operations over a structure of object types without modifying them; common in compilers.
  • Interpreter: evaluate expressions of a small language.

  Choosing

  Problem	Pattern
  Many alternative algorithms or rules	Strategy
  React to changes without coupling	Observer
  Undo, queue, log or retry actions	Command
  Behaviour depends on lifecycle state	State
  Same steps, different details	Template Method
  A pipeline of optional processing steps	Chain of Responsibility

  Remember: with first-class functions many of these become callbacks, closures and tables. Use the lightest form that keeps the code clear.

  Real-world reading

  Related reading

  • Cape: Event Stream Processing Framework at DropboxDropbox engineers · DropboxCase Studies
  • NATS ServerNATS · CNCF / NATSGitHub
  • Samza: Stream Processing System for Latency Insighs at LinkedInLinkedIn engineers · LinkedInCase Studies
  • Autoscaling AWS Step Functions Activities at YelpYelp engineers · YelpCase Studies

  Show 6 moreShow fewer

  • Distributed Messaging, Queuing, and Event StreamingArxivPapers
  • Steady State: Always Put Logs on Separate DiskMicrosoft engineers · MicrosoftCase Studies
  • Self-Serve Reporting Platform on Hadoop at LinkedInLinkedInPapers
  • Ambry: Distributed Immutable Object Store at LinkedInLinkedIn engineers · LinkedInCase Studies
  • Benchmarking Streaming Computation Engines at YahooYahoo engineers · YahooCase Studies
  • NodeJS High Availability at YahooYahoo engineers · YahooCase Studies

  Open this lesson with its quiz and exercise