Samza: Stream Processing System for Latency Insighs at LinkedIn

Ideas you will meet

Event Driven Architecture

Loosely coupled services reacting to events.

Distributed Systems

Consensus, time, failure and the fundamentals of scale-out.

What we already know about this

Our own lessons on these ideas. Read them here before you start, so it makes sense.

• Real-World Systems · 11 minDesign an AI coding assistantHow an editor assistant suggests code in under a second: gathering context, retrieval, the model, streaming and keeping private code safe.Read hereClose

  The problem

  An assistant inside the editor that completes code as you type, answers questions in a chat panel, and can edit files. The hard constraints:

  • Latency: inline suggestions must appear in roughly a few hundred milliseconds, or the user has already moved on.
  • Relevance: good suggestions need the right context from a large codebase, not just the current line.
  • Cost: language-model inference is expensive per request, and typing generates a request storm.
  • Privacy and security: customer source code and secrets are sensitive.
  • Quality: wrong or insecure suggestions must be limited and measurable.

  Architecture

  100%

  Figure SDA-RWS-18-1 · AI coding assistant architecture · © forcsguy.com

  1. The editor extension

  Runs in the user's editor. It watches typing, debounces requests (waits a few tens of milliseconds), cancels outdated requests when the user continues typing, and streams the suggestion into the editor as ghost text. It also handles accept, reject and partial accept, and enforces local privacy rules (excluded files, secret detection).

  2. Building context

  Prompt = instructions + context + the code around the cursor. Context comes from:

  • The current file: the code before the cursor (prefix) and after it (suffix, for fill-in-the-middle).
  • Open tabs and recently edited files.
  • Imports and related files: definitions of functions and types being used.
  • Retrieved snippets from the wider project.

  All of it must fit the model's context window and a token budget, so the builder ranks pieces by relevance and trims.

  3. Retrieval over a codebase

  A repository is too big to include whole, so it is indexed:

  1. Split files into chunks (functions, classes, blocks).
  2. Compute embeddings (vectors capturing meaning) and store them in a vector index, together with a symbol and dependency graph (what calls what).
  3. At request time, search with the current code as the query and combine semantic results with structural ones (definitions, callers, tests).
  4. Update the index incrementally as files change, and keep per-user and per-organisation isolation.

  4. Model serving

  • A language model generates tokens. Serving systems batch many requests together, reuse a KV cache of already-processed prompt tokens, and stream output so the first words show immediately.
  • Use smaller, faster models for inline completion and larger ones for chat and complex edits.
  • Cache repeated prompts and completions where context is identical, and prefix-cache shared prompt parts.
  • Autoscale GPU capacity, and route by region for latency.

  5. Post-processing and safety

  • Trim the output to a sensible stopping point (end of block or line).
  • Filter secrets, personal data and known insecure patterns; optionally check for verbatim public code matches and licence concerns.
  • Reject low-confidence suggestions instead of showing noise.
  • For agents that edit files or run commands, require explicit user approval and run actions in a sandbox.

  100%

  Figure SDA-RWS-18-2 · How a code model is trained and judged · © forcsguy.com

  Latency budget

  For inline suggestions, a budget like this keeps the experience smooth:

  Step	Target
  Debounce and local context	about 30 ms
  Network to the service	about 30 ms
  Retrieval and prompt building	about 30 ms
  Model: time to first token	about 150 ms
  Streaming and display	continuous

  Everything else (retrieval, indexing, safety checks) must fit inside it or run asynchronously ahead of time.

  Privacy, security and trust

  • Send only what is needed, respect ignore files, and redact secrets.
  • Provide enterprise controls: private deployment or isolated tenancy, no training on customer code, short retention, audit logs.
  • Encrypt in transit and at rest; strong authentication and per-user quotas.
  • Defend against prompt injection (malicious text in files or web pages trying to instruct the model): treat retrieved content as data, limit tool permissions, require confirmation for risky actions.

  Measuring quality

  • Offline evaluation: run the model on held-out code and tests, measure pass rates and exact-match similarity.
  • Online metrics: acceptance rate of suggestions, characters kept after acceptance, latency percentiles, and errors.
  • A/B test changes to prompts, models and retrieval, and watch for regressions in security and licence filters.
  • Collect feedback signals (thumbs up or down) while respecting privacy.

  What this design reuses

  Debouncing and cancellation, caching, rate limiting and quotas, queues and batching, vector search, streaming responses and careful privacy controls: the same building blocks from the rest of this course, applied to a latency-sensitive AI workload.

  Real-world reading

  Related reading

  • Secret Detector: Preventing Secrets in Source Code at YelpYelp engineers · YelpCase Studies
  • Samza: Stream Processing System for Latency Insighs at LinkedInLinkedIn engineers · LinkedInCase Studies
  • Use The Index, LukeMarkus Winand · use-the-index-luke.comCourses
  • Backend Developer Roadmaproadmap.sh community · roadmap.shCourses

  Show 6 moreShow fewer

  • Natural Language Processing and Content Analysis (2 parts) at Condé NastCondé Nast engineers · Condé NastCase Studies
  • Machine Learning for Indexing Text from Billions of Images at DropboxDropbox engineers · DropboxCase Studies
  • Optimizing Web Servers for High Throughput and Low Latency at DropboxDropbox engineers · DropboxCase Studies
  • Scalable User Privacy at SpotifySpotify engineers · SpotifyCase Studies
  • Privacy-Preserving Analytics and Reporting at LinkedInLinkedIn engineers · LinkedInCase Studies
  • Learning with Privacy at Scale at AppleScale at Apple engineers · Scale at AppleCase Studies

  Open this lesson with its quiz and exercise

• Big Data and Cloud Data Platforms · 12 minBatch and stream processingMapReduce and Spark for large batch jobs, and streaming concepts: event time, windows, watermarks and exactly-once results.Read hereClose

  Two ways to process data

  • Batch: process a bounded dataset (yesterday's logs) as a job that starts, runs and finishes. High throughput, results later.
  • Stream: process an unbounded flow of events continuously as they arrive. Low latency, harder to get right.

  Most platforms need both: stream for fresh insight, batch for heavy recomputation and history.

  Before: batch: a big job on a schedule

  100%

  After: stream: process each event

  100%

  What changed

  • Results arrive in seconds instead of hours
  • Late and duplicate events must be handled
  • Batch stays useful for reprocessing history

  Batch: the MapReduce idea

  Word count is the classic example.

  100%

  Figure SDA-BDP-02-1 · MapReduce word count flow · © forcsguy.com

  1. Map: each worker reads a slice and emits key-value pairs (("system", 1)).
  2. Shuffle: the framework groups all pairs with the same key and sends them to the same reducer. This is the expensive step (network and sorting).
  3. Reduce: each reducer combines the values for its keys (("system", 4812)).

  Failures are handled by re-running tasks, which is possible because inputs are immutable and tasks have no side effects.

  Spark improved this model: it keeps intermediate data in memory, expresses jobs as chains of transformations on DataFrames (with a query optimiser), and offers SQL, machine learning and streaming APIs on one engine. Today you usually write Spark or SQL, not raw MapReduce.

  Image: Unknown authorUnknown author · Public domain · Source · Wikimedia Commons: Ford assembly line - 1913

  Stream processing

  A stream processor reads events from a log (such as Kafka), transforms them, and writes results to another log, a database or a dashboard.

  Common operations: filter and enrich, aggregate, join streams, and detect patterns.

  Time is the hard part

  • Processing time: when the system sees the event.
  • Event time: when it actually happened.

  Phones go offline, networks lag, and events arrive late and out of order. Computing "clicks per minute" by arrival time gives wrong answers. Use event time.

  Windows

  Aggregations need boundaries:

  Window	Meaning
  Tumbling	Fixed, non-overlapping (every minute)
  Sliding	Overlapping (last 5 minutes, updated each minute)
  Session	Grouped by activity, closed by a gap of inactivity

  Watermarks and late data

  A watermark is the system's estimate of "we have seen everything up to event time T". When it passes the end of a window, the window can close and emit a result. Choosing the delay trades latency against completeness. Events arriving after the watermark are late: drop them, send them to a side output, or update the earlier result.

  State and failures

  Windows and joins need state. Engines such as Flink keep state locally and checkpoint it periodically to durable storage, together with the position in the input log. After a failure they restore the checkpoint and replay from that position.

  Delivery guarantees

  • At-most-once: may lose events.
  • At-least-once: may duplicate on replay.
  • Exactly-once effect: achieved by replayable sources, checkpointed state and idempotent or transactional sinks, so a replay does not change results.

  Architectures that combine both

  • Lambda architecture: a batch layer computes accurate views from all data, a speed layer gives fast approximations; the two are merged. Powerful but means writing logic twice.
  • Kappa architecture: treat everything as a stream. Keep the log long enough to replay history through the same code when logic changes. Simpler, if your log and engine can handle it.
  • Modern engines and the dataflow model increasingly unify batch and streaming under one API.

  100%

  Figure SDA-BDP-02-2 · Lambda and Kappa architectures · © forcsguy.com

  Choosing

  Need	Choice
  Nightly reports, big joins, model training on history	Batch (Spark, SQL warehouse)
  Fraud detection, live dashboards, alerting, real-time features	Streaming (Flink, Kafka Streams, Spark Structured Streaming)
  Both with minimal duplication	Streaming core with replayable log, or a unified engine

  Real-world reading

  Related reading

  • MapReduce: Simplified Data Processing on Large Clusters (paper)Dean and Ghemawat · GooglePapers
  • Cape: Event Stream Processing Framework at DropboxDropbox engineers · DropboxCase Studies
  • Pagelets Parallelize Server-side Processing at YelpYelp engineers · YelpCase Studies
  • Optimizing CAL Report Hadoop MapReduce Jobs at eBayeBay engineers · eBayCase Studies

  Show 6 moreShow fewer

  • Bullet: Forward-Looking Query Engine for Streaming Data at YahooYahoo engineers · YahooCase Studies
  • Samza: Stream Processing System for Latency Insighs at LinkedInLinkedIn engineers · LinkedInCase Studies
  • Netflix: What Happens When You Press Play?High Scalability engineers · High ScalabilityBlogs
  • CMU Database GroupCarnegie Mellon UniversityYouTube
  • Omid: Transaction Processing Platform at YahooYahoo engineers · YahooCase Studies
  • Big Data Processing at UberUberPapers

  Open this lesson with its quiz and exercise

• Software Design Essentials · 11 minBehavioural patterns: Strategy, Observer, Command, StatePatterns for how objects cooperate: swapping algorithms, reacting to events, turning actions into objects and changing behaviour with state.Read hereClose

  The goal

  Behavioural patterns distribute responsibility between objects and define how they communicate.

  Strategy

  Define a family of interchangeable algorithms behind an interface and pick one at runtime.

  type Pricing = (subtotalCents: number) => number;
  
  const regular: Pricing = (s) => s;
  const member:  Pricing = (s) => Math.round(s * 0.9);
  const blackFriday: Pricing = (s) => Math.round(s * 0.7);
  
  function checkout(subtotal: number, pricing: Pricing) { return pricing(subtotal); }
  checkout(10_000, member);   // the strategy is passed in
  

  Replaces long if/else or switch chains and supports the open/closed principle. In languages with first-class functions the "strategy" is simply a function you pass in.

  Observer (publish and subscribe)

  Objects subscribe to a subject and are notified when it changes, without the subject knowing who listens.

  class Emitter<T> {
    private handlers = new Set<(v: T) => void>();
    on(h: (v: T) => void) { this.handlers.add(h); return () => this.handlers.delete(h); } // returns "unsubscribe"
    emit(v: T) { this.handlers.forEach((h) => h(v)); }
  }
  const cart = new Emitter<number>();
  const stop = cart.on((count) => updateBadge(count));
  cart.emit(3);
  stop();                                              // always clean up
  

  Used for UI events, domain events and message buses. Watch for leaks (forgotten unsubscribes), ordering surprises and cascades of updates that are hard to follow.

  Observer as a picture

  100%

  Figure SDA-SDE-05-1 · One subject, many observers · © forcsguy.com

  Command

  Turn a request into an object containing everything needed to run it. That lets you queue, log, retry, schedule and undo it.

  interface Command { run(): void; undo(): void }
  
  class InsertText implements Command {
    constructor(private doc: Doc, private at: number, private text: string) {}
    run()  { this.doc.insert(this.at, this.text); }
    undo() { this.doc.remove(this.at, this.text.length); }
  }
  
  const history: Command[] = [];
  function perform(c: Command) { c.run(); history.push(c); }
  function undo() { history.pop()?.undo(); }
  

  Editors, job queues, macros and transaction logs all use it.

  State

  Let an object change behaviour when its internal state changes, by delegating to a state object (or a transition table).

  type State = "new" | "paid" | "shipped" | "cancelled";
  const next: Record<State, Partial<Record<"pay" | "ship" | "cancel", State>>> = {
    new:       { pay: "paid", cancel: "cancelled" },
    paid:      { ship: "shipped", cancel: "cancelled" },
    shipped:   {},
    cancelled: {},
  };
  function apply(state: State, action: "pay" | "ship" | "cancel"): State {
    const to = next[state][action];
    if (!to) throw new Error(`Cannot ${action} an order that is ${state}`);
    return to;
  }
  

  Makes illegal transitions explicit and impossible, replacing scattered flags and nested conditions.

  Template Method

  Define the skeleton of an algorithm in one place and let steps vary.

  abstract class Importer {
    run() { const raw = this.read(); const rows = this.parse(raw); this.save(rows); } // fixed order
    protected abstract read(): string;
    protected abstract parse(raw: string): Row[];
    protected save(rows: Row[]) { /* default */ }
  }
  

  Frameworks use it widely ("override these hooks"). With functions, you can pass the varying steps in instead of subclassing.

  Chain of Responsibility

  Pass a request along a chain of handlers until one handles it (or each adds something). Middleware pipelines in web servers are this pattern.

  type Handler = (req: Req, next: () => Promise<Res>) => Promise<Res>;
  // authenticate -> rateLimit -> validate -> handle
  

  Iterator and others

  • Iterator: walk a collection without exposing its structure. In most languages this is built in (for...of, generators).
  • Mediator: one object coordinates many (a chat room, a form controller) so participants do not reference each other.
  • Memento: capture and restore an object's state (undo snapshots).
  • Visitor: add new operations over a structure of object types without modifying them; common in compilers.
  • Interpreter: evaluate expressions of a small language.

  Choosing

  Problem	Pattern
  Many alternative algorithms or rules	Strategy
  React to changes without coupling	Observer
  Undo, queue, log or retry actions	Command
  Behaviour depends on lifecycle state	State
  Same steps, different details	Template Method
  A pipeline of optional processing steps	Chain of Responsibility

  Remember: with first-class functions many of these become callbacks, closures and tables. Use the lightest form that keeps the code clear.

  Real-world reading

  Related reading

  • Cape: Event Stream Processing Framework at DropboxDropbox engineers · DropboxCase Studies
  • NATS ServerNATS · CNCF / NATSGitHub
  • Samza: Stream Processing System for Latency Insighs at LinkedInLinkedIn engineers · LinkedInCase Studies
  • Autoscaling AWS Step Functions Activities at YelpYelp engineers · YelpCase Studies

  Show 6 moreShow fewer

  • Distributed Messaging, Queuing, and Event StreamingArxivPapers
  • Steady State: Always Put Logs on Separate DiskMicrosoft engineers · MicrosoftCase Studies
  • Self-Serve Reporting Platform on Hadoop at LinkedInLinkedInPapers
  • Ambry: Distributed Immutable Object Store at LinkedInLinkedIn engineers · LinkedInCase Studies
  • Benchmarking Streaming Computation Engines at YahooYahoo engineers · YahooCase Studies
  • NodeJS High Availability at YahooYahoo engineers · YahooCase Studies

  Open this lesson with its quiz and exercise