Self-Serve Reporting Platform on Hadoop at LinkedIn

Ideas you will meet

Machine Learning Systems

Recommendations, ranking, feature stores and serving models at scale.

What we already know about this

Our own lessons on these ideas. Read them here before you start, so it makes sense.

• Big Data and Cloud Data Platforms · 10 minChoosing a cloud provider for a big data solutionA fair comparison of the main data services on AWS, Azure and Google Cloud, and a decision process that avoids choosing by brand.Read hereClose

  Start from needs, not logos

  No provider is best for everyone. The right choice depends on where your data and applications already are, your team's skills, what workloads you run and how you will pay. Marketing comparisons and generic benchmarks matter less than a test on your own data.

  The same building blocks, different names

  Need	AWS	Microsoft Azure	Google Cloud
  Object storage (the lake)	S3	Azure Data Lake Storage Gen2 (Blob Storage)	Cloud Storage
  Managed Spark and Hadoop	EMR (and Glue jobs)	Azure Databricks, Synapse Spark, HDInsight	Dataproc
  Data warehouse	Redshift	Synapse Analytics (and Microsoft Fabric)	BigQuery
  Query files in place	Athena	Synapse serverless SQL	BigQuery external tables
  Streaming ingest	Kinesis, MSK (managed Kafka)	Event Hubs	Pub/Sub
  Stream processing	Managed Flink, Kinesis analytics	Stream Analytics	Dataflow
  Orchestration	Step Functions, MWAA (managed Airflow), Glue workflows	Data Factory	Cloud Composer (managed Airflow), Workflows
  Catalogue and governance	Glue Data Catalog, Lake Formation	Microsoft Purview	Dataplex, Data Catalog

  Independent platforms such as Snowflake and Databricks run on all three and are common choices when you want a multi-cloud or cloud-neutral platform.

  100%

  Figure SDA-BDP-04-1 · The same building blocks, different names · © forcsguy.com

  100%

  Figure SDA-BDP-04-2 · The stages of a data platform, whichever cloud you choose · © forcsguy.com

  Criteria that matter

  1. Data gravity and egress: where the data already is. Moving petabytes is slow and incurs transfer fees.
  2. Existing ecosystem: identity, networking, security tooling and applications already on a cloud reduce integration work.
  3. Team skills: a SQL-first team may prefer a managed warehouse; an engineering-heavy team may prefer Spark and open formats.
  4. Workload fit: interactive BI, large batch, streaming, machine learning, geospatial. Each service has strengths.
  5. Cost model: warehouses charge differently: per data scanned (good for occasional queries), per cluster or capacity time (good for steady heavy use), plus storage and egress. Model your pattern.
  6. Performance and scale: test with your real queries, since results depend on data shape.
  7. Governance, security and compliance: data residency, encryption, fine-grained access, auditing, certifications for your industry.
  8. Openness and lock-in: prefer open file and table formats (Parquet, Iceberg, Delta) and standard SQL so you can switch engines. Proprietary features give convenience at the cost of portability.
  9. Reliability and support: regions available, service limits, support plans.
  10. Operational burden: managed and serverless services reduce work but cost more per unit.

  A sensible process

  1. Write down requirements: data volume and growth, latency needs, users, workloads, compliance.
  2. Shortlist two or three options, including your current cloud.
  3. Proof of concept with a representative slice of data and your real queries; measure speed, ease and cost.
  4. Model total cost over a year or more: storage, compute, transfer, support and people.
  5. Decide with a weighted scorecard, and record why.
  6. Plan the exit: open formats and a clean data layer make a future change feasible.

  Common mistakes

  • Choosing on brand or a single benchmark.
  • Ignoring egress and cross-region transfer costs.
  • Underestimating the people and operations cost of a self-managed cluster.
  • Loading everything into a proprietary format with no way out.
  • Skipping governance until the platform is already messy.
  • Building for petabytes when you have gigabytes: a single powerful database or a small warehouse may be enough.

  Real-world reading

  Related reading

  • The Architecture of Open Source ApplicationsVarious authors · aosabook.orgBooks
  • eBay: Building Mission-Critical Multi-Data Center Applications with MongoDBMongodb engineers · MongodbCase Studies
  • Improving Search Engine Efficiency by over 25% at eBayeBay engineers · eBayCase Studies
  • Bullet: Forward-Looking Query Engine for Streaming Data at YahooYahoo engineers · YahooCase Studies

  Show 6 moreShow fewer

  • Dremel: Interactive Analysis of Web-Scale Datasets (paper)Melnik et al. · GooglePapers
  • Reasons for Choosing S3 over HDFS at DatabricksDatabricks engineers · DatabricksCase Studies
  • Self-Serve Reporting Platform on Hadoop at LinkedInLinkedInPapers
  • Workload isolation using shuffle shardingAmazon Builders' Library · Amazon Web ServicesBlogs
  • Altus Care: Applying a Chatbot to Platform Engineering at eBayeBay engineers · eBayCase Studies
  • Microsoft Azure BlogMicrosoft engineers · MicrosoftBlogs

  Open this lesson with its quiz and exercise

• Software Design Essentials · 11 minBehavioural patterns: Strategy, Observer, Command, StatePatterns for how objects cooperate: swapping algorithms, reacting to events, turning actions into objects and changing behaviour with state.Read hereClose

  The goal

  Behavioural patterns distribute responsibility between objects and define how they communicate.

  Strategy

  Define a family of interchangeable algorithms behind an interface and pick one at runtime.

  type Pricing = (subtotalCents: number) => number;
  
  const regular: Pricing = (s) => s;
  const member:  Pricing = (s) => Math.round(s * 0.9);
  const blackFriday: Pricing = (s) => Math.round(s * 0.7);
  
  function checkout(subtotal: number, pricing: Pricing) { return pricing(subtotal); }
  checkout(10_000, member);   // the strategy is passed in
  

  Replaces long if/else or switch chains and supports the open/closed principle. In languages with first-class functions the "strategy" is simply a function you pass in.

  Observer (publish and subscribe)

  Objects subscribe to a subject and are notified when it changes, without the subject knowing who listens.

  class Emitter<T> {
    private handlers = new Set<(v: T) => void>();
    on(h: (v: T) => void) { this.handlers.add(h); return () => this.handlers.delete(h); } // returns "unsubscribe"
    emit(v: T) { this.handlers.forEach((h) => h(v)); }
  }
  const cart = new Emitter<number>();
  const stop = cart.on((count) => updateBadge(count));
  cart.emit(3);
  stop();                                              // always clean up
  

  Used for UI events, domain events and message buses. Watch for leaks (forgotten unsubscribes), ordering surprises and cascades of updates that are hard to follow.

  Observer as a picture

  100%

  Figure SDA-SDE-05-1 · One subject, many observers · © forcsguy.com

  Command

  Turn a request into an object containing everything needed to run it. That lets you queue, log, retry, schedule and undo it.

  interface Command { run(): void; undo(): void }
  
  class InsertText implements Command {
    constructor(private doc: Doc, private at: number, private text: string) {}
    run()  { this.doc.insert(this.at, this.text); }
    undo() { this.doc.remove(this.at, this.text.length); }
  }
  
  const history: Command[] = [];
  function perform(c: Command) { c.run(); history.push(c); }
  function undo() { history.pop()?.undo(); }
  

  Editors, job queues, macros and transaction logs all use it.

  State

  Let an object change behaviour when its internal state changes, by delegating to a state object (or a transition table).

  type State = "new" | "paid" | "shipped" | "cancelled";
  const next: Record<State, Partial<Record<"pay" | "ship" | "cancel", State>>> = {
    new:       { pay: "paid", cancel: "cancelled" },
    paid:      { ship: "shipped", cancel: "cancelled" },
    shipped:   {},
    cancelled: {},
  };
  function apply(state: State, action: "pay" | "ship" | "cancel"): State {
    const to = next[state][action];
    if (!to) throw new Error(`Cannot ${action} an order that is ${state}`);
    return to;
  }
  

  Makes illegal transitions explicit and impossible, replacing scattered flags and nested conditions.

  Template Method

  Define the skeleton of an algorithm in one place and let steps vary.

  abstract class Importer {
    run() { const raw = this.read(); const rows = this.parse(raw); this.save(rows); } // fixed order
    protected abstract read(): string;
    protected abstract parse(raw: string): Row[];
    protected save(rows: Row[]) { /* default */ }
  }
  

  Frameworks use it widely ("override these hooks"). With functions, you can pass the varying steps in instead of subclassing.

  Chain of Responsibility

  Pass a request along a chain of handlers until one handles it (or each adds something). Middleware pipelines in web servers are this pattern.

  type Handler = (req: Req, next: () => Promise<Res>) => Promise<Res>;
  // authenticate -> rateLimit -> validate -> handle
  

  Iterator and others

  • Iterator: walk a collection without exposing its structure. In most languages this is built in (for...of, generators).
  • Mediator: one object coordinates many (a chat room, a form controller) so participants do not reference each other.
  • Memento: capture and restore an object's state (undo snapshots).
  • Visitor: add new operations over a structure of object types without modifying them; common in compilers.
  • Interpreter: evaluate expressions of a small language.

  Choosing

  Problem	Pattern
  Many alternative algorithms or rules	Strategy
  React to changes without coupling	Observer
  Undo, queue, log or retry actions	Command
  Behaviour depends on lifecycle state	State
  Same steps, different details	Template Method
  A pipeline of optional processing steps	Chain of Responsibility

  Remember: with first-class functions many of these become callbacks, closures and tables. Use the lightest form that keeps the code clear.

  Real-world reading

  Related reading

  • Cape: Event Stream Processing Framework at DropboxDropbox engineers · DropboxCase Studies
  • NATS ServerNATS · CNCF / NATSGitHub
  • Samza: Stream Processing System for Latency Insighs at LinkedInLinkedIn engineers · LinkedInCase Studies
  • Autoscaling AWS Step Functions Activities at YelpYelp engineers · YelpCase Studies

  Show 6 moreShow fewer

  • Distributed Messaging, Queuing, and Event StreamingArxivPapers
  • Steady State: Always Put Logs on Separate DiskMicrosoft engineers · MicrosoftCase Studies
  • Self-Serve Reporting Platform on Hadoop at LinkedInLinkedInPapers
  • Ambry: Distributed Immutable Object Store at LinkedInLinkedIn engineers · LinkedInCase Studies
  • Benchmarking Streaming Computation Engines at YahooYahoo engineers · YahooCase Studies
  • NodeJS High Availability at YahooYahoo engineers · YahooCase Studies

  Open this lesson with its quiz and exercise

• Low-Level Design Practice · 10 minShopping cartCarts, inventory and pricing rules: adding items, applying discounts, checking stock and turning a cart into an order safely.Read hereClose

  The problem

  Shoppers add and remove products, apply coupons, see a total, and check out. The system must check stock, compute the price correctly and create an order.

  Entities

  100%

  Figure SDA-LLD-11-1 · Shopping cart classes · © forcsguy.com

  Cart versus order

  A cart is mutable and may be stale. An order is a frozen record of what was bought, at what price, with which discounts. Checkout converts one into the other.

  100%

  Figure SDA-LLD-11-2 · Checkout flow · © forcsguy.com

  Pricing as ordered rules

  type Line = { sku: string; qty: number; unit: number };
  interface PricingRule { apply(lines: Line[]): { reason: string; cents: number }[] }
  
  const buy2get1: PricingRule = {
    apply: (lines) => lines.filter(l => l.sku === "TSHIRT" && l.qty >= 3)
      .map(l => ({ reason: "Buy 2 get 1", cents: -Math.floor(l.qty / 3) * l.unit })),
  };
  const coupon10: PricingRule = {
    apply: (lines) => [{ reason: "10% coupon", cents: -Math.round(lines.reduce((s, l) => s + l.qty * l.unit, 0) * 0.1) }],
  };
  
  function total(lines: Line[], rules: PricingRule[]) {
    const sub = lines.reduce((s, l) => s + l.qty * l.unit, 0);
    const adj = rules.flatMap(r => r.apply(lines));
    return { subtotal: sub, adjustments: adj, total: sub + adj.reduce((s, a) => s + a.cents, 0) };
  }
  

  Decide and document precedence (do percentage discounts apply before or after fixed ones?) and whether discounts stack. Keep a record of which rules applied so the invoice can explain the price.

  Inventory safely

  // conditional update: succeeds only if enough stock remains
  function reserve(sku: string, qty: number): boolean {
    const row = stock.get(sku)!;
    if (row.available < qty) return false;
    row.available -= qty; row.reserved += qty; return true;   // atomic in a real database: UPDATE ... WHERE available >= qty
  }
  

  Design notes

  • Guest and signed-in carts: merge a guest cart into the user's cart on login.
  • Abandoned carts: stock is not reserved by a cart, only by checkout; reservations expire.
  • Idempotent checkout: a double click must not create two orders (use a checkout key).
  • Money is integer minor units; taxes and shipping are further rules or services.
  • Extensions: wishlists, saved-for-later, per-region pricing, bundles.

  Real-world reading

  Related reading

  • Electronic Arts SEED BlogElectronic Arts engineers · Electronic ArtsBlogs
  • AWS Compute BlogAmazon Web Services engineers · Amazon Web ServicesBlogs
  • PricingNet: Pricing Modelling with Neural Networks at SkyscannerSkyscanner engineers · SkyscannerCase Studies
  • Self-Serve Reporting Platform on Hadoop at LinkedInLinkedInPapers

  Show 6 moreShow fewer

  • Vortex: Monitoring Server Applications at DropboxDropbox engineers · DropboxCase Studies
  • Determinism (4 parts) in League of Legends ServerRiotgames engineers · RiotgamesCase Studies
  • Cassandra for Keeping Counts In Sync at SoundCloudSoundCloud engineers · SoundCloudCase Studies
  • Neural Architecture Search (NAS) for Prohibited Item Detection at MercariMercari engineers · MercariCase Studies
  • Kogan.com Dev BlogKogan.com engineers · Kogan.comBlogs
  • HackerEarth Engineering BlogHackerEarth engineers · HackerEarthBlogs

  Open this lesson with its quiz and exercise