Apache Flink: Concepts overview

Ideas you will meet

Event Driven Architecture

Loosely coupled services reacting to events.

Distributed Systems

Consensus, time, failure and the fundamentals of scale-out.

What we already know about this

Our own lessons on these ideas. Read them here before you start, so it makes sense.

• Software Design Essentials · 12 minThe SOLID principlesFive guidelines for classes and modules that are easy to understand, extend and test, with before-and-after examples.Read hereClose

  What SOLID is for

  SOLID is a set of five guidelines (collected by Robert C. Martin) that help keep code easy to change. They fight three common problems:

  • Rigidity: one change forces edits in many places.
  • Fragility: a change breaks something unrelated.
  • Immobility: you cannot reuse a part without dragging everything with it.

  They are guidelines, not laws: apply them where they remove pain.

  100%

  Figure SDA-SDE-02-1 · What SOLID is for · © forcsguy.com

  S: Single Responsibility Principle

  A module should have one reason to change.

  Group code that changes for the same reason, and separate code that changes for different reasons.

  // Before: three reasons to change (rules, formatting, storage)
  class Report {
    total(items: Item[]) { /* business rule */ }
    toHtml(items: Item[]) { /* presentation */ }
    save(items: Item[]) { /* database */ }
  }
  
  // After: each part changes independently
  class ReportCalculator { total(items: Item[]) { /* ... */ } }
  class ReportHtmlView  { render(total: number) { /* ... */ } }
  class ReportRepository { save(items: Item[]) { /* ... */ } }
  

  O: Open/Closed Principle

  Software should be open for extension, closed for modification.

  Add behaviour by adding code, not by editing tested code. Typically use an abstraction and several implementations.

  // Before: every new payment type edits this function
  function fee(method: string, amount: number) {
    if (method === "card") return amount * 0.03;
    if (method === "bank") return 0.5;
    return 0;
  }
  
  // After: add a new class; the caller never changes
  interface FeePolicy { fee(amount: number): number }
  class CardFee implements FeePolicy { fee(a: number) { return a * 0.03; } }
  class BankFee implements FeePolicy { fee(_a: number) { return 0.5; } }
  const total = (p: FeePolicy, amount: number) => amount + p.fee(amount);
  

  L: Liskov Substitution Principle

  A subtype must be usable wherever its base type is expected, without surprises.

  If code works with Bird, it must still work when given any subclass. A classic violation:

  class Bird { fly() { /* ... */ } }
  class Penguin extends Bird { fly() { throw new Error("cannot fly"); } } // breaks callers
  

  The fix is a better model: separate Bird from FlyingBird, or use composition. Signs of a violation: subclasses that throw for inherited methods, strengthen preconditions, or weaken guarantees.

  I: Interface Segregation Principle

  Clients should not be forced to depend on methods they do not use.

  Prefer several small, focused interfaces to one large one.

  // Before: a read-only report is forced to implement write methods
  interface Storage { read(): string; write(s: string): void; delete(): void }
  
  // After
  interface Reader { read(): string }
  interface Writer { write(s: string): void }
  

  D: Dependency Inversion Principle

  High-level policy should not depend on low-level details. Both should depend on abstractions.

  The business rule owns an interface; the database, email service or queue implements it.

  interface Mailer { send(to: string, body: string): Promise<void> }
  
  class SignUp {
    constructor(private mailer: Mailer) {}              // depends on the abstraction
    async run(email: string) { await this.mailer.send(email, "Welcome"); }
  }
  // Production: new SignUp(new SmtpMailer())    Tests: new SignUp(new FakeMailer())
  

  This is the basis of dependency injection, makes code testable (swap in fakes) and keeps rules independent of technology. It underlies clean and hexagonal architecture (a later lesson).

  Using SOLID well

  • Start with simple code, and refactor when a change shows you the pain.
  • Do not create an interface for every class "for SOLID". An abstraction needs at least two plausible implementations or a real need to isolate a dependency (a database, the clock, the network).
  • Composition over inheritance: assemble behaviour from small parts rather than deep class trees.
  • Small functions and modules are easier to keep single-purpose.

  Real-world reading

  Related reading

  • Software Engineering DailySoftware Engineering DailyPodcasts & Newsletters
  • Apache Flink: Concepts overviewApache Flink · Apache Software FoundationDocumentation
  • Martin Fowler: Software Architecture GuideMartin FowlerBlogs
  • Moving Core Business Search to Elasticsearch at YelpYelp engineers · YelpCase Studies

  Show 6 moreShow fewer

  • Explaining Low-Level Systems (OS, Network/Protocol, Database, Storage)Wustl engineers · WustlBlogs
  • O'Reilly MySQL CE 2011: Jeremy Cole, "Big and Small Data at @Twitter"TwitterYouTube
  • PostgreSQL: Using EXPLAINPostgreSQL Global Development Group · PostgreSQLDocumentation
  • Managing Software Dependency at Scale at LinkedInScale at LinkedIn engineers · Scale at LinkedInCase Studies
  • Top 10 Common Large-Scale Software Architectural Patterns in a NutshellTowardsdatascience engineers · TowardsdatascienceBlogs
  • Single Repository at GoogleGoogle engineers · GoogleCase Studies

  Open this lesson with its quiz and exercise