Brooklin: Distributed Service for Near Real-Time Data Streaming at LinkedIn

Ideas you will meet

WebSockets

Persistent bidirectional connections for real-time features.

Event Driven Architecture

Loosely coupled services reacting to events.

Distributed Systems

Consensus, time, failure and the fundamentals of scale-out.

What we already know about this

Our own lessons on these ideas. Read them here before you start, so it makes sense.

• Core Concepts: How Things Actually Work · 7 minPolling, long polling, SSE and WebSocketsFour ways to get live updates to a browser, and how to choose between them.Read hereClose

  The problem

  HTTP is request and response: the client asks. How does the server tell a browser something now?

  The four options

  Technique	How	Good	Weak
  Short polling	Client asks every N seconds	Trivial, works everywhere	Wasteful, delayed by N
  Long polling	Client asks; the server holds the request until there is news (or a timeout), then the client asks again	Near real time, plain HTTP	A held request per client, awkward at scale
  Server-sent events (SSE)	One long HTTP response streams events	Simple, auto-reconnect, text over HTTP	One-way only
  WebSockets	HTTP upgrades to a two-way connection	Low latency both ways	Stateful connections, more infrastructure

  100%

  Figure SDA-CORE-08-1 · The four options · © forcsguy.com

  Before: polling: keep asking

  100%

  After: WebSocket: server pushes

  100%

  What changed

  • No wasted requests when nothing has changed
  • Updates arrive the moment they happen
  • The server must hold many open connections

  Choosing

  • Updates are rare or delays are fine: polling.
  • The server pushes, the client mostly listens (feeds, scores, notifications, progress): SSE.
  • Both sides send often (chat, games, collaborative editing): WebSockets.

  Running persistent connections

  Connections are long-lived, so capacity is about how many connections a server can hold, not requests per second.

  • Put connection handling in servers that are separate from your stateless API.
  • Use heartbeats to detect dead connections, and reconnect with backoff and jitter.
  • Track which server holds which user so messages can be routed (a presence or session store), and use a pub/sub system between servers.
  • When a client reconnects, let it ask for what it missed (events after id X), because connections drop.
  • Authenticate on connect and again when tokens expire.

  More on this site

  Related reading

  • RFC 6455: The WebSocket ProtocolIETFDocumentation
  • MDN: WebSockets APIMDN Contributors · MozillaDocumentation
  • Uber's Real-Time Push PlatformUber Engineering · UberCase Studies
  • How Discord Scaled Elixir to 5,000,000 Concurrent UsersDiscordCase Studies

  Show 6 moreShow fewer

  • The WhatsApp Architecture Facebook Bought for $19BHigh ScalabilityCase Studies
  • Erlang at Facebook (slides)Eugene Letuchy · FacebookCase Studies
  • Slack EngineeringSlack engineers · SlackBlogs
  • Real-time MessagingSlack Engineering · SlackCase Studies
  • How Uber Scales Their Real-time Market Platform - High Scalability -High Scalability engineers · High ScalabilityCase Studies
  • Real-Time Predictions at DoorDashDoorDash engineers · DoorDashCase Studies

  Open this lesson with its quiz and exercise

• System Design Foundations · 11 minQueues, streams and asynchronous workHow to take slow or bursty work off the request path, and what delivery guarantees really mean.Read hereClose

  Not everything has to happen now

  A user clicks "Sign up". Creating the account must happen before you reply. Sending the welcome email, resizing the avatar and updating dashboards do not. Doing them inside the request makes it slow and fragile: if the email service hiccups, signup fails.

  Image: W.carter · CC0 · Source · Wikimedia Commons: People waiting in line at Stena Line ferry terminal, Frederikshavn

  Asynchronous processing moves such work out of the request. The request records what must happen and returns; other workers do it shortly after.

  100%

  Figure SDA-FND-08-1 · Asynchronous work via a queue · © forcsguy.com

  What a queue gives you

  • Decoupling: the producer does not know or care who consumes.
  • Load levelling: a spike of 10,000 jobs is stored and worked off at a steady rate instead of crashing the consumer.
  • Resilience: if a consumer is down, messages wait.
  • Independent scaling: add more workers when the queue grows.

  The price is eventual completion: the work is done soon, not now, and the system has more moving parts.

  Before: everything in the request

  100%

  After: slow work goes through a queue

  100%

  What changed

  • The user gets a reply straight away
  • A slow email service no longer breaks signup
  • Spikes pile up in the queue instead of crashing servers

  Queues versus streams

  • A message queue (RabbitMQ, SQS) delivers each message to one consumer, which then removes it. Great for task distribution.
  • A log or stream (Kafka, Kinesis) keeps messages in order for a retention period. Many independent consumers can read it, and each can replay from any point. Great for event pipelines and for feeding several systems from one source of truth.

  Delivery guarantees

  Networks and consumers fail, so a broker cannot give every guarantee for free.

  Guarantee	Meaning	Risk
  At-most-once	Delivered zero or one time	May lose messages
  At-least-once	Delivered one or more times	May duplicate messages
  Exactly-once	Effect happens exactly once	Costly, and usually achieved by at-least-once plus idempotency

  The practical approach: use at-least-once delivery and make handlers idempotent.

  100%

  Figure SDA-FND-08-2 · Why the moment of acknowledgement decides the guarantee · © forcsguy.com

  Idempotency in practice

  Make repeating an operation harmless:

  • Prefer "set the order status to paid" over "add 10 to the balance".
  • Attach a unique idempotency key to a request and remember the keys already processed.
  • Use database constraints (a unique key) so a duplicate insert fails safely.

  Failure handling

  • Retries with backoff: wait longer between attempts (1 s, 2 s, 4 s...) with a little random jitter so many consumers do not retry in lockstep.
  • Dead-letter queue: after N failures, move the message aside so it cannot block the rest, and alert a human.
  • Ordering: strict global order is expensive. Ordering is usually guaranteed only within a partition, so pick a key (like user id) so related events land in the same one.
  • Back pressure: if consumers fall behind, the queue grows. Monitor its depth and age, and scale workers or slow producers.

  A worked example: sizing workers and watching the backlog

  A photo app creates thumbnails in the background. Each job takes 2 seconds of one worker. Normal traffic is 50 uploads per second.

  • Workers needed at the average: 50 jobs/s x 2 s = 100 workers busy all the time. Plan for about 70% utilisation, so roughly 143 workers, which can do about 71 jobs a second.
  • A spike: a campaign sends 500 uploads per second for 60 seconds: 30,000 jobs. The queue accepts them instantly; the workers keep draining at about 71 per second. The backlog grows by 500 - 71 = 429 per second, to about 25,700 jobs by the end of the spike.
  • Recovery: after the spike, arrivals fall back to 50 per second and the workers drain a net 71 - 50 = 21 per second. Clearing 25,700 jobs takes about 1,200 seconds, 20 minutes. That is the real cost of a spike: not an outage, but a delay.

  This is why the metric to watch is queue age (how long the oldest message has waited), not just depth. Depth of 25,000 can be fine for a thumbnail and disastrous for a payment confirmation. Decide how stale each job may become, and alert on that.

  100%

  Figure SDA-FND-08-3 · A queue turns a spike into a delay · © forcsguy.com

  What happens to a message that keeps failing

  Workers fail: a bug, a missing record, a dependency down. Without a plan, one bad message can block everything or loop forever.

  1. A worker receives a message. The queue hides it for a visibility timeout (say 60 seconds) instead of deleting it.
  2. If the worker finishes, it acknowledges and the message is deleted.
  3. If the worker crashes or fails, no acknowledgement arrives, the timeout expires and the message becomes visible again for another try.
  4. Retries use exponential backoff with jitter so a struggling dependency is not hammered.
  5. After a fixed number of attempts (often 3 to 10) the message moves to a dead-letter queue so it stops blocking others. A person or a tool inspects it, fixes the cause, and replays it.

  100%

  Figure SDA-FND-08-4 · Retries and the dead-letter queue · © forcsguy.com

  Because a message can be delivered more than once, every consumer must be idempotent: processing it twice must have the same effect as once. Typical tools are a unique message id recorded in a table (insert first, skip on conflict), or operations that are naturally repeatable ("set status to shipped" rather than "add one").

  Mistakes people make

  • Non-idempotent consumers. "Charge the card" run twice is a double charge. Use an idempotency key.
  • Unbounded retries. A permanently failing message that retries forever wastes capacity. Cap attempts and dead-letter.
  • Ignoring queue age. Alert on how old the oldest message is, per queue.
  • One queue for everything. A flood of low-priority jobs starves urgent ones. Separate queues, or priorities, by importance.
  • Assuming order. Most queues do not guarantee order across consumers. If order matters per user or per order, partition by that key.
  • Huge messages. Put big payloads in object storage and send a reference.
  • Using a queue when the caller needs the answer now. Queues make work asynchronous; if the user is waiting for the result, they only add delay.

  Try it yourself

  In the System Builder, open the challenge "A burst of writes": run it without a queue and then with one, and watch the "Writes waiting in queue" chart. For a database-backed version with retries and a dead-letter state, read the SKIP LOCKED queue lesson.

  More on this site

  Related reading

  • Confluent Developer: Apache Kafka 101ConfluentCourses
  • Confluent: Apache Kafka TalksConfluentYouTube
  • Making retries safe with idempotent APIsAmazon Builders' Library · Amazon Web ServicesBlogs
  • Apache Kafka DocumentationApache Kafka · Apache Software FoundationDocumentation

  Show 6 moreShow fewer

  • Pattern: Transactional outboxChris Richardson · microservices.ioDocumentation
  • Confluent BlogConfluent engineers · ConfluentBlogs
  • Scaling Slack's Job QueueSlack Engineering · SlackCase Studies
  • Kafka on PaaSTA: Running Kafka on Kubernetes at Yelp (2 parts)Yelp engineers · YelpCase Studies
  • Kafka on Kubernetes at ShopifyShopify engineers · ShopifyCase Studies
  • Brooklin: Distributed Service for Near Real-Time Data Streaming at LinkedInLinkedIn engineers · LinkedInCase Studies

  Open this lesson with its quiz and exercise