Real-time Presence Platform at LinkedIn

Ideas you will meet

WebSockets

Persistent bidirectional connections for real-time features.

What we already know about this

Our own lessons on these ideas. Read them here before you start, so it makes sense.

• Core Concepts: How Things Actually Work · 7 minPolling, long polling, SSE and WebSocketsFour ways to get live updates to a browser, and how to choose between them.Read hereClose

  The problem

  HTTP is request and response: the client asks. How does the server tell a browser something now?

  The four options

  Technique	How	Good	Weak
  Short polling	Client asks every N seconds	Trivial, works everywhere	Wasteful, delayed by N
  Long polling	Client asks; the server holds the request until there is news (or a timeout), then the client asks again	Near real time, plain HTTP	A held request per client, awkward at scale
  Server-sent events (SSE)	One long HTTP response streams events	Simple, auto-reconnect, text over HTTP	One-way only
  WebSockets	HTTP upgrades to a two-way connection	Low latency both ways	Stateful connections, more infrastructure

  100%

  Figure SDA-CORE-08-1 · The four options · © forcsguy.com

  Before: polling: keep asking

  100%

  After: WebSocket: server pushes

  100%

  What changed

  • No wasted requests when nothing has changed
  • Updates arrive the moment they happen
  • The server must hold many open connections

  Choosing

  • Updates are rare or delays are fine: polling.
  • The server pushes, the client mostly listens (feeds, scores, notifications, progress): SSE.
  • Both sides send often (chat, games, collaborative editing): WebSockets.

  Running persistent connections

  Connections are long-lived, so capacity is about how many connections a server can hold, not requests per second.

  • Put connection handling in servers that are separate from your stateless API.
  • Use heartbeats to detect dead connections, and reconnect with backoff and jitter.
  • Track which server holds which user so messages can be routed (a presence or session store), and use a pub/sub system between servers.
  • When a client reconnects, let it ask for what it missed (events after id X), because connections drop.
  • Authenticate on connect and again when tokens expire.

  More on this site

  Related reading

  • RFC 6455: The WebSocket ProtocolIETFDocumentation
  • MDN: WebSockets APIMDN Contributors · MozillaDocumentation
  • Uber's Real-Time Push PlatformUber Engineering · UberCase Studies
  • How Discord Scaled Elixir to 5,000,000 Concurrent UsersDiscordCase Studies

  Show 6 moreShow fewer

  • The WhatsApp Architecture Facebook Bought for $19BHigh ScalabilityCase Studies
  • Erlang at Facebook (slides)Eugene Letuchy · FacebookCase Studies
  • Slack EngineeringSlack engineers · SlackBlogs
  • Real-time MessagingSlack Engineering · SlackCase Studies
  • How Uber Scales Their Real-time Market Platform - High Scalability -High Scalability engineers · High ScalabilityCase Studies
  • Real-Time Predictions at DoorDashDoorDash engineers · DoorDashCase Studies

  Open this lesson with its quiz and exercise

• Classic System Designs · 12 minDesign cloud file storage and syncStore files for millions of users and keep devices in sync: block-level chunking, deduplication, metadata, conflicts and sharing.Read hereClose

  The problem

  A cloud file storage service (think of a personal drive) stores each user's files, makes them available on all their devices, keeps them in sync and lets them share.

  Requirements

  • Upload, download and sync files on web, desktop and mobile.
  • Sync quickly across devices; show file history and allow restore.
  • Sharing with permissions.
  • Reliable: never lose a file; handle large files and flaky connections.
  • Efficient in bandwidth and storage.

  Estimates

  Suppose 50 million users with 10 GB each: about 500 PB of data. That makes storage cost and upload bandwidth central. It also means you must avoid re-uploading unchanged data.

  High-level design

  Separate file content from file metadata:

  100%

  Figure SDA-CSD-11-1 · Metadata and block storage architecture · © forcsguy.com

  • Metadata (names, folders, versions, permissions, ordered list of block hashes): a relational database with strong consistency, since many operations need correct, ordered updates.
  • File content: stored as blocks in distributed object storage, replicated for durability.

  Growing from one server

  100%

  Figure SDA-CSD-11-2 · From one disk to a scalable storage design · © forcsguy.com

  100%

  Figure SDA-CSD-11-3 · Replication within and across regions · © forcsguy.com

  Blocks make everything better

  The client splits each file into fixed-size blocks (for example 4 MB), computes a hash per block, and uploads only the blocks the server does not already have.

  100%

  Figure SDA-CSD-11-4 · File as an ordered list of block hashes · © forcsguy.com

  Benefits:

  • Delta sync: after an edit only the changed blocks are uploaded.
  • Resumable uploads and downloads.
  • Deduplication: blocks with the same hash are stored once, even across users (with care for privacy).
  • Cheap versions: a version is just a list of block hashes, so history costs little.
  • Blocks can be compressed and encrypted before storage.

  Before: upload the whole file

  100%

  After: upload changed blocks only

  100%

  What changed

  • Editing one page of a large file sends a few megabytes, not all of it
  • Identical blocks are stored once
  • A broken upload can resume from the last block

  Sync flow

  1. The client detects a local change, splits, hashes and asks the server which blocks are missing.
  2. It uploads missing blocks, then commits a new file version in the metadata service.
  3. The notification service tells the user's other devices (through a long-lived connection or long polling) that something changed.
  4. Those devices fetch the new metadata and download only blocks they lack.

  100%

  Figure SDA-CSD-11-5 · A sync conflict · © forcsguy.com

  Conflicts

  If two devices change the same file from the same base version, the second commit finds the base is out of date. The safe rule is to keep both: accept the first, and save the second as a "conflicted copy" so nothing is lost. Never overwrite silently.

  Metadata, consistency and notifications

  Data model. A user has devices (each with a push id) and a root namespace; a file row holds the latest state; a file-version table keeps read-only history so past revisions stay trustworthy; a block table maps each version to its ordered blocks. Any version can be rebuilt by joining its blocks in order.

  Strong consistency for metadata. Two clients must never see different file lists. Caches default to eventual consistency, so keep cache replicas in step with the primary and invalidate the cache on every database write. A relational database gives atomic, isolated updates natively, which is why it suits metadata better than a store where you would have to build those guarantees yourself.

  Why long polling can be enough for notifications. Updates flow only server to client and are infrequent, so a persistent two-way channel is more than needed; a client holds a request open, gets told "something changed", fetches metadata, then reopens. (WebSocket is the better fit for chat-like two-way traffic.)

  Upload sequence. Two things happen at once: the client registers metadata (status pending, others are told a file is arriving) and sends blocks through the block servers; when storage confirms the upload, a callback flips the status to uploaded and others are notified so they can pull metadata and then the blocks.

  Failure notes. A notification server may hold a million connections; if it dies, all of them reconnect elsewhere at once, which is slow, so spread reconnects out. Offline clients are served from a backup queue of pending changes. Block servers' unfinished jobs are taken over by others.

  Trade-off: uploading straight to storage is faster (one hop) but means chunking, compression and encryption must be reimplemented and trusted on every client platform, and clients can be tampered with; a central block tier keeps that logic in one place.

  Image: U.S. Department of Agriculture · Public domain · Source · Wikimedia Commons: Magnetic tape library at USDA

  The block pipeline and delta sync

  100%

  Figure SDA-CSD-11-6 · Block server pipeline · © forcsguy.com

  100%

  Figure SDA-CSD-11-7 · Delta sync: only changed blocks · © forcsguy.com

  Reliability and cost

  • Durability: store blocks in several locations, with checksums verified on read and repaired by background scrubbing.
  • Hot and cold tiers: recent files on fast storage, old versions and rarely used data moved to cheaper cold storage.
  • Limit version history (by count or age) to control cost.
  • Metadata scale: shard the metadata database by user id; most operations touch one user's data.
  • Large files: parallel block uploads, resumable and retried independently.

  Upload and download as sequences

  100%

  Figure SDA-CSD-11-8 · Upload with notification to other devices · © forcsguy.com

  100%

  Figure SDA-CSD-11-9 · Download after a change elsewhere · © forcsguy.com

  Sharing and security

  • Permissions (view, edit) stored with the file or folder, checked on every request.
  • Share links with optional expiry and password.
  • Encrypt in transit and at rest; consider client-side encryption for stronger privacy (which limits server-side features such as search and dedup).
  • Audit logs for access and changes.

  Worked numbers: why blocks and deltas matter

  Files are cut into blocks of about 4 MB, each identified by a hash of its content.

  • Editing a large file: a user changes one paragraph in a 1 GB file. Only the block(s) that contain the change differ, so the client uploads about 4 MB instead of 1 GB, roughly 0.4% of the data, and the sync finishes in seconds.
  • Deduplication: if a block's hash already exists in storage (the same file uploaded by another user, or a copy within your account), the server stores nothing new and just records that this file version uses that block. Popular files cost almost nothing to store again.
  • Raw capacity: with 10 million users x 10 GB the system holds about 100 PB. Three full copies would need 300 PB; erasure coding (for example 10 data pieces plus 4 parity pieces, a 1.4x overhead) needs about 140 PB while surviving the loss of any 4 pieces. That difference is tens of millions of dollars a year, so cold data is usually erasure coded, and hot data may be replicated.
  • Metadata (file names, folder trees, versions, block lists) is tiny by comparison but is read and written constantly, so it lives in a separate, strongly consistent database, not with the blocks.

  100%

  Figure SDA-CSD-11-10 · Delta sync: upload only the changed block · © forcsguy.com

  When two devices edit the same file

  A laptop is offline and edits plan.docx; meanwhile a phone edits the same file and syncs first. When the laptop reconnects it has a version based on a now-outdated parent. The server notices because the upload says "I am based on version 7" while the latest is 8.

  For files in general the safe approach is: do not silently overwrite. Keep the first writer's version as current and save the second as a conflicted copy ("plan (Laptop's conflicted copy).docx"), then let the user merge. For real-time collaborative documents, a different technique (operational transforms or CRDTs) merges edits automatically, but that is a separate design.

  100%

  Figure SDA-CSD-11-11 · Detecting a conflict by version · © forcsguy.com

  Mistakes people make

  • Uploading whole files. It wastes bandwidth and makes large files unusable on poor connections. Use blocks, deltas and resumable uploads.
  • Trusting client timestamps to decide the winner. Clocks differ; compare versions, not times.
  • Treating a rename or move as delete plus create. It re-uploads the data. Metadata should record moves without touching blocks.
  • Polling for changes. It wastes battery and server capacity. Use a long-lived notification channel that says "something changed, pull your list".
  • Mixing metadata and block storage. They have different access patterns and consistency needs.
  • Forgetting security. Encrypt in transit and at rest, check permissions on every request, expire share links, and scan uploads for malware if you serve them to others.

  Try it yourself

  Read the consistent hashing lesson for placing blocks across servers, and build the consistent hash ring lab to place block ids on storage nodes. The Builder's write-burst challenge shows why uploads and metadata updates benefit from a queue.

  More on this site

  Related reading

  • How Discord Scaled Elixir to 5,000,000 Concurrent UsersDiscordCase Studies
  • CAP Twelve Years Later: How the Rules Have ChangedEric Brewer · InfoQBlogs
  • Amazon S3 strong consistencyAmazon Web ServicesDocumentation
  • Designing Data-Intensive ApplicationsMartin Kleppmann · O'ReillyBooks

  Show 6 moreShow fewer

  • Concourse: Generating Personalized Content Notifications in Near-Real-Time at LinkedInLinkedIn engineers · LinkedInCase Studies
  • Rewriting the heart of our sync engineDropboxCase Studies
  • Fast Order Search using Data Pipeline and Elasticsearch at YelpYelp engineers · YelpCase Studies
  • Pagelets Parallelize Server-side Processing at YelpYelp engineers · YelpCase Studies
  • Real-time Presence Platform at LinkedInLinkedIn engineers · LinkedInCase Studies
  • GitHub Engineering BlogGitHub engineers · GitHubBlogs

  Open this lesson with its quiz and exercise