Intelligent DNS based load balancing at Dropbox

Ideas you will meet

Load Balancer

Spread traffic across servers with L4/L7 balancing, health checks and algorithms.

DNS

How names resolve, TTLs, and DNS-based traffic routing.

What we already know about this

Our own lessons on these ideas. Read them here before you start, so it makes sense.

• System Design Foundations · 11 minNetworking, APIs and CDNsWhat happens between a click and your server: DNS, HTTP, API styles, and how a CDN moves content closer to users.Read hereClose

  The journey of a request

  Before your code runs, a lot happens. When a browser loads https://shop.example.com:

  100%

  Figure SDA-FND-09-1 · Request journey: DNS, CDN, origin · © forcsguy.com

  1. DNS turns the name into an IP address. Answers are cached at several levels for a time given by the record's TTL, so changes spread gradually.
  2. A connection is opened (TCP) and secured (TLS). Each step costs round trips, so long distances hurt.
  3. HTTP carries the request and response.

  Latency is mostly physics: a round trip across an ocean costs over 100 ms no matter how fast your code is. The only fixes are fewer round trips and moving data closer to the user.

  Image: Syced · CC0 · Source · Wikimedia Commons: Fujikura optic fiber at Science Museum

  Name lookup is a short chain of questions, each answered by a more specific server:

  100%

  Figure SDA-FND-09-2 · How a name becomes an address · © forcsguy.com

  Image: Howe's Adventures & Achievements of Americans · Public domain · Source · Wikimedia Commons: Transatlantic submarine cable map

  CDNs

  A content delivery network is a worldwide set of servers (edges) that keep copies of your content close to users.

  • Great for static files: images, video, scripts, stylesheets, and often whole pages that rarely change.
  • Cuts latency, absorbs traffic spikes and shields your origin from load.
  • Controlled by cache headers (Cache-Control: max-age=...) and purge APIs when content changes.
  • Use a versioned file name (app.3f9a.js) so you can cache for a year and still ship updates instantly.

  100%

  Figure SDA-FND-09-3 · Content delivery in tiers · © forcsguy.com

  Before: everyone goes to the origin

  100%

  After: a CDN serves copies near users

  100%

  What changed

  • Users get files from a nearby location
  • The origin handles far fewer requests
  • Changed files need invalidation or versioned names

  Choosing an API style

  Style	Strength	Watch out for
  REST over HTTP/JSON	Simple, cacheable, universal	Over-fetching, many round trips for complex screens
  GraphQL	Client asks for exactly the fields it needs	Query cost control, caching is harder
  gRPC	Fast binary protocol, strong contracts	Needs tooling, awkward in browsers
  SOAP	XML messages with a formal contract (WSDL) and built-in standards for security and transactions, still common in banks and older enterprise systems	Verbose, heavier tooling, rarely chosen for new public APIs
  WebSockets / SSE	Server can push live updates	Long-lived connections need extra capacity and care

  A good default is REST for public APIs, gRPC between internal services if you need speed, and WebSockets only where live push is truly needed.

  Designing good APIs

  • Version them so old clients keep working.
  • Paginate list endpoints (cursor-based pagination stays stable as data changes).
  • Make writes idempotent with idempotency keys, so a retry does not double-charge.
  • Return clear errors with correct status codes.
  • Authenticate and rate limit at the edge, often in an API gateway that also handles routing, logging and TLS.

  100%

  Figure SDA-FND-09-4 · API gateway in front of services · © forcsguy.com

  More on this site

  Related reading

  • High Performance Browser NetworkingIlya Grigorik · O'ReillyBooks
  • HTTP/3: the past, the present, and the futureCloudflareBlogs
  • MDN: HTTP cachingMDN contributors · MozillaDocumentation
  • Cloudflare BlogCloudflare engineers · CloudflareBlogs

  Show 6 moreShow fewer

  • The Road to QUICCloudflareBlogs
  • Flannel: An Application-Level Edge Cache to Make Slack ScaleSlack Engineering · SlackCase Studies
  • Cloudflare Learning Center: What is a CDN?CloudflareDocumentation
  • Cloudflare EngineeringCloudflare engineers · CloudflareBlogs
  • Justin.tv's Live Video Broadcasting Architecture - High Scalability -High Scalability engineers · High ScalabilityCase Studies
  • Service Workers at SlackSlack engineers · SlackCase Studies

  Open this lesson with its quiz and exercise

• System Design Foundations · 10 minFrom one server to many: scaling and load balancingVertical versus horizontal scaling, why stateless servers matter, and how a load balancer spreads traffic.Read hereClose

  Two ways to grow

  When one server is not enough you have two options.

  Image: Derrick Coetzee · CC0 · Source · Wikimedia Commons: Technician with laptop working on server rack at NERSC

  • Vertical scaling ("scale up"): use a bigger machine with more CPU, memory or disk. It is simple because nothing in the design changes.
  • Horizontal scaling ("scale out"): add more machines and share the work between them. It needs more design effort but has no hard ceiling.

  	Vertical	Horizontal
  Effort	Low	Higher
  Ceiling	Hardware limit	Practically none
  Failure	One machine is a single point of failure	Survives losing a machine
  Cost curve	Gets steep for the largest machines	Roughly linear with commodity machines

  Real systems use both: scale up until it gets expensive or risky, then scale out.

  100%

  Figure SDA-FND-02-1 · Scale up versus scale out · © forcsguy.com

  What a load balancer does

  Once you have several servers, something must decide which one handles each request. That is the load balancer.

  100%

  Figure SDA-FND-02-2 · Load balancer in front of a server pool · © forcsguy.com

  It does three jobs:

  1. Distributes requests so no server is overloaded.
  2. Checks health. It probes each server regularly and stops sending traffic to ones that fail.
  3. Hides the pool. Clients talk to one address, and servers can be added or removed without clients noticing.

  Common strategies

  • Round robin: take turns. Simple and fine when requests are similar.
  • Least connections: send to the server with the fewest active requests. Better when request cost varies.
  • Hashing (for example on user id or IP): the same key goes to the same server. Useful when a server keeps a local cache for that key.
  • Weighted: bigger machines get a bigger share.

  Load balancers work at different layers. A layer 4 balancer routes by IP and port and is very fast. A layer 7 balancer reads the HTTP request, so it can route by URL path or header and terminate TLS.

  A security benefit comes with this layout: only the balancer needs a public address. The servers behind it use private addresses, reachable from inside the network but not from the internet, so clients cannot bypass the balancer or attack a server directly. If one server goes offline, the balancer sends its traffic to the others and a replacement can join the pool.

  Before: one server

  100%

  After: load balancer and several servers

  100%

  What changed

  • Traffic is shared across servers
  • One server failing no longer takes the site down
  • You can add capacity without downtime

  Stateless servers make scaling easy

  A server is stateless when it keeps nothing about a user between requests. Everything it needs comes from the request or from a shared store.

  100%

  Figure SDA-FND-02-3 · Stateful versus stateless servers · © forcsguy.com

  With stateless servers any request can go to any server, so you can add servers at will, replace a failed one without losing data, and run auto-scaling. The state does not disappear. It moves to a database, a cache or a session store, which you then have to scale and protect, which is what the next lessons are about.

  Putting it together

  A typical first scalable layout:

  1. Users reach a load balancer.
  2. It spreads requests over a pool of stateless application servers.
  3. Servers read and write a shared database (and later a cache).
  4. Static files such as images and scripts are served from a CDN so the servers do not carry that traffic (lesson 9).

  This design handles a surprisingly large amount of traffic. The first thing to give way is usually the database, which is where we go next.

  A worked example: how many servers do you need?

  Capacity planning is arithmetic you can do on a napkin. Say your product has 5 million daily users and each makes about 20 requests a day.

  1. Requests per day: 5,000,000 x 20 = 100 million.
  2. Per second on average: 100,000,000 / 86,400 = about 1,160 requests per second.
  3. Traffic is not flat. A common rule of thumb is that the busiest second is 2 to 3 times the average, so plan for about 3,500 per second.
  4. One server handles about 400 requests per second flat out, but you never want servers at 100%: queues build and latency explodes near full load. Plan for roughly 70% utilisation, so 280 requests per second each.
  5. 3,500 / 280 = 12.5, so 13 servers carry the peak.
  6. Add spare capacity so you survive failures: with 13 servers, losing 2 leaves 11, which carry only 3,080 requests per second. Plan for N+2: 15 servers.

  100%

  Figure SDA-FND-02-4 · From users to a server count · © forcsguy.com

  The same method works for any tier. Only the "requests per server" number changes, and you should measure it with a load test rather than guess.

  Choosing a balancing method with real numbers

  Round robin is fair in the number of requests, not in the work. Imagine three servers and a stream where most requests take 10 ms but every fifth takes 1 second (a report export, say). Round robin gives each server the same count, but by chance one server collects several slow requests at once, its queue grows, and its users wait while the other two are nearly idle.

  Least connections notices: the server with the most open requests gets fewer new ones, so the slow requests spread out as they arrive. It costs the balancer a little bookkeeping and helps most when request costs vary a lot. Use plain round robin when requests are similar and servers identical; use least connections, or "power of two choices" (pick two servers at random and send to the less busy one), when they are not. For caches and stateful workloads, consistent hashing keeps the same key on the same server so its memory stays useful.

  Health checks and draining

  A balancer is only as good as its knowledge of which servers are alive.

  • Active health checks call a lightweight endpoint (for example /health) every few seconds. After two or three failures in a row, the server leaves rotation; after a few successes it returns. Make the check reflect real readiness: a server that is up but cannot reach its database should fail the check.
  • Passive checks watch real traffic: too many errors or timeouts from one server and it is ejected.
  • Connection draining matters at deploys. Before stopping a server, tell the balancer to send it no new requests, let in-flight requests finish (typically 30 to 60 seconds), then stop it. Without draining, every deploy drops a few user requests.

  100%

  Figure SDA-FND-02-5 · Rolling out a new version without dropping requests · © forcsguy.com

  Mistakes people make

  • Treating the balancer as magic. It is a server too and can be a single point of failure. Run at least two, or use a managed one.
  • Adding servers behind a single database. App servers scale out easily; if the database is the bottleneck, more app servers only make it fail faster.
  • Keeping state on the server. Sessions, uploads and caches held in one server's memory break the moment a request lands elsewhere. Move them to a shared store.
  • Sizing for the average. The system fails at peak, which is when it matters.
  • Never testing failure. Kill a server during a load test. If users notice, you have found a gap before they did.

  Try it yourself

  Open the System Builder challenge "One server is not enough". Run the test on one server and watch it fail, add a load balancer without adding servers, then add servers and see where the numbers settle. Or skip the challenge and open a load-balanced system on the free canvas, then change the wiring yourself.

  More on this site

  Related reading

  • Maglev: A Fast and Reliable Software Network Load Balancer (paper)Eisenbud et al. · GooglePapers
  • NGINX HTTP Load BalancingNGINX · F5 / NGINXDocumentation
  • Google SRE Book: Load Balancing in the DatacenterGoogle SRE team · GoogleBlogs
  • Scaling Load Balancing Infra to Support 1.3 Billion Users at Facebook - Patrick Shuff, Production Engineer at FacebookFacebookYouTube

  Show 6 moreShow fewer

  • Load Balancing infrastructure to support more than 1.3 billion users at FacebookFacebook engineers · FacebookCase Studies
  • Implementing health checksAmazon Builders' Library · Amazon Web ServicesBlogs
  • Awesome System Design ResourcesAshish Pratap Singh · ashishps1GitHub
  • Load Balancing at GithubGithub engineers · GithubCase Studies
  • Google Cloud Architecture FrameworkGoogle · Google CloudDocumentation
  • HAProxyHAProxy Technologies · HAProxyDocumentation

  Open this lesson with its quiz and exercise

• Classic System Designs · 13 minDesign a video streaming serviceUpload, process and stream video to millions: transcoding pipelines, adaptive bitrate and using a CDN without going bankrupt.Read hereClose

  The problem

  A video streaming service lets users upload videos and lets many others watch them smoothly on any device and connection.

  Requirements

  • Upload large files reliably.
  • Playback starts quickly and does not buffer.
  • Works on phones, TVs and browsers, on fast and slow networks.
  • Huge scale, with traffic dominated by viewing, not uploading.
  • Keep cost under control.

  Estimates that shape the design

  • Video files are big: a few GB uploaded, then several versions stored.
  • Views outnumber uploads by orders of magnitude, and bandwidth is the main bill.

  High-level design

  Two separate flows: upload and process, and stream.

  100%

  Figure SDA-CSD-10-1 · Upload-and-process and streaming flows · © forcsguy.com

  Upload

  • The client asks the API for an upload URL and sends the file directly to object storage, in chunks that can resume after a network drop. Your application servers never carry the video bytes.
  • When the upload finishes, storage emits an event that starts processing.

  Processing pipeline

  Raw video cannot be streamed to everyone. The pipeline:

  1. Splits the video into short segments, so they can be processed in parallel.
  2. Transcodes each into several resolutions and bitrates (for example 240p to 4K) and several formats and codecs.
  3. Creates thumbnails, subtitles, and sometimes watermarks.
  4. Runs checks: content moderation, copyright matching, virus scanning.
  5. Writes a manifest listing the available versions.
  6. Updates the metadata (title, status) so the video becomes visible.

  Model it as a graph of tasks run by a workflow engine over a queue and a pool of workers, so steps parallelise, retry independently and scale with load.

  100%

  Figure SDA-CSD-10-2 · Transcoding task graph · © forcsguy.com

  Inside the transcoding pipeline

  Transcoding is heavy and varies by creator (some want watermarks, some supply thumbnails, some upload very high resolution). So define the work as a task graph (a DAG): the original is split into video, audio and metadata; the video stage fans out into encodings at several resolutions and bitrates, a thumbnail and optionally a watermark; the audio stage encodes separately; a final stage merges. Independent branches run in parallel.

  The parts that make it run:

  • Pre-processor splits the video into short independently playable chunks (aligned to groups of pictures), builds the graph from configuration, and keeps the chunks in temporary storage so a failed step can retry without starting over.
  • Graph scheduler cuts the graph into stages and puts the tasks on a queue.
  • Resource manager keeps a priority queue of tasks, a priority queue of worker availability and a list of what is running, and pairs the most important task with the best-suited worker.
  • Workers run specific tasks; temporary storage holds metadata in memory and media in blob storage and is cleaned when the job ends.

  Two vocabulary points: a container (.mp4, .mov) is the wrapper holding video, audio and metadata; a codec (H.264, VP9, HEVC) compresses the content. Raw footage can be hundreds of GB per hour, which is the first reason to transcode; the others are device compatibility and adapting quality to bandwidth.

  Cloud services instead of building everything

  100%

  Figure SDA-CSD-10-3 · Two paths: video and everything else · © forcsguy.com

  100%

  Figure SDA-CSD-10-4 · Upload flow with completion events · © forcsguy.com

  100%

  Figure SDA-CSD-10-5 · Metadata is updated in parallel with the upload · © forcsguy.com

  Streaming

  • The player downloads the manifest, then fetches small segments over plain HTTP (the HLS and MPEG-DASH formats work this way).
  • Adaptive bitrate: the player measures its throughput and switches between quality levels segment by segment, so playback adapts to the connection instead of stopping.
  • Segments are static files, ideal for CDN caching.

  100%

  Figure SDA-CSD-10-6 · One video, many encodings · © forcsguy.com

  100%

  Figure SDA-CSD-10-7 · Popular videos from the CDN, the rest from storage servers · © forcsguy.com

  The CDN and cost

  CDN traffic is expensive. Practical tactics:

  • Serve popular videos from edge caches; keep the long tail only at the origin or in a regional cache.
  • Pre-position new releases that you expect to be popular.
  • Choose bitrates carefully, and use efficient codecs for popular content (extra encoding cost is repaid by lower bandwidth).
  • Large services run their own cache servers inside ISPs to cut transit cost.

  Before: serve from the origin

  100%

  After: serve from a CDN

  100%

  What changed

  • Viewers stream from a nearby edge, so less buffering
  • The origin handles only cache misses
  • Bandwidth cost falls because edges absorb the traffic

  Metadata and other parts

  • A database for video metadata, ownership, comments and counts, with caches for the hot reads.
  • Search and recommendations as separate services fed by events.
  • View counts are written to a queue and aggregated, not updated row by row on every view.
  • Security: signed URLs for private videos, and DRM for premium content.

  Speed, safety and cost, in detail

  • Parallel and resumable uploads: the client cuts the file into chunks and uploads them independently, so a drop resumes at the last chunk. Put upload points near users (a CDN works as an upload edge).
  • Loose coupling: a queue between download, encode and publish steps lets each stage work as soon as items arrive instead of waiting for the previous stage to finish a whole file.
  • Pre-signed upload URLs: the API issues a short-lived address that authorises one upload to one location, so clients never hold storage credentials.
  • Protecting content: DRM, encrypted streams with an authorisation check at playback, and visible watermarks.
  • Cost: a rough sum shows CDN delivery alone could cost on the order of a hundred thousand dollars a day at this scale, and views follow a long tail. So serve only the popular videos from the CDN, serve the rest from your own storage servers, keep fewer encoded versions of rarely watched videos (encode short ones on demand), keep regional favourites in their region, and at very large scale partner with internet providers or run your own delivery network. Always check your own viewing data first.

  Errors. Separate recoverable errors (a chunk fails to encode: retry a few times) from non-recoverable ones (malformed file: stop the job and return a clear error). A playbook per component: retry uploads and encodes; if an old client cannot chunk, chunk on the server; regenerate a graph or reschedule a task if the pre-processor or scheduler fails; use replicas for queues, caches and databases; promote a replica if a primary database fails.

  Live streaming differs in needing lower latency (a different protocol), less parallelism (chunks are small and processed as they arrive) and error handling that cannot afford long retries. Takedowns need both upload-time checks and user flagging.

  100%

  Figure SDA-CSD-10-8 · Chunked, resumable and regional uploads · © forcsguy.com

  100%

  Figure SDA-CSD-10-9 · Pre-signed upload URL · © forcsguy.com

  Failure handling

  Every processing task is idempotent and tracked, so a failed or retried segment does not corrupt the output. Failures go to a dead-letter queue and set a visible error status. Use multiple regions for storage and CDN origins so viewing survives a regional outage.

  Worked numbers: why a CDN is not optional

  Suppose 5 million viewers are watching at the same time, each at an average of 3 Mbit/s.

  • Egress: 5,000,000 x 3 Mbit/s = 15 Tbit/s. No single data centre network, and no sensible origin, can send that. Video is served from edge caches close to viewers; the origin only sends each popular segment to each edge once.
  • Why caching works so well: video segments are the same files for everyone watching, and demand is highly skewed. A popular show may be 90%+ cache hits at the edge, so the origin sees a small fraction of the traffic.
  • Upload side: if 1 million videos are uploaded a day at an average of 300 MB, that is 300 TB a day of originals, before the several encoded versions of each (often 5 to 8 renditions, which together can take as much space as the original or more). The long tail of rarely watched videos is the main storage cost, which is why some systems encode extra renditions only when a video becomes popular.

  100%

  Figure SDA-CSD-10-10 · Fast transcoding by splitting a video into segments · © forcsguy.com

  How the player picks the quality

  Streaming uses adaptive bitrate (ABR): the video exists in several renditions (for example 240p to 1080p), each cut into the same short segments, and the player decides per segment which one to request.

  1. The player downloads the playlist (manifest) listing the renditions and segments.
  2. It starts with a conservative rendition so playback begins quickly.
  3. For each segment it measures how fast the download went and how full its buffer is.
  4. If the buffer is healthy and the network fast, it steps up a rendition; if the buffer is draining, it steps down before the video stalls.

  Because every rendition has identical segment boundaries, switching is seamless. Viewers see a slightly softer picture instead of a spinning wheel.

  100%

  Figure SDA-CSD-10-11 · The player adapts to the network segment by segment · © forcsguy.com

  Mistakes people make

  • Encoding every rendition for every upload. Most videos are barely watched. Encode a base set first and add more when a video proves popular.
  • Serving from the origin. Without a CDN the origin saturates and far-away viewers buffer.
  • Cold caches on a launch. A new popular release hits empty edges. Pre-warm the CDN for expected hits.
  • Uploads that cannot resume. Large files over mobile networks fail often; use chunked, resumable uploads.
  • Forgetting access control. Public segment URLs are easily shared. Use signed, expiring URLs, and DRM where licences require it.
  • Ignoring start-up time. Viewers leave if playback takes more than a couple of seconds. Short first segments, a good first rendition and a nearby edge matter more than peak quality.

  Try it yourself

  Open the System Builder's viral link challenge to see how much traffic a CDN takes off your servers, and the WebRTC lesson for when low-latency live video needs a different approach than segment-based streaming.

  More on this site

  Related reading

  • HTTP/3: the past, the present, and the futureCloudflareBlogs
  • Cloudflare BlogCloudflare engineers · CloudflareBlogs
  • The Road to QUICCloudflareBlogs
  • Cloudflare EngineeringCloudflare engineers · CloudflareBlogs

  Show 6 moreShow fewer

  • High Performance Browser NetworkingIlya Grigorik · O'ReillyBooks
  • MDN: HTTP cachingMDN contributors · MozillaDocumentation
  • Meta Engineering BlogMeta Engineering · MetaBlogs
  • Flannel: An Application-Level Edge Cache to Make Slack ScaleSlack Engineering · SlackCase Studies
  • Netflix ResearchNetflix engineers · NetflixBlogs
  • Nvidia Engineering BlogNvidia engineers · NvidiaBlogs

  Open this lesson with its quiz and exercise