Erlang Solutions Engineering Blog

Ideas you will meet

Distributed Systems

Consensus, time, failure and the fundamentals of scale-out.

What we already know about this

Our own lessons on these ideas. Read them here before you start, so it makes sense.

• Classic System Designs · 13 minDesign a video streaming serviceUpload, process and stream video to millions: transcoding pipelines, adaptive bitrate and using a CDN without going bankrupt.Read hereClose

  The problem

  A video streaming service lets users upload videos and lets many others watch them smoothly on any device and connection.

  Requirements

  • Upload large files reliably.
  • Playback starts quickly and does not buffer.
  • Works on phones, TVs and browsers, on fast and slow networks.
  • Huge scale, with traffic dominated by viewing, not uploading.
  • Keep cost under control.

  Estimates that shape the design

  • Video files are big: a few GB uploaded, then several versions stored.
  • Views outnumber uploads by orders of magnitude, and bandwidth is the main bill.

  High-level design

  Two separate flows: upload and process, and stream.

  100%

  Figure SDA-CSD-10-1 · Upload-and-process and streaming flows · © forcsguy.com

  Upload

  • The client asks the API for an upload URL and sends the file directly to object storage, in chunks that can resume after a network drop. Your application servers never carry the video bytes.
  • When the upload finishes, storage emits an event that starts processing.

  Processing pipeline

  Raw video cannot be streamed to everyone. The pipeline:

  1. Splits the video into short segments, so they can be processed in parallel.
  2. Transcodes each into several resolutions and bitrates (for example 240p to 4K) and several formats and codecs.
  3. Creates thumbnails, subtitles, and sometimes watermarks.
  4. Runs checks: content moderation, copyright matching, virus scanning.
  5. Writes a manifest listing the available versions.
  6. Updates the metadata (title, status) so the video becomes visible.

  Model it as a graph of tasks run by a workflow engine over a queue and a pool of workers, so steps parallelise, retry independently and scale with load.

  100%

  Figure SDA-CSD-10-2 · Transcoding task graph · © forcsguy.com

  Inside the transcoding pipeline

  Transcoding is heavy and varies by creator (some want watermarks, some supply thumbnails, some upload very high resolution). So define the work as a task graph (a DAG): the original is split into video, audio and metadata; the video stage fans out into encodings at several resolutions and bitrates, a thumbnail and optionally a watermark; the audio stage encodes separately; a final stage merges. Independent branches run in parallel.

  The parts that make it run:

  • Pre-processor splits the video into short independently playable chunks (aligned to groups of pictures), builds the graph from configuration, and keeps the chunks in temporary storage so a failed step can retry without starting over.
  • Graph scheduler cuts the graph into stages and puts the tasks on a queue.
  • Resource manager keeps a priority queue of tasks, a priority queue of worker availability and a list of what is running, and pairs the most important task with the best-suited worker.
  • Workers run specific tasks; temporary storage holds metadata in memory and media in blob storage and is cleaned when the job ends.

  Two vocabulary points: a container (.mp4, .mov) is the wrapper holding video, audio and metadata; a codec (H.264, VP9, HEVC) compresses the content. Raw footage can be hundreds of GB per hour, which is the first reason to transcode; the others are device compatibility and adapting quality to bandwidth.

  Cloud services instead of building everything

  100%

  Figure SDA-CSD-10-3 · Two paths: video and everything else · © forcsguy.com

  100%

  Figure SDA-CSD-10-4 · Upload flow with completion events · © forcsguy.com

  100%

  Figure SDA-CSD-10-5 · Metadata is updated in parallel with the upload · © forcsguy.com

  Streaming

  • The player downloads the manifest, then fetches small segments over plain HTTP (the HLS and MPEG-DASH formats work this way).
  • Adaptive bitrate: the player measures its throughput and switches between quality levels segment by segment, so playback adapts to the connection instead of stopping.
  • Segments are static files, ideal for CDN caching.

  100%

  Figure SDA-CSD-10-6 · One video, many encodings · © forcsguy.com

  100%

  Figure SDA-CSD-10-7 · Popular videos from the CDN, the rest from storage servers · © forcsguy.com

  The CDN and cost

  CDN traffic is expensive. Practical tactics:

  • Serve popular videos from edge caches; keep the long tail only at the origin or in a regional cache.
  • Pre-position new releases that you expect to be popular.
  • Choose bitrates carefully, and use efficient codecs for popular content (extra encoding cost is repaid by lower bandwidth).
  • Large services run their own cache servers inside ISPs to cut transit cost.

  Before: serve from the origin

  100%

  After: serve from a CDN

  100%

  What changed

  • Viewers stream from a nearby edge, so less buffering
  • The origin handles only cache misses
  • Bandwidth cost falls because edges absorb the traffic

  Metadata and other parts

  • A database for video metadata, ownership, comments and counts, with caches for the hot reads.
  • Search and recommendations as separate services fed by events.
  • View counts are written to a queue and aggregated, not updated row by row on every view.
  • Security: signed URLs for private videos, and DRM for premium content.

  Speed, safety and cost, in detail

  • Parallel and resumable uploads: the client cuts the file into chunks and uploads them independently, so a drop resumes at the last chunk. Put upload points near users (a CDN works as an upload edge).
  • Loose coupling: a queue between download, encode and publish steps lets each stage work as soon as items arrive instead of waiting for the previous stage to finish a whole file.
  • Pre-signed upload URLs: the API issues a short-lived address that authorises one upload to one location, so clients never hold storage credentials.
  • Protecting content: DRM, encrypted streams with an authorisation check at playback, and visible watermarks.
  • Cost: a rough sum shows CDN delivery alone could cost on the order of a hundred thousand dollars a day at this scale, and views follow a long tail. So serve only the popular videos from the CDN, serve the rest from your own storage servers, keep fewer encoded versions of rarely watched videos (encode short ones on demand), keep regional favourites in their region, and at very large scale partner with internet providers or run your own delivery network. Always check your own viewing data first.

  Errors. Separate recoverable errors (a chunk fails to encode: retry a few times) from non-recoverable ones (malformed file: stop the job and return a clear error). A playbook per component: retry uploads and encodes; if an old client cannot chunk, chunk on the server; regenerate a graph or reschedule a task if the pre-processor or scheduler fails; use replicas for queues, caches and databases; promote a replica if a primary database fails.

  Live streaming differs in needing lower latency (a different protocol), less parallelism (chunks are small and processed as they arrive) and error handling that cannot afford long retries. Takedowns need both upload-time checks and user flagging.

  100%

  Figure SDA-CSD-10-8 · Chunked, resumable and regional uploads · © forcsguy.com

  100%

  Figure SDA-CSD-10-9 · Pre-signed upload URL · © forcsguy.com

  Failure handling

  Every processing task is idempotent and tracked, so a failed or retried segment does not corrupt the output. Failures go to a dead-letter queue and set a visible error status. Use multiple regions for storage and CDN origins so viewing survives a regional outage.

  Worked numbers: why a CDN is not optional

  Suppose 5 million viewers are watching at the same time, each at an average of 3 Mbit/s.

  • Egress: 5,000,000 x 3 Mbit/s = 15 Tbit/s. No single data centre network, and no sensible origin, can send that. Video is served from edge caches close to viewers; the origin only sends each popular segment to each edge once.
  • Why caching works so well: video segments are the same files for everyone watching, and demand is highly skewed. A popular show may be 90%+ cache hits at the edge, so the origin sees a small fraction of the traffic.
  • Upload side: if 1 million videos are uploaded a day at an average of 300 MB, that is 300 TB a day of originals, before the several encoded versions of each (often 5 to 8 renditions, which together can take as much space as the original or more). The long tail of rarely watched videos is the main storage cost, which is why some systems encode extra renditions only when a video becomes popular.

  100%

  Figure SDA-CSD-10-10 · Fast transcoding by splitting a video into segments · © forcsguy.com

  How the player picks the quality

  Streaming uses adaptive bitrate (ABR): the video exists in several renditions (for example 240p to 1080p), each cut into the same short segments, and the player decides per segment which one to request.

  1. The player downloads the playlist (manifest) listing the renditions and segments.
  2. It starts with a conservative rendition so playback begins quickly.
  3. For each segment it measures how fast the download went and how full its buffer is.
  4. If the buffer is healthy and the network fast, it steps up a rendition; if the buffer is draining, it steps down before the video stalls.

  Because every rendition has identical segment boundaries, switching is seamless. Viewers see a slightly softer picture instead of a spinning wheel.

  100%

  Figure SDA-CSD-10-11 · The player adapts to the network segment by segment · © forcsguy.com

  Mistakes people make

  • Encoding every rendition for every upload. Most videos are barely watched. Encode a base set first and add more when a video proves popular.
  • Serving from the origin. Without a CDN the origin saturates and far-away viewers buffer.
  • Cold caches on a launch. A new popular release hits empty edges. Pre-warm the CDN for expected hits.
  • Uploads that cannot resume. Large files over mobile networks fail often; use chunked, resumable uploads.
  • Forgetting access control. Public segment URLs are easily shared. Use signed, expiring URLs, and DRM where licences require it.
  • Ignoring start-up time. Viewers leave if playback takes more than a couple of seconds. Short first segments, a good first rendition and a nearby edge matter more than peak quality.

  Try it yourself

  Open the System Builder's viral link challenge to see how much traffic a CDN takes off your servers, and the WebRTC lesson for when low-latency live video needs a different approach than segment-based streaming.

  More on this site

  Related reading

  • HTTP/3: the past, the present, and the futureCloudflareBlogs
  • Cloudflare BlogCloudflare engineers · CloudflareBlogs
  • The Road to QUICCloudflareBlogs
  • Cloudflare EngineeringCloudflare engineers · CloudflareBlogs

  Show 6 moreShow fewer

  • High Performance Browser NetworkingIlya Grigorik · O'ReillyBooks
  • MDN: HTTP cachingMDN contributors · MozillaDocumentation
  • Meta Engineering BlogMeta Engineering · MetaBlogs
  • Flannel: An Application-Level Edge Cache to Make Slack ScaleSlack Engineering · SlackCase Studies
  • Netflix ResearchNetflix engineers · NetflixBlogs
  • Nvidia Engineering BlogNvidia engineers · NvidiaBlogs

  Open this lesson with its quiz and exercise