Natural Language Processing and Content Analysis (2 parts) at Condé Nast

Ideas you will meet

Distributed Systems

Consensus, time, failure and the fundamentals of scale-out.

Machine Learning Systems

Recommendations, ranking, feature stores and serving models at scale.

What we already know about this

Our own lessons on these ideas. Read them here before you start, so it makes sense.

• Core Concepts: How Things Actually Work · 6 minLatency, throughput and bandwidthThree numbers that are often confused, and why a fast network can still feel slow.Read hereClose

  Three different questions

  • Latency: how long does one request take from start to finish? Measured in milliseconds.
  • Bandwidth: how much could the pipe carry per second? A capacity, measured in bits per second.
  • Throughput: how much actually flows per second, measured in requests or bytes per second. It is at most the bandwidth, and often much less.

  A motorway analogy: bandwidth is the number of lanes, latency is the time one car needs to drive the length, and throughput is how many cars arrive per hour. Widening the road (more lanes) lets more cars through but does not make any single car arrive sooner.

  Why it matters in design

  Many speed problems are really latency problems, and more bandwidth will not fix them:

  Symptom	Likely cause	Lever
  Page makes 80 small requests and feels slow	Round-trip latency adds up	Fewer requests, bundling, HTTP/2, caching
  Users far away complain	Distance	CDN, regional servers
  Big downloads are slow	Throughput or bandwidth	Compression, parallel transfers, bigger pipe
  System is slow only at busy times	Throughput limit	Scale out, queue, shed load

  Image: Syced · CC0 · Source · Wikimedia Commons: Peabird Ethernet switch in 2003

  Where the time goes

  100%

  Figure SDA-CORE-01-1 · A latency budget for one page · © forcsguy.com

  Tail latency

  Averages hide the worst experiences. If 1% of requests take 3 seconds, a page that makes 50 calls will hit one of those slow calls most of the time. Track percentiles (p95, p99), and design to keep the slowest requests under control with timeouts and hedged requests.

  Latency budget

  Break a target (say 300 ms for a page) into a budget: network 80 ms, application 60 ms, database 50 ms, rendering 80 ms, slack 30 ms. When the total exceeds the target you know where to cut.

  Real-world reading

  Related reading

  • Optimizing Web Servers for High Throughput and Low Latency at DropboxDropbox engineers · DropboxCase Studies
  • Natural Language Processing and Content Analysis (2 parts) at Condé NastCondé Nast engineers · Condé NastCase Studies
  • Finding Kafka throughput limit in infrastructure at DropboxDropbox engineers · DropboxCase Studies
  • Scaling Gradient Boosted Trees for Click-Through-Rate Prediction at YelpYelp engineers · YelpCase Studies

  Show 6 moreShow fewer

  • Machine Learning (2 parts) at Condé NastCondé Nast engineers · Condé NastCase Studies
  • DataSift Architecture: Realtime Datamining at 120,000 Tweets Per Second - High Scalability -High Scalability engineers · High ScalabilityCase Studies
  • Deploying Brotli for Static Content at DropboxDropbox engineers · DropboxCase Studies
  • How Twitter Handles 3,000 Images Per Second - High Scalability -High Scalability engineers · High ScalabilityCase Studies
  • ESPN's Architecture at Scale - Operating at 100,000 Duh Nuh Nuhs Per Second - High Scalability -High Scalability engineers · High ScalabilityCase Studies
  • Architecture NotesArchitecture Notes · architecturenotes.coPodcasts & Newsletters

  Open this lesson with its quiz and exercise

• Real-World Systems · 9 minUploading large files reliablyChunked, resumable, direct-to-storage uploads: how to move gigabytes over unreliable networks without tying up your servers.Read hereClose

  Why large uploads are hard

  A 5 GB file on a normal request has many ways to fail:

  • A single long connection dies on a flaky network, losing everything.
  • Timeouts in load balancers and proxies cut off long requests.
  • Your application servers are tied up carrying bytes, and memory or disk fills.
  • A failure at 99% means starting again.
  • Users want a progress bar and the ability to pause and resume.

  The design in one picture

  100%

  Figure SDA-RWS-17-1 · Chunked, resumable upload flow · © forcsguy.com

  Before: upload through your server

  100%

  After: upload straight to storage

  100%

  What changed

  • Your servers no longer carry large files
  • A failed chunk is retried, not the whole file
  • Pre-signed URLs keep storage private

  Step by step

  1. Initiate. The client tells the API the file name, size and type. The API checks permissions and limits, and creates an upload session with a unique id.
  2. Direct to storage. The API returns pre-signed URLs: short-lived, scoped links that let the client upload straight to object storage. Your servers never carry the file.
  3. Chunk. The client splits the file into parts (for example 5 to 100 MB; object stores typically accept thousands of parts per object) and uploads them, in parallel.
  4. Verify. Each part carries a checksum (MD5, CRC32C or SHA-256). The storage service or your server checks it, and a bad part is simply re-sent.
  5. Resume. The session remembers which parts are done. After a drop the client asks which parts exist and sends only the missing ones. Open protocols such as tus standardise resumable uploads.
  6. Complete. The client sends the list of parts; storage assembles the object.
  7. Post-process asynchronously. An event triggers virus scanning, type verification, thumbnails or transcoding, and metadata updates. The user sees "processing" until it finishes.

  Checking each part

  When a part arrives, the store replies with a checksum tag. The client keeps the list of part numbers and tags, and sends it with the final "complete" request, so the store can verify and assemble exactly the parts you meant, in order. A failed part is re-sent alone; assembly of a very large object can take a few minutes, so treat "complete" as asynchronous from the user's point of view.

  Smart extras

  • Deduplication: hash the file (or chunks) on the client first. If the content already exists in storage, skip the upload entirely and just add a reference (as in cloud file storage services).
  • Parallelism and adaptive chunk size: larger parts on fast networks, smaller on poor ones.
  • Compression for compressible data before upload.
  • Upload acceleration: send to the nearest edge location, which forwards over the provider's backbone.
  • Progress and cancellation: the client tracks bytes per part; cancelling aborts the session.

  Safety and cost

  • Limit size, type and number of concurrent uploads per user, and expire pre-signed URLs quickly.
  • Validate on the server, not only in the client: check file type by content, scan for malware, and store uploads in a quarantine area until cleared.
  • Use lifecycle rules to delete incomplete multipart uploads after a few days, since unfinished parts cost money until removed.
  • Encrypt in transit and at rest; do not let user-provided names choose storage paths.
  • Rate limit session creation to prevent abuse.

  Failure handling checklist

  Failure	Handling
  Connection drops mid-part	Retry that part with backoff
  Client closes the app	Session persists; resume later by listing stored parts
  Corrupted part	Checksum mismatch, re-upload that part
  Duplicate completion request	Idempotent: completing twice returns the same result
  Abandoned upload	Lifecycle rule cleans up the leftovers
  Processing fails	Retry from the event queue, mark the file as failed, tell the user

  Real-world reading

  Related reading

  • Determinism (4 parts) in League of Legends ServerRiotgames engineers · RiotgamesCase Studies
  • Implementing health checksAmazon Builders' Library · Amazon Web ServicesBlogs
  • The Network is ReliableKyle Kingsbury · aphyr.comBlogs
  • Bulkheads: Partition and Tolerate Failure in One PartSkife engineers · SkifeCase Studies

  Show 6 moreShow fewer

  • Natural Language Processing and Content Analysis (2 parts) at Condé NastCondé Nast engineers · Condé NastCase Studies
  • Autoscaling Bigtable Clusters based on CPU Load at SpotifySpotify engineers · SpotifyCase Studies
  • Ambry: Distributed Immutable Object Store at LinkedInLinkedIn engineers · LinkedInCase Studies
  • Big Data Processing (2 parts) at SpotifySpotify engineers · SpotifyCase Studies
  • Container (8 parts) at Riot GamesRiot Games engineers · Riot GamesCase Studies
  • Architecture of Chat Service (3 parts) at Riot GamesRiot Games engineers · Riot GamesCase Studies

  Open this lesson with its quiz and exercise

• Real-World Systems · 11 minDesign an AI coding assistantHow an editor assistant suggests code in under a second: gathering context, retrieval, the model, streaming and keeping private code safe.Read hereClose

  The problem

  An assistant inside the editor that completes code as you type, answers questions in a chat panel, and can edit files. The hard constraints:

  • Latency: inline suggestions must appear in roughly a few hundred milliseconds, or the user has already moved on.
  • Relevance: good suggestions need the right context from a large codebase, not just the current line.
  • Cost: language-model inference is expensive per request, and typing generates a request storm.
  • Privacy and security: customer source code and secrets are sensitive.
  • Quality: wrong or insecure suggestions must be limited and measurable.

  Architecture

  100%

  Figure SDA-RWS-18-1 · AI coding assistant architecture · © forcsguy.com

  1. The editor extension

  Runs in the user's editor. It watches typing, debounces requests (waits a few tens of milliseconds), cancels outdated requests when the user continues typing, and streams the suggestion into the editor as ghost text. It also handles accept, reject and partial accept, and enforces local privacy rules (excluded files, secret detection).

  2. Building context

  Prompt = instructions + context + the code around the cursor. Context comes from:

  • The current file: the code before the cursor (prefix) and after it (suffix, for fill-in-the-middle).
  • Open tabs and recently edited files.
  • Imports and related files: definitions of functions and types being used.
  • Retrieved snippets from the wider project.

  All of it must fit the model's context window and a token budget, so the builder ranks pieces by relevance and trims.

  3. Retrieval over a codebase

  A repository is too big to include whole, so it is indexed:

  1. Split files into chunks (functions, classes, blocks).
  2. Compute embeddings (vectors capturing meaning) and store them in a vector index, together with a symbol and dependency graph (what calls what).
  3. At request time, search with the current code as the query and combine semantic results with structural ones (definitions, callers, tests).
  4. Update the index incrementally as files change, and keep per-user and per-organisation isolation.

  4. Model serving

  • A language model generates tokens. Serving systems batch many requests together, reuse a KV cache of already-processed prompt tokens, and stream output so the first words show immediately.
  • Use smaller, faster models for inline completion and larger ones for chat and complex edits.
  • Cache repeated prompts and completions where context is identical, and prefix-cache shared prompt parts.
  • Autoscale GPU capacity, and route by region for latency.

  5. Post-processing and safety

  • Trim the output to a sensible stopping point (end of block or line).
  • Filter secrets, personal data and known insecure patterns; optionally check for verbatim public code matches and licence concerns.
  • Reject low-confidence suggestions instead of showing noise.
  • For agents that edit files or run commands, require explicit user approval and run actions in a sandbox.

  100%

  Figure SDA-RWS-18-2 · How a code model is trained and judged · © forcsguy.com

  Latency budget

  For inline suggestions, a budget like this keeps the experience smooth:

  Step	Target
  Debounce and local context	about 30 ms
  Network to the service	about 30 ms
  Retrieval and prompt building	about 30 ms
  Model: time to first token	about 150 ms
  Streaming and display	continuous

  Everything else (retrieval, indexing, safety checks) must fit inside it or run asynchronously ahead of time.

  Privacy, security and trust

  • Send only what is needed, respect ignore files, and redact secrets.
  • Provide enterprise controls: private deployment or isolated tenancy, no training on customer code, short retention, audit logs.
  • Encrypt in transit and at rest; strong authentication and per-user quotas.
  • Defend against prompt injection (malicious text in files or web pages trying to instruct the model): treat retrieved content as data, limit tool permissions, require confirmation for risky actions.

  Measuring quality

  • Offline evaluation: run the model on held-out code and tests, measure pass rates and exact-match similarity.
  • Online metrics: acceptance rate of suggestions, characters kept after acceptance, latency percentiles, and errors.
  • A/B test changes to prompts, models and retrieval, and watch for regressions in security and licence filters.
  • Collect feedback signals (thumbs up or down) while respecting privacy.

  What this design reuses

  Debouncing and cancellation, caching, rate limiting and quotas, queues and batching, vector search, streaming responses and careful privacy controls: the same building blocks from the rest of this course, applied to a latency-sensitive AI workload.

  Real-world reading

  Related reading

  • Secret Detector: Preventing Secrets in Source Code at YelpYelp engineers · YelpCase Studies
  • Samza: Stream Processing System for Latency Insighs at LinkedInLinkedIn engineers · LinkedInCase Studies
  • Use The Index, LukeMarkus Winand · use-the-index-luke.comCourses
  • Backend Developer Roadmaproadmap.sh community · roadmap.shCourses

  Show 6 moreShow fewer

  • Natural Language Processing and Content Analysis (2 parts) at Condé NastCondé Nast engineers · Condé NastCase Studies
  • Machine Learning for Indexing Text from Billions of Images at DropboxDropbox engineers · DropboxCase Studies
  • Optimizing Web Servers for High Throughput and Low Latency at DropboxDropbox engineers · DropboxCase Studies
  • Scalable User Privacy at SpotifySpotify engineers · SpotifyCase Studies
  • Privacy-Preserving Analytics and Reporting at LinkedInLinkedIn engineers · LinkedInCase Studies
  • Learning with Privacy at Scale at AppleScale at Apple engineers · Scale at AppleCase Studies

  Open this lesson with its quiz and exercise