Scalable User Privacy at Spotify

Ideas you will meet

Distributed Systems

Consensus, time, failure and the fundamentals of scale-out.

Web Security

OWASP risks, XSS, CSRF, CORS and secure design.

What we already know about this

Our own lessons on these ideas. Read them here before you start, so it makes sense.

• Real-World Systems · 11 minDesign an AI coding assistantHow an editor assistant suggests code in under a second: gathering context, retrieval, the model, streaming and keeping private code safe.Read hereClose

  The problem

  An assistant inside the editor that completes code as you type, answers questions in a chat panel, and can edit files. The hard constraints:

  • Latency: inline suggestions must appear in roughly a few hundred milliseconds, or the user has already moved on.
  • Relevance: good suggestions need the right context from a large codebase, not just the current line.
  • Cost: language-model inference is expensive per request, and typing generates a request storm.
  • Privacy and security: customer source code and secrets are sensitive.
  • Quality: wrong or insecure suggestions must be limited and measurable.

  Architecture

  100%

  Figure SDA-RWS-18-1 · AI coding assistant architecture · © forcsguy.com

  1. The editor extension

  Runs in the user's editor. It watches typing, debounces requests (waits a few tens of milliseconds), cancels outdated requests when the user continues typing, and streams the suggestion into the editor as ghost text. It also handles accept, reject and partial accept, and enforces local privacy rules (excluded files, secret detection).

  2. Building context

  Prompt = instructions + context + the code around the cursor. Context comes from:

  • The current file: the code before the cursor (prefix) and after it (suffix, for fill-in-the-middle).
  • Open tabs and recently edited files.
  • Imports and related files: definitions of functions and types being used.
  • Retrieved snippets from the wider project.

  All of it must fit the model's context window and a token budget, so the builder ranks pieces by relevance and trims.

  3. Retrieval over a codebase

  A repository is too big to include whole, so it is indexed:

  1. Split files into chunks (functions, classes, blocks).
  2. Compute embeddings (vectors capturing meaning) and store them in a vector index, together with a symbol and dependency graph (what calls what).
  3. At request time, search with the current code as the query and combine semantic results with structural ones (definitions, callers, tests).
  4. Update the index incrementally as files change, and keep per-user and per-organisation isolation.

  4. Model serving

  • A language model generates tokens. Serving systems batch many requests together, reuse a KV cache of already-processed prompt tokens, and stream output so the first words show immediately.
  • Use smaller, faster models for inline completion and larger ones for chat and complex edits.
  • Cache repeated prompts and completions where context is identical, and prefix-cache shared prompt parts.
  • Autoscale GPU capacity, and route by region for latency.

  5. Post-processing and safety

  • Trim the output to a sensible stopping point (end of block or line).
  • Filter secrets, personal data and known insecure patterns; optionally check for verbatim public code matches and licence concerns.
  • Reject low-confidence suggestions instead of showing noise.
  • For agents that edit files or run commands, require explicit user approval and run actions in a sandbox.

  100%

  Figure SDA-RWS-18-2 · How a code model is trained and judged · © forcsguy.com

  Latency budget

  For inline suggestions, a budget like this keeps the experience smooth:

  Step	Target
  Debounce and local context	about 30 ms
  Network to the service	about 30 ms
  Retrieval and prompt building	about 30 ms
  Model: time to first token	about 150 ms
  Streaming and display	continuous

  Everything else (retrieval, indexing, safety checks) must fit inside it or run asynchronously ahead of time.

  Privacy, security and trust

  • Send only what is needed, respect ignore files, and redact secrets.
  • Provide enterprise controls: private deployment or isolated tenancy, no training on customer code, short retention, audit logs.
  • Encrypt in transit and at rest; strong authentication and per-user quotas.
  • Defend against prompt injection (malicious text in files or web pages trying to instruct the model): treat retrieved content as data, limit tool permissions, require confirmation for risky actions.

  Measuring quality

  • Offline evaluation: run the model on held-out code and tests, measure pass rates and exact-match similarity.
  • Online metrics: acceptance rate of suggestions, characters kept after acceptance, latency percentiles, and errors.
  • A/B test changes to prompts, models and retrieval, and watch for regressions in security and licence filters.
  • Collect feedback signals (thumbs up or down) while respecting privacy.

  What this design reuses

  Debouncing and cancellation, caching, rate limiting and quotas, queues and batching, vector search, streaming responses and careful privacy controls: the same building blocks from the rest of this course, applied to a latency-sensitive AI workload.

  Real-world reading

  Related reading

  • Secret Detector: Preventing Secrets in Source Code at YelpYelp engineers · YelpCase Studies
  • Samza: Stream Processing System for Latency Insighs at LinkedInLinkedIn engineers · LinkedInCase Studies
  • Use The Index, LukeMarkus Winand · use-the-index-luke.comCourses
  • Backend Developer Roadmaproadmap.sh community · roadmap.shCourses

  Show 6 moreShow fewer

  • Natural Language Processing and Content Analysis (2 parts) at Condé NastCondé Nast engineers · Condé NastCase Studies
  • Machine Learning for Indexing Text from Billions of Images at DropboxDropbox engineers · DropboxCase Studies
  • Optimizing Web Servers for High Throughput and Low Latency at DropboxDropbox engineers · DropboxCase Studies
  • Scalable User Privacy at SpotifySpotify engineers · SpotifyCase Studies
  • Privacy-Preserving Analytics and Reporting at LinkedInLinkedIn engineers · LinkedInCase Studies
  • Learning with Privacy at Scale at AppleScale at Apple engineers · Scale at AppleCase Studies

  Open this lesson with its quiz and exercise

• Low-Level Design Practice · 10 minSocial feed (Instagram-style)Users, follows, posts and likes: a feed model with the Observer pattern for notifications and a strategy for ordering.Read hereClose

  The problem

  Users follow each other, publish photo posts, like and comment, and see a home feed of posts from the accounts they follow. This lesson models the classes and rules. Scaling the feed (fan-out strategies) is covered in the news-feed lesson of the classic designs course.

  Entities

  100%

  Figure SDA-LLD-15-1 · Social feed classes · © forcsguy.com

  Publishing and the feed

  100%

  Figure SDA-LLD-15-2 · Publishing a post · © forcsguy.com

  interface FeedStrategy { build(userId: string, limit: number): Post[] }
  
  const newestFirst: FeedStrategy = {
    build(userId, limit) {
      const followees = follows.acceptedFolloweesOf(userId);
      return posts.byAuthors(followees).filter(p => canView(userId, p))
        .sort((a, b) => b.createdAt - a.createdAt).slice(0, limit);
    },
  };
  
  function canView(viewer: string, p: Post): boolean {          // one place for visibility rules
    const author = users.get(p.authorId);
    return !author.isPrivate || author.id === viewer || follows.isAccepted(viewer, author.id);
  }
  

  Putting visibility in one function means the feed, profile and search all agree about who may see what.

  Likes and comments

  • A like is unique per (user, post); enforce it with a uniqueness rule, not just a check, so double taps do not create two likes.
  • Keep a like count on the post for fast display and update it with each like or unlike, accepting that the count may briefly lag.
  • Comments are ordered by time and can have a parent id for replies.

  Design notes

  • Pagination uses a cursor (the time or id of the last item) so new posts do not shift pages.
  • Deleting a post must remove it from feeds and counts; a soft delete flag filtered at read time is the usual first step.
  • Blocking and muting are visibility rules too.
  • Extensions: stories with expiry, hashtags, tagging, search, and a ranking strategy that mixes recency with interest.

  Real-world reading

  Related reading

  • Self-Ranking Search with Elasticsearch at WattpadWattpad engineers · WattpadCase Studies
  • Monitoring Powers High Availability for LinkedIn FeedUSENIX engineers · USENIXCase Studies
  • Search Federation Architecture at LinkedIn (2018)LinkedIn engineers · LinkedInCase Studies
  • Scaling the Hadoop Distributed File System at LinkedInLinkedIn engineers · LinkedInCase Studies

  Show 6 moreShow fewer

  • Galene: Search Architecture of LinkedInLinkedIn engineers · LinkedInCase Studies
  • Infrastructure to Run User Forecasts at SpotifySpotify engineers · SpotifyCase Studies
  • Scalable User Privacy at SpotifySpotify engineers · SpotifyCase Studies
  • Moving Food Feed from Redis to Cassandra at ZomatoZomato engineers · ZomatoCase Studies
  • Clustering Similar Stories Using LDA at FlipboardFlipboard engineers · FlipboardCase Studies
  • Data Orchestration Service at SpotifySpotify engineers · SpotifyCase Studies

  Open this lesson with its quiz and exercise