Cookie preferences

Essential cookies keep this site working. Optional Google Analytics cookies help us understand page visits and improve the site. You can change your choice anytime. Cookie policy

Skip to content
ForCSGuy System Design Atlas home

Library / Case Studies

Experimentation Platform (2 parts) at Spotify

So you’ve read of our two-part series about the new Experimentation Platform we’ve built at Spotify, and now know why we decided to invest in a new platform. In Part II, you’ll get a more detailed look at how we assigned users to experiments, how we analyze results and ensure test integrity.

IntermediateMachine Learning Systems

By Spotify engineers · Spotify · Added 2026-10-12

Open original source

All content belongs to Spotify engineers / Spotify. We only link to it and give credit. Report a problem

Learn this case study

About 37 min left. Your work is saved as you go.

What you are about to study

Spotify explains a real system. You should know the basics: caching, queues and replication.

Ideas you will meet

Machine Learning Systems
Recommendations, ranking, feature stores and serving models at scale.

What we already know about this

Our own lessons on these ideas. Read them here before you start, so it makes sense.

  • Messaging and Microservice Patterns · 12 minService discovery, service mesh and sidecarsHow services find each other as instances come and go, and how sidecar proxies and a mesh move retries, security and telemetry out of application code.Read here

    The problem: moving targets

    In a microservice system, instances are created and destroyed constantly by autoscaling, deployments and failures. A caller cannot rely on a fixed address. It needs an answer to "where are the healthy instances of the orders service right now?"

    Service discovery

    Instances register themselves (or are registered by the platform) in a registry; callers look up the current list.

    Figure SDA-MSV-04-1 · Service registry and lookup · © forcsguy.com

    • Client-side discovery: the caller queries the registry and balances itself (smart client library).
    • Server-side discovery: the caller talks to a load balancer or the platform's service address, which knows the instances (Kubernetes Services and cluster DNS work this way; see the Kubernetes lesson).
    • DNS-based discovery is simple but constrained by caching.
    • Registries must be highly available (often built on consensus; see the consensus lesson), and the system should keep working with the last known list if the registry is briefly unreachable.

    Sidecars

    A sidecar is a helper process or container deployed alongside each service instance (in Kubernetes, in the same pod). All the service's network traffic goes through it. It can take over cross-cutting jobs so application code does not have to implement them in every language:

    • retries, timeouts and circuit breaking (see the resilience lesson),
    • mutual TLS (encrypting and authenticating service-to-service traffic),
    • load balancing and service discovery,
    • metrics, logs and tracing headers,
    • access policy.

    Other sidecar uses: log shippers, config reloaders, secret agents.

    Service mesh

    A service mesh is a dedicated infrastructure layer made of two parts:

    • Data plane: the fleet of sidecar proxies (often Envoy) that carry the traffic.
    • Control plane: configures and secures the proxies, distributes certificates and policy (for example Istio, Linkerd, Consul).

    Figure SDA-MSV-04-2 · Control plane and data plane · © forcsguy.com

    What a mesh enables:

    • Traffic management: weighted routing for canary releases, mirroring, fault injection, timeouts and retries by route.
    • Security: automatic mutual TLS and service-level authorisation, giving each service a cryptographic identity.
    • Observability: uniform metrics, traces and access logs for every call, without code changes.

    Costs and alternatives

    • Resources and latency: a proxy per instance adds CPU, memory and a small latency per hop.
    • Operational complexity: another system to upgrade and debug; misconfiguration can break everything.
    • Learning curve and sometimes opaque behaviour.
    • Newer designs reduce per-pod cost (node-level or "ambient" proxies).

    Alternatives: a shared library for resilience and telemetry (language-bound, needs upgrades everywhere), a managed platform's built-in features, or an API gateway for north-south traffic only (see the load balancing lesson).

    When a mesh makes sense

    Many services, several languages, strong needs for mutual TLS or consistent policy, and a platform team able to run it. For a handful of services, platform defaults and a good library are usually enough. As ever, adopt the pattern for a problem you have.

    Real-world reading

    Related reading

    Show 6 more

    Open this lesson with its quiz and exercise

Want the architecture first? Read our Spotify case study, with its diagram.

Continue to Spotify

Go deeper on this

A path through videos, articles, other case studies and code on the same ideas as Experimentation Platform (2 parts) at Spotify.

  1. ReadEngineers explain it in their own words.

  2. WatchHow others solve the same problem, on video.

    Videos belong to their creators and play from YouTube. More on YouTube (opens in a new tab)

  3. See it in another companyThe same ideas, a different system.

Keep learning

Loading community…