Active-Active for Multi-Regional Resiliency

Ideas you will meet

DNS

How names resolve, TTLs, and DNS-based traffic routing.

Load Balancer

Spread traffic across servers with L4/L7 balancing, health checks and algorithms.

Replication

Leader/follower, multi-leader and leaderless replication.

What we already know about this

Our own lessons on these ideas. Read them here before you start, so it makes sense.

• Databases Deep Dive · 12 minReplication topologies and multi-region databasesSingle-leader, multi-leader and leaderless replication, how they handle conflicts and lag, and what changes across regions.Read hereClose

  Why replicate

  Replication keeps copies of data on several machines for three reasons: availability (survive a node failure), read scaling (serve reads from replicas) and latency (put data near users). The hard part is keeping copies consistent while writes continue. (Basics are in the replication and sharding lesson of the foundations course; this lesson compares the topologies.)

  Single-leader (primary and replicas)

  One node, the leader (primary or master), accepts writes and ships its changes to followers (replicas), which serve reads.

  100%

  Figure SDA-DDV-04-1 · Single-leader replication · © forcsguy.com

  • Synchronous replication waits for a follower before acknowledging: safer, slower, and a stalled follower can block writes. Asynchronous is faster but a failover can lose the latest writes. A common compromise: one synchronous follower, others asynchronous.
  • Failover must detect failure, choose the most up-to-date replica, redirect clients and ensure the old leader does not come back believing it still leads (split brain).
  • Replication lag causes stale reads. Techniques: read-your-writes (route a user's reads to the leader shortly after they write), monotonic reads (keep a user on one replica), and checking replica position.

  Multi-leader

  Several nodes accept writes (often one per region) and replicate to each other. Benefits: local write latency and continued operation if a region is cut off. Cost: write conflicts.

  Conflict handling options:

  • Avoid them: route all writes for one record to one leader (for example by user or region).
  • Last write wins by timestamp: simple, but silently drops one of two concurrent updates and depends on clocks.
  • Merge values (union of sets, counters) using mergeable data types (CRDTs; see the collaborative editing lesson).
  • Custom resolution in application code, or ask the user.

  Multi-leader also appears in offline-capable apps (each device is a leader) and collaborative editing.

  Before: single leader

  100%

  After: multi-leader across regions

  100%

  What changed

  • Users in each region write to a nearby database
  • Concurrent edits to the same row need conflict handling
  • More moving parts to run and monitor

  Leaderless (Dynamo-style)

  Any replica accepts writes; clients write to several replicas and read from several, using quorums (W + R > N) to get fresh data (see the key-value store lesson). Background anti-entropy and read repair fix stale copies; versions or vector clocks detect conflicts. It favours availability and is typical of Cassandra and DynamoDB-style stores.

  100%

  Figure SDA-DDV-04-2 · Choosing a replication model · © forcsguy.com

  Multi-region designs

  • Active-passive: one region serves writes; another is a warm standby. Simple; failover loses a little data (the replication lag) and takes minutes.
  • Active-active: several regions serve traffic. Requires either partitioning data by home region (each record has one home, cross-region reads are replicas) or true multi-leader conflict handling.
  • Global databases use consensus across regions (see the consensus lesson) for strong consistency at the cost of cross-region write latency; some offer tunable consistency per query.
  • Latency numbers matter: a cross-ocean round trip is around 100 ms or more, so synchronous cross-region replication adds that to every write.
  • Data residency laws may require keeping certain users' data in a region.

  Partitioning and replication together

  Real systems shard data into partitions and replicate each partition (a leader per partition spreads write load across nodes). Rebalancing, hot partitions and cross-partition transactions are the new problems (see sharding and the consistent-hashing lesson).

  Master/slave naming

  You will see "master/slave", "primary/replica" and "leader/follower" for the same single-leader idea; most current documentation prefers the last two.

  Checklist

  • What happens to writes if the leader or a region is lost, and how long?
  • How much data can we lose (RPO) and how long can we be down (RTO)?
  • How do we handle stale reads from replicas?
  • Who resolves conflicts, and how?
  • Have we practised failover?

  Real-world reading

  Related reading

  • Active-Active for Multi-Regional ResiliencyNetflix Engineering · NetflixCase Studies
  • Designing Data-Intensive ApplicationsMartin Kleppmann · O'ReillyBooks
  • Read Consistency with Database Replicas at ShopifyShopify engineers · ShopifyCase Studies
  • Uber Schemaless: A Scalable MySQL DatastoreUber Engineering · UberCase Studies

  Show 6 moreShow fewer

  • Mitigating MySQL Replication Lag and Reducing Read Load at GithubGithub engineers · GithubCase Studies
  • Cassandra architecture overviewApache Cassandra · Apache Software FoundationDocumentation
  • How Discord Stores Trillions of MessagesBo Ingram · DiscordBlogs
  • Spanner: Google's Globally Distributed Database (paper)Corbett et al. · GooglePapers
  • Scaling Multitenant Architecture Across Multiple Data Centres at Shopify - Weingarten, Engineering Lead at ShopifyShopifyYouTube
  • MIT 6.824: Distributed SystemsRobert Morris · MITCourses

  Open this lesson with its quiz and exercise

• Advanced Distributed Systems · 12 minLoad balancing: layer 4, layer 7 and gatewaysWhat a load balancer can see and do at each layer, the algorithms behind it, and how API gateways add routing, aggregation and policy.Read hereClose

  What a load balancer sees

  A load balancer spreads incoming work across servers, removes failed servers and makes scaling invisible to clients. What it can do depends on which layer it works at.

  	Layer 4 (transport)	Layer 7 (application)
  Sees	IP addresses, ports, TCP/UDP connections	HTTP method, URL, headers, cookies, body
  Decides by	Connection details	Request content
  TLS	Passes through, or terminates in some products	Usually terminates TLS
  Speed and cost	Very fast, low overhead	More work per request
  Typical features	Spread connections, health checks	Path routing, header rules, sticky sessions, rewrites, caching, WAF rules
  Examples	Network load balancers, LVS, HAProxy in TCP mode	NGINX, Envoy, HAProxy HTTP mode, cloud application balancers

  100%

  Figure SDA-ADS-05-1 · Layers of balancing in front of an app · © forcsguy.com

  Many designs use both: an L4 tier for raw scale and an L7 tier for smart routing.

  Before: layer 4: ports and addresses

  100%

  After: layer 7: reads the request

  100%

  What changed

  • Layer 7 can route by path, header or cookie
  • Layer 7 can terminate TLS and retry safely
  • Layer 4 is simpler and faster, with less to configure

  Choosing a target: algorithms

  • Round robin / weighted round robin: simple rotation; weights favour bigger machines.
  • Least connections (and weighted): send to the server with fewest active requests; good when requests vary in length.
  • Least response time: combine active load with measured latency.
  • IP hash / consistent hashing: the same client or key goes to the same server, useful for caches and sticky state (see consistent hashing).
  • Resource-based: servers report load and the balancer follows.

  Health checks (active probes and passive error tracking) remove unhealthy servers; connection draining lets in-flight requests finish before a server is removed during a deploy.

  Global and DNS balancing

  Across regions, DNS-based or global server load balancing directs users to the nearest or healthiest region (see the DNS and CDN lessons). DNS answers are cached, so failover takes as long as the TTL; anycast and global balancers react faster.

  API gateway: policy at the edge

  An API gateway is an L7 component specialised for APIs. Beyond routing it commonly provides:

  • Authentication and authorisation (validate tokens, apply scopes),
  • Rate limiting and quotas (see the rate-limiter lesson),
  • Request and response transformation, versioning, and protocol translation,
  • Aggregation: one client call fans out to several services and merges results, saving round trips for mobile clients,
  • Caching, logging, metrics and tracing in one place.

  100%

  Figure SDA-ADS-05-2 · Gateway routing and aggregation · © forcsguy.com

  Risks: the gateway is a single entry point (run several instances, keep it stateless) and can become a bottleneck or a place where business logic piles up. Keep it to cross-cutting concerns. A variation is a backend for frontend: one gateway per client type (web, mobile) tailored to its needs.

  Inside the cluster

  Between services, balancing is done by client-side libraries or sidecar proxies that know the healthy instances from service discovery, so no central balancer is needed for east-west traffic. See the microservices lessons for service mesh ideas.

  Failure habits

  • Run at least two of every balancer and gateway instance across zones.
  • Keep timeouts and retry budgets (see the resilience lesson) so the balancer does not amplify an outage.
  • Use slow start for new instances so they warm up before taking a full share.

  Real-world reading

  Related reading

  • Maglev: A Fast and Reliable Software Network Load Balancer (paper)Eisenbud et al. · GooglePapers
  • NGINX HTTP Load BalancingNGINX · F5 / NGINXDocumentation
  • Scaling Your API with Rate LimitersPaul Tarjan · StripeBlogs
  • Active-Active for Multi-Regional ResiliencyNetflix Engineering · NetflixCase Studies

  Show 6 moreShow fewer

  • ByteByteGo: System Design Video SeriesAlex Xu · ByteByteGoYouTube
  • Rate Limiting with NGINXNGINX · F5 / NGINXBlogs
  • Implementing health checksAmazon Builders' Library · Amazon Web ServicesBlogs
  • Google SRE Book: Load Balancing in the DatacenterGoogle SRE team · GoogleBlogs
  • Traffic Steering using Rum DNS at LinkedInLinkedIn engineers · LinkedInCase Studies
  • Google SRE BookBetsy Beyer et al. · GoogleCourses

  Open this lesson with its quiz and exercise

• System Design Foundations · 11 minNetworking, APIs and CDNsWhat happens between a click and your server: DNS, HTTP, API styles, and how a CDN moves content closer to users.Read hereClose

  The journey of a request

  Before your code runs, a lot happens. When a browser loads https://shop.example.com:

  100%

  Figure SDA-FND-09-1 · Request journey: DNS, CDN, origin · © forcsguy.com

  1. DNS turns the name into an IP address. Answers are cached at several levels for a time given by the record's TTL, so changes spread gradually.
  2. A connection is opened (TCP) and secured (TLS). Each step costs round trips, so long distances hurt.
  3. HTTP carries the request and response.

  Latency is mostly physics: a round trip across an ocean costs over 100 ms no matter how fast your code is. The only fixes are fewer round trips and moving data closer to the user.

  Image: Syced · CC0 · Source · Wikimedia Commons: Fujikura optic fiber at Science Museum

  Name lookup is a short chain of questions, each answered by a more specific server:

  100%

  Figure SDA-FND-09-2 · How a name becomes an address · © forcsguy.com

  Image: Howe's Adventures & Achievements of Americans · Public domain · Source · Wikimedia Commons: Transatlantic submarine cable map

  CDNs

  A content delivery network is a worldwide set of servers (edges) that keep copies of your content close to users.

  • Great for static files: images, video, scripts, stylesheets, and often whole pages that rarely change.
  • Cuts latency, absorbs traffic spikes and shields your origin from load.
  • Controlled by cache headers (Cache-Control: max-age=...) and purge APIs when content changes.
  • Use a versioned file name (app.3f9a.js) so you can cache for a year and still ship updates instantly.

  100%

  Figure SDA-FND-09-3 · Content delivery in tiers · © forcsguy.com

  Before: everyone goes to the origin

  100%

  After: a CDN serves copies near users

  100%

  What changed

  • Users get files from a nearby location
  • The origin handles far fewer requests
  • Changed files need invalidation or versioned names

  Choosing an API style

  Style	Strength	Watch out for
  REST over HTTP/JSON	Simple, cacheable, universal	Over-fetching, many round trips for complex screens
  GraphQL	Client asks for exactly the fields it needs	Query cost control, caching is harder
  gRPC	Fast binary protocol, strong contracts	Needs tooling, awkward in browsers
  SOAP	XML messages with a formal contract (WSDL) and built-in standards for security and transactions, still common in banks and older enterprise systems	Verbose, heavier tooling, rarely chosen for new public APIs
  WebSockets / SSE	Server can push live updates	Long-lived connections need extra capacity and care

  A good default is REST for public APIs, gRPC between internal services if you need speed, and WebSockets only where live push is truly needed.

  Designing good APIs

  • Version them so old clients keep working.
  • Paginate list endpoints (cursor-based pagination stays stable as data changes).
  • Make writes idempotent with idempotency keys, so a retry does not double-charge.
  • Return clear errors with correct status codes.
  • Authenticate and rate limit at the edge, often in an API gateway that also handles routing, logging and TLS.

  100%

  Figure SDA-FND-09-4 · API gateway in front of services · © forcsguy.com

  More on this site

  Related reading

  • High Performance Browser NetworkingIlya Grigorik · O'ReillyBooks
  • HTTP/3: the past, the present, and the futureCloudflareBlogs
  • MDN: HTTP cachingMDN contributors · MozillaDocumentation
  • Cloudflare BlogCloudflare engineers · CloudflareBlogs

  Show 6 moreShow fewer

  • The Road to QUICCloudflareBlogs
  • Flannel: An Application-Level Edge Cache to Make Slack ScaleSlack Engineering · SlackCase Studies
  • Cloudflare Learning Center: What is a CDN?CloudflareDocumentation
  • Cloudflare EngineeringCloudflare engineers · CloudflareBlogs
  • Justin.tv's Live Video Broadcasting Architecture - High Scalability -High Scalability engineers · High ScalabilityCase Studies
  • Service Workers at SlackSlack engineers · SlackCase Studies

  Open this lesson with its quiz and exercise