Replication topologies and multi-region databases

Why replicate

Replication keeps copies of data on several machines for three reasons: availability (survive a node failure), read scaling (serve reads from replicas) and latency (put data near users). The hard part is keeping copies consistent while writes continue. (Basics are in the replication and sharding lesson of the foundations course; this lesson compares the topologies.)

Single-leader (primary and replicas)

One node, the leader (primary or master), accepts writes and ships its changes to followers (replicas), which serve reads.

100%

Figure SDA-DDV-04-1 · Single-leader replication · © forcsguy.com

• Synchronous replication waits for a follower before acknowledging: safer, slower, and a stalled follower can block writes. Asynchronous is faster but a failover can lose the latest writes. A common compromise: one synchronous follower, others asynchronous.
• Failover must detect failure, choose the most up-to-date replica, redirect clients and ensure the old leader does not come back believing it still leads (split brain).
• Replication lag causes stale reads. Techniques: read-your-writes (route a user's reads to the leader shortly after they write), monotonic reads (keep a user on one replica), and checking replica position.

Multi-leader

Several nodes accept writes (often one per region) and replicate to each other. Benefits: local write latency and continued operation if a region is cut off. Cost: write conflicts.

Conflict handling options:

• Avoid them: route all writes for one record to one leader (for example by user or region).
• Last write wins by timestamp: simple, but silently drops one of two concurrent updates and depends on clocks.
• Merge values (union of sets, counters) using mergeable data types (CRDTs; see the collaborative editing lesson).
• Custom resolution in application code, or ask the user.

Multi-leader also appears in offline-capable apps (each device is a leader) and collaborative editing.

Leaderless (Dynamo-style)

Any replica accepts writes; clients write to several replicas and read from several, using quorums (W + R > N) to get fresh data (see the key-value store lesson). Background anti-entropy and read repair fix stale copies; versions or vector clocks detect conflicts. It favours availability and is typical of Cassandra and DynamoDB-style stores.

100%

Figure SDA-DDV-04-2 · Choosing a replication model · © forcsguy.com

Multi-region designs

• Active-passive: one region serves writes; another is a warm standby. Simple; failover loses a little data (the replication lag) and takes minutes.
• Active-active: several regions serve traffic. Requires either partitioning data by home region (each record has one home, cross-region reads are replicas) or true multi-leader conflict handling.
• Global databases use consensus across regions (see the consensus lesson) for strong consistency at the cost of cross-region write latency; some offer tunable consistency per query.
• Latency numbers matter: a cross-ocean round trip is around 100 ms or more, so synchronous cross-region replication adds that to every write.
• Data residency laws may require keeping certain users' data in a region.

Partitioning and replication together

Real systems shard data into partitions and replicate each partition (a leader per partition spreads write load across nodes). Rebalancing, hot partitions and cross-partition transactions are the new problems (see sharding and the consistent-hashing lesson).

Master/slave naming

You will see "master/slave", "primary/replica" and "leader/follower" for the same single-leader idea; most current documentation prefers the last two.

Checklist

• What happens to writes if the leader or a region is lost, and how long?
• How much data can we lose (RPO) and how long can we be down (RTO)?
• How do we handle stale reads from replicas?
• Who resolves conflicts, and how?
• Have we practised failover?

Real-world reading