Black-Box Auditing: Verifying End-to-End Replication Integrity between MySQL and Redshift at Yelp

Ideas you will meet

Replication

Leader/follower, multi-leader and leaderless replication.

Databases

Schemas, indexes, query plans and choosing a store.

What we already know about this

Our own lessons on these ideas. Read them here before you start, so it makes sense.

• Core Concepts: How Things Actually Work · 7 minDurability: replication and erasure codingHow storage systems survive disk failures without storing three full copies of everything.Read hereClose

  Why durability is hard

  Disks fail every day in a large fleet. Durability is the probability data is not lost over time, and cloud object storage advertises numbers like eleven nines. How?

  Image: Mk2010 · CC BY 4.0 · Source · Wikimedia Commons: Hard disk drive platter, Samsung MP0402H

  Technique 1: replication

  Keep N full copies on different machines, racks and zones. Simple and fast:

  • Any copy can serve a read.
  • Recovery is just copying a replica.
  • Cost: 3 copies mean 3x storage.

  Technique 2: erasure coding

  Split an object into k data pieces, compute m parity pieces (using maths such as Reed-Solomon), and store all k + m pieces on different machines. Any k of the k + m pieces can rebuild the original.

  100%

  Figure SDA-CORE-14-1 · Technique 2: erasure coding · © forcsguy.com

  With (6, 3) you store 9 pieces for 6 pieces of data: 1.5x overhead, and you can lose any 3 disks. That beats 3x replication on both counts.

  100%

  Figure SDA-CORE-14-2 · Surviving two failures and rebuilding · © forcsguy.com

  The trade-offs

  	Replication	Erasure coding
  Storage overhead	high (3x)	low (about 1.2x to 1.5x)
  Read	from one copy, fast	read several pieces, fast if all data pieces present
  Repair	copy one replica	read many pieces and recompute
  Small updates	easy	costly (re-encode)
  Best for	hot, small, frequently changed data	cold, large, rarely changed objects

  Making it real

  • Placement: put pieces in different failure domains (disk, server, rack, zone) so one event cannot take many.
  • Detection and repair: monitor continuously, and rebuild lost pieces before more fail. The faster the repair, the higher the durability.
  • Checksums: verify data on read and in background scrubbing to catch silent corruption.
  • Tiering: hot data replicated; older data converted to erasure coding.
  • Versioning and backups protect against human error, which replication does not (a delete replicates too).

  More on this site

  Related reading

  • Uber Schemaless: A Scalable MySQL DatastoreUber Engineering · UberCase Studies
  • How Discord Stores Trillions of MessagesBo Ingram · DiscordBlogs
  • Jordan Has No Life: Systems Design Concepts SeriesJordan · Jordan Has No LifeYouTube
  • Inside the Magic PocketDropboxCase Studies

  Show 6 moreShow fewer

  • Kafka Source & DocsApache Software Foundation · apacheGitHub
  • Black-Box Auditing: Verifying End-to-End Replication Integrity between MySQL and Redshift at YelpYelp engineers · YelpCase Studies
  • Apache CassandraApache Software Foundation · GitHubGitHub
  • Types of Replication - SQL ServerMicrosoft engineers · MicrosoftCase Studies
  • Concept && CodingShrayansh Jain · Concept && CodingYouTube
  • Accelerator: Fast Data Processing Framework at eBayeBay engineers · eBayCase Studies

  Open this lesson with its quiz and exercise

• Databases Deep Dive · 12 minReplication topologies and multi-region databasesSingle-leader, multi-leader and leaderless replication, how they handle conflicts and lag, and what changes across regions.Read hereClose

  Why replicate

  Replication keeps copies of data on several machines for three reasons: availability (survive a node failure), read scaling (serve reads from replicas) and latency (put data near users). The hard part is keeping copies consistent while writes continue. (Basics are in the replication and sharding lesson of the foundations course; this lesson compares the topologies.)

  Single-leader (primary and replicas)

  One node, the leader (primary or master), accepts writes and ships its changes to followers (replicas), which serve reads.

  100%

  Figure SDA-DDV-04-1 · Single-leader replication · © forcsguy.com

  • Synchronous replication waits for a follower before acknowledging: safer, slower, and a stalled follower can block writes. Asynchronous is faster but a failover can lose the latest writes. A common compromise: one synchronous follower, others asynchronous.
  • Failover must detect failure, choose the most up-to-date replica, redirect clients and ensure the old leader does not come back believing it still leads (split brain).
  • Replication lag causes stale reads. Techniques: read-your-writes (route a user's reads to the leader shortly after they write), monotonic reads (keep a user on one replica), and checking replica position.

  Multi-leader

  Several nodes accept writes (often one per region) and replicate to each other. Benefits: local write latency and continued operation if a region is cut off. Cost: write conflicts.

  Conflict handling options:

  • Avoid them: route all writes for one record to one leader (for example by user or region).
  • Last write wins by timestamp: simple, but silently drops one of two concurrent updates and depends on clocks.
  • Merge values (union of sets, counters) using mergeable data types (CRDTs; see the collaborative editing lesson).
  • Custom resolution in application code, or ask the user.

  Multi-leader also appears in offline-capable apps (each device is a leader) and collaborative editing.

  Before: single leader

  100%

  After: multi-leader across regions

  100%

  What changed

  • Users in each region write to a nearby database
  • Concurrent edits to the same row need conflict handling
  • More moving parts to run and monitor

  Leaderless (Dynamo-style)

  Any replica accepts writes; clients write to several replicas and read from several, using quorums (W + R > N) to get fresh data (see the key-value store lesson). Background anti-entropy and read repair fix stale copies; versions or vector clocks detect conflicts. It favours availability and is typical of Cassandra and DynamoDB-style stores.

  100%

  Figure SDA-DDV-04-2 · Choosing a replication model · © forcsguy.com

  Multi-region designs

  • Active-passive: one region serves writes; another is a warm standby. Simple; failover loses a little data (the replication lag) and takes minutes.
  • Active-active: several regions serve traffic. Requires either partitioning data by home region (each record has one home, cross-region reads are replicas) or true multi-leader conflict handling.
  • Global databases use consensus across regions (see the consensus lesson) for strong consistency at the cost of cross-region write latency; some offer tunable consistency per query.
  • Latency numbers matter: a cross-ocean round trip is around 100 ms or more, so synchronous cross-region replication adds that to every write.
  • Data residency laws may require keeping certain users' data in a region.

  Partitioning and replication together

  Real systems shard data into partitions and replicate each partition (a leader per partition spreads write load across nodes). Rebalancing, hot partitions and cross-partition transactions are the new problems (see sharding and the consistent-hashing lesson).

  Master/slave naming

  You will see "master/slave", "primary/replica" and "leader/follower" for the same single-leader idea; most current documentation prefers the last two.

  Checklist

  • What happens to writes if the leader or a region is lost, and how long?
  • How much data can we lose (RPO) and how long can we be down (RTO)?
  • How do we handle stale reads from replicas?
  • Who resolves conflicts, and how?
  • Have we practised failover?

  Real-world reading

  Related reading

  • Active-Active for Multi-Regional ResiliencyNetflix Engineering · NetflixCase Studies
  • Designing Data-Intensive ApplicationsMartin Kleppmann · O'ReillyBooks
  • Read Consistency with Database Replicas at ShopifyShopify engineers · ShopifyCase Studies
  • Uber Schemaless: A Scalable MySQL DatastoreUber Engineering · UberCase Studies

  Show 6 moreShow fewer

  • Mitigating MySQL Replication Lag and Reducing Read Load at GithubGithub engineers · GithubCase Studies
  • Cassandra architecture overviewApache Cassandra · Apache Software FoundationDocumentation
  • How Discord Stores Trillions of MessagesBo Ingram · DiscordBlogs
  • Spanner: Google's Globally Distributed Database (paper)Corbett et al. · GooglePapers
  • Scaling Multitenant Architecture Across Multiple Data Centres at Shopify - Weingarten, Engineering Lead at ShopifyShopifyYouTube
  • MIT 6.824: Distributed SystemsRobert Morris · MITCourses

  Open this lesson with its quiz and exercise

• Advanced Distributed Systems · 12 minRedis: persistence, replication, Sentinel and ClusterWhat keeps an in-memory database from losing data and from being a single point of failure: snapshots, the append-only file, replicas, Sentinel and hash slots.Read hereClose

  Fast because it is in memory, durable because of logs

  Redis keeps its dataset in RAM (the reason it is so fast; see the Redis and Memcached lesson). To survive a restart it offers two persistence mechanisms, which can be combined.

  Snapshots (RDB)

  Redis periodically writes a point-in-time snapshot of the whole dataset to disk, using a forked child process so the main process keeps serving. Snapshots are compact and quick to load, but anything written since the last snapshot is lost on a crash.

  Append-only file (AOF)

  Redis appends every write command to a log. On restart it replays the log. The fsync policy sets the loss window:

  Policy	Behaviour	Risk
  always	Flush after every command	Slowest, almost no loss
  everysec	Flush once per second	At most about a second lost (common default)
  no	Let the OS decide	Fastest, larger loss

  The AOF grows, so Redis periodically rewrites it into a compact form. Many deployments use AOF with everysec, sometimes plus snapshots for fast backups.

  100%

  Figure SDA-ADS-04-1 · Redis persistence choices · © forcsguy.com

  Replication and Sentinel

  A Redis primary replicates to one or more replicas asynchronously. Replicas serve reads (possibly slightly stale) and can take over. Sentinel is a small set of monitor processes that:

  1. watch the primary and replicas,
  2. agree by quorum that the primary is down,
  3. promote a replica and tell clients where the new primary is.

  100%

  Figure SDA-ADS-04-2 · Sentinel failover · © forcsguy.com

  Because replication is asynchronous, a failover can lose the last few acknowledged writes. Redis offers settings to reduce this (for example requiring a minimum number of replicas to acknowledge), at the cost of availability.

  Redis Cluster: sharding

  When data or traffic outgrow one node, Redis Cluster shards the keyspace into 16,384 hash slots. A key's slot is a hash of the key modulo that number; each node owns a range of slots, and each primary has replicas. Clients learn the slot map and talk to the right node; a wrong guess gets a redirect.

  • Resharding moves slots (and their keys) between nodes while the cluster runs.
  • Multi-key operations only work when all keys are in the same slot. Hash tags (a {tag} part of the key) force related keys into one slot.
  • Failover is built in: replicas of a failed primary are promoted by the cluster.
  • This is the "fixed number of slots" cousin of consistent hashing (see the consistent-hashing lesson).

  Which setup when

  Need	Choice
  Pure cache, loss acceptable	Single node or replicas, little or no persistence
  Must survive restart	AOF (everysec) and snapshots
  Must survive node loss	Replica plus Sentinel
  Data larger than one node, or very high throughput	Redis Cluster

  Also keep memory in mind: set a max memory and an eviction policy (see the caching lessons) so Redis behaves predictably when full, and watch for large keys and slow commands, which block a single-threaded executor.

  Real-world reading

  Related reading

  • Redis DocumentationRedis · Redis Ltd.Documentation
  • Redis persistenceRedis · Redis Ltd.Documentation
  • Redis UniversityRedis · Redis Ltd.Courses
  • Redis BlogRedis engineers · Redis Ltd.Blogs

  Show 6 moreShow fewer

  • Scale with Redis ClusterRedis · Redis Ltd.Documentation
  • RedisRedis · Redis Ltd.GitHub
  • Failover with Redis Sentinel at VintedVinted engineers · VintedCase Studies
  • Memory Optimization in Redis at WattpadWattpad engineers · WattpadCase Studies
  • Speeding Up Redis with Compression at DoorDashDoorDash engineers · DoorDashCase Studies
  • Availability in Globally Distributed Storage Systems at GoogleGooglePapers

  Open this lesson with its quiz and exercise