Replication
Leader/follower, multi-leader and leaderless replication.
All topics · 2 lessons · 31 resources
Learn Replication on System Design Atlas
- 01Replication and shardingTwo ways to spread data over many machines: copy it for safety and read speed, or split it for capacity.System Design Foundations · 11 min read · start here
- 02Replication topologies and multi-region databasesSingle-leader, multi-leader and leaderless replication, how they handle conflicts and lag, and what changes across regions.Databases Deep Dive · 12 min read
Related topics
These share a lesson with Replication.
Curated Replication resources
Books
- Concurrent and Distributed Systems (Cambridge lecture notes)Eight lectures on time, broadcast, replication, consensus and transactions, with notes, slides and video. Free from the author under CC BY-SA.University of Cambridge
- Distributed Systems for Fun and ProfitA short, readable tour of time, consistency and replication in distributed systems, free to read online from the author.mixu.net
- Designing Data-Intensive ApplicationsThe standard book on data systems: storage engines, replication, partitioning, transactions, batch and stream processing.O'Reilly
Papers
- Amazon Aurora: Design Considerations for High Throughput Cloud-Native Relational Databases (paper)Why Aurora moves the redo log to a distributed storage layer, so replication and crash recovery become cheap.Amazon Web Services
- In Search of an Understandable Consensus Algorithm (Raft paper)The Raft paper: leader election, log replication and safety explained so that engineers can implement them.Stanford University
- Megastore: Providing Scalable, Highly Available Storage (paper)Synchronous replication across data centres with Paxos while keeping a transactional data model.Google
- Spanner: Google's Globally Distributed Database (paper)A globally distributed database that gives strongly consistent transactions by combining consensus groups with clocks whose uncertainty is bounded.Google
- The Google File System (paper)How Google built a file system for huge files and constant machine failures: one master for metadata, many chunk servers, and append-heavy workloads. The ancestor of HDFS.Google
GitHub
- Apache CassandraThe source of the wide-column, leaderless database.GitHub
- CockroachDBA distributed SQL database built on ranges, Raft and MVCC.GitHub
- Kafka Source & DocsApache Kafka source code: read the log, replication and consumer-group internals first-hand.apache
- The System Design PrimerThe most-starred system design study guide: scalability concepts, trade-offs and Anki flashcards.donnemartin
Blogs
- AWS Database BlogGuidance on choosing, migrating and tuning relational and NoSQL databases.Amazon Web Services
- Cockroach Labs BlogDistributed SQL internals: consensus, transactions and multi-region design.Cockroach Labs
- Patterns of Distributed SystemsA catalogue of named building blocks (replicated log, lease, quorum, gossip, high-water mark) found in systems like Kafka and Cassandra.Martin Fowler
- How Discord Stores Trillions of MessagesA migration story from MongoDB to Cassandra to ScyllaDB at huge scale.Discord
Case Studies
- Black-Box Auditing: Verifying End-to-End Replication Integrity between MySQL and Redshift at YelpBlack-Box Auditing: Verifying End-to-End Replication Integrity between MySQL and Redshift Jacob Park, Software Engineering Intern Apr 12, 2018 Since Yelp introduced its real-time streaming data infrastructure, “Data Pipeline”, it has...Yelp
- High-performance Replicated Log Service at TwitterBuilding DistributedLog: High-performance replicated log serviceTwitter
- Inside the Magic PocketHow Dropbox built its own exabyte-scale block storage, including durability and cost choices.Dropbox
- Mitigating MySQL Replication Lag and Reducing Read Load at GithubAt GitHub, we use MySQL as the main database technology backing our services.Github
- MySQL High Availability at GitHubHow GitHub detects primary failure and promotes a replica safely using orchestration and service discovery.GitHub
- Read Consistency with Database Replicas at ShopifyAt Shopify, we’ve long used database replication for redundancy and failure recovery, but only fairly recently started to explore the potential of replicas as an alternative read-only data source for applications.Shopify
- Types of Replication - SQL ServerLearn about the different types of replication that SQL Server provides for use in distributed applications.Microsoft
- ZippyDB: Facebook's key-value storeFacebook's strongly consistent key-value store built on RocksDB and Paxos, with data sharded and replicated.Meta
- Active-Active for Multi-Regional ResiliencyHow Netflix routes traffic across regions and survives regional outages.Netflix
- Uber Schemaless: A Scalable MySQL DatastoreHow Uber built an append-only, sharded datastore on top of MySQL.Uber
- Amazon Dynamo: Highly Available Key-Value StoreWerner Vogels' write-up of the design that inspired DynamoDB, Cassandra and Riak.Amazon