The system design library
Search and filter every curated system design video, article, repository and paper in one place.
94 resources
Concurrent and Distributed Systems (Cambridge lecture notes)
Eight lectures on time, broadcast, replication, consensus and transactions, with notes, slides and video. Free from the author under CC BY-SA.
BooksIntermediateFree from the authorBy Martin Kleppmann · University of Cambridge
Distributed Systems for Fun and Profit
A short, readable tour of time, consistency and replication in distributed systems, free to read online from the author.
BooksBeginnerFree from the authorBy Mikito Takada · mixu.net
All Things Distributed
The Amazon CTO's blog on architecture, scale and the history of AWS services.
BlogsIntermediateBy Werner Vogels · Amazon
Amazon Aurora: Design Considerations for High Throughput Cloud-Native Relational Databases (paper)
Why Aurora moves the redo log to a distributed storage layer, so replication and crash recovery become cheap.
PapersAdvancedRead hereBy Verbitski et al. · Amazon Web Services
Amazon S3 strong consistency
What strong read-after-write consistency means for S3 and how it simplifies applications.
DocumentationIntermediateBy Amazon Web Services
Apache Cassandra
The source of the wide-column, leaderless database.
GitHubAdvancedBy Apache Software Foundation · GitHub
Apache ZooKeeper
The source of the coordination service.
GitHubAdvancedBy Apache Software Foundation · GitHub
Architecture of Nubank
Edward Wible and Rafael Ferreira discuss the key elements that make Nubank tick for millions of customers every day, including transaction authorization, messaging infrastructure (Kafka), real-time double-entry accounting (Datomic), cust…
Case StudiesIntermediateBy InfoQ engineers · InfoQ
Awesome Distributed Systems
A curated list of distributed-systems papers, books, talks and courses.
GitHubAdvancedBy theanalyst · GitHub
AWS Database Blog
Guidance on choosing, migrating and tuning relational and NoSQL databases.
BlogsIntermediateBy AWS database teams · Amazon Web Services
Black-Box Auditing: Verifying End-to-End Replication Integrity between MySQL and Redshift at Yelp
Black-Box Auditing: Verifying End-to-End Replication Integrity between MySQL and Redshift Jacob Park, Software Engineering Intern Apr 12, 2018 Since Yelp introduced its real-time streaming data infrastructure, “Data Pipeline”, it has...
Case StudiesIntermediateRead hereBy Yelp engineers · Yelp
BookKeeper: Distributed Log Storage at Yahoo
Apache Software Foundation recently announced that BookKeeper has been promoted as an Apache Top Level Project. BookKeeper is near and dear to Yahoo: it was actually created here in 2009 to provide horizontally scalable, reliable, replic…
Case StudiesIntermediateBy Yahoo engineers · Yahoo
Building and Deploying MySQL Raft at Meta
We’re rolling out MySQL Raft with the aim to eventually replace our current MySQL semisynchronous databases. The biggest win of MySQL Raft was simplification of the operation and making MyS…
Case StudiesAdvancedRead hereBy Meta engineers · Meta
CAP Twelve Years Later: How the Rules Have Changed
The author of the CAP conjecture revisits it and explains the trade-offs more carefully.
BlogsAdvancedRead hereBy Eric Brewer · InfoQ
Cassandra architecture overview
The ring, replication, tunable consistency and the storage engine of Cassandra.
DocumentationIntermediateBy Apache Cassandra · Apache Software Foundation
Cassandra: A Decentralized Structured Storage System (paper)
The paper describing Cassandra's design: a Dynamo-style ring and replication with a Bigtable-style data model and storage engine.
PapersIntermediateRead hereBy Lakshman and Malik · Facebook
Clock Skew when Syncing Password Payloads at Drobbox
How Dropbox Passwords fixed sync errors from clock skew by replacing a two-way timestamp merge with a three-way merge against a base payload.
Case StudiesIntermediateRead hereBy Drobbox engineers · Drobbox
Cockroach Labs Blog
Distributed SQL internals: consensus, transactions and multi-region design.
BlogsAdvancedBy Cockroach Labs engineers · Cockroach Labs
CockroachDB
A distributed SQL database built on ranges, Raft and MVCC.
GitHubAdvancedBy Cockroach Labs · GitHub
Consistent Hashing and Random Trees (paper)
The original consistent-hashing paper, written for distributing web caches.
PapersAdvancedRead hereBy Karger et al. · MIT
Cross Shard Transactions at 10 Million RPS at Dropbox
How Dropbox added cross-shard transactions to Edgestore, its metadata store, with a modified two-phase commit proven on a live system at 10 million requests a second.
Case StudiesAdvancedRead hereBy Dropbox engineers · Dropbox
Data Checking at Dropbox
A SREcon talk on designing consistency checkers that verify data at rest: goals, components, alert triage and auto-remediation.
Case StudiesIntermediateBy Dropbox engineers · Dropbox
Database Internals
How storage engines and distributed databases work, from B-trees and LSM-trees to consensus and replication.
BooksAdvancedSold by publisherBy Alex Petrov · O'Reilly
Designing Data-Intensive Applications
The standard book on data systems: storage engines, replication, partitioning, transactions, batch and stream processing.
BooksIntermediateSold by publisherBy Martin Kleppmann · O'Reilly