Database Sharding
Partition data across nodes by key, range or directory.
All topics · 3 lessons · 26 resources
Learn Database Sharding on System Design Atlas
- 01Replication and shardingTwo ways to spread data over many machines: copy it for safety and read speed, or split it for capacity.System Design Foundations · 11 min read · start here
- 02Design consistent hashingHow to spread keys over servers so that adding or removing a server moves only a small share of them.Classic System Designs · 10 min read
- 03Sharding a search index: scatter, gather and mergeHow a search index is split across machines, why queries fan out to every shard, how top-k results are merged, and what goes wrong with deep paging and uneven shards.Search, Analytics and Geo · 11 min read
Related topics
These share a lesson with Database Sharding.
Curated Database Sharding resources
YouTube
- System Design InterviewA small set of detailed videos on how distributed systems and common interview designs work.System Design Interview
- Hello Interview: System Design WalkthroughsStructured mock-interview style designs with explicit trade-off discussion.Hello Interview
- Gaurav Sen: System Design PlaylistInterview-oriented walkthroughs of consistent hashing, load balancing, sharding and classic designs.Gaurav Sen (YouTube)
Papers
- Bigtable: A Distributed Storage System for Structured Data (paper)Google's wide-column store: tablets split by key range, a commit log plus sorted files on disk, and the design later copied by HBase and Cassandra.Google
- Consistent Hashing and Random Trees (paper)The original consistent-hashing paper, written for distributing web caches.MIT
Case Studies
- Cross Shard Transactions at 10 Million RPS at DropboxHow Dropbox added cross-shard transactions to Edgestore, its metadata store, with a modified two-phase commit proven on a live system at 10 million requests a second.Dropbox
- How Discord Stores Billions of MessagesThe data model and partitioning of Discord's message store on Cassandra, and the hot-partition problems it met.Discord
- MongoDB at Baidu: Multi-Tenant Cluster Storing 200+ Billion Documents across 160 ShardsLearn how MongoDB powers real-time applications and intelligent experiences for critical use cases across industries.Mongodb
- Scaling Services with Shard Manager at FacebookOver the years, as we’ve expanded in scale and functionalities, Facebook has evolved from a basic web server architecture into a complex one with thousands of services working behind the scenes. It…Facebook
- Sharding MySQL at QuoraTeam: Nagavamsi (Vamsi) Ponnekanti, Lingduo Kong, Hwan Seung Yeo (manager) In this blog post, we discuss how Quora was able to scale our usage of MySQL to meet the growing requirements of our content. We’ll focus especially on the challenge of sharding the data stored in MySQL at scale. MySQL...Quora
- Sharding MySQL at TwilioA real-world engineering write-up from Twilio on relational Databases.Twilio
- Sharding out Elasticsearch at VintedHow Vinted diagnosed problems with its Elasticsearch shards and fixed them with metrics and automation.Vinted
- Sharding Postgres at FigmaOur nine month journey to horizontally shard Figma’s Postgres stack, and the key to unlocking (nearly) infinite scalability.Figma
- Sharding Postgres at NotionWith an effort to make Notion faster and more reliable for years to come — we migrated Notion’s PostgreSQL monolith into a horizontally-partitioned database fleet.Notion
- Uber Schemaless: A Scalable MySQL DatastoreHow Uber built an append-only, sharded datastore on top of MySQL.Uber
- YouTube ArchitectureA look at YouTube's early scaling decisions around MySQL, caching and video serving.High Scalability
Documentation
Blogs
- PlanetScale BlogMySQL scaling, sharding with Vitess and database internals.PlanetScale
- Shopify EngineeringScaling a very large commerce platform: sharding, resilience, flash sales and payments.Shopify
- Workload isolation using shuffle shardingA technique that assigns each customer a random small subset of servers, so one bad actor affects very few others.Amazon Web Services
- How Discord Stores Trillions of MessagesA migration story from MongoDB to Cassandra to ScyllaDB at huge scale.Discord
- Uber Engineering BlogHow Uber builds dispatch, maps, data platforms and real-time infrastructure.Uber