Storage Engines
Write-ahead logs, B-trees and LSM-trees.
All topics · 2 lessons · 22 resources
Learn Storage Engines on System Design Atlas
- 01Write-ahead logs, B-trees and LSM-treesHow databases make writes durable and fast: the log first, then two ways of organising data on disk and the trade-offs between them.Advanced Distributed Systems · 13 min read · start here
- 02Bitcask: a tiny log-structured key-value storeHow an append-only data file plus an in-memory key directory gives one-seek reads and fast writes, how deletes and merges work, and the limits of the design.Storage, Queues and Real-time · 11 min read
Curated Storage Engines resources
Books
Papers
- Amazon Aurora: Design Considerations for High Throughput Cloud-Native Relational Databases (paper)Why Aurora moves the redo log to a distributed storage layer, so replication and crash recovery become cheap.Amazon Web Services
- Bigtable: A Distributed Storage System for Structured Data (paper)Google's wide-column store: tablets split by key range, a commit log plus sorted files on disk, and the design later copied by HBase and Cassandra.Google
- Bitcask: a log-structured hash table for fast key/value dataThe design note behind Bitcask: append-only files, an in-memory key directory and merging.Basho Technologies
- The Google File System (paper)How Google built a file system for huge files and constant machine failures: one master for metadata, many chunk servers, and append-heavy workloads. The ancestor of HDFS.Google
- The Log-Structured Merge-Tree (paper)The original LSM-tree paper behind write-optimised stores such as Cassandra and RocksDB.UMass Boston
- USENIX ConferencesThe proceedings of top systems conferences (OSDI, NSDI, ATC), mostly free to read.USENIX
GitHub
Blogs
- ClickHouse BlogLearn about the latest ClickHouse tips, tricks and company announcements.ClickHouse
- Datadog Engineering BlogHow an observability company builds high-volume time-series and tracing systems.Datadog
- Dropbox Tech BlogInfrastructure, storage and sync write-ups from Dropbox.Dropbox
- RocksDB Engineering BlogRocksDB is an embeddable persistent key-value store for fast storage.RocksDB
Case Studies
- HaloDB: Embedded Key-Value Storage Engine at Yahooyahoodevelopers: By Arjun Mannaly, Senior Software Engineer At Oath, multiple ad platforms use a high throughput, low latency distributed key-value database that runs in data centers all over the world. The database stores billions of records and handles millions of read and write requests per second at millisecond…Yahoo
- Inside the Magic PocketHow Dropbox built its own exabyte-scale block storage, including durability and cost choices.Dropbox
Documentation
- How SQLite WorksA tour of how a small, widely deployed database stores data and stays crash safe.SQLite
- MinIO documentationAn S3-compatible object store: erasure coding, buckets and deployment.MinIO
- Redis persistenceSnapshots versus the append-only file, and the durability trade-offs of each.Redis Ltd.