Sherpa: Distributed NoSQL Key-Value Store at Yahoo

Ideas you will meet

Distributed Systems

Consensus, time, failure and the fundamentals of scale-out.

Databases

Schemas, indexes, query plans and choosing a store.

What we already know about this

Our own lessons on these ideas. Read them here before you start, so it makes sense.

• System Design Foundations · 11 minChoosing and using a databaseRelational versus NoSQL, what an index really does, and how to decide between them without following fashion.Read hereClose

  The two big families

  Relational databases (PostgreSQL, MySQL) store data in tables with rows and columns, linked by keys. You query them with SQL and can join tables. Their key promise is ACID transactions:

  • Atomicity: a group of changes happens entirely or not at all.
  • Consistency: rules such as foreign keys always hold.
  • Isolation: concurrent transactions do not see each other's half-finished work.
  • Durability: once committed, data survives a crash.

  NoSQL databases is a loose label for stores that give up some relational features for scale or flexibility:

  Type	Shape	Examples	Good for
  Key-value	key maps to a blob	Redis, DynamoDB	caches, sessions, huge lookups
  Document	JSON-like documents	MongoDB, Firestore	content with varied fields
  Wide-column	rows with flexible columns	Cassandra, HBase	massive write volumes, time series
  Graph	nodes and edges	Neo4j	relationship-heavy queries

  A quick map from the kind of question you ask to the kind of store that answers it best:

  100%

  Figure SDA-FND-04-1 · Which kind of store fits the questions you ask · © forcsguy.com

  Image: MarkBuckawicki · CC0 · Source · Wikimedia Commons: Wooden Card Catalog Furniture

  Indexes: how reads get fast

  Without an index, finding rows where email = '[email protected]' means reading every row. With an index on email the database keeps a sorted structure (usually a B-tree) and finds the row in a few steps, even among millions.

  100%

  Figure SDA-FND-04-2 · Indexed lookup versus full scan · © forcsguy.com

  The trade-off: each index takes space and every write must update it. Index the columns you filter, join and sort on, and do not index everything.

  Useful habits:

  • Look at how the database plans a query (EXPLAIN) before guessing.
  • A composite index on (user_id, created_at) serves queries on user_id alone or on both, but not on created_at alone. Column order matters.
  • Very low-variety columns (like a true/false flag) rarely benefit from an index.

  Where relational databases get strained

  A single relational database handles a lot, but you may hit limits:

  • Read load: add copies that serve reads (lesson 5).
  • Write load or data size: split the data across several databases, called sharding (lesson 5).
  • Shape of data: huge volumes of simple records, or extremely flexible documents, may fit another model better.

  Choosing: ask about the access pattern

  1. What are the most frequent queries? By key, by range, by search, by relationship?
  2. Which guarantees are non-negotiable? Strict consistency for money, or "eventually correct" for like counts?
  3. How big and how fast will it grow?
  4. What does the team already know how to operate?

  A worked example: what an index is worth

  A table holds 100 million orders, and the app asks "show this customer's orders" thousands of times a second.

  • Without an index the database reads every row to find matches: a sequential scan. At, say, 200 bytes a row that is 20 GB read from disk or cache for a single query, taking seconds. Do that 1,000 times a second and the database collapses.
  • With a B-tree index on customer_id the database walks a shallow tree. A page holds a few hundred keys, so 100 million rows need only about 4 levels: root, two internal levels, a leaf. That is roughly 4 page reads to find the row locations, then a few more to fetch the rows. Most of the upper levels live in memory, so the real cost is one or two disk reads, in milliseconds.

  The price: every index must be updated on every insert, update and delete. A table with 5 indexes turns one logical insert into about 6 physical writes, and the indexes take space (often as much as the table). So index what your queries actually filter, join and sort on, and not much else.

  100%

  Figure SDA-FND-04-3 · Finding one row among a hundred million with a B-tree · © forcsguy.com

  What a transaction does for you

  A transaction groups several changes so they succeed or fail together. A transfer of 100 from account A to account B is two updates, and it must never be applied halfway. Relational databases give transactions the ACID properties:

  • Atomic: all of it happens or none of it.
  • Consistent: rules such as "balance never negative" are enforced.
  • Isolated: concurrent transactions do not see each other's half-finished work (to a degree you choose).
  • Durable: once the database says "committed", the change survives a crash.

  Durability comes from the write-ahead log (WAL): before changing the data pages, the database appends the change to a log and forces it to disk. If the machine crashes, restart replays the log and the committed work reappears.

  100%

  Figure SDA-FND-04-4 · Why a committed write survives a crash · © forcsguy.com

  You can build a toy version in the write-ahead log lab.

  Mistakes people make

  • No index on the column you filter by, especially foreign keys. Use EXPLAIN to see whether a query scans.
  • Too many indexes. Each one slows writes and uses memory. Drop the ones nobody uses.
  • The N+1 query. Loading 100 orders and then running one query per order for its items is 101 round trips. Load them together with a join or a single IN query.
  • Money as floating point. Use integers (cents) or exact decimals.
  • Choosing NoSQL to avoid thinking about schema. If your data is relational, you will rebuild joins and constraints by hand, badly.
  • Treating the database as a queue or a log. Polling a hot table for work is a common cause of lock contention (see the SKIP LOCKED lesson for doing it properly).
  • Never testing a restore. A backup you have not restored is a hope, not a backup.

  Try it yourself

  Open the indexes in depth lesson and the query optimisation lesson next. In the System Builder, see how much load a cache and replicas take off a single database.

  Real-world reading

  Related reading

  • Amazon Aurora: Design Considerations for High Throughput Cloud-Native Relational Databases (paper)Verbitski et al. · Amazon Web ServicesPapers
  • Minimizing Read-Write MySQL Downtime at YelpYelp engineers · YelpCase Studies
  • Sherpa: Distributed NoSQL Key-Value Store at YahooYahoo engineers · YahooCase Studies
  • Use The Index, LukeMarkus Winand · use-the-index-luke.comCourses

  Show 6 moreShow fewer

  • Fast Order Search using Data Pipeline and Elasticsearch at YelpYelp engineers · YelpCase Studies
  • MPH: Fast and Compact Immutable Key-Value Stores at IndeedIndeed engineers · IndeedCase Studies
  • Espresso: Distributed Document Store at LinkedInLinkedIn engineers · LinkedInCase Studies
  • MongoDB at Baidu: Multi-Tenant Cluster Storing 200+ Billion Documents across 160 ShardsMongodb engineers · MongodbCase Studies
  • Relational DatabasesMysql engineers · MysqlCase Studies
  • Document DatabasesMicrosoft engineers · MicrosoftCase Studies

  Open this lesson with its quiz and exercise

• Big Data and Cloud Data Platforms · 11 minHow big data evolved, and the papers behind itFrom single databases to Hadoop, Spark, streaming and lakehouses: a timeline of ideas and the research papers that started each one.Read hereClose

  Why a new wave kept appearing

  Each generation of data tools answered a limit of the previous one. Understanding the problem behind each wave makes the tools easier to place.

  Image: Trower, NASA · Public domain · Source · Wikimedia Commons: Columbia Supercomputer - NASA Advanced Supercomputing Facility

  A timeline

  100%

  Figure SDA-BDP-01-1 · A timeline · © forcsguy.com

  1. Before: relational databases and data warehouses

  Data fit on one big server. SQL and data warehouses served reports. Limits: scaling up got very expensive, and unstructured data (web logs, clicks) did not fit neatly.

  2. Web scale (around 2003 to 2006)

  Search companies had far more data than any single machine could hold. Google described how it coped, and open-source projects copied the ideas.

  • GFS (Google File System): a distributed file system on commodity machines that expects failures.
  • MapReduce: a model to process huge datasets in parallel.
  • Hadoop (HDFS plus MapReduce) brought both to everyone.

  3. NoSQL (about 2006 to 2010)

  Web applications needed massive scale and high availability that relational systems struggled to give.

  • Bigtable: a sparse, sorted, wide-column store; inspired HBase and Cassandra's data model.
  • Dynamo: an always-writable key-value store with consistent hashing and eventual consistency; inspired Cassandra, Riak and DynamoDB.

  4. Faster and friendlier processing (2010 onward)

  • Dremel: interactive SQL over huge datasets using columnar storage and tree-shaped execution; the idea behind BigQuery and later engines, and the Parquet format.
  • Spark (resilient distributed datasets): keeps data in memory between steps, much faster than disk-based MapReduce for iterative work, with friendly APIs.

  5. Streaming

  Waiting for nightly batches was too slow. Kafka made a durable, replayable log the backbone of event pipelines, and engines such as Flink and the Dataflow model handled continuous computation with proper treatment of event time, windows and late data.

  6. The cloud

  Cloud providers separated storage (cheap object storage) from compute (elastic clusters or serverless queries). Managed warehouses (BigQuery, Redshift, Snowflake and others) removed cluster management.

  7. The lakehouse

  Data lakes were cheap but unreliable (no transactions, messy files). Table formats (Delta Lake, Apache Iceberg, Apache Hudi) add ACID transactions, schema evolution and time travel on top of files in object storage, so one copy of data can serve both analytics and machine learning.

  100%

  Figure SDA-BDP-01-2 · Key dates, in order · © forcsguy.com

  The classic papers

  These are the research papers most often cited. Search the title to find each; most are freely available.

  Paper	Authors (and year)	The idea
  The Google File System	Ghemawat, Gobioff, Leung (2003)	Large-file storage across commodity machines, built to tolerate failures
  MapReduce: Simplified Data Processing on Large Clusters	Dean, Ghemawat (2004)	A simple map and reduce model with automatic parallelism and fault tolerance
  Bigtable: A Distributed Storage System for Structured Data	Chang and others (2006)	A sorted, sparse, multi-dimensional map for structured data at scale
  Dynamo: Amazon's Highly Available Key-value Store	DeCandia and others (2007)	Consistent hashing, quorums, vector clocks and eventual consistency
  Dremel: Interactive Analysis of Web-Scale Datasets	Melnik and others (2010)	Columnar nested data and tree execution for fast SQL
  Kafka: a Distributed Messaging System for Log Processing	Kreps, Narkhede, Rao (2011)	A partitioned, replicated log as the backbone for event data
  Resilient Distributed Datasets (Spark)	Zaharia and others (2012)	In-memory, fault-tolerant parallel computing
  Spanner: Google's Globally-Distributed Database	Corbett and others (2012)	Strongly consistent, globally distributed SQL using synchronised clocks
  The Dataflow Model	Akidau and others (2015)	A unified model for batch and streaming with event time and windows
  Delta Lake: High-Performance ACID Table Storage over Cloud Object Stores	Armbrust and others (2020)	Transactional tables on object storage: the lakehouse idea

  A good companion is Jay Kreps's essay "The Log: What every software engineer should know about real-time data's unifying abstraction" (2013).

  What to take away

  • Scale-out on cheap machines (GFS, MapReduce) made big data practical.
  • Availability versus consistency choices (Dynamo, Bigtable, Spanner) shaped databases.
  • Columnar storage (Dremel, Parquet) made analytics fast.
  • Logs and streams (Kafka, dataflow) made data continuous.
  • Separating storage from compute and open table formats define today's platforms.

  Real-world reading

  Related reading

  • Dremel: Interactive Analysis of Web-Scale Datasets (paper)Melnik et al. · GooglePapers
  • Cassandra architecture overviewApache Cassandra · Apache Software FoundationDocumentation
  • Fast Order Search using Data Pipeline and Elasticsearch at YelpYelp engineers · YelpCase Studies
  • Apache CassandraApache Software Foundation · GitHubGitHub

  Show 6 moreShow fewer

  • Amazon S3 strong consistencyAmazon Web ServicesDocumentation
  • Sherpa: Distributed NoSQL Key-Value Store at YahooYahoo engineers · YahooCase Studies
  • Elasticsearch GuideElasticDocumentation
  • MapReduce: Simplified Data Processing on Large Clusters (paper)Dean and Ghemawat · GooglePapers
  • HBase at SalesforceSalesforce engineers · SalesforceCase Studies
  • Columnar DatabasesAmazon Web Services engineers · Amazon Web ServicesCase Studies

  Open this lesson with its quiz and exercise

• Databases Deep Dive · 12 minReplication topologies and multi-region databasesSingle-leader, multi-leader and leaderless replication, how they handle conflicts and lag, and what changes across regions.Read hereClose

  Why replicate

  Replication keeps copies of data on several machines for three reasons: availability (survive a node failure), read scaling (serve reads from replicas) and latency (put data near users). The hard part is keeping copies consistent while writes continue. (Basics are in the replication and sharding lesson of the foundations course; this lesson compares the topologies.)

  Single-leader (primary and replicas)

  One node, the leader (primary or master), accepts writes and ships its changes to followers (replicas), which serve reads.

  100%

  Figure SDA-DDV-04-1 · Single-leader replication · © forcsguy.com

  • Synchronous replication waits for a follower before acknowledging: safer, slower, and a stalled follower can block writes. Asynchronous is faster but a failover can lose the latest writes. A common compromise: one synchronous follower, others asynchronous.
  • Failover must detect failure, choose the most up-to-date replica, redirect clients and ensure the old leader does not come back believing it still leads (split brain).
  • Replication lag causes stale reads. Techniques: read-your-writes (route a user's reads to the leader shortly after they write), monotonic reads (keep a user on one replica), and checking replica position.

  Multi-leader

  Several nodes accept writes (often one per region) and replicate to each other. Benefits: local write latency and continued operation if a region is cut off. Cost: write conflicts.

  Conflict handling options:

  • Avoid them: route all writes for one record to one leader (for example by user or region).
  • Last write wins by timestamp: simple, but silently drops one of two concurrent updates and depends on clocks.
  • Merge values (union of sets, counters) using mergeable data types (CRDTs; see the collaborative editing lesson).
  • Custom resolution in application code, or ask the user.

  Multi-leader also appears in offline-capable apps (each device is a leader) and collaborative editing.

  Before: single leader

  100%

  After: multi-leader across regions

  100%

  What changed

  • Users in each region write to a nearby database
  • Concurrent edits to the same row need conflict handling
  • More moving parts to run and monitor

  Leaderless (Dynamo-style)

  Any replica accepts writes; clients write to several replicas and read from several, using quorums (W + R > N) to get fresh data (see the key-value store lesson). Background anti-entropy and read repair fix stale copies; versions or vector clocks detect conflicts. It favours availability and is typical of Cassandra and DynamoDB-style stores.

  100%

  Figure SDA-DDV-04-2 · Choosing a replication model · © forcsguy.com

  Multi-region designs

  • Active-passive: one region serves writes; another is a warm standby. Simple; failover loses a little data (the replication lag) and takes minutes.
  • Active-active: several regions serve traffic. Requires either partitioning data by home region (each record has one home, cross-region reads are replicas) or true multi-leader conflict handling.
  • Global databases use consensus across regions (see the consensus lesson) for strong consistency at the cost of cross-region write latency; some offer tunable consistency per query.
  • Latency numbers matter: a cross-ocean round trip is around 100 ms or more, so synchronous cross-region replication adds that to every write.
  • Data residency laws may require keeping certain users' data in a region.

  Partitioning and replication together

  Real systems shard data into partitions and replicate each partition (a leader per partition spreads write load across nodes). Rebalancing, hot partitions and cross-partition transactions are the new problems (see sharding and the consistent-hashing lesson).

  Master/slave naming

  You will see "master/slave", "primary/replica" and "leader/follower" for the same single-leader idea; most current documentation prefers the last two.

  Checklist

  • What happens to writes if the leader or a region is lost, and how long?
  • How much data can we lose (RPO) and how long can we be down (RTO)?
  • How do we handle stale reads from replicas?
  • Who resolves conflicts, and how?
  • Have we practised failover?

  Real-world reading

  Related reading

  • Active-Active for Multi-Regional ResiliencyNetflix Engineering · NetflixCase Studies
  • Designing Data-Intensive ApplicationsMartin Kleppmann · O'ReillyBooks
  • Read Consistency with Database Replicas at ShopifyShopify engineers · ShopifyCase Studies
  • Uber Schemaless: A Scalable MySQL DatastoreUber Engineering · UberCase Studies

  Show 6 moreShow fewer

  • Mitigating MySQL Replication Lag and Reducing Read Load at GithubGithub engineers · GithubCase Studies
  • Cassandra architecture overviewApache Cassandra · Apache Software FoundationDocumentation
  • How Discord Stores Trillions of MessagesBo Ingram · DiscordBlogs
  • Spanner: Google's Globally Distributed Database (paper)Corbett et al. · GooglePapers
  • Scaling Multitenant Architecture Across Multiple Data Centres at Shopify - Weingarten, Engineering Lead at ShopifyShopifyYouTube
  • MIT 6.824: Distributed SystemsRobert Morris · MITCourses

  Open this lesson with its quiz and exercise