Workload isolation using shuffle sharding

Ideas you will meet

Resilience

Timeouts, retries, circuit breakers, bulkheads and load shedding.

Database Sharding

Partition data across nodes by key, range or directory.

What we already know about this

Our own lessons on these ideas. Read them here before you start, so it makes sense.

• Advanced Distributed Systems · 13 minResilience: timeouts, retries, bulkheads, backpressure and circuit breakersHow to stop one slow or failing dependency from taking everything down: the patterns, how they fit together and where each one goes wrong.Read hereClose

  The core problem

  In a system of many services, slowness is more dangerous than failure. A dependency that fails fast is easy to handle. One that responds in 30 seconds makes callers hold threads, connections and memory until they too stop responding: a cascading failure. Resilience patterns limit how far a problem can spread.

  The toolbox

  100%

  Figure SDA-ADS-06-1 · Resilience layers around a call · © forcsguy.com

  Timeouts

  Every network call needs a timeout, including connect, read and overall. Pick from measured latency (for example slightly above the 99th percentile), not a guess, and make the caller's budget smaller than its own caller's so work is abandoned in an orderly way. Propagate deadlines across calls.

  Retries

  Retry only transient failures and only idempotent operations (or use idempotency keys). Use exponential backoff with jitter (random spread) so clients do not retry in lockstep, cap the attempts, and apply a retry budget (for example no more than 10 percent extra traffic). Retry at one layer, not at every layer, or attempts multiply.

  Circuit breaker

  A breaker watches failures or slow calls. Closed: normal. After too many failures it opens: calls fail immediately (or use the fallback) so the dependency can recover and callers stop wasting resources. After a cool-down it goes half-open: a few trial calls decide whether to close again.

  100%

  Figure SDA-ADS-06-2 · Circuit breaker states · © forcsguy.com

  Bulkheads

  Partition resources so one dependency cannot consume them all: separate thread pools or connection pools per dependency, separate queues per tenant or per priority, or separate instances for critical paths. A slow recommendation service then cannot starve checkout.

  Backpressure and load shedding

  If producers outpace consumers, work piles up. Without limits the buffer grows until memory is exhausted or latency becomes huge. Options:

  • Bounded queues that reject or block when full.
  • Signalling upstream to slow down (credit-based flow in streaming systems, TCP windows, reactive streams).
  • Load shedding: deliberately drop low-priority work early (return 429 or 503 quickly) to protect the rest.
  • Rate limiting at the edge (see the rate-limiter lesson).

  Failing fast and visibly under overload is better than accepting everything and failing slowly.

  Fallbacks and graceful degradation

  Return something useful when a dependency fails: cached data, a default, a simpler algorithm, or hide a non-essential section. Decide which features are essential before an incident, and test the degraded mode.

  How they interact

  • A timeout is the minimum; without it nothing else works.
  • Retries inside a breaker inside a bulkhead is the usual nesting.
  • Everything needs metrics: breaker state, rejected calls, queue depth, timeout counts. Alert on them.
  • Test it: inject latency and failures (chaos experiments) to confirm the patterns work before production does it for you.

  Common mistakes

  Retries with no backoff; timeouts longer than the caller's; unbounded queues; a shared thread pool for everything; breakers that never close; fallbacks that call the same failing service; and testing only the happy path.

  Real-world reading

  Related reading

  • Timeouts, retries, and backoff with jitterAmazon Builders' Library · Amazon Web ServicesBlogs
  • Using load shedding to avoid overloadAmazon Builders' Library · Amazon Web ServicesBlogs
  • Circuit BreakerMartin FowlerBlogs
  • Circuit Breaker patternMicrosoft · Microsoft AzureDocumentation

  Show 6 moreShow fewer

  • Google SRE Book: Addressing Cascading FailuresGoogle SRE team · GoogleBlogs
  • Chaos Monkey (source)Netflix · GitHubGitHub
  • Bulkheads: Partition and Tolerate Failure in One PartSkife engineers · SkifeCase Studies
  • Protector: Circuit Breaker for Time Series Databases at TrivagoTrivago engineers · TrivagoCase Studies
  • Autoscaling Jenkins at TrivagoTrivago engineers · TrivagoCase Studies
  • Troubleshooting a Connection Timeout Issue with tcp_tw_recycle Enabled at eBayeBay engineers · eBayCase Studies

  Open this lesson with its quiz and exercise

• Core Concepts: How Things Actually Work · 9 minTransactions and isolation levelsHow databases keep concurrent updates from corrupting each other, and the anomalies each isolation level allows.Read hereClose

  What a transaction is

  A transaction groups several operations so they are treated as one unit. The classic guarantees are ACID:

  • Atomicity: all or nothing.
  • Consistency: rules (constraints) always hold afterwards.
  • Isolation: concurrent transactions do not see each other's half-done work.
  • Durability: once committed, it survives a crash.

  Isolation is the subtle one, because perfect isolation is expensive.

  What can go wrong when transactions overlap

  Anomaly	What happens
  Dirty read	You read data another transaction has not committed (and may roll back)
  Non-repeatable read	You read a row twice and it changed in between
  Phantom read	You run the same query twice and new matching rows appear
  Lost update	Two transactions read, then both write, and one overwrites the other

  Isolation levels

  Databases offer levels that trade safety for speed:

  Level	Prevents	Still allows
  Read uncommitted	nothing much	dirty reads and more
  Read committed (a common default)	dirty reads	non-repeatable and phantom reads
  Repeatable read	dirty and non-repeatable reads	phantoms (varies by database)
  Serializable	all of the above	nothing, but slower or more retries

  The exact behaviour differs between databases, so read your database's documentation instead of assuming.

  100%

  Figure SDA-CORE-02-1 · Isolation levels · © forcsguy.com

  How databases implement it

  • Locks: readers and writers wait for each other. Simple, but can block and deadlock.
  • MVCC (multi-version concurrency control): each transaction sees a consistent snapshot while writers create new versions. Readers do not block writers. Used by PostgreSQL, MySQL (InnoDB) and others.

  Two mechanisms keep concurrent transactions apart. Locks make others wait; row versions let readers carry on:

  100%

  Figure SDA-CORE-02-2 · Locks versus row versions · © forcsguy.com

  A multi-version example

  Say row X holds 100 at version 1. Transaction A begins and takes a snapshot. Transaction B updates X to 150, creating version 2, and commits. A still reads 100 because its snapshot predates version 2, while a transaction started afterwards reads 150. Readers never block writers; old versions are cleaned up once no snapshot needs them.

  Practical advice

  • Keep transactions short. Long ones hold locks and snapshots and hurt everyone.
  • Prefer conditional updates (WHERE seats > 0) over read-then-write in application code.
  • Be ready to retry on serialization failures and deadlocks.
  • Across several databases or services there is no single transaction: use patterns such as sagas (later lessons).

  Real-world reading

  Related reading

  • Workload isolation using shuffle shardingAmazon Builders' Library · Amazon Web ServicesBlogs
  • How Discord Stores Billions of MessagesDiscordCase Studies
  • Cross Shard Transactions at 10 Million RPS at DropboxDropbox engineers · DropboxCase Studies
  • HaloDB: Embedded Key-Value Storage Engine at YahooYahoo engineers · YahooCase Studies

  Show 6 moreShow fewer

  • Explaining Low-Level Systems (OS, Network/Protocol, Database, Storage)Wustl engineers · WustlBlogs
  • GitHub EngineeringGitHub engineers · GitHubBlogs
  • Profiling C++ (Part 1: Optimization, Part 2: Measurement and Analysis) at Riot GamesRiot Games engineers · Riot GamesCase Studies
  • VitessCNCF / Vitess · GitHubGitHub
  • 8th Light Engineering Blog8th Light engineers · 8th LightBlogs
  • Service Mesh at SnapSnap engineers · SnapCase Studies

  Open this lesson with its quiz and exercise

• Big Data and Cloud Data Platforms · 10 minChoosing a cloud provider for a big data solutionA fair comparison of the main data services on AWS, Azure and Google Cloud, and a decision process that avoids choosing by brand.Read hereClose

  Start from needs, not logos

  No provider is best for everyone. The right choice depends on where your data and applications already are, your team's skills, what workloads you run and how you will pay. Marketing comparisons and generic benchmarks matter less than a test on your own data.

  The same building blocks, different names

  Need	AWS	Microsoft Azure	Google Cloud
  Object storage (the lake)	S3	Azure Data Lake Storage Gen2 (Blob Storage)	Cloud Storage
  Managed Spark and Hadoop	EMR (and Glue jobs)	Azure Databricks, Synapse Spark, HDInsight	Dataproc
  Data warehouse	Redshift	Synapse Analytics (and Microsoft Fabric)	BigQuery
  Query files in place	Athena	Synapse serverless SQL	BigQuery external tables
  Streaming ingest	Kinesis, MSK (managed Kafka)	Event Hubs	Pub/Sub
  Stream processing	Managed Flink, Kinesis analytics	Stream Analytics	Dataflow
  Orchestration	Step Functions, MWAA (managed Airflow), Glue workflows	Data Factory	Cloud Composer (managed Airflow), Workflows
  Catalogue and governance	Glue Data Catalog, Lake Formation	Microsoft Purview	Dataplex, Data Catalog

  Independent platforms such as Snowflake and Databricks run on all three and are common choices when you want a multi-cloud or cloud-neutral platform.

  100%

  Figure SDA-BDP-04-1 · The same building blocks, different names · © forcsguy.com

  100%

  Figure SDA-BDP-04-2 · The stages of a data platform, whichever cloud you choose · © forcsguy.com

  Criteria that matter

  1. Data gravity and egress: where the data already is. Moving petabytes is slow and incurs transfer fees.
  2. Existing ecosystem: identity, networking, security tooling and applications already on a cloud reduce integration work.
  3. Team skills: a SQL-first team may prefer a managed warehouse; an engineering-heavy team may prefer Spark and open formats.
  4. Workload fit: interactive BI, large batch, streaming, machine learning, geospatial. Each service has strengths.
  5. Cost model: warehouses charge differently: per data scanned (good for occasional queries), per cluster or capacity time (good for steady heavy use), plus storage and egress. Model your pattern.
  6. Performance and scale: test with your real queries, since results depend on data shape.
  7. Governance, security and compliance: data residency, encryption, fine-grained access, auditing, certifications for your industry.
  8. Openness and lock-in: prefer open file and table formats (Parquet, Iceberg, Delta) and standard SQL so you can switch engines. Proprietary features give convenience at the cost of portability.
  9. Reliability and support: regions available, service limits, support plans.
  10. Operational burden: managed and serverless services reduce work but cost more per unit.

  A sensible process

  1. Write down requirements: data volume and growth, latency needs, users, workloads, compliance.
  2. Shortlist two or three options, including your current cloud.
  3. Proof of concept with a representative slice of data and your real queries; measure speed, ease and cost.
  4. Model total cost over a year or more: storage, compute, transfer, support and people.
  5. Decide with a weighted scorecard, and record why.
  6. Plan the exit: open formats and a clean data layer make a future change feasible.

  Common mistakes

  • Choosing on brand or a single benchmark.
  • Ignoring egress and cross-region transfer costs.
  • Underestimating the people and operations cost of a self-managed cluster.
  • Loading everything into a proprietary format with no way out.
  • Skipping governance until the platform is already messy.
  • Building for petabytes when you have gigabytes: a single powerful database or a small warehouse may be enough.

  Real-world reading

  Related reading

  • The Architecture of Open Source ApplicationsVarious authors · aosabook.orgBooks
  • eBay: Building Mission-Critical Multi-Data Center Applications with MongoDBMongodb engineers · MongodbCase Studies
  • Improving Search Engine Efficiency by over 25% at eBayeBay engineers · eBayCase Studies
  • Bullet: Forward-Looking Query Engine for Streaming Data at YahooYahoo engineers · YahooCase Studies

  Show 6 moreShow fewer

  • Dremel: Interactive Analysis of Web-Scale Datasets (paper)Melnik et al. · GooglePapers
  • Reasons for Choosing S3 over HDFS at DatabricksDatabricks engineers · DatabricksCase Studies
  • Self-Serve Reporting Platform on Hadoop at LinkedInLinkedInPapers
  • Workload isolation using shuffle shardingAmazon Builders' Library · Amazon Web ServicesBlogs
  • Altus Care: Applying a Chatbot to Platform Engineering at eBayeBay engineers · eBayCase Studies
  • Microsoft Azure BlogMicrosoft engineers · MicrosoftBlogs

  Open this lesson with its quiz and exercise