Resiliency and Disaster Recovery with Kafka at eBay

Ideas you will meet

Kafka

Distributed commit log for event streaming at scale.

Resilience

Timeouts, retries, circuit breakers, bulkheads and load shedding.

What we already know about this

Our own lessons on these ideas. Read them here before you start, so it makes sense.

• Core Concepts: How Things Actually Work · 7 minWhy Kafka is fastSequential disk writes, batching, zero-copy and partitions: the design choices behind its throughput.Read hereClose

  What Kafka is

  Kafka is a distributed commit log: producers append messages to topics, and consumers read them at their own pace. Messages are kept for a retention period, not deleted when read, so many consumers can read the same data, and replay it.

  Why it is so fast

  1. Sequential I/O. Each partition is an append-only log. Writing to the end of a file and reading in order is vastly faster than random access.
  2. The page cache. Kafka relies on the operating system's cache rather than its own heap, so recent data is served from memory without extra copying or garbage collection.
  3. Batching. Producers and brokers group many messages per request, and data is compressed per batch, amortising network and disk costs.
  4. Zero-copy transfer. The broker can send data from the file cache to the network socket without copying it through application memory.
  5. Partitioning. A topic is split into partitions spread over brokers, so reads and writes happen in parallel.
  6. A simple model. The broker mostly appends and serves byte ranges. Consumers track their own position (the offset), so the broker does little bookkeeping per message.

  100%

  Figure SDA-CORE-10-1 · Why it is so fast · © forcsguy.com

  100%

  Figure SDA-CORE-10-2 · Zero-copy: fewer trips through memory · © forcsguy.com

  Ordering and parallelism

  • Order is guaranteed within a partition, not across a whole topic.
  • The message key decides the partition, so related events (same customer) stay in order.
  • A consumer group shares partitions: each partition is read by exactly one consumer in the group. More partitions allow more parallel consumers.

  Reliability

  • Each partition is replicated to several brokers: one leader and followers.
  • Producers choose an acknowledgment level (acks): faster with fewer, safer with all in-sync replicas.
  • Consumers commit offsets after processing. If they crash, they resume from the last committed offset, so processing is at-least-once and handlers should be idempotent.

  When to use it, and when not

  Good for event streams, log and metric pipelines, and decoupling systems with replayable history. A plain task queue (RabbitMQ, SQS) can be simpler when you only need work distribution and no replay.

  More on this site

  Related reading

  • Kafka Source & DocsApache Software Foundation · apacheGitHub
  • LinkedIn Engineering BlogLinkedIn engineers · LinkedInBlogs
  • Confluent BlogConfluent engineers · ConfluentBlogs
  • Uber Engineering BlogUber Engineering · UberBlogs

  Show 6 moreShow fewer

  • Finding Kafka throughput limit in infrastructure at DropboxDropbox engineers · DropboxCase Studies
  • Kafka - Message BrokerKleppmannPapers
  • LinkedIn Engineering: DataLinkedIn engineers · LinkedInBlogs
  • Kafka on PaaSTA: Running Kafka on Kubernetes at Yelp (2 parts)Yelp engineers · YelpCase Studies
  • Freeletics Engineering BlogFreeletics engineers · FreeleticsBlogs
  • Findmypast Engineering BlogFindmypast engineers · FindmypastBlogs

  Open this lesson with its quiz and exercise

• Advanced Distributed Systems · 13 minKafka internals: segments, replication and consumer groupsWhat is inside a topic: partitions as append-only logs split into segments, how followers keep copies, and how consumer groups share the work.Read hereClose

  A topic is a set of logs

  A Kafka topic is split into partitions. Each partition is an append-only log: new records go on the end and get an increasing offset. Order is guaranteed only within a partition; records with the same key go to the same partition, so per-key order holds. Consumers track how far they have read by offset. (Why this design is fast is covered in the "Why is Kafka fast" lesson.)

  On disk: segments

  A partition's log is stored as a series of segment files. The newest segment is the one being written; older segments are immutable. Each segment has companion index files that map offsets (and timestamps) to positions in the file, so a consumer can jump to an offset quickly.

  • Retention is applied by deleting whole old segments (by age or total size), which is cheap.
  • Compaction (for some topics) keeps only the latest record per key, turning the log into a changelog of current state.

  100%

  Figure SDA-ADS-03-1 · Partitions, segments and replicas · © forcsguy.com

  Replication and durability

  Each partition has a replication factor (commonly 3): one leader and followers. Followers fetch from the leader. The in-sync replicas (ISR) are the followers that are caught up.

  A producer chooses an acknowledgement level:

  Setting	Meaning	Trade-off
  acks=0	Do not wait	Fastest, may lose data
  acks=1	Leader has written it	Loses data if the leader fails before replicating
  acks=all	All in-sync replicas have it	Safest, slower

  With acks=all and a minimum in-sync replicas setting, a write succeeds only if enough copies exist, trading availability for durability. If the leader fails, a new leader is chosen from the in-sync replicas, so acknowledged writes survive. (Older versions used ZooKeeper for cluster metadata; newer ones use a built-in Raft-based controller; the idea of consensus on metadata is the same as in the consensus lesson.)

  Consumer groups and rebalancing

  Consumers sharing a group id split a topic's partitions: each partition is read by one member at a time, so parallelism is limited by the number of partitions. Different groups each read the full topic independently.

  100%

  Figure SDA-ADS-03-2 · Consumer group rebalance · © forcsguy.com

  Offsets are committed back to Kafka. Commit after processing for at-least-once delivery (duplicates possible, so make consumers idempotent). Rebalances happen when members join, leave, crash or are too slow (missing heartbeats or taking too long to poll). Frequent rebalances hurt throughput, so keep processing per batch short and group membership stable.

  Practical rules

  • Partition count is a design decision: it caps consumer parallelism, and increasing it later changes the key-to-partition mapping.
  • Key choice controls ordering and balance: pick a key with many distinct values.
  • Monitor consumer lag per partition, under-replicated partitions and request latency.
  • Idempotent producers and transactions exist for stronger guarantees (exactly-once within Kafka); end-to-end exactly-once still needs idempotent consumers or transactional sinks.
  • Stream processing (Kafka Streams, Flink) builds on the same consumer mechanics; see the big-data platforms course.

  Real-world reading

  Related reading

  • Exactly-once Semantics Is Possible: Here's How Apache Kafka Does ItNeha Narkhede · ConfluentBlogs
  • Kafka Source & DocsApache Software Foundation · apacheGitHub
  • Confluent Developer: Apache Kafka 101ConfluentCourses
  • Confluent: Apache Kafka TalksConfluentYouTube

  Show 6 moreShow fewer

  • Jordan Has No Life: Systems Design Concepts SeriesJordan · Jordan Has No LifeYouTube
  • Apache Kafka DocumentationApache Kafka · Apache Software FoundationDocumentation
  • Apache CassandraApache Software Foundation · GitHubGitHub
  • Cassandra architecture overviewApache Cassandra · Apache Software FoundationDocumentation
  • REA Group Engineering BlogREA Group engineers · REA GroupBlogs
  • Repo-Topix: Topic Extraction Framework at GithubGithub engineers · GithubCase Studies

  Open this lesson with its quiz and exercise

• Advanced Distributed Systems · 13 minResilience: timeouts, retries, bulkheads, backpressure and circuit breakersHow to stop one slow or failing dependency from taking everything down: the patterns, how they fit together and where each one goes wrong.Read hereClose

  The core problem

  In a system of many services, slowness is more dangerous than failure. A dependency that fails fast is easy to handle. One that responds in 30 seconds makes callers hold threads, connections and memory until they too stop responding: a cascading failure. Resilience patterns limit how far a problem can spread.

  The toolbox

  100%

  Figure SDA-ADS-06-1 · Resilience layers around a call · © forcsguy.com

  Timeouts

  Every network call needs a timeout, including connect, read and overall. Pick from measured latency (for example slightly above the 99th percentile), not a guess, and make the caller's budget smaller than its own caller's so work is abandoned in an orderly way. Propagate deadlines across calls.

  Retries

  Retry only transient failures and only idempotent operations (or use idempotency keys). Use exponential backoff with jitter (random spread) so clients do not retry in lockstep, cap the attempts, and apply a retry budget (for example no more than 10 percent extra traffic). Retry at one layer, not at every layer, or attempts multiply.

  Circuit breaker

  A breaker watches failures or slow calls. Closed: normal. After too many failures it opens: calls fail immediately (or use the fallback) so the dependency can recover and callers stop wasting resources. After a cool-down it goes half-open: a few trial calls decide whether to close again.

  100%

  Figure SDA-ADS-06-2 · Circuit breaker states · © forcsguy.com

  Bulkheads

  Partition resources so one dependency cannot consume them all: separate thread pools or connection pools per dependency, separate queues per tenant or per priority, or separate instances for critical paths. A slow recommendation service then cannot starve checkout.

  Backpressure and load shedding

  If producers outpace consumers, work piles up. Without limits the buffer grows until memory is exhausted or latency becomes huge. Options:

  • Bounded queues that reject or block when full.
  • Signalling upstream to slow down (credit-based flow in streaming systems, TCP windows, reactive streams).
  • Load shedding: deliberately drop low-priority work early (return 429 or 503 quickly) to protect the rest.
  • Rate limiting at the edge (see the rate-limiter lesson).

  Failing fast and visibly under overload is better than accepting everything and failing slowly.

  Fallbacks and graceful degradation

  Return something useful when a dependency fails: cached data, a default, a simpler algorithm, or hide a non-essential section. Decide which features are essential before an incident, and test the degraded mode.

  How they interact

  • A timeout is the minimum; without it nothing else works.
  • Retries inside a breaker inside a bulkhead is the usual nesting.
  • Everything needs metrics: breaker state, rejected calls, queue depth, timeout counts. Alert on them.
  • Test it: inject latency and failures (chaos experiments) to confirm the patterns work before production does it for you.

  Common mistakes

  Retries with no backoff; timeouts longer than the caller's; unbounded queues; a shared thread pool for everything; breakers that never close; fallbacks that call the same failing service; and testing only the happy path.

  Real-world reading

  Related reading

  • Timeouts, retries, and backoff with jitterAmazon Builders' Library · Amazon Web ServicesBlogs
  • Using load shedding to avoid overloadAmazon Builders' Library · Amazon Web ServicesBlogs
  • Circuit BreakerMartin FowlerBlogs
  • Circuit Breaker patternMicrosoft · Microsoft AzureDocumentation

  Show 6 moreShow fewer

  • Google SRE Book: Addressing Cascading FailuresGoogle SRE team · GoogleBlogs
  • Chaos Monkey (source)Netflix · GitHubGitHub
  • Bulkheads: Partition and Tolerate Failure in One PartSkife engineers · SkifeCase Studies
  • Protector: Circuit Breaker for Time Series Databases at TrivagoTrivago engineers · TrivagoCase Studies
  • Autoscaling Jenkins at TrivagoTrivago engineers · TrivagoCase Studies
  • Troubleshooting a Connection Timeout Issue with tcp_tw_recycle Enabled at eBayeBay engineers · eBayCase Studies

  Open this lesson with its quiz and exercise