REA Group Engineering Blog

Ideas you will meet

Distributed Systems

Consensus, time, failure and the fundamentals of scale-out.

What we already know about this

Our own lessons on these ideas. Read them here before you start, so it makes sense.

• Advanced Distributed Systems · 13 minKafka internals: segments, replication and consumer groupsWhat is inside a topic: partitions as append-only logs split into segments, how followers keep copies, and how consumer groups share the work.Read hereClose

  A topic is a set of logs

  A Kafka topic is split into partitions. Each partition is an append-only log: new records go on the end and get an increasing offset. Order is guaranteed only within a partition; records with the same key go to the same partition, so per-key order holds. Consumers track how far they have read by offset. (Why this design is fast is covered in the "Why is Kafka fast" lesson.)

  On disk: segments

  A partition's log is stored as a series of segment files. The newest segment is the one being written; older segments are immutable. Each segment has companion index files that map offsets (and timestamps) to positions in the file, so a consumer can jump to an offset quickly.

  • Retention is applied by deleting whole old segments (by age or total size), which is cheap.
  • Compaction (for some topics) keeps only the latest record per key, turning the log into a changelog of current state.

  100%

  Figure SDA-ADS-03-1 · Partitions, segments and replicas · © forcsguy.com

  Replication and durability

  Each partition has a replication factor (commonly 3): one leader and followers. Followers fetch from the leader. The in-sync replicas (ISR) are the followers that are caught up.

  A producer chooses an acknowledgement level:

  Setting	Meaning	Trade-off
  acks=0	Do not wait	Fastest, may lose data
  acks=1	Leader has written it	Loses data if the leader fails before replicating
  acks=all	All in-sync replicas have it	Safest, slower

  With acks=all and a minimum in-sync replicas setting, a write succeeds only if enough copies exist, trading availability for durability. If the leader fails, a new leader is chosen from the in-sync replicas, so acknowledged writes survive. (Older versions used ZooKeeper for cluster metadata; newer ones use a built-in Raft-based controller; the idea of consensus on metadata is the same as in the consensus lesson.)

  Consumer groups and rebalancing

  Consumers sharing a group id split a topic's partitions: each partition is read by one member at a time, so parallelism is limited by the number of partitions. Different groups each read the full topic independently.

  100%

  Figure SDA-ADS-03-2 · Consumer group rebalance · © forcsguy.com

  Offsets are committed back to Kafka. Commit after processing for at-least-once delivery (duplicates possible, so make consumers idempotent). Rebalances happen when members join, leave, crash or are too slow (missing heartbeats or taking too long to poll). Frequent rebalances hurt throughput, so keep processing per batch short and group membership stable.

  Practical rules

  • Partition count is a design decision: it caps consumer parallelism, and increasing it later changes the key-to-partition mapping.
  • Key choice controls ordering and balance: pick a key with many distinct values.
  • Monitor consumer lag per partition, under-replicated partitions and request latency.
  • Idempotent producers and transactions exist for stronger guarantees (exactly-once within Kafka); end-to-end exactly-once still needs idempotent consumers or transactional sinks.
  • Stream processing (Kafka Streams, Flink) builds on the same consumer mechanics; see the big-data platforms course.

  Real-world reading

  Related reading

  • Exactly-once Semantics Is Possible: Here's How Apache Kafka Does ItNeha Narkhede · ConfluentBlogs
  • Kafka Source & DocsApache Software Foundation · apacheGitHub
  • Confluent Developer: Apache Kafka 101ConfluentCourses
  • Confluent: Apache Kafka TalksConfluentYouTube

  Show 6 moreShow fewer

  • Jordan Has No Life: Systems Design Concepts SeriesJordan · Jordan Has No LifeYouTube
  • Apache Kafka DocumentationApache Kafka · Apache Software FoundationDocumentation
  • Apache CassandraApache Software Foundation · GitHubGitHub
  • Cassandra architecture overviewApache Cassandra · Apache Software FoundationDocumentation
  • REA Group Engineering BlogREA Group engineers · REA GroupBlogs
  • Repo-Topix: Topic Extraction Framework at GithubGithub engineers · GithubCase Studies

  Open this lesson with its quiz and exercise