Kafka internals: segments, replication and consumer groups

A topic is a set of logs

A Kafka topic is split into partitions. Each partition is an append-only log: new records go on the end and get an increasing offset. Order is guaranteed only within a partition; records with the same key go to the same partition, so per-key order holds. Consumers track how far they have read by offset. (Why this design is fast is covered in the "Why is Kafka fast" lesson.)

On disk: segments

A partition's log is stored as a series of segment files. The newest segment is the one being written; older segments are immutable. Each segment has companion index files that map offsets (and timestamps) to positions in the file, so a consumer can jump to an offset quickly.

• Retention is applied by deleting whole old segments (by age or total size), which is cheap.
• Compaction (for some topics) keeps only the latest record per key, turning the log into a changelog of current state.

100%

Figure SDA-ADS-03-1 · Partitions, segments and replicas · © forcsguy.com

Replication and durability

Each partition has a replication factor (commonly 3): one leader and followers. Followers fetch from the leader. The in-sync replicas (ISR) are the followers that are caught up.

A producer chooses an acknowledgement level:

Setting	Meaning	Trade-off
acks=0	Do not wait	Fastest, may lose data
acks=1	Leader has written it	Loses data if the leader fails before replicating
acks=all	All in-sync replicas have it	Safest, slower

With acks=all and a minimum in-sync replicas setting, a write succeeds only if enough copies exist, trading availability for durability. If the leader fails, a new leader is chosen from the in-sync replicas, so acknowledged writes survive. (Older versions used ZooKeeper for cluster metadata; newer ones use a built-in Raft-based controller; the idea of consensus on metadata is the same as in the consensus lesson.)

Consumer groups and rebalancing

Consumers sharing a group id split a topic's partitions: each partition is read by one member at a time, so parallelism is limited by the number of partitions. Different groups each read the full topic independently.

100%

Figure SDA-ADS-03-2 · Consumer group rebalance · © forcsguy.com

Offsets are committed back to Kafka. Commit after processing for at-least-once delivery (duplicates possible, so make consumers idempotent). Rebalances happen when members join, leave, crash or are too slow (missing heartbeats or taking too long to poll). Frequent rebalances hurt throughput, so keep processing per batch short and group membership stable.

Practical rules

• Partition count is a design decision: it caps consumer parallelism, and increasing it later changes the key-to-partition mapping.
• Key choice controls ordering and balance: pick a key with many distinct values.
• Monitor consumer lag per partition, under-replicated partitions and request latency.
• Idempotent producers and transactions exist for stronger guarantees (exactly-once within Kafka); end-to-end exactly-once still needs idempotent consumers or transactional sinks.
• Stream processing (Kafka Streams, Flink) builds on the same consumer mechanics; see the big-data platforms course.

Real-world reading