Why Kafka is fast

What Kafka is

Kafka is a distributed commit log: producers append messages to topics, and consumers read them at their own pace. Messages are kept for a retention period, not deleted when read, so many consumers can read the same data, and replay it.

Why it is so fast

1. Sequential I/O. Each partition is an append-only log. Writing to the end of a file and reading in order is vastly faster than random access.
2. The page cache. Kafka relies on the operating system's cache rather than its own heap, so recent data is served from memory without extra copying or garbage collection.
3. Batching. Producers and brokers group many messages per request, and data is compressed per batch, amortising network and disk costs.
4. Zero-copy transfer. The broker can send data from the file cache to the network socket without copying it through application memory.
5. Partitioning. A topic is split into partitions spread over brokers, so reads and writes happen in parallel.
6. A simple model. The broker mostly appends and serves byte ranges. Consumers track their own position (the offset), so the broker does little bookkeeping per message.

100%

Figure SDA-CORE-10-1 · Why it is so fast · © forcsguy.com

100%

Figure SDA-CORE-10-2 · Zero-copy: fewer trips through memory · © forcsguy.com

Ordering and parallelism

• Order is guaranteed within a partition, not across a whole topic.
• The message key decides the partition, so related events (same customer) stay in order.
• A consumer group shares partitions: each partition is read by exactly one consumer in the group. More partitions allow more parallel consumers.

Reliability

• Each partition is replicated to several brokers: one leader and followers.
• Producers choose an acknowledgment level (acks): faster with fewer, safer with all in-sync replicas.
• Consumers commit offsets after processing. If they crash, they resume from the last committed offset, so processing is at-least-once and handlers should be idempotent.

When to use it, and when not

Good for event streams, log and metric pipelines, and decoupling systems with replayable history. A plain task queue (RabbitMQ, SQS) can be simpler when you only need work distribution and no replay.

More on this site