REA Group Blog

Ideas you will meet

Distributed Systems

Consensus, time, failure and the fundamentals of scale-out.

What we already know about this

Our own lessons on these ideas. Read them here before you start, so it makes sense.

• Core Concepts: How Things Actually Work · 7 minWhy Kafka is fastSequential disk writes, batching, zero-copy and partitions: the design choices behind its throughput.Read hereClose

  What Kafka is

  Kafka is a distributed commit log: producers append messages to topics, and consumers read them at their own pace. Messages are kept for a retention period, not deleted when read, so many consumers can read the same data, and replay it.

  Why it is so fast

  1. Sequential I/O. Each partition is an append-only log. Writing to the end of a file and reading in order is vastly faster than random access.
  2. The page cache. Kafka relies on the operating system's cache rather than its own heap, so recent data is served from memory without extra copying or garbage collection.
  3. Batching. Producers and brokers group many messages per request, and data is compressed per batch, amortising network and disk costs.
  4. Zero-copy transfer. The broker can send data from the file cache to the network socket without copying it through application memory.
  5. Partitioning. A topic is split into partitions spread over brokers, so reads and writes happen in parallel.
  6. A simple model. The broker mostly appends and serves byte ranges. Consumers track their own position (the offset), so the broker does little bookkeeping per message.

  100%

  Figure SDA-CORE-10-1 · Why it is so fast · © forcsguy.com

  100%

  Figure SDA-CORE-10-2 · Zero-copy: fewer trips through memory · © forcsguy.com

  Ordering and parallelism

  • Order is guaranteed within a partition, not across a whole topic.
  • The message key decides the partition, so related events (same customer) stay in order.
  • A consumer group shares partitions: each partition is read by exactly one consumer in the group. More partitions allow more parallel consumers.

  Reliability

  • Each partition is replicated to several brokers: one leader and followers.
  • Producers choose an acknowledgment level (acks): faster with fewer, safer with all in-sync replicas.
  • Consumers commit offsets after processing. If they crash, they resume from the last committed offset, so processing is at-least-once and handlers should be idempotent.

  When to use it, and when not

  Good for event streams, log and metric pipelines, and decoupling systems with replayable history. A plain task queue (RabbitMQ, SQS) can be simpler when you only need work distribution and no replay.

  More on this site

  Related reading

  • Kafka Source & DocsApache Software Foundation · apacheGitHub
  • LinkedIn Engineering BlogLinkedIn engineers · LinkedInBlogs
  • Confluent BlogConfluent engineers · ConfluentBlogs
  • Uber Engineering BlogUber Engineering · UberBlogs

  Show 6 moreShow fewer

  • Finding Kafka throughput limit in infrastructure at DropboxDropbox engineers · DropboxCase Studies
  • Kafka - Message BrokerKleppmannPapers
  • LinkedIn Engineering: DataLinkedIn engineers · LinkedInBlogs
  • Kafka on PaaSTA: Running Kafka on Kubernetes at Yelp (2 parts)Yelp engineers · YelpCase Studies
  • Freeletics Engineering BlogFreeletics engineers · FreeleticsBlogs
  • Findmypast Engineering BlogFindmypast engineers · FindmypastBlogs

  Open this lesson with its quiz and exercise