Designing Event-Driven Systems

Ideas you will meet

Event Driven Architecture

Loosely coupled services reacting to events.

Kafka

Distributed commit log for event streaming at scale.

What we already know about this

Our own lessons on these ideas. Read them here before you start, so it makes sense.

• Core Concepts: How Things Actually Work · 7 minWhy Kafka is fastSequential disk writes, batching, zero-copy and partitions: the design choices behind its throughput.Read hereClose

  What Kafka is

  Kafka is a distributed commit log: producers append messages to topics, and consumers read them at their own pace. Messages are kept for a retention period, not deleted when read, so many consumers can read the same data, and replay it.

  Why it is so fast

  1. Sequential I/O. Each partition is an append-only log. Writing to the end of a file and reading in order is vastly faster than random access.
  2. The page cache. Kafka relies on the operating system's cache rather than its own heap, so recent data is served from memory without extra copying or garbage collection.
  3. Batching. Producers and brokers group many messages per request, and data is compressed per batch, amortising network and disk costs.
  4. Zero-copy transfer. The broker can send data from the file cache to the network socket without copying it through application memory.
  5. Partitioning. A topic is split into partitions spread over brokers, so reads and writes happen in parallel.
  6. A simple model. The broker mostly appends and serves byte ranges. Consumers track their own position (the offset), so the broker does little bookkeeping per message.

  100%

  Figure SDA-CORE-10-1 · Why it is so fast · © forcsguy.com

  100%

  Figure SDA-CORE-10-2 · Zero-copy: fewer trips through memory · © forcsguy.com

  Ordering and parallelism

  • Order is guaranteed within a partition, not across a whole topic.
  • The message key decides the partition, so related events (same customer) stay in order.
  • A consumer group shares partitions: each partition is read by exactly one consumer in the group. More partitions allow more parallel consumers.

  Reliability

  • Each partition is replicated to several brokers: one leader and followers.
  • Producers choose an acknowledgment level (acks): faster with fewer, safer with all in-sync replicas.
  • Consumers commit offsets after processing. If they crash, they resume from the last committed offset, so processing is at-least-once and handlers should be idempotent.

  When to use it, and when not

  Good for event streams, log and metric pipelines, and decoupling systems with replayable history. A plain task queue (RabbitMQ, SQS) can be simpler when you only need work distribution and no replay.

  More on this site

  Related reading

  • Kafka Source & DocsApache Software Foundation · apacheGitHub
  • LinkedIn Engineering BlogLinkedIn engineers · LinkedInBlogs
  • Confluent BlogConfluent engineers · ConfluentBlogs
  • Uber Engineering BlogUber Engineering · UberBlogs

  Show 6 moreShow fewer

  • Finding Kafka throughput limit in infrastructure at DropboxDropbox engineers · DropboxCase Studies
  • Kafka - Message BrokerKleppmannPapers
  • LinkedIn Engineering: DataLinkedIn engineers · LinkedInBlogs
  • Kafka on PaaSTA: Running Kafka on Kubernetes at Yelp (2 parts)Yelp engineers · YelpCase Studies
  • Freeletics Engineering BlogFreeletics engineers · FreeleticsBlogs
  • Findmypast Engineering BlogFindmypast engineers · FindmypastBlogs

  Open this lesson with its quiz and exercise

• Advanced Distributed Systems · 13 minKafka internals: segments, replication and consumer groupsWhat is inside a topic: partitions as append-only logs split into segments, how followers keep copies, and how consumer groups share the work.Read hereClose

  A topic is a set of logs

  A Kafka topic is split into partitions. Each partition is an append-only log: new records go on the end and get an increasing offset. Order is guaranteed only within a partition; records with the same key go to the same partition, so per-key order holds. Consumers track how far they have read by offset. (Why this design is fast is covered in the "Why is Kafka fast" lesson.)

  On disk: segments

  A partition's log is stored as a series of segment files. The newest segment is the one being written; older segments are immutable. Each segment has companion index files that map offsets (and timestamps) to positions in the file, so a consumer can jump to an offset quickly.

  • Retention is applied by deleting whole old segments (by age or total size), which is cheap.
  • Compaction (for some topics) keeps only the latest record per key, turning the log into a changelog of current state.

  100%

  Figure SDA-ADS-03-1 · Partitions, segments and replicas · © forcsguy.com

  Replication and durability

  Each partition has a replication factor (commonly 3): one leader and followers. Followers fetch from the leader. The in-sync replicas (ISR) are the followers that are caught up.

  A producer chooses an acknowledgement level:

  Setting	Meaning	Trade-off
  acks=0	Do not wait	Fastest, may lose data
  acks=1	Leader has written it	Loses data if the leader fails before replicating
  acks=all	All in-sync replicas have it	Safest, slower

  With acks=all and a minimum in-sync replicas setting, a write succeeds only if enough copies exist, trading availability for durability. If the leader fails, a new leader is chosen from the in-sync replicas, so acknowledged writes survive. (Older versions used ZooKeeper for cluster metadata; newer ones use a built-in Raft-based controller; the idea of consensus on metadata is the same as in the consensus lesson.)

  Consumer groups and rebalancing

  Consumers sharing a group id split a topic's partitions: each partition is read by one member at a time, so parallelism is limited by the number of partitions. Different groups each read the full topic independently.

  100%

  Figure SDA-ADS-03-2 · Consumer group rebalance · © forcsguy.com

  Offsets are committed back to Kafka. Commit after processing for at-least-once delivery (duplicates possible, so make consumers idempotent). Rebalances happen when members join, leave, crash or are too slow (missing heartbeats or taking too long to poll). Frequent rebalances hurt throughput, so keep processing per batch short and group membership stable.

  Practical rules

  • Partition count is a design decision: it caps consumer parallelism, and increasing it later changes the key-to-partition mapping.
  • Key choice controls ordering and balance: pick a key with many distinct values.
  • Monitor consumer lag per partition, under-replicated partitions and request latency.
  • Idempotent producers and transactions exist for stronger guarantees (exactly-once within Kafka); end-to-end exactly-once still needs idempotent consumers or transactional sinks.
  • Stream processing (Kafka Streams, Flink) builds on the same consumer mechanics; see the big-data platforms course.

  Real-world reading

  Related reading

  • Exactly-once Semantics Is Possible: Here's How Apache Kafka Does ItNeha Narkhede · ConfluentBlogs
  • Kafka Source & DocsApache Software Foundation · apacheGitHub
  • Confluent Developer: Apache Kafka 101ConfluentCourses
  • Confluent: Apache Kafka TalksConfluentYouTube

  Show 6 moreShow fewer

  • Jordan Has No Life: Systems Design Concepts SeriesJordan · Jordan Has No LifeYouTube
  • Apache Kafka DocumentationApache Kafka · Apache Software FoundationDocumentation
  • Apache CassandraApache Software Foundation · GitHubGitHub
  • Cassandra architecture overviewApache Cassandra · Apache Software FoundationDocumentation
  • REA Group Engineering BlogREA Group engineers · REA GroupBlogs
  • Repo-Topix: Topic Extraction Framework at GithubGithub engineers · GithubCase Studies

  Open this lesson with its quiz and exercise

• Messaging and Microservice Patterns · 12 minMessage brokers comparedQueues versus logs, and how Kafka, RabbitMQ, ActiveMQ, SQS, Pub/Sub and Pulsar differ, so you can pick one for a job.Read hereClose

  Why brokers

  A message broker sits between producers and consumers so they do not need to be up at the same time, can run at different speeds and can scale separately. It gives you buffering (absorb bursts), decoupling, retries and fan-out. (The idea is introduced in the queues and async lesson of the foundations course.)

  Two families

  	Queue-style broker	Log-style broker
  Model	Messages are routed to queues, delivered to a consumer, acknowledged and removed	Messages are appended to a retained log; consumers track their own position
  Replay	Not normally	Yes: re-read from any retained offset
  Consumers	Competing workers share one queue	Several groups read the same log independently
  Ordering	Per queue, loosened by competing workers	Per partition
  Strengths	Rich routing, per-message acks, priorities, delays	Huge throughput, history, stream processing
  Examples	RabbitMQ, ActiveMQ, Amazon SQS	Kafka, Pulsar (also queue mode), Kinesis

  100%

  Figure SDA-MSV-01-1 · Queue versus log · © forcsguy.com

  The common brokers

  • Kafka: distributed log; partitions give parallelism and per-key order; very high throughput; consumer groups; the foundation of many event pipelines and stream processors. More to operate (or use a managed service). Internals in the Kafka lesson of the advanced distributed systems course.
  • RabbitMQ (and ActiveMQ): classic brokers speaking AMQP and other protocols. Exchanges route messages to queues by rules (direct, topic, fanout, headers). Per-message acknowledgements, priorities, delays and dead letter handling. Excellent for task queues and flexible routing at moderate scale.
  • Amazon SQS: fully managed queue. Standard queues scale enormously with at-least-once delivery and best-effort ordering; FIFO queues keep order and deduplicate, at lower throughput. Visibility timeout and dead letter queues are built in. Pairs with SNS for fan-out.
  • Google Pub/Sub (and similar cloud services): managed, global pub/sub with subscriptions, acknowledgement deadlines and optional ordering keys.
  • Apache Pulsar: log-style with separated compute (brokers) and storage (BookKeeper), multi-tenancy, geo-replication, and both streaming and queue-style subscriptions.

  Delivery guarantees

  • At most once: may lose, never duplicates (acknowledge before processing). Fine for metrics.
  • At least once: no loss, possible duplicates (acknowledge after processing); the practical default. Consumers must be idempotent.
  • Exactly once: really means at-least-once delivery plus deduplication or transactions, available within some systems (for example Kafka transactions), but end-to-end it requires idempotent consumers or transactional sinks.

  Choosing

  100%

  Figure SDA-MSV-01-2 · Which broker family · © forcsguy.com

  Need	Lean towards
  Simple background jobs, no ops	SQS or a cloud queue
  Flexible routing, priorities, delays	RabbitMQ
  High-volume event streaming, replay	Kafka or Pulsar
  Fan-out to many consumers in a cloud	Pub/Sub or SNS plus queues
  Stream processing and joins	Kafka with Flink or Kafka Streams

  Questions that decide it

  How many messages per second, and how large? Do you need ordering (and by what key)? Do you need replay? How many independent consumers? What latency? Who will operate it? Start with the simplest tool that meets the needs; moving to a streaming platform when you truly need replay and scale is easier than operating one you did not need.

  Real-world reading

  Related reading

  • Exactly-once Semantics Is Possible: Here's How Apache Kafka Does ItNeha Narkhede · ConfluentBlogs
  • RabbitMQ TutorialsRabbitMQ · Broadcom / VMwareDocumentation
  • Confluent Developer: Apache Kafka 101ConfluentCourses
  • Confluent: Apache Kafka TalksConfluentYouTube

  Show 6 moreShow fewer

  • Apache Kafka DocumentationApache Kafka · Apache Software FoundationDocumentation
  • Delaying Asynchronous Message Processing with RabbitMQ at IndeedIndeed engineers · IndeedCase Studies
  • Confluent BlogConfluent engineers · ConfluentBlogs
  • Kafka Source & DocsApache Software Foundation · apacheGitHub
  • Deduplication at SegmentSegment engineers · SegmentCase Studies
  • Kafka at LinkedInLinkedIn engineers · LinkedInCase Studies

  Open this lesson with its quiz and exercise