Event-driven architecture

Kafka-style partitions, keys, consumer groups and lag.

On the roadmap:Advanced · step 5: Event-driven design

Intermediateabout 10 to 15 min6 experiments

Experiments

New here? A few short steps show what happens, one at a time.

Settings

Partitions4

Consumers in the group2

Live traffic

Messages sent per beat3

Capacity: 2 messages per beat (2 busy consumers × 1). Traffic is above capacity: the lag will keep growing.

Messages per consumer per tick1

Message key

Key “alice” goes to partition 3 of 4.

Dead-letter queue (set a message aside after 3 failed tries)

Total lag

Stuck behind a failure

Handled out of order

Rebalances

Messages with the same key always land in the same partition, so one consumer sees them in order. Add consumers (up to the number of partitions) to drain lag faster.

Partitions (waiting messages are the consumer lag)

P1 to C1

lag 0

P2 to C2

lag 0

P3 to C1

lag 0

P4 to C2

lag 0

Nothing is waiting. Press “Produce 1” or “Burst of 12” above, or switch on Live traffic, and watch the lines fill.

In plain words

A topic is split into partitions, each an ordered line of messages. Producers add messages. A group of consumers shares the partitions, and each partition is read by at most one consumer in the group. Lag is the number of messages waiting.

Why it matters

Splitting into partitions is what lets a stream be processed in parallel, and giving messages a key is what keeps related ones in order. The same design explains the classic problems: idle consumers, growing lag, one bad message blocking a line, and order breaking when partitions change.

How real systems do it

The simulation is a simplified model. Here is what production systems add or do.

In Kafka, messages with the same key go to the same partition, and order is guaranteed only within a partition, not across them.
A partition is read by at most one consumer in a group, so extra consumers sit idle. With 3 partitions and 5 consumers, 3 work and 2 wait, ready to take over if one fails.
Adding partitions to a topic that uses keys changes where keys hash to, so per-key order can break while the old messages drain.
Queues such as Amazon SQS can move a message to a dead-letter queue after it has been received a set number of times (maxReceiveCount), so one poison message cannot block the rest.