Cookie preferences

Essential cookies keep this site working. Optional Google Analytics cookies help us understand page visits and improve the site. You can change your choice anytime. Cookie policy

Skip to content
ForCSGuy System Design Atlas home

Library / Case Studies

Kafka at Trello

A real-world engineering write-up from Trello on distributed Messaging, Queuing, and Event Streaming.

IntermediateKafkaEvent Driven ArchitectureDistributed Systems

By Trello engineers · Trello · Added 2026-10-12

Open original source

All content belongs to Trello engineers / Trello. We only link to it and give credit. Report a problem

Learn this case study

About 37 min left. Your work is saved as you go.

What you are about to study

Trello explains a real system. You should know the basics: caching, queues and replication.

Ideas you will meet

Kafka
Distributed commit log for event streaming at scale.
Event Driven Architecture
Loosely coupled services reacting to events.
Distributed Systems
Consensus, time, failure and the fundamentals of scale-out.

What we already know about this

Our own lessons on these ideas. Read them here before you start, so it makes sense.

  • Messaging and Microservice Patterns · 12 minMessage brokers comparedQueues versus logs, and how Kafka, RabbitMQ, ActiveMQ, SQS, Pub/Sub and Pulsar differ, so you can pick one for a job.Read here

    Why brokers

    A message broker sits between producers and consumers so they do not need to be up at the same time, can run at different speeds and can scale separately. It gives you buffering (absorb bursts), decoupling, retries and fan-out. (The idea is introduced in the queues and async lesson of the foundations course.)

    Two families

    Queue-style brokerLog-style broker
    ModelMessages are routed to queues, delivered to a consumer, acknowledged and removedMessages are appended to a retained log; consumers track their own position
    ReplayNot normallyYes: re-read from any retained offset
    ConsumersCompeting workers share one queueSeveral groups read the same log independently
    OrderingPer queue, loosened by competing workersPer partition
    StrengthsRich routing, per-message acks, priorities, delaysHuge throughput, history, stream processing
    ExamplesRabbitMQ, ActiveMQ, Amazon SQSKafka, Pulsar (also queue mode), Kinesis

    Figure SDA-MSV-01-1 · Queue versus log · © forcsguy.com

    The common brokers

    • Kafka: distributed log; partitions give parallelism and per-key order; very high throughput; consumer groups; the foundation of many event pipelines and stream processors. More to operate (or use a managed service). Internals in the Kafka lesson of the advanced distributed systems course.
    • RabbitMQ (and ActiveMQ): classic brokers speaking AMQP and other protocols. Exchanges route messages to queues by rules (direct, topic, fanout, headers). Per-message acknowledgements, priorities, delays and dead letter handling. Excellent for task queues and flexible routing at moderate scale.
    • Amazon SQS: fully managed queue. Standard queues scale enormously with at-least-once delivery and best-effort ordering; FIFO queues keep order and deduplicate, at lower throughput. Visibility timeout and dead letter queues are built in. Pairs with SNS for fan-out.
    • Google Pub/Sub (and similar cloud services): managed, global pub/sub with subscriptions, acknowledgement deadlines and optional ordering keys.
    • Apache Pulsar: log-style with separated compute (brokers) and storage (BookKeeper), multi-tenancy, geo-replication, and both streaming and queue-style subscriptions.

    Delivery guarantees

    • At most once: may lose, never duplicates (acknowledge before processing). Fine for metrics.
    • At least once: no loss, possible duplicates (acknowledge after processing); the practical default. Consumers must be idempotent.
    • Exactly once: really means at-least-once delivery plus deduplication or transactions, available within some systems (for example Kafka transactions), but end-to-end it requires idempotent consumers or transactional sinks.

    Choosing

    Figure SDA-MSV-01-2 · Which broker family · © forcsguy.com

    NeedLean towards
    Simple background jobs, no opsSQS or a cloud queue
    Flexible routing, priorities, delaysRabbitMQ
    High-volume event streaming, replayKafka or Pulsar
    Fan-out to many consumers in a cloudPub/Sub or SNS plus queues
    Stream processing and joinsKafka with Flink or Kafka Streams

    Questions that decide it

    How many messages per second, and how large? Do you need ordering (and by what key)? Do you need replay? How many independent consumers? What latency? Who will operate it? Start with the simplest tool that meets the needs; moving to a streaming platform when you truly need replay and scale is easier than operating one you did not need.

    Real-world reading

    Related reading

    Show 6 more

    Open this lesson with its quiz and exercise

  • Core Concepts: How Things Actually Work · 7 minWhy Kafka is fastSequential disk writes, batching, zero-copy and partitions: the design choices behind its throughput.Read here

    What Kafka is

    Kafka is a distributed commit log: producers append messages to topics, and consumers read them at their own pace. Messages are kept for a retention period, not deleted when read, so many consumers can read the same data, and replay it.

    Why it is so fast

    1. Sequential I/O. Each partition is an append-only log. Writing to the end of a file and reading in order is vastly faster than random access.
    2. The page cache. Kafka relies on the operating system's cache rather than its own heap, so recent data is served from memory without extra copying or garbage collection.
    3. Batching. Producers and brokers group many messages per request, and data is compressed per batch, amortising network and disk costs.
    4. Zero-copy transfer. The broker can send data from the file cache to the network socket without copying it through application memory.
    5. Partitioning. A topic is split into partitions spread over brokers, so reads and writes happen in parallel.
    6. A simple model. The broker mostly appends and serves byte ranges. Consumers track their own position (the offset), so the broker does little bookkeeping per message.

    Figure SDA-CORE-10-1 · Why it is so fast · © forcsguy.com

    Figure SDA-CORE-10-2 · Zero-copy: fewer trips through memory · © forcsguy.com

    Ordering and parallelism

    • Order is guaranteed within a partition, not across a whole topic.
    • The message key decides the partition, so related events (same customer) stay in order.
    • A consumer group shares partitions: each partition is read by exactly one consumer in the group. More partitions allow more parallel consumers.

    Reliability

    • Each partition is replicated to several brokers: one leader and followers.
    • Producers choose an acknowledgment level (acks): faster with fewer, safer with all in-sync replicas.
    • Consumers commit offsets after processing. If they crash, they resume from the last committed offset, so processing is at-least-once and handlers should be idempotent.

    When to use it, and when not

    Good for event streams, log and metric pipelines, and decoupling systems with replayable history. A plain task queue (RabbitMQ, SQS) can be simpler when you only need work distribution and no replay.

    More on this site

    Related reading

    Show 6 more

    Open this lesson with its quiz and exercise

  • Advanced Distributed Systems · 13 minKafka internals: segments, replication and consumer groupsWhat is inside a topic: partitions as append-only logs split into segments, how followers keep copies, and how consumer groups share the work.Read here

    A topic is a set of logs

    A Kafka topic is split into partitions. Each partition is an append-only log: new records go on the end and get an increasing offset. Order is guaranteed only within a partition; records with the same key go to the same partition, so per-key order holds. Consumers track how far they have read by offset. (Why this design is fast is covered in the "Why is Kafka fast" lesson.)

    On disk: segments

    A partition's log is stored as a series of segment files. The newest segment is the one being written; older segments are immutable. Each segment has companion index files that map offsets (and timestamps) to positions in the file, so a consumer can jump to an offset quickly.

    • Retention is applied by deleting whole old segments (by age or total size), which is cheap.
    • Compaction (for some topics) keeps only the latest record per key, turning the log into a changelog of current state.

    Figure SDA-ADS-03-1 · Partitions, segments and replicas · © forcsguy.com

    Replication and durability

    Each partition has a replication factor (commonly 3): one leader and followers. Followers fetch from the leader. The in-sync replicas (ISR) are the followers that are caught up.

    A producer chooses an acknowledgement level:

    SettingMeaningTrade-off
    acks=0Do not waitFastest, may lose data
    acks=1Leader has written itLoses data if the leader fails before replicating
    acks=allAll in-sync replicas have itSafest, slower

    With acks=all and a minimum in-sync replicas setting, a write succeeds only if enough copies exist, trading availability for durability. If the leader fails, a new leader is chosen from the in-sync replicas, so acknowledged writes survive. (Older versions used ZooKeeper for cluster metadata; newer ones use a built-in Raft-based controller; the idea of consensus on metadata is the same as in the consensus lesson.)

    Consumer groups and rebalancing

    Consumers sharing a group id split a topic's partitions: each partition is read by one member at a time, so parallelism is limited by the number of partitions. Different groups each read the full topic independently.

    Figure SDA-ADS-03-2 · Consumer group rebalance · © forcsguy.com

    Offsets are committed back to Kafka. Commit after processing for at-least-once delivery (duplicates possible, so make consumers idempotent). Rebalances happen when members join, leave, crash or are too slow (missing heartbeats or taking too long to poll). Frequent rebalances hurt throughput, so keep processing per batch short and group membership stable.

    Practical rules

    • Partition count is a design decision: it caps consumer parallelism, and increasing it later changes the key-to-partition mapping.
    • Key choice controls ordering and balance: pick a key with many distinct values.
    • Monitor consumer lag per partition, under-replicated partitions and request latency.
    • Idempotent producers and transactions exist for stronger guarantees (exactly-once within Kafka); end-to-end exactly-once still needs idempotent consumers or transactional sinks.
    • Stream processing (Kafka Streams, Flink) builds on the same consumer mechanics; see the big-data platforms course.

    Real-world reading

    Related reading

    Show 6 more

    Open this lesson with its quiz and exercise

Continue to Trello

Go deeper on this

A path through videos, articles, other case studies and code on the same ideas as Kafka at Trello.

  1. Warm upGet the vocabulary first.

  2. WatchHow others solve the same problem, on video.

    Videos belong to their creators and play from YouTube. More on YouTube (opens in a new tab)

  3. ReadEngineers explain it in their own words.

  4. See it in another companyThe same ideas, a different system.

  5. Read the codeSee how it looks when it is real.

Keep learning

Loading community…