Kafka
Distributed commit log for event streaming at scale.
All topics · 3 lessons · 22 resources
Learn Kafka on System Design Atlas
- 01Why Kafka is fastSequential disk writes, batching, zero-copy and partitions: the design choices behind its throughput.Core Concepts: How Things Actually Work · 7 min read · start here
- 02Message brokers comparedQueues versus logs, and how Kafka, RabbitMQ, ActiveMQ, SQS, Pub/Sub and Pulsar differ, so you can pick one for a job.Messaging and Microservice Patterns · 12 min read
- 03Kafka internals: segments, replication and consumer groupsWhat is inside a topic: partitions as append-only logs split into segments, how followers keep copies, and how consumer groups share the work.Advanced Distributed Systems · 13 min read
Curated Kafka resources
Blogs
- Confluent BlogKafka design, event streaming patterns and stream processing.Confluent
- Exactly-once Semantics Is Possible: Here's How Apache Kafka Does ItHow idempotent producers and transactions give exactly-once processing inside Kafka.Confluent
- LinkedIn Engineering BlogPosts from the team that created Kafka and many other data systems: streaming, search, feeds and infrastructure.LinkedIn
- LinkedIn Engineering: DataThe engineering and technology blog of LinkedIn: posts on how it designs, builds and scales its systems.LinkedIn
- Uber Engineering BlogHow Uber builds dispatch, maps, data platforms and real-time infrastructure.Uber
Case Studies
- Finding Kafka throughput limit in infrastructure at DropboxHow Dropbox's Jetstream team built an automated platform to measure the maximum throughput of its Kafka clusters, finding about 60 MB/s per broker.Dropbox
- Kafka at LinkedInIf data is the lifeblood of high technology, Apache Kafka is the circulatory system in use at LinkedIn. We use Kafka for moving every type of data around between systems, and it touches virtually every server, every day. The complexity of the infrastructure, as well as the reasoning behind choices tLinkedIn
- Kafka at SalesforceApache Kafka came to the rescue by inspiring us to leverage the time-ordered event log paradigmSalesforce
- Kafka at TrelloA real-world engineering write-up from Trello on distributed Messaging, Queuing, and Event Streaming.Trello
- Kafka at YelpBillions of Messages a Day - Yelp's Real-time Data Pipeline Justin C., Software Engineer Jul 14, 2016 This post is part of a series covering Yelp's real-time streaming data infrastructure....Yelp
- Kafka on Kubernetes at ShopifyIn the Beginning, There Was the Data Center Shopify is a leading multi-channel commerce platform that powers over 600,000 businesses in approximately 175 countries. We first adopted Apache Kafka as our data bus for reliable messaging in 2014 and mainly used it for collecting events and log aggregation across the…Shopify
- Kafka on PaaSTA: Running Kafka on Kubernetes at Yelp (2 parts)Kafka on PaaSTA: Running Kafka on Kubernetes at Yelp (Part 2 - Migration) Lennart Rudolph, Software Engineer Mar 3, 2022 In a previous post we detailed the architecture and motivation...Yelp
- Migrating Kafka's Zookeeper with No Downtime at YelpMigrating Kafka's Zookeeper With No Downtime Toby Cole, Engineering Manager Jan 17, 2019 Here at Yelp we use Kafka extensively. In fact, we send billions of messages a day through our various...Yelp
- Resiliency and Disaster Recovery with Kafka at eBayHigh availability, resilience and scalability have become the standard when building user-facing applications. Cloud computing has been a contributing factor to that.eBay
- Scaling Slack's Job QueueHow Slack moved from a Redis-only job queue to one buffered with Kafka to survive spikes.Slack