The system design library
Search and filter every curated system design video, article, repository and paper in one place.
22 resources
Designing Event-Driven Systems
Event sourcing, CQRS and using event streams as the source of truth, with Kafka. A free ebook from the author and Confluent.
BooksIntermediateFree from the authorBy Ben Stopford · Confluent
Confluent Blog
Kafka design, event streaming patterns and stream processing.
BlogsIntermediateBy Confluent engineers · Confluent
Exactly-once Semantics Is Possible: Here's How Apache Kafka Does It
How idempotent producers and transactions give exactly-once processing inside Kafka.
BlogsAdvancedBy Neha Narkhede · Confluent
Finding Kafka throughput limit in infrastructure at Dropbox
How Dropbox's Jetstream team built an automated platform to measure the maximum throughput of its Kafka clusters, finding about 60 MB/s per broker.
Case StudiesIntermediateRead hereBy Dropbox engineers · Dropbox
Kafka - Message Broker
Apache Kafka is a scalable message broker , and Apache Samza is a stream processing framework built upon Kafka. They are widely used as infrastructure for implementing personalized online services and real-time predictive analytics.
PapersIntermediateBy Kleppmann
Kafka at LinkedIn
If data is the lifeblood of high technology, Apache Kafka is the circulatory system in use at LinkedIn. We use Kafka for moving every type of data around between systems, and it touches virtually every server, every day. The complexity o…
Case StudiesIntermediateRead hereBy LinkedIn engineers · LinkedIn
Kafka at Salesforce
Apache Kafka came to the rescue by inspiring us to leverage the time-ordered event log paradigm
Case StudiesIntermediateRead hereBy Salesforce engineers · Salesforce
Kafka at Trello
A real-world engineering write-up from Trello on distributed Messaging, Queuing, and Event Streaming.
Case StudiesIntermediateBy Trello engineers · Trello
Kafka at Yelp
Billions of Messages a Day - Yelp's Real-time Data Pipeline Justin C., Software Engineer Jul 14, 2016 This post is part of a series covering Yelp's real-time streaming data infrastructure....
Case StudiesIntermediateRead hereBy Yelp engineers · Yelp
Kafka on Kubernetes at Shopify
In the Beginning, There Was the Data Center Shopify is a leading multi-channel commerce platform that powers over 600,000 businesses in approximately 175 countries. We first adopted Apache Kafka as our data bus for reliable messaging in …
Case StudiesIntermediateRead hereBy Shopify engineers · Shopify
Kafka on PaaSTA: Running Kafka on Kubernetes at Yelp (2 parts)
Kafka on PaaSTA: Running Kafka on Kubernetes at Yelp (Part 2 - Migration) Lennart Rudolph, Software Engineer Mar 3, 2022 In a previous post we detailed the architecture and motivation...
Case StudiesIntermediateRead hereBy Yelp engineers · Yelp
LinkedIn Engineering Blog
Posts from the team that created Kafka and many other data systems: streaming, search, feeds and infrastructure.
BlogsIntermediateBy LinkedIn engineers · LinkedIn
LinkedIn Engineering: Data
The engineering and technology blog of LinkedIn: posts on how it designs, builds and scales its systems.
BlogsIntermediateBy LinkedIn engineers · LinkedIn
Migrating Kafka's Zookeeper with No Downtime at Yelp
Migrating Kafka's Zookeeper With No Downtime Toby Cole, Engineering Manager Jan 17, 2019 Here at Yelp we use Kafka extensively. In fact, we send billions of messages a day through our various...
Case StudiesIntermediateRead hereBy Yelp engineers · Yelp
Resiliency and Disaster Recovery with Kafka at eBay
High availability, resilience and scalability have become the standard when building user-facing applications. Cloud computing has been a contributing factor to that.
Case StudiesIntermediateRead hereBy eBay engineers · eBay
Scaling Slack's Job Queue
How Slack moved from a Redis-only job queue to one buffered with Kafka to survive spikes.
Case StudiesAdvancedRead hereBy Slack Engineering · Slack
Confluent Developer: Apache Kafka 101
Short free video course on events, topics, partitions, producers and consumers.
CoursesBeginnerBy Confluent
Uber Engineering Blog
How Uber builds dispatch, maps, data platforms and real-time infrastructure.
BlogsAdvancedBy Uber Engineering · Uber
Apache Kafka Documentation
Official reference for topics, partitions, producers, consumers and delivery semantics.
DocumentationIntermediateBy Apache Kafka · Apache Software Foundation
Kafka Source & Docs
Apache Kafka source code: read the log, replication and consumer-group internals first-hand.
GitHubAdvancedBy Apache Software Foundation · apache
Confluent: Apache Kafka Talks
Official talks and tutorials on event streaming, Kafka internals and stream processing.
YouTubeIntermediateBy Confluent
Jordan Has No Life: Systems Design Concepts Series
Deep, whiteboard-style dives into replication, consensus and stream processing.
YouTubeAdvancedBy Jordan · Jordan Has No Life