The system design library
Search and filter every curated system design video, article, repository and paper in one place.
48 resources
Designing Event-Driven Systems
Event sourcing, CQRS and using event streams as the source of truth, with Kafka. A free ebook from the author and Confluent.
BooksIntermediateFree from the authorBy Ben Stopford · Confluent
Apache Flink: Concepts overview
Stateful stream processing concepts: time, windows, state and fault tolerance.
DocumentationAdvancedBy Apache Flink · Apache Software Foundation
Avoiding insurmountable queue backlogs
Why queues can accumulate work faster than it can be done, and design choices that keep recovery fast.
BlogsAdvancedBy Amazon Builders' Library · Amazon Web Services
Benchmarking Streaming Computation Engines at Yahoo
(Yahoo Storm Team in alphabetical order) Sanket Chintapalli, Derek Dagit, Bobby Evans, Reza Farivar, Tom Graves, Mark Holderbaugh, Zhuo Liu, Kyle Nusbaum, Kishorkumar Patil, Boyang Jerry Peng and Paul Poulosky. DISCLAIMER: Dec 17th 2015 …
Case StudiesIntermediateBy Yahoo engineers · Yahoo
Brooklin: Distributed Service for Near Real-Time Data Streaming at LinkedIn
Brooklin—a distributed service for streaming data in near real-time and at scale—has been running in production at LinkedIn since 2016, powering thousands of data streams and over 2 trillion messages per day. Today, we are pleased to ann…
Case StudiesIntermediateRead hereBy LinkedIn engineers · LinkedIn
Bullet: Forward-Looking Query Engine for Streaming Data at Yahoo
A real-world engineering write-up from Yahoo on distributed Messaging, Queuing, and Event Streaming.
Case StudiesIntermediateBy Yahoo engineers · Yahoo
Cape: Event Stream Processing Framework at Dropbox
We launched Cape towards the end of 2016 to replace Livefill, which we last talked about in 2014. In this post, we’ll give an overview of Cape and its key features, high-level architecture, and varied use cases at Dropbox.
Case StudiesIntermediateRead hereBy Dropbox engineers · Dropbox
Confluent Blog
Kafka design, event streaming patterns and stream processing.
BlogsIntermediateBy Confluent engineers · Confluent
CQRS
A short, clear explanation of separating the model for updates from the model for reads, and when not to.
BlogsIntermediateBy Martin Fowler
CQRS pattern
When and how to separate read and write models, with trade-offs.
DocumentationIntermediateBy Microsoft · Microsoft Azure
Deduplication at Segment
The single requirement of all data pipelines is that they cannot lose data. Data can usually be delayed or re-ordered–but never dropped.
Case StudiesIntermediateRead hereBy Segment engineers · Segment
Delaying Asynchronous Message Processing with RabbitMQ at Indeed
By implementing a delay queue for our message processing, we improved the resilience of our internal systems and were able to deliver jobs more reliably.
Case StudiesIntermediateRead hereBy Indeed engineers · Indeed
Distributed Messaging, Queuing, and Event Streaming
This paper surveys the message brokers that are in vogue today for distributed communication. Their primary goal is to facilitate the construction of decentralized topologies without single points of failure, enabling fault tolerance and…
PapersIntermediateBy Arxiv
Drools documentation
Documentation for a widely used open-source business rules engine.
DocumentationIntermediateBy Drools community · Drools (KIE)
Event Sourcing
Storing every change as an event and deriving current state from them.
BlogsIntermediateBy Martin Fowler
Finding Kafka throughput limit in infrastructure at Dropbox
How Dropbox's Jetstream team built an automated platform to measure the maximum throughput of its Kafka clusters, finding about 60 MB/s per broker.
Case StudiesIntermediateRead hereBy Dropbox engineers · Dropbox
Improving Stream Data Quality With Protobuf Schema Validation at Deliveroo
The requirements for fast and reliable data pipelines are growing quickly at Deliveroo as the business continues to grow and innovate. We have delivered an event streaming platform which gives strong guarantees on data quality, using Apa…
Case StudiesIntermediateRead hereBy Deliveroo engineers · Deliveroo
Kafka - Message Broker
Apache Kafka is a scalable message broker , and Apache Samza is a stream processing framework built upon Kafka. They are widely used as infrastructure for implementing personalized online services and real-time predictive analytics.
PapersIntermediateBy Kleppmann
Kafka at LinkedIn
If data is the lifeblood of high technology, Apache Kafka is the circulatory system in use at LinkedIn. We use Kafka for moving every type of data around between systems, and it touches virtually every server, every day. The complexity o…
Case StudiesIntermediateRead hereBy LinkedIn engineers · LinkedIn
Kafka at Salesforce
Apache Kafka came to the rescue by inspiring us to leverage the time-ordered event log paradigm
Case StudiesIntermediateRead hereBy Salesforce engineers · Salesforce
Kafka at Trello
A real-world engineering write-up from Trello on distributed Messaging, Queuing, and Event Streaming.
Case StudiesIntermediateBy Trello engineers · Trello
Kafka at Yelp
Billions of Messages a Day - Yelp's Real-time Data Pipeline Justin C., Software Engineer Jul 14, 2016 This post is part of a series covering Yelp's real-time streaming data infrastructure....
Case StudiesIntermediateRead hereBy Yelp engineers · Yelp
Kafka on Kubernetes at Shopify
In the Beginning, There Was the Data Center Shopify is a leading multi-channel commerce platform that powers over 600,000 businesses in approximately 175 countries. We first adopted Apache Kafka as our data bus for reliable messaging in …
Case StudiesIntermediateRead hereBy Shopify engineers · Shopify
Kafka on PaaSTA: Running Kafka on Kubernetes at Yelp (2 parts)
Kafka on PaaSTA: Running Kafka on Kubernetes at Yelp (Part 2 - Migration) Lennart Rudolph, Software Engineer Mar 3, 2022 In a previous post we detailed the architecture and motivation...
Case StudiesIntermediateRead hereBy Yelp engineers · Yelp