The system design library
Search and filter every curated system design video, article, repository and paper in one place.
88 resources
Designing Event-Driven Systems
Event sourcing, CQRS and using event streams as the source of truth, with Kafka. A free ebook from the author and Confluent.
BooksIntermediateFree from the authorBy Ben Stopford · Confluent
Scalable Web Architecture and Distributed Systems
A full chapter on the building blocks of large websites: availability, performance, reliability, scalability, caches, proxies, indexes, load balancers and queues. Free to read here under CC BY 3.0.
BlogsBeginner40 minRead hereFree from the authorBy Kate Matsudaira · The Architecture of Open Source Applications
30x Performance Improvements on MySQLStreamer at Yelp
Making 30x performance improvements on Yelp’s MySQLStreamer Abrar Sheikh, Software Engineer Feb 5, 2018 Introduction MySQLStreamer is an important application in Yelp’s Data Pipeline infrastructure. It’s responsible for streaming high-vo…
Case StudiesIntermediateRead hereBy Yelp engineers · Yelp
Apache Flink: Concepts overview
Stateful stream processing concepts: time, windows, state and fault tolerance.
DocumentationAdvancedBy Apache Flink · Apache Software Foundation
Apache Spark Documentation
Overview and guides for Spark's batch and streaming engine.
DocumentationIntermediateBy Apache Spark · Apache Software Foundation
Architecture of Nubank
Edward Wible and Rafael Ferreira discuss the key elements that make Nubank tick for millions of customers every day, including transaction authorization, messaging infrastructure (Kafka), real-time double-entry accounting (Datomic), cust…
Case StudiesIntermediateBy InfoQ engineers · InfoQ
Autoscaling Pub-Sub Consumers at Spotify
is responsible for delivering hundreds of billions of events every day. Most of the events are generated as a response to a user action, such as playing a song, following an artist or clicking on an ad. All in all, more than 300 differen…
Case StudiesIntermediateRead hereBy Spotify engineers · Spotify
Avoiding insurmountable queue backlogs
Why queues can accumulate work faster than it can be done, and design choices that keep recovery fast.
BlogsAdvancedBy Amazon Builders' Library · Amazon Web Services
Benchmarking Streaming Computation Engines at Yahoo
(Yahoo Storm Team in alphabetical order) Sanket Chintapalli, Derek Dagit, Bobby Evans, Reza Farivar, Tom Graves, Mark Holderbaugh, Zhuo Liu, Kyle Nusbaum, Kishorkumar Patil, Boyang Jerry Peng and Paul Poulosky. DISCLAIMER: Dec 17th 2015 …
Case StudiesIntermediateBy Yahoo engineers · Yahoo
Black-Box Auditing: Verifying End-to-End Replication Integrity between MySQL and Redshift at Yelp
Black-Box Auditing: Verifying End-to-End Replication Integrity between MySQL and Redshift Jacob Park, Software Engineering Intern Apr 12, 2018 Since Yelp introduced its real-time streaming data infrastructure, “Data Pipeline”, it has...
Case StudiesIntermediateRead hereBy Yelp engineers · Yelp
Brooklin: Distributed Service for Near Real-Time Data Streaming at LinkedIn
Brooklin—a distributed service for streaming data in near real-time and at scale—has been running in production at LinkedIn since 2016, powering thousands of data streams and over 2 trillion messages per day. Today, we are pleased to ann…
Case StudiesIntermediateRead hereBy LinkedIn engineers · LinkedIn
Bullet: Forward-Looking Query Engine for Streaming Data at Yahoo
A real-world engineering write-up from Yahoo on distributed Messaging, Queuing, and Event Streaming.
Case StudiesIntermediateBy Yahoo engineers · Yahoo
Cape: Event Stream Processing Framework at Dropbox
We launched Cape towards the end of 2016 to replace Livefill, which we last talked about in 2014. In this post, we’ll give an overview of Cape and its key features, high-level architecture, and varied use cases at Dropbox.
Case StudiesIntermediateRead hereBy Dropbox engineers · Dropbox
CDN Services at Spotify
Spotify built its business on flawless content delivery. Our streaming platform serves up more than 50 million tracks (plus an array of images and other assets) to more than 230 million monthly active users around the world — making us o…
Case StudiesIntermediateRead hereBy Spotify engineers · Spotify
Challenges with distributed systems
A tour of why distributed systems are hard: partial failure, unreliable networks, queues and retries.
BlogsIntermediateBy Amazon Builders' Library · Amazon Web Services
Confluent Blog
Kafka design, event streaming patterns and stream processing.
BlogsIntermediateBy Confluent engineers · Confluent
CQRS
A short, clear explanation of separating the model for updates from the model for reads, and when not to.
BlogsIntermediateBy Martin Fowler
CQRS pattern
When and how to separate read and write models, with trade-offs.
DocumentationIntermediateBy Microsoft · Microsoft Azure
Cross-Region Caching Library at Wix
At the Wix server infrastructure group we work daily on a set of tools that help make the lives of our backend developers easier, while also helping them deliver their code to production faster and safer. Wix handles huge traffic with mo…
Case StudiesIntermediateRead hereBy Wix engineers · Wix
Deduplication at Segment
The single requirement of all data pipelines is that they cannot lose data. Data can usually be delayed or re-ordered–but never dropped.
Case StudiesIntermediateRead hereBy Segment engineers · Segment
Delaying Asynchronous Message Processing with RabbitMQ at Indeed
By implementing a delay queue for our message processing, we improved the resilience of our internal systems and were able to deliver jobs more reliably.
Case StudiesIntermediateRead hereBy Indeed engineers · Indeed
Distributed Messaging, Queuing, and Event Streaming
This paper surveys the message brokers that are in vogue today for distributed communication. Their primary goal is to facilitate the construction of decentralized topologies without single points of failure, enabling fault tolerance and…
PapersIntermediateBy Arxiv
Distributed Rate Limiting at Allegro
In our services ecosystem it’s usually the case that services can handle a limited amount of requests per second. We show how we introduced a new algorithm for our publish-subscribe queue system. The road to production deployment highlig…
Case StudiesIntermediateRead hereBy Allegro engineers · Allegro
Distributed Task Scheduling (3 parts) at PagerDuty
In an earlier post, we discussed how PagerDuty has built a distributed task scheduler using Akka, Kafka, and Cassandra. This open-source Scala library schedules and...
Case StudiesIntermediateRead hereBy PagerDuty engineers · PagerDuty