Event Driven Architecture
Loosely coupled services reacting to events.
All topics · 3 lessons · 48 resources
Learn Event Driven Architecture on System Design Atlas
- 01Queues, streams and asynchronous workHow to take slow or bursty work off the request path, and what delivery guarantees really mean.System Design Foundations · 11 min read · start here
- 02Event-driven architecture: sagas, CQRS and event sourcingHow services coordinate without distributed transactions: events, sagas with compensation, separate read and write models, and storing changes instead of state.Messaging and Microservice Patterns · 14 min read
- 03How microservices work togetherSynchronous calls, events, sagas and discovery: how separate services cooperate without becoming one tangled system.Real-World Systems · 11 min read
Related topics
These share a lesson with Event Driven Architecture.
Curated Event Driven Architecture resources
Books
Documentation
- Apache Flink: Concepts overviewStateful stream processing concepts: time, windows, state and fault tolerance.Apache Software Foundation
- CQRS patternWhen and how to separate read and write models, with trade-offs.Microsoft Azure
- Drools documentationDocumentation for a widely used open-source business rules engine.Drools (KIE)
- Pattern: SagaManaging a business transaction across services with local transactions and compensations.microservices.io
- Pattern: Transactional outboxPublishing events reliably by saving them in the same database transaction as the change.microservices.io
- RabbitMQ TutorialsHands-on tutorials for work queues, pub/sub, routing and RPC patterns.Broadcom / VMware
- Apache Kafka DocumentationOfficial reference for topics, partitions, producers, consumers and delivery semantics.Apache Software Foundation
- Azure Architecture CenterReference architectures, cloud design patterns and best-practice guides.Microsoft Azure
Blogs
- Avoiding insurmountable queue backlogsWhy queues can accumulate work faster than it can be done, and design choices that keep recovery fast.Amazon Web Services
- Confluent BlogKafka design, event streaming patterns and stream processing.Confluent
- CQRSA short, clear explanation of separating the model for updates from the model for reads, and when not to.Martin Fowler
- Event SourcingStoring every change as an event and deriving current state from them.Martin Fowler
- Rules Engine (Martin Fowler)A critical look at when business rules engines help and when they hurt.martinfowler.com
Case Studies
- Benchmarking Streaming Computation Engines at Yahoo(Yahoo Storm Team in alphabetical order) Sanket Chintapalli, Derek Dagit, Bobby Evans, Reza Farivar, Tom Graves, Mark Holderbaugh, Zhuo Liu, Kyle Nusbaum, Kishorkumar Patil, Boyang Jerry Peng and Paul Poulosky. DISCLAIMER: Dec 17th 2015 data-artisans has pointed out to us that we accidentally left on some debugging…Yahoo
- Brooklin: Distributed Service for Near Real-Time Data Streaming at LinkedInBrooklin—a distributed service for streaming data in near real-time and at scale—has been running in production at LinkedIn since 2016, powering thousands of data streams and over 2 trillion messages per day. Today, we are pleased to announce the open-sourcing of Brooklin and that the source code is available in…LinkedIn
- Bullet: Forward-Looking Query Engine for Streaming Data at YahooA real-world engineering write-up from Yahoo on distributed Messaging, Queuing, and Event Streaming.Yahoo
- Cape: Event Stream Processing Framework at DropboxWe launched Cape towards the end of 2016 to replace Livefill, which we last talked about in 2014. In this post, we’ll give an overview of Cape and its key features, high-level architecture, and varied use cases at Dropbox.Dropbox
- Deduplication at SegmentThe single requirement of all data pipelines is that they cannot lose data. Data can usually be delayed or re-ordered–but never dropped.Segment
- Delaying Asynchronous Message Processing with RabbitMQ at IndeedBy implementing a delay queue for our message processing, we improved the resilience of our internal systems and were able to deliver jobs more reliably.Indeed
- Finding Kafka throughput limit in infrastructure at DropboxHow Dropbox's Jetstream team built an automated platform to measure the maximum throughput of its Kafka clusters, finding about 60 MB/s per broker.Dropbox
- Improving Stream Data Quality With Protobuf Schema Validation at DeliverooThe requirements for fast and reliable data pipelines are growing quickly at Deliveroo as the business continues to grow and innovate. We have delivered an event streaming platform which gives strong guarantees on data quality, using Apache Kafka® and Protocol Buffers.Deliveroo
- Kafka at LinkedInIf data is the lifeblood of high technology, Apache Kafka is the circulatory system in use at LinkedIn. We use Kafka for moving every type of data around between systems, and it touches virtually every server, every day. The complexity of the infrastructure, as well as the reasoning behind choices tLinkedIn
- Kafka at SalesforceApache Kafka came to the rescue by inspiring us to leverage the time-ordered event log paradigmSalesforce
- Kafka at TrelloA real-world engineering write-up from Trello on distributed Messaging, Queuing, and Event Streaming.Trello
- Kafka at YelpBillions of Messages a Day - Yelp's Real-time Data Pipeline Justin C., Software Engineer Jul 14, 2016 This post is part of a series covering Yelp's real-time streaming data infrastructure....Yelp
- Kafka on Kubernetes at ShopifyIn the Beginning, There Was the Data Center Shopify is a leading multi-channel commerce platform that powers over 600,000 businesses in approximately 175 countries. We first adopted Apache Kafka as our data bus for reliable messaging in 2014 and mainly used it for collecting events and log aggregation across the…Shopify
- Kafka on PaaSTA: Running Kafka on Kubernetes at Yelp (2 parts)Kafka on PaaSTA: Running Kafka on Kubernetes at Yelp (Part 2 - Migration) Lennart Rudolph, Software Engineer Mar 3, 2022 In a previous post we detailed the architecture and motivation...Yelp
- Messaging Service at Riot GamesHey there! My name is Michal 0xDEADB33F Ptaszek, and I’m a software architect at Riot. Today I would like to talk about communication. But not the kind of communication you’re probably thinking of. I want to talk about the other, more exci…Riot Games
- Messaging System Model at DropboxThe MSM helped us build a unified event-driven system capable of orchestrating a wide range of asynchronous tasks and meeting future needs, especially as we focus on AI.Dropbox
- Migrating Kafka's Zookeeper with No Downtime at YelpMigrating Kafka's Zookeeper With No Downtime Toby Cole, Engineering Manager Jan 17, 2019 Here at Yelp we use Kafka extensively. In fact, we send billions of messages a day through our various...Yelp
- Pulsar: Pub-Sub Messaging at Scale at YahooBy Joe Francis and Matteo Merli, Yahoo Platforms Pub-sub messaging is a very common design pattern that is increasingly found in distributed systems powering Internet applications. These applications provide real-time services, and need publish-latencies of 5ms on average and no more than 15ms at the 99th…Scale at Yahoo
- Qmessage: Distributed, Asynchronous Task Queue at QuoraAuthors: Bandish Chheda and Yiming Li Today, we're excited to discuss Qmessage, our distributed, asynchronous task queue. We use Qmessage to handle tasks where a response is not immediately needed and would require too much time to execute during a web or API request, such as delivering answers...Quora
- Samza: Stream Processing System for Latency Insighs at LinkedInTraditional request response metrics are fairly easy to measure. There is a clearly defined start time (when the request is received) and a clearly defined end time (when the last byte of the response is sent and the connection is closed).LinkedIn
- Scaling Job Queue with Redis at SlackSlack uses a job queue system for business logic that is too time-consuming to run in the context of a web request. This system is a critical component of our architecture, used for every Slack message post, push notification, URL unfurl, calendar reminder, and billing calculation. On our busiest days, the system processes over 1.4…Slack
- Scaling Push Messaging for Millions of Devices at NetflixSusheel Aroskar talks about Zuul Push, a scalable push notification service that handles millions of "always-on" persistent connections from all the Netflix apps running. He covers the design of the Zuul Push server and reviews the design details of the back-end message routing infrastructure that lets any Netflix…Netflix
- Scaling Slack's Job QueueHow Slack moved from a Redis-only job queue to one buffered with Kafka to survive spikes.Slack
- Stream Data DeduplicationIn computing, data deduplication is a technique for eliminating duplicate copies of repeating data. Successful implementation of the technique can improve storage utilization, which may in turn lower capital expenditure by reducing the overall amount of storage media required to meet storage capacity needs. It can…Wikipedia
- Uber's Real-Time Push PlatformHow Uber pushes live updates to millions of phones instead of having apps poll.Uber
Papers
- Distributed Messaging, Queuing, and Event StreamingThis paper surveys the message brokers that are in vogue today for distributed communication. Their primary goal is to facilitate the construction of decentralized topologies without single points of failure, enabling fault tolerance and high availability.Arxiv
- Kafka - Message BrokerApache Kafka is a scalable message broker , and Apache Samza is a stream processing framework built upon Kafka. They are widely used as infrastructure for implementing personalized online services and real-time predictive analytics.Kleppmann
- The Dataflow Model (paper)A unified model for batch and stream processing with windows, triggers and event time.Google