Event Sourcing

Ideas you will meet

Event Driven Architecture

Loosely coupled services reacting to events.

CQRS

Separate read and write models to scale each independently.

What we already know about this

Our own lessons on these ideas. Read them here before you start, so it makes sense.

• Messaging and Microservice Patterns · 14 minEvent-driven architecture: sagas, CQRS and event sourcingHow services coordinate without distributed transactions: events, sagas with compensation, separate read and write models, and storing changes instead of state.Read hereClose

  Events as the connective tissue

  In an event-driven design, services announce facts that happened ("OrderPlaced", "PaymentCaptured") and others react. This decouples services in time and knowledge: the publisher does not know who listens. The price is that the flow is spread over many services, so you need discipline about event design, ordering, retries (previous lesson) and observability.

  Three patterns come up constantly.

  Saga: transactions across services

  In a microservice system each service owns its database, so a single ACID transaction across them is not available (two-phase commit across services is slow and fragile). A saga is a sequence of local transactions, each followed by an event or command that triggers the next. If a step fails, the saga runs compensating actions for the steps already done.

  100%

  Figure SDA-MSV-03-1 · Order saga with compensation · © forcsguy.com

  Two styles:

  • Choreography: services react to each other's events with no central controller. Simple for short flows; hard to follow and to change as it grows (nobody owns the whole flow).
  • Orchestration: a coordinator (saga orchestrator or a workflow engine) tells each service what to do and tracks state. Easier to understand, monitor and change; the coordinator is an extra component to build and keep available.

  Rules: every step and every compensation must be idempotent; saga state must be persisted so it survives crashes; and be explicit about isolation, since other requests can see intermediate states (use a "pending" status and semantic locks).

  CQRS: separate reads from writes

  Command Query Responsibility Segregation uses different models for changing data (commands) and for reading it (queries).

  100%

  Figure SDA-MSV-03-2 · CQRS with a read model · © forcsguy.com

  Why: reads and writes often differ in shape (normalised writes, denormalised reads) and scale (reads vastly outnumber writes). Cost: extra moving parts and eventual consistency. Use it where the gap between read and write needs is real, not as a default. A light form (one database, separate query code and views) captures much of the benefit.

  Event sourcing: store the changes

  Instead of storing the current state, store the sequence of events that led to it, as an append-only log. Current state is derived by replaying events (with snapshots for speed).

  type Event = { type: "Deposited" | "Withdrawn"; amount: number };
  const apply = (balance: number, e: Event) => e.type === "Deposited" ? balance + e.amount : balance - e.amount;
  const balance = events.reduce(apply, 0);          // state = fold over history
  

  Benefits: a perfect audit trail, ability to rebuild any view (including new ones) from history, debugging by replay, and natural fit with CQRS projections.

  Costs: more complex to build and query, event schema evolution over years, handling of mistakes by compensating events (never edit history), eventual consistency of views, and privacy rules (erasing personal data in an immutable log needs design, such as encryption keys per person). Event sourcing suits domains where history is the point (finance, ledgers, workflows); it is overkill for plain CRUD. (A ledger in the payment lessons is the same idea.)

  Using them together, and not

  Situation	Reach for
  Business process spanning several services	Saga (orchestrated if non-trivial)
  Read load and shape differ greatly from write	CQRS
  Need full history and rebuildable views	Event sourcing
  Simple CRUD within one service	None of these

  These patterns add power and complexity. Start with the simplest design (a modular monolith or a few services with synchronous calls), and adopt them where a concrete problem demands, ideally one at a time.

  Event design tips

  • Name events in the past tense after business facts.
  • Include ids, timestamps and a version; avoid leaking internal tables.
  • Prefer event-carried state for what consumers need, to avoid chatty callbacks, but beware coupling.
  • Keep a schema registry and evolve compatibly.
  • Make the flow observable: carry a correlation id through every event (see the observability lesson).

  Real-world reading

  Related reading

  • Event SourcingMartin FowlerBlogs
  • CQRSMartin FowlerBlogs
  • CQRS patternMicrosoft · Microsoft AzureDocumentation
  • Microservices PatternsChris Richardson · ManningBooks

  Show 6 moreShow fewer

  • Azure Architecture CenterMicrosoft · Microsoft AzureDocumentation
  • Pattern: SagaChris Richardson · microservices.ioDocumentation
  • 10 Tips for Building Resilient Payment SystemsShopify Engineering · ShopifyCase Studies
  • Microservices Patterns CatalogChris Richardson · microservices.ioDocumentation
  • Zendesk BlogZendesk engineers · ZendeskBlogs
  • LogFeeder: Log Collection System at YelpYelp engineers · YelpCase Studies

  Open this lesson with its quiz and exercise

• Classic System Designs · 10 minDesign a notification systemSend push, SMS and email notifications reliably at scale: queues per channel, user preferences, retries and avoiding spam.Read hereClose

  The problem

  Applications tell users things: "your order shipped", "someone replied", "your code is 482913". A notification system delivers such messages over several channels:

  • Mobile push through Apple (APNs) and Google (FCM) services.
  • SMS through a telecom provider.
  • Email through an email provider.
  • Sometimes in-app messages or webhooks.

  Requirements: soft real time (seconds, not hours), high volume with spikes, no lost notifications, no duplicate floods, user control over what they receive.

  High-level design

  100%

  Figure SDA-CSD-06-1 · Notification pipeline with a queue per channel · © forcsguy.com

  1. Other services call the notification service (an API or an event) with who and what.
  2. It validates, applies preferences, renders the message from a template and picks the channel.
  3. It places a job on a per-channel queue.
  4. Workers call the third-party provider and record the result.

  Putting a queue between each stage absorbs spikes and lets you scale the channels independently.

  100%

  Figure SDA-CSD-06-2 · Should this event become a notification? · © forcsguy.com

  Image: Océanos y dados · CC0 · Source · Wikimedia Commons: Smartphone Use

  Channels and their providers

  100%

  Figure SDA-CSD-06-3 · One provider per channel · © forcsguy.com

  Collecting contact details

  100%

  Figure SDA-CSD-06-4 · Contact information flow and tables · © forcsguy.com

  Before: one shared queue

  100%

  After: separate lanes

  100%

  What changed

  • A marketing blast no longer delays login codes
  • Each channel scales and fails on its own
  • Priorities can be set per lane

  Supporting data

  Data	Purpose
  Contact info and device tokens	where to send
  User preferences	channels allowed, topics, quiet hours, language
  Templates	consistent text, translations
  Notification log	state per notification (queued, sent, failed), for debugging and dedupe

  Reliability

  • Persist first: write the notification to the log before queueing, so a crash does not lose it.
  • At-least-once delivery from the queue, plus a dedupe check (notification_id already sent?) so retries do not double send.
  • Retries with exponential backoff and jitter; after repeated failures move to a dead-letter queue and alert.
  • Provider failures: use a second provider as a fallback for critical channels such as SMS verification codes.
  • Invalid tokens: providers tell you when a device token is dead; remove it.

  Protecting users and the system

  • Preferences and opt-out are mandatory for marketing messages (and legally required in many places).
  • Rate limit per user and per topic, and collapse bursts into a digest ("12 new comments").
  • Priority: security codes and payment alerts jump the queue ahead of marketing.
  • Scheduling: delay non-urgent notifications until a sensible local time.

  Why the design looks this way

  A first version with one notification server has three weaknesses: it is a single point of failure, everything scales together (database, cache and sending), and slow third-party calls drag down the whole thing at peak. The fixes are the ones in the design: move state out of the server, run several servers, and put a queue per channel between the API and the workers so an outage at one provider (say the SMS vendor) only backs up its own queue.

  Other points worth knowing: providers differ by region (a push service available in one market may not be in another), so keep each provider behind a plug-in interface. Only internal services or verified clients may call the send API, using an app key and secret, or the system becomes a spam relay. Notifications may be delayed or reordered but must never be silently lost, so persist each one before sending and retry from the stored record. And the single most useful health signal is the number of queued notifications: if it keeps growing, add workers.

  Life of a notification

  100%

  Figure SDA-CSD-06-5 · Notification states for tracking · © forcsguy.com

  100%

  Figure SDA-CSD-06-6 · The complete notification path · © forcsguy.com

  Observability

  Track sent, delivered, opened and failed counts per channel and per provider, plus queue depth and age. A growing queue age is the earliest sign something is wrong. Keep a trail so support can answer "did this user get the message?".

  Worked numbers: a steady stream and a blast

  Take 10 million users who each get about 5 notifications a day across push, email and in-app: 50 million a day, about 580 per second on average. That is easy. The hard case is a campaign blast: a push to all 10 million users within 10 minutes means 16,700 sends per second, and the push providers rate-limit each connection, so you need many parallel workers and connections, and you must prevent the blast from delaying a login code.

  Log volume: at about 200 bytes per delivery record, 50 million a day is 10 GB a day. Keeping 30 days of detail is about 300 GB, which is manageable; keep aggregates much longer.

  Provider failure: a push provider that fails for 10 minutes during a blast would otherwise cause 10 million retries at once. The queue absorbs it, workers back off, and delivery continues when the provider recovers.

  100%

  Figure SDA-CSD-06-7 · Separate lanes for urgent and bulk messages · © forcsguy.com

  One notification, start to finish

  1. Event: the order service publishes "order 4711 shipped" with the user id.
  2. Decide: the notification service loads the user's preferences (allowed channels, quiet hours, language), checks the per-user limit (not more than a few promotional messages a day) and the dedupe key (order-4711-shipped), and renders the template.
  3. Enqueue: a message is placed on the right queue with an idempotency key.
  4. Send: a worker calls the provider. On success it records "sent". On a temporary error it retries with backoff. On a permanent error (for example, an invalid device token) it removes the token and records the failure.
  5. Track: delivery and open receipts update the record; the data feeds dashboards and "did not receive" support tools.

  100%

  Figure SDA-CSD-06-8 · Deduplicate before sending · © forcsguy.com

  Mistakes people make

  • Sharing one queue between login codes and marketing. A blast delays the code people are waiting for. Use separate lanes.
  • No per-user rate limit. A bug that emits an event in a loop can send a user a hundred messages.
  • Ignoring preferences and legal rules. Unsubscribe, quiet hours and consent are requirements, not nice-to-haves.
  • Keeping dead device tokens. Providers tell you when a token is invalid. Remove it, or your sender reputation and efficiency suffer.
  • Retry storms. Back off with jitter and cap retries, or you will hammer a provider that is already failing.
  • Time zones. "9 am local time" needs the user's zone, and the send time must be scheduled per zone.

  Try it yourself

  Use the System Builder to see a burst absorbed by a queue, and read the rule engine lesson for how triggers, cooldowns and idempotent actions fit the "decide" step.

  More on this site

  Related reading

  • Avoiding insurmountable queue backlogsAmazon Builders' Library · Amazon Web ServicesBlogs
  • Timeouts, retries, and backoff with jitterAmazon Builders' Library · Amazon Web ServicesBlogs
  • Event SourcingMartin FowlerBlogs
  • Uber's Real-Time Push PlatformUber Engineering · UberCase Studies

  Show 6 moreShow fewer

  • NATS ServerNATS · CNCF / NATSGitHub
  • CQRSMartin FowlerBlogs
  • CQRS patternMicrosoft · Microsoft AzureDocumentation
  • Scaling Push Messaging for Millions of Devices at NetflixNetflix engineers · NetflixCase Studies
  • Reference Architectures for Big Data SystemsCmu engineers · CmuCase Studies
  • Messaging System Model at DropboxDropbox engineers · DropboxCase Studies

  Open this lesson with its quiz and exercise

• Real-World Systems · 13 minDesign a stock exchangeAn order book, a matching engine and a sequencer: how an exchange matches buyers and sellers fairly in microseconds.Read hereClose

  The problem

  An exchange lets buyers and sellers trade. Participants submit orders; the exchange matches them and reports the trades.

  Key properties:

  • Fairness: orders are handled in a strict, auditable order.
  • Very low latency: microseconds from order to acknowledgment.
  • High throughput: hundreds of thousands to millions of orders per second at peaks.
  • Correctness and durability: money is involved. No lost or duplicated trades.
  • Availability: an exchange outage halts a market.

  The order book

  For each symbol the order book holds resting orders:

  • Bids (buy orders), highest price first.
  • Asks (sell orders), lowest price first.
  • Within a price level, orders queue by arrival time.

  Common order types: limit (a worst price you accept; unmatched part rests in the book) and market (take the best available now).

  Matching: price-time priority

  An incoming buy matches the lowest-priced asks it can afford; among equal prices the earliest order matches first. Matching repeats until the order is filled or no longer matches, and any remainder rests in the book.

  Image: Thomas J. O'Halloran · Public domain · Source · Wikimedia Commons: NY stock exchange traders floor LC-U9-10548-6

  Architecture

  100%

  Figure SDA-RWS-05-1 · Exchange: gateway, sequencer, matching engine · © forcsguy.com

  1. Gateways authenticate participants, validate orders, and run pre-trade risk checks (limits, available funds).
  2. The sequencer gives every event a sequence number: one agreed total order, the basis of fairness and recovery.
  3. A matching engine applies events in order to the in-memory order book and emits trades.
  4. Everything is written to a durable event log. Executions go to participants, and a market data stream tells everyone about prices and depth.
  5. Clearing and settlement later move money and securities between accounts.

  100%

  Figure SDA-RWS-05-2 · An order book with a buy order sweeping the sell side · © forcsguy.com

  How it gets so fast

  • In-memory order books and no database in the critical path.
  • Single-threaded matching per symbol group: no locks, deterministic behaviour. Parallelism comes from partitioning symbols across engines.
  • Event sourcing: state is the result of applying an ordered log. Persist the event first (to a memory-mapped file or replicated log), then process.
  • Pre-allocated memory, ring buffers, no garbage-collection pauses, and careful cache-friendly data structures.
  • Hardware and network: servers co-located in the exchange's data centre, fast network cards and kernel-bypass networking, precise time sources.

  100%

  Figure SDA-RWS-05-3 · A fast engine on one machine: shared log and busy loops · © forcsguy.com

  Reliability

  • A hot standby applies the same event log and can take over within moments. Use sequence numbers to detect gaps and avoid duplicate processing.
  • Replicate the log across machines or sites before acknowledging, trading a little latency for durability.
  • Idempotent order identifiers prevent a resubmitted order executing twice.
  • Circuit breakers halt trading when prices move too fast.

  Market data fairness

  Everyone should see the same information at about the same time. Exchanges publish through multicast feeds, give all participants equal-length cables inside the data centre, and timestamp events at the matching engine. A separate, slower snapshot and retransmission service helps receivers that miss packets.

  What is out of scope here

  Clearing houses, settlement cycles, regulation and surveillance are large systems of their own. The core design is the sequencer plus deterministic matching engine plus durable log.

  Real-world reading

  Related reading

  • Event SourcingMartin FowlerBlogs
  • Monolith FirstMartin FowlerBlogs
  • Rewriting the heart of our sync engineDropboxCase Studies
  • Cape: Event Stream Processing Framework at DropboxDropbox engineers · DropboxCase Studies

  Show 6 moreShow fewer

  • Improving Search Engine Efficiency by over 25% at eBayeBay engineers · eBayCase Studies
  • Finding Kafka throughput limit in infrastructure at DropboxDropbox engineers · DropboxCase Studies
  • Bullet: Forward-Looking Query Engine for Streaming Data at YahooYahoo engineers · YahooCase Studies
  • MicroservicesJames Lewis and Martin Fowler · Martin FowlerBlogs
  • The LMAX ArchitectureMartin FowlerCase Studies
  • Microservices and OrchestrationMartinfowler engineers · MartinfowlerCase Studies

  Open this lesson with its quiz and exercise