End-to-End Arguments in System Design (paper)

Ideas you will meet

Distributed Systems

Consensus, time, failure and the fundamentals of scale-out.

Idempotency

Making retries safe so duplicates do no harm.

What we already know about this

Our own lessons on these ideas. Read them here before you start, so it makes sense.

• Real-World Systems · 13 minDesign a payment systemTaking a customer's payment safely: the payment service, the ledger, retries and idempotency, asynchronous results and reconciliation.Read hereClose

  The problem

  Let customers pay for orders, and make sure the right amount moves, exactly once, and every cent can be accounted for.

  Requirements

  • Correctness above all: no double charges, no lost payments, no negative balances.
  • Reliability with partial failures: providers time out, webhooks arrive late.
  • Security: protect card data, prevent fraud.
  • Auditability: complete, immutable records.
  • Support several payment methods and providers.

  Components

  100%

  Figure SDA-RWS-07-1 · Payment system components · © forcsguy.com

  • Checkout collects payment details using the provider's hosted fields or SDK, so raw card numbers never touch your servers (you get a token).
  • Payment service owns the payment lifecycle and talks to providers.
  • Payments database stores each payment, its state and its idempotency key.
  • Ledger records money movements as balanced entries (see below).
  • Event bus tells other services (orders, emails) that a payment succeeded or failed.
  • Webhook handler receives the provider's asynchronous updates.
  • Reconciliation compares your records with the provider's.

  The payment flow

  1. The order service asks the payment service to create a payment with an idempotency key (usually the order id).
  2. The payment is saved as created, then submitted to the provider and moved to pending.
  3. The customer may complete an extra check (3-D Secure) at their bank.
  4. The provider reports the result, often asynchronously by webhook. The service verifies the webhook's signature, updates the state (authorised, captured, failed) and publishes an event.
  5. The order service reacts to the event (ship, or release stock).

  State machine

  Make states and allowed transitions explicit. A payment cannot jump from failed to captured or be captured twice. Reject invalid transitions and make every handler idempotent, because webhooks and retries can repeat.

  Idempotency

  Every request that creates or changes money carries an idempotency key. The service stores the key with the outcome; a repeat returns the stored outcome without acting. This protects against:

  • a customer double-clicking Pay,
  • a client retrying after a timeout,
  • the provider sending the same webhook twice.

  100%

  Figure SDA-RWS-07-2 · Retrying without paying twice · © forcsguy.com

  Image: P199 · Public domain · Source · Wikimedia Commons: Bank vault 1901

  Before: a retry can charge twice

  100%

  After: an idempotency key makes retries safe

  100%

  What changed

  • A repeated request returns the stored result
  • Network timeouts no longer risk a double charge
  • The key and result are stored with the payment

  The ledger

  Record every movement as double-entry lines: each transaction debits one account and credits another by the same amount (customer funds, merchant balance, fees, refunds). Entries are append-only and amounts are integers in minor units (cents) with a currency. Balances are derived from entries, and the books always sum to zero, which makes errors visible. Write the ledger entries in the same database transaction as the payment state change, so they cannot disagree.

  Example: a purchase in the ledger

  A customer pays 200 for an order. The ledger records one transaction with two balanced lines:

  Account	Debit	Credit
  Customer funds	200
  Merchant balance		200

  Debits equal credits, so the books stay balanced. A later refund is a new transaction that reverses these lines. The original entries are never edited.

  Handling uncertainty

  If a call to the provider times out you do not know what happened. Do not retry blindly and do not mark it failed:

  1. Keep the payment pending.
  2. Query the provider with the same key, or wait for the webhook.
  3. A reconciliation job later compares all pending and recent payments with the provider's report and settles any that are stuck.

  100%

  Figure SDA-RWS-07-3 · Authorise, clear, settle: three layers behind one payment · © forcsguy.com

  Reconciliation: the safety net

  Even with idempotency and a ledger, records in your system, the provider and the bank drift apart. Reconciliation compares them and flags the differences. Typical pains and answers:

  • Different formats. Dates, amounts and ids look different in each system, so normalise them into one shape first.
  • Volume. Nightly batch jobs are enough for most firms; a streaming job gives near-real-time matching.
  • Cut-off time. A payment stamped 23:59:55 by you may be stamped just after midnight by the provider. Mark such mismatches as temporary breaks and re-match them against the next day's file; only unresolved ones need a human.
  • Chained ids. Carry the order id through to the provider payment id and on to the payout id, so one purchase can be traced across all its events (purchase, fee, payout, and a currency conversion where one applies).

  Reconciliation is still needed when you believe you have exactly-once behaviour, because something outside your code can always go wrong.

  Security and fraud

  • Tokenise card data, encrypt secrets, and never log sensitive fields.
  • Verify webhook signatures and restrict who can call internal APIs.
  • Risk checks: velocity limits, device and address signals, rules and models; send risky payments for review.
  • Protect against replay by rejecting old timestamps and reused identifiers.

  Threats and the defence for each

  Threat	Typical defence
  Eavesdropping on requests and responses	HTTPS everywhere
  Tampering with data in transit	Encryption plus integrity checks and signatures
  Impersonating a server (man in the middle)	Certificates and certificate validation
  Losing payment data	Replication across zones and regular backups
  Floods of traffic (DDoS)	Rate limiting, firewalls and edge protection
  Stolen card numbers	Tokenisation: store and use tokens, not real card numbers
  Compliance obligations	PCI DSS controls for anyone handling branded cards
  Fraud	Address and card-code checks, device and behaviour analysis, risk scoring

  Scale

  Payments are write-heavy but not huge compared with social apps. Prioritise consistency and durability: use a strongly consistent database, a single writer for each payment, and a queue for the slow side effects. Shard by payment or merchant id if needed.

  Real-world reading

  Related reading

  • Designing robust and predictable APIs with idempotencyBrandur Leach · StripeBlogs
  • 10 Tips for Building Resilient Payment SystemsShopify Engineering · ShopifyCase Studies
  • Stripe Engineering BlogStripe engineers · StripeBlogs
  • Making retries safe with idempotent APIsAmazon Builders' Library · Amazon Web ServicesBlogs

  Show 6 moreShow fewer

  • Stripe's payments APIs: the first 10 yearsStripeCase Studies
  • Exactly-once Semantics Is Possible: Here's How Apache Kafka Does ItNeha Narkhede · ConfluentBlogs
  • Klarna BlogKlarna engineers · KlarnaBlogs
  • Square BlogBlock (Square) engineers · Block (Square)Blogs
  • Stripe Engineering BlogStripe engineers · StripeBlogs
  • Grofers Engineering BlogGrofers engineers · GrofersBlogs

  Open this lesson with its quiz and exercise

• Core Concepts: How Things Actually Work · 9 minHow online payments workThe parties involved, why idempotency is essential, and how a ledger and reconciliation keep money correct.Read hereClose

  The players

  A card payment involves several parties:

  • Customer and their issuing bank (which holds the account).
  • Merchant and their acquiring bank.
  • A card network (such as Visa or Mastercard) connecting banks.
  • A payment service provider (PSP) that handles the complexity for merchants.

  Most businesses do not touch card numbers: they use a PSP's hosted form or tokens, which shrinks security and compliance scope (PCI DSS).

  Image: Syced · CC0 · Source · Wikimedia Commons: LaCuCa card reader - payment terminal at supermarket register

  A typical flow

  100%

  Figure SDA-CORE-15-1 · A typical flow · © forcsguy.com

  Steps are usually authorise (reserve funds) then capture (take the money), then settlement between banks days later.

  Idempotency

  Networks time out, users double-click, services retry. Without protection, a retry can charge twice. Solve it with an idempotency key: a unique value per intended payment, sent with the request. The server stores the result under that key, and any repeat returns the stored result instead of acting again.

  Model the money as a ledger

  Never just update a balance field. Record every movement as entries in an append-only ledger using double-entry bookkeeping: each transaction has a debit and an equal credit.

  • Balances are computed from entries, and always sum to zero across the system, so errors stand out.
  • You get a full audit trail.
  • Use integer minor units (cents), never floating point, for amounts, and store the currency.

  State machines and asynchronous reality

  A payment moves through states: created, pending, authorised, captured, failed, refunded. Make transitions explicit and allow only valid ones. Many results arrive asynchronously by webhook, so your system must accept webhooks idempotently, verify their signatures, and tolerate them arriving late or out of order.

  Reconciliation

  Your records, the provider's and the bank's will occasionally disagree (a timeout hid a success, a webhook was lost). A scheduled reconciliation job compares them, flags mismatches and drives them to a resolution. Treat unexplained differences as incidents.

  Reliability and security

  • Keep the write path simple and durable; do slow work (emails, analytics) asynchronously.
  • Use strong consistency for balances and payment state.
  • Protect against fraud with limits, velocity checks and risk scoring.
  • Never log card data or secrets; use tokenisation and encryption.

  Real-world reading

  Related reading

  • Designing robust and predictable APIs with idempotencyBrandur Leach · StripeBlogs
  • 10 Tips for Building Resilient Payment SystemsShopify Engineering · ShopifyCase Studies
  • Stripe Engineering BlogStripe engineers · StripeBlogs
  • Making retries safe with idempotent APIsAmazon Builders' Library · Amazon Web ServicesBlogs

  Show 6 moreShow fewer

  • Stripe's payments APIs: the first 10 yearsStripeCase Studies
  • Klarna BlogKlarna engineers · KlarnaBlogs
  • Square BlogBlock (Square) engineers · Block (Square)Blogs
  • Coinbase BlogCoinbase engineers · CoinbaseBlogs
  • Asana Engineering BlogAsana engineers · AsanaBlogs
  • Auth0 Engineering BlogAuth0 engineers · Auth0Blogs

  Open this lesson with its quiz and exercise