Asana Engineering Blog

Ideas you will meet

Distributed Systems

Consensus, time, failure and the fundamentals of scale-out.

What we already know about this

Our own lessons on these ideas. Read them here before you start, so it makes sense.

• Core Concepts: How Things Actually Work · 9 minHow online payments workThe parties involved, why idempotency is essential, and how a ledger and reconciliation keep money correct.Read hereClose

  The players

  A card payment involves several parties:

  • Customer and their issuing bank (which holds the account).
  • Merchant and their acquiring bank.
  • A card network (such as Visa or Mastercard) connecting banks.
  • A payment service provider (PSP) that handles the complexity for merchants.

  Most businesses do not touch card numbers: they use a PSP's hosted form or tokens, which shrinks security and compliance scope (PCI DSS).

  Image: Syced · CC0 · Source · Wikimedia Commons: LaCuCa card reader - payment terminal at supermarket register

  A typical flow

  100%

  Figure SDA-CORE-15-1 · A typical flow · © forcsguy.com

  Steps are usually authorise (reserve funds) then capture (take the money), then settlement between banks days later.

  Idempotency

  Networks time out, users double-click, services retry. Without protection, a retry can charge twice. Solve it with an idempotency key: a unique value per intended payment, sent with the request. The server stores the result under that key, and any repeat returns the stored result instead of acting again.

  Model the money as a ledger

  Never just update a balance field. Record every movement as entries in an append-only ledger using double-entry bookkeeping: each transaction has a debit and an equal credit.

  • Balances are computed from entries, and always sum to zero across the system, so errors stand out.
  • You get a full audit trail.
  • Use integer minor units (cents), never floating point, for amounts, and store the currency.

  State machines and asynchronous reality

  A payment moves through states: created, pending, authorised, captured, failed, refunded. Make transitions explicit and allow only valid ones. Many results arrive asynchronously by webhook, so your system must accept webhooks idempotently, verify their signatures, and tolerate them arriving late or out of order.

  Reconciliation

  Your records, the provider's and the bank's will occasionally disagree (a timeout hid a success, a webhook was lost). A scheduled reconciliation job compares them, flags mismatches and drives them to a resolution. Treat unexplained differences as incidents.

  Reliability and security

  • Keep the write path simple and durable; do slow work (emails, analytics) asynchronously.
  • Use strong consistency for balances and payment state.
  • Protect against fraud with limits, velocity checks and risk scoring.
  • Never log card data or secrets; use tokenisation and encryption.

  Real-world reading

  Related reading

  • Designing robust and predictable APIs with idempotencyBrandur Leach · StripeBlogs
  • 10 Tips for Building Resilient Payment SystemsShopify Engineering · ShopifyCase Studies
  • Stripe Engineering BlogStripe engineers · StripeBlogs
  • Making retries safe with idempotent APIsAmazon Builders' Library · Amazon Web ServicesBlogs

  Show 6 moreShow fewer

  • Stripe's payments APIs: the first 10 yearsStripeCase Studies
  • Klarna BlogKlarna engineers · KlarnaBlogs
  • Square BlogBlock (Square) engineers · Block (Square)Blogs
  • Coinbase BlogCoinbase engineers · CoinbaseBlogs
  • Asana Engineering BlogAsana engineers · AsanaBlogs
  • Auth0 Engineering BlogAuth0 engineers · Auth0Blogs

  Open this lesson with its quiz and exercise