Designing robust and predictable APIs with idempotency

Ideas you will meet

Idempotency

Making retries safe so duplicates do no harm.

Payments

Idempotency, ledgers and reliable money movement.

What we already know about this

Our own lessons on these ideas. Read them here before you start, so it makes sense.

• Core Concepts: How Things Actually Work · 9 minHow online payments workThe parties involved, why idempotency is essential, and how a ledger and reconciliation keep money correct.Read hereClose

  The players

  A card payment involves several parties:

  • Customer and their issuing bank (which holds the account).
  • Merchant and their acquiring bank.
  • A card network (such as Visa or Mastercard) connecting banks.
  • A payment service provider (PSP) that handles the complexity for merchants.

  Most businesses do not touch card numbers: they use a PSP's hosted form or tokens, which shrinks security and compliance scope (PCI DSS).

  Image: Syced · CC0 · Source · Wikimedia Commons: LaCuCa card reader - payment terminal at supermarket register

  A typical flow

  100%

  Figure SDA-CORE-15-1 · A typical flow · © forcsguy.com

  Steps are usually authorise (reserve funds) then capture (take the money), then settlement between banks days later.

  Idempotency

  Networks time out, users double-click, services retry. Without protection, a retry can charge twice. Solve it with an idempotency key: a unique value per intended payment, sent with the request. The server stores the result under that key, and any repeat returns the stored result instead of acting again.

  Model the money as a ledger

  Never just update a balance field. Record every movement as entries in an append-only ledger using double-entry bookkeeping: each transaction has a debit and an equal credit.

  • Balances are computed from entries, and always sum to zero across the system, so errors stand out.
  • You get a full audit trail.
  • Use integer minor units (cents), never floating point, for amounts, and store the currency.

  State machines and asynchronous reality

  A payment moves through states: created, pending, authorised, captured, failed, refunded. Make transitions explicit and allow only valid ones. Many results arrive asynchronously by webhook, so your system must accept webhooks idempotently, verify their signatures, and tolerate them arriving late or out of order.

  Reconciliation

  Your records, the provider's and the bank's will occasionally disagree (a timeout hid a success, a webhook was lost). A scheduled reconciliation job compares them, flags mismatches and drives them to a resolution. Treat unexplained differences as incidents.

  Reliability and security

  • Keep the write path simple and durable; do slow work (emails, analytics) asynchronously.
  • Use strong consistency for balances and payment state.
  • Protect against fraud with limits, velocity checks and risk scoring.
  • Never log card data or secrets; use tokenisation and encryption.

  Real-world reading

  Related reading

  • Designing robust and predictable APIs with idempotencyBrandur Leach · StripeBlogs
  • 10 Tips for Building Resilient Payment SystemsShopify Engineering · ShopifyCase Studies
  • Stripe Engineering BlogStripe engineers · StripeBlogs
  • Making retries safe with idempotent APIsAmazon Builders' Library · Amazon Web ServicesBlogs

  Show 6 moreShow fewer

  • Stripe's payments APIs: the first 10 yearsStripeCase Studies
  • Klarna BlogKlarna engineers · KlarnaBlogs
  • Square BlogBlock (Square) engineers · Block (Square)Blogs
  • Coinbase BlogCoinbase engineers · CoinbaseBlogs
  • Asana Engineering BlogAsana engineers · AsanaBlogs
  • Auth0 Engineering BlogAuth0 engineers · Auth0Blogs

  Open this lesson with its quiz and exercise

• Real-World Systems · 13 minDesign a payment systemTaking a customer's payment safely: the payment service, the ledger, retries and idempotency, asynchronous results and reconciliation.Read hereClose

  The problem

  Let customers pay for orders, and make sure the right amount moves, exactly once, and every cent can be accounted for.

  Requirements

  • Correctness above all: no double charges, no lost payments, no negative balances.
  • Reliability with partial failures: providers time out, webhooks arrive late.
  • Security: protect card data, prevent fraud.
  • Auditability: complete, immutable records.
  • Support several payment methods and providers.

  Components

  100%

  Figure SDA-RWS-07-1 · Payment system components · © forcsguy.com

  • Checkout collects payment details using the provider's hosted fields or SDK, so raw card numbers never touch your servers (you get a token).
  • Payment service owns the payment lifecycle and talks to providers.
  • Payments database stores each payment, its state and its idempotency key.
  • Ledger records money movements as balanced entries (see below).
  • Event bus tells other services (orders, emails) that a payment succeeded or failed.
  • Webhook handler receives the provider's asynchronous updates.
  • Reconciliation compares your records with the provider's.

  The payment flow

  1. The order service asks the payment service to create a payment with an idempotency key (usually the order id).
  2. The payment is saved as created, then submitted to the provider and moved to pending.
  3. The customer may complete an extra check (3-D Secure) at their bank.
  4. The provider reports the result, often asynchronously by webhook. The service verifies the webhook's signature, updates the state (authorised, captured, failed) and publishes an event.
  5. The order service reacts to the event (ship, or release stock).

  State machine

  Make states and allowed transitions explicit. A payment cannot jump from failed to captured or be captured twice. Reject invalid transitions and make every handler idempotent, because webhooks and retries can repeat.

  Idempotency

  Every request that creates or changes money carries an idempotency key. The service stores the key with the outcome; a repeat returns the stored outcome without acting. This protects against:

  • a customer double-clicking Pay,
  • a client retrying after a timeout,
  • the provider sending the same webhook twice.

  100%

  Figure SDA-RWS-07-2 · Retrying without paying twice · © forcsguy.com

  Image: P199 · Public domain · Source · Wikimedia Commons: Bank vault 1901

  Before: a retry can charge twice

  100%

  After: an idempotency key makes retries safe

  100%

  What changed

  • A repeated request returns the stored result
  • Network timeouts no longer risk a double charge
  • The key and result are stored with the payment

  The ledger

  Record every movement as double-entry lines: each transaction debits one account and credits another by the same amount (customer funds, merchant balance, fees, refunds). Entries are append-only and amounts are integers in minor units (cents) with a currency. Balances are derived from entries, and the books always sum to zero, which makes errors visible. Write the ledger entries in the same database transaction as the payment state change, so they cannot disagree.

  Example: a purchase in the ledger

  A customer pays 200 for an order. The ledger records one transaction with two balanced lines:

  Account	Debit	Credit
  Customer funds	200
  Merchant balance		200

  Debits equal credits, so the books stay balanced. A later refund is a new transaction that reverses these lines. The original entries are never edited.

  Handling uncertainty

  If a call to the provider times out you do not know what happened. Do not retry blindly and do not mark it failed:

  1. Keep the payment pending.
  2. Query the provider with the same key, or wait for the webhook.
  3. A reconciliation job later compares all pending and recent payments with the provider's report and settles any that are stuck.

  100%

  Figure SDA-RWS-07-3 · Authorise, clear, settle: three layers behind one payment · © forcsguy.com

  Reconciliation: the safety net

  Even with idempotency and a ledger, records in your system, the provider and the bank drift apart. Reconciliation compares them and flags the differences. Typical pains and answers:

  • Different formats. Dates, amounts and ids look different in each system, so normalise them into one shape first.
  • Volume. Nightly batch jobs are enough for most firms; a streaming job gives near-real-time matching.
  • Cut-off time. A payment stamped 23:59:55 by you may be stamped just after midnight by the provider. Mark such mismatches as temporary breaks and re-match them against the next day's file; only unresolved ones need a human.
  • Chained ids. Carry the order id through to the provider payment id and on to the payout id, so one purchase can be traced across all its events (purchase, fee, payout, and a currency conversion where one applies).

  Reconciliation is still needed when you believe you have exactly-once behaviour, because something outside your code can always go wrong.

  Security and fraud

  • Tokenise card data, encrypt secrets, and never log sensitive fields.
  • Verify webhook signatures and restrict who can call internal APIs.
  • Risk checks: velocity limits, device and address signals, rules and models; send risky payments for review.
  • Protect against replay by rejecting old timestamps and reused identifiers.

  Threats and the defence for each

  Threat	Typical defence
  Eavesdropping on requests and responses	HTTPS everywhere
  Tampering with data in transit	Encryption plus integrity checks and signatures
  Impersonating a server (man in the middle)	Certificates and certificate validation
  Losing payment data	Replication across zones and regular backups
  Floods of traffic (DDoS)	Rate limiting, firewalls and edge protection
  Stolen card numbers	Tokenisation: store and use tokens, not real card numbers
  Compliance obligations	PCI DSS controls for anyone handling branded cards
  Fraud	Address and card-code checks, device and behaviour analysis, risk scoring

  Scale

  Payments are write-heavy but not huge compared with social apps. Prioritise consistency and durability: use a strongly consistent database, a single writer for each payment, and a queue for the slow side effects. Shard by payment or merchant id if needed.

  Real-world reading

  Related reading

  • Designing robust and predictable APIs with idempotencyBrandur Leach · StripeBlogs
  • 10 Tips for Building Resilient Payment SystemsShopify Engineering · ShopifyCase Studies
  • Stripe Engineering BlogStripe engineers · StripeBlogs
  • Making retries safe with idempotent APIsAmazon Builders' Library · Amazon Web ServicesBlogs

  Show 6 moreShow fewer

  • Stripe's payments APIs: the first 10 yearsStripeCase Studies
  • Exactly-once Semantics Is Possible: Here's How Apache Kafka Does ItNeha Narkhede · ConfluentBlogs
  • Klarna BlogKlarna engineers · KlarnaBlogs
  • Square BlogBlock (Square) engineers · Block (Square)Blogs
  • Stripe Engineering BlogStripe engineers · StripeBlogs
  • Grofers Engineering BlogGrofers engineers · GrofersBlogs

  Open this lesson with its quiz and exercise

• Core Concepts: How Things Actually Work · 8 minWhere online payments are headingInstant payments, open banking, wallets and tokens, stronger authentication and new money types: what is changing and what it means for system design.Read hereClose

  Why this matters to a system designer

  Payment systems are changing from card-centric, batch and slow, to real-time, account-based and programmable. Each trend changes your architecture, so it helps to know the direction. The items below describe trends in the industry, not predictions about any one product.

  Trend 1: instant, always-on payments

  Many countries now run instant payment schemes: UPI (India), Pix (Brazil), Faster Payments (UK), SEPA Instant (Europe), FedNow (United States). Funds move between bank accounts in seconds, at any hour, including weekends.

  Design consequences:

  • Settlement is final and fast, so fraud checks must happen in real time, before sending.
  • Your system must run 24/7 with no overnight batch window.
  • Confirmation arrives asynchronously, so you need pending states, webhooks and reconciliation (see the payment design lesson).

  Trend 2: open banking and account-to-account payments

  Regulations such as the EU's PSD2 and similar rules elsewhere require banks to offer APIs so that, with the customer's consent, a regulated third party can see account data or initiate a payment directly from the bank account. This enables card-free checkout at lower cost.

  Design consequences: you integrate with consent flows and bank-specific quirks, handle redirects and expiring consents, and treat the bank as an unreliable dependency.

  Trend 3: wallets, network tokens and tokenisation

  Apple Pay, Google Pay and similar wallets store a token instead of the card number. Card networks also issue network tokens for saved cards that can be refreshed when a card is reissued. A stolen token is of little use elsewhere.

  Design consequences: store tokens, not card numbers; handle lifecycle updates (card replaced, token suspended); and rely on the provider's PCI-compliant fields.

  Trend 4: stronger authentication and less fraud

  Strong customer authentication (for example 3-D Secure 2) adds a risk-based extra check only when needed. Passkeys and device biometrics replace passwords and one-time codes. Machine learning scores transactions in milliseconds using device, behaviour and network signals.

  Design consequences: payment flows need extra steps and failure handling, and a risk engine on the critical path with tight latency budgets.

  Trend 5: embedded finance and flexible credit

  Payments are built into other products (marketplaces, software platforms, ride apps), and pay later options split a purchase into instalments. Platforms handle onboarding of sellers, split payouts and compliance.

  Design consequences: multi-party ledgers, payouts, KYC and risk for sub-merchants, and refunds that touch several accounts.

  Trend 6: richer messages and faster cross-border

  The ISO 20022 standard brings structured, data-rich payment messages to banks worldwide, improving reconciliation and screening. Cross-border networks are working to shorten the chain of intermediaries and improve tracking (see the cross-border lesson).

  Trend 7: new forms of money

  Stablecoins and central bank digital currencies (CBDCs) are being explored, piloted or regulated in different places. Treat them as additional payment rails with their own settlement, custody and regulatory models, not a replacement for the fundamentals.

  A ledger-based wallet on a blockchain keeps balances as shared, append-only records that many parties verify, which can shorten settlement. The trade-offs are throughput, finality rules, regulation and the loss of simple reversal, so treat it as another rail with its own failure modes, not a replacement for the ledger discipline above.

  The payment rails side by side

  100%

  Figure SDA-CORE-21-1 · Several rails into one payment service · © forcsguy.com

  What stays the same

  Whatever the rail, the engineering core is stable: idempotency, a double-entry ledger, a clear state machine, asynchronous confirmation, reconciliation, strong security, and observability. Build those well, and adding a new payment method becomes an integration, not a rewrite.

  Real-world reading

  Related reading

  • Stripe's payments APIs: the first 10 yearsStripeCase Studies
  • Stripe Engineering BlogStripe engineers · StripeBlogs
  • Square BlogBlock (Square) engineers · Block (Square)Blogs
  • Klarna BlogKlarna engineers · KlarnaBlogs

  Show 6 moreShow fewer

  • Coinbase BlogCoinbase engineers · CoinbaseBlogs
  • Nubank BlogNubank engineers · NubankBlogs
  • Kraken BlogKraken engineers · KrakenBlogs
  • Capital One BlogCapital One engineers · Capital OneBlogs
  • Automated Fake Account Detection at LinkedInLinkedIn engineers · LinkedInCase Studies
  • Squarespace Engineering BlogSquarespace engineers · SquarespaceBlogs

  Open this lesson with its quiz and exercise