Making retries safe with idempotent APIs

Ideas you will meet

Idempotency

Making retries safe so duplicates do no harm.

Resilience

Timeouts, retries, circuit breakers, bulkheads and load shedding.

What we already know about this

Our own lessons on these ideas. Read them here before you start, so it makes sense.

• Core Concepts: How Things Actually Work · 9 minHow online payments workThe parties involved, why idempotency is essential, and how a ledger and reconciliation keep money correct.Read hereClose

  The players

  A card payment involves several parties:

  • Customer and their issuing bank (which holds the account).
  • Merchant and their acquiring bank.
  • A card network (such as Visa or Mastercard) connecting banks.
  • A payment service provider (PSP) that handles the complexity for merchants.

  Most businesses do not touch card numbers: they use a PSP's hosted form or tokens, which shrinks security and compliance scope (PCI DSS).

  Image: Syced · CC0 · Source · Wikimedia Commons: LaCuCa card reader - payment terminal at supermarket register

  A typical flow

  100%

  Figure SDA-CORE-15-1 · A typical flow · © forcsguy.com

  Steps are usually authorise (reserve funds) then capture (take the money), then settlement between banks days later.

  Idempotency

  Networks time out, users double-click, services retry. Without protection, a retry can charge twice. Solve it with an idempotency key: a unique value per intended payment, sent with the request. The server stores the result under that key, and any repeat returns the stored result instead of acting again.

  Model the money as a ledger

  Never just update a balance field. Record every movement as entries in an append-only ledger using double-entry bookkeeping: each transaction has a debit and an equal credit.

  • Balances are computed from entries, and always sum to zero across the system, so errors stand out.
  • You get a full audit trail.
  • Use integer minor units (cents), never floating point, for amounts, and store the currency.

  State machines and asynchronous reality

  A payment moves through states: created, pending, authorised, captured, failed, refunded. Make transitions explicit and allow only valid ones. Many results arrive asynchronously by webhook, so your system must accept webhooks idempotently, verify their signatures, and tolerate them arriving late or out of order.

  Reconciliation

  Your records, the provider's and the bank's will occasionally disagree (a timeout hid a success, a webhook was lost). A scheduled reconciliation job compares them, flags mismatches and drives them to a resolution. Treat unexplained differences as incidents.

  Reliability and security

  • Keep the write path simple and durable; do slow work (emails, analytics) asynchronously.
  • Use strong consistency for balances and payment state.
  • Protect against fraud with limits, velocity checks and risk scoring.
  • Never log card data or secrets; use tokenisation and encryption.

  Real-world reading

  Related reading

  • Designing robust and predictable APIs with idempotencyBrandur Leach · StripeBlogs
  • 10 Tips for Building Resilient Payment SystemsShopify Engineering · ShopifyCase Studies
  • Stripe Engineering BlogStripe engineers · StripeBlogs
  • Making retries safe with idempotent APIsAmazon Builders' Library · Amazon Web ServicesBlogs

  Show 6 moreShow fewer

  • Stripe's payments APIs: the first 10 yearsStripeCase Studies
  • Klarna BlogKlarna engineers · KlarnaBlogs
  • Square BlogBlock (Square) engineers · Block (Square)Blogs
  • Coinbase BlogCoinbase engineers · CoinbaseBlogs
  • Asana Engineering BlogAsana engineers · AsanaBlogs
  • Auth0 Engineering BlogAuth0 engineers · Auth0Blogs

  Open this lesson with its quiz and exercise

• Real-World Systems · 13 minDesign a payment systemTaking a customer's payment safely: the payment service, the ledger, retries and idempotency, asynchronous results and reconciliation.Read hereClose

  The problem

  Let customers pay for orders, and make sure the right amount moves, exactly once, and every cent can be accounted for.

  Requirements

  • Correctness above all: no double charges, no lost payments, no negative balances.
  • Reliability with partial failures: providers time out, webhooks arrive late.
  • Security: protect card data, prevent fraud.
  • Auditability: complete, immutable records.
  • Support several payment methods and providers.

  Components

  100%

  Figure SDA-RWS-07-1 · Payment system components · © forcsguy.com

  • Checkout collects payment details using the provider's hosted fields or SDK, so raw card numbers never touch your servers (you get a token).
  • Payment service owns the payment lifecycle and talks to providers.
  • Payments database stores each payment, its state and its idempotency key.
  • Ledger records money movements as balanced entries (see below).
  • Event bus tells other services (orders, emails) that a payment succeeded or failed.
  • Webhook handler receives the provider's asynchronous updates.
  • Reconciliation compares your records with the provider's.

  The payment flow

  1. The order service asks the payment service to create a payment with an idempotency key (usually the order id).
  2. The payment is saved as created, then submitted to the provider and moved to pending.
  3. The customer may complete an extra check (3-D Secure) at their bank.
  4. The provider reports the result, often asynchronously by webhook. The service verifies the webhook's signature, updates the state (authorised, captured, failed) and publishes an event.
  5. The order service reacts to the event (ship, or release stock).

  State machine

  Make states and allowed transitions explicit. A payment cannot jump from failed to captured or be captured twice. Reject invalid transitions and make every handler idempotent, because webhooks and retries can repeat.

  Idempotency

  Every request that creates or changes money carries an idempotency key. The service stores the key with the outcome; a repeat returns the stored outcome without acting. This protects against:

  • a customer double-clicking Pay,
  • a client retrying after a timeout,
  • the provider sending the same webhook twice.

  100%

  Figure SDA-RWS-07-2 · Retrying without paying twice · © forcsguy.com

  Image: P199 · Public domain · Source · Wikimedia Commons: Bank vault 1901

  Before: a retry can charge twice

  100%

  After: an idempotency key makes retries safe

  100%

  What changed

  • A repeated request returns the stored result
  • Network timeouts no longer risk a double charge
  • The key and result are stored with the payment

  The ledger

  Record every movement as double-entry lines: each transaction debits one account and credits another by the same amount (customer funds, merchant balance, fees, refunds). Entries are append-only and amounts are integers in minor units (cents) with a currency. Balances are derived from entries, and the books always sum to zero, which makes errors visible. Write the ledger entries in the same database transaction as the payment state change, so they cannot disagree.

  Example: a purchase in the ledger

  A customer pays 200 for an order. The ledger records one transaction with two balanced lines:

  Account	Debit	Credit
  Customer funds	200
  Merchant balance		200

  Debits equal credits, so the books stay balanced. A later refund is a new transaction that reverses these lines. The original entries are never edited.

  Handling uncertainty

  If a call to the provider times out you do not know what happened. Do not retry blindly and do not mark it failed:

  1. Keep the payment pending.
  2. Query the provider with the same key, or wait for the webhook.
  3. A reconciliation job later compares all pending and recent payments with the provider's report and settles any that are stuck.

  100%

  Figure SDA-RWS-07-3 · Authorise, clear, settle: three layers behind one payment · © forcsguy.com

  Reconciliation: the safety net

  Even with idempotency and a ledger, records in your system, the provider and the bank drift apart. Reconciliation compares them and flags the differences. Typical pains and answers:

  • Different formats. Dates, amounts and ids look different in each system, so normalise them into one shape first.
  • Volume. Nightly batch jobs are enough for most firms; a streaming job gives near-real-time matching.
  • Cut-off time. A payment stamped 23:59:55 by you may be stamped just after midnight by the provider. Mark such mismatches as temporary breaks and re-match them against the next day's file; only unresolved ones need a human.
  • Chained ids. Carry the order id through to the provider payment id and on to the payout id, so one purchase can be traced across all its events (purchase, fee, payout, and a currency conversion where one applies).

  Reconciliation is still needed when you believe you have exactly-once behaviour, because something outside your code can always go wrong.

  Security and fraud

  • Tokenise card data, encrypt secrets, and never log sensitive fields.
  • Verify webhook signatures and restrict who can call internal APIs.
  • Risk checks: velocity limits, device and address signals, rules and models; send risky payments for review.
  • Protect against replay by rejecting old timestamps and reused identifiers.

  Threats and the defence for each

  Threat	Typical defence
  Eavesdropping on requests and responses	HTTPS everywhere
  Tampering with data in transit	Encryption plus integrity checks and signatures
  Impersonating a server (man in the middle)	Certificates and certificate validation
  Losing payment data	Replication across zones and regular backups
  Floods of traffic (DDoS)	Rate limiting, firewalls and edge protection
  Stolen card numbers	Tokenisation: store and use tokens, not real card numbers
  Compliance obligations	PCI DSS controls for anyone handling branded cards
  Fraud	Address and card-code checks, device and behaviour analysis, risk scoring

  Scale

  Payments are write-heavy but not huge compared with social apps. Prioritise consistency and durability: use a strongly consistent database, a single writer for each payment, and a queue for the slow side effects. Shard by payment or merchant id if needed.

  Real-world reading

  Related reading

  • Designing robust and predictable APIs with idempotencyBrandur Leach · StripeBlogs
  • 10 Tips for Building Resilient Payment SystemsShopify Engineering · ShopifyCase Studies
  • Stripe Engineering BlogStripe engineers · StripeBlogs
  • Making retries safe with idempotent APIsAmazon Builders' Library · Amazon Web ServicesBlogs

  Show 6 moreShow fewer

  • Stripe's payments APIs: the first 10 yearsStripeCase Studies
  • Exactly-once Semantics Is Possible: Here's How Apache Kafka Does ItNeha Narkhede · ConfluentBlogs
  • Klarna BlogKlarna engineers · KlarnaBlogs
  • Square BlogBlock (Square) engineers · Block (Square)Blogs
  • Stripe Engineering BlogStripe engineers · StripeBlogs
  • Grofers Engineering BlogGrofers engineers · GrofersBlogs

  Open this lesson with its quiz and exercise

• Advanced Distributed Systems · 13 minResilience: timeouts, retries, bulkheads, backpressure and circuit breakersHow to stop one slow or failing dependency from taking everything down: the patterns, how they fit together and where each one goes wrong.Read hereClose

  The core problem

  In a system of many services, slowness is more dangerous than failure. A dependency that fails fast is easy to handle. One that responds in 30 seconds makes callers hold threads, connections and memory until they too stop responding: a cascading failure. Resilience patterns limit how far a problem can spread.

  The toolbox

  100%

  Figure SDA-ADS-06-1 · Resilience layers around a call · © forcsguy.com

  Timeouts

  Every network call needs a timeout, including connect, read and overall. Pick from measured latency (for example slightly above the 99th percentile), not a guess, and make the caller's budget smaller than its own caller's so work is abandoned in an orderly way. Propagate deadlines across calls.

  Retries

  Retry only transient failures and only idempotent operations (or use idempotency keys). Use exponential backoff with jitter (random spread) so clients do not retry in lockstep, cap the attempts, and apply a retry budget (for example no more than 10 percent extra traffic). Retry at one layer, not at every layer, or attempts multiply.

  Circuit breaker

  A breaker watches failures or slow calls. Closed: normal. After too many failures it opens: calls fail immediately (or use the fallback) so the dependency can recover and callers stop wasting resources. After a cool-down it goes half-open: a few trial calls decide whether to close again.

  100%

  Figure SDA-ADS-06-2 · Circuit breaker states · © forcsguy.com

  Bulkheads

  Partition resources so one dependency cannot consume them all: separate thread pools or connection pools per dependency, separate queues per tenant or per priority, or separate instances for critical paths. A slow recommendation service then cannot starve checkout.

  Backpressure and load shedding

  If producers outpace consumers, work piles up. Without limits the buffer grows until memory is exhausted or latency becomes huge. Options:

  • Bounded queues that reject or block when full.
  • Signalling upstream to slow down (credit-based flow in streaming systems, TCP windows, reactive streams).
  • Load shedding: deliberately drop low-priority work early (return 429 or 503 quickly) to protect the rest.
  • Rate limiting at the edge (see the rate-limiter lesson).

  Failing fast and visibly under overload is better than accepting everything and failing slowly.

  Fallbacks and graceful degradation

  Return something useful when a dependency fails: cached data, a default, a simpler algorithm, or hide a non-essential section. Decide which features are essential before an incident, and test the degraded mode.

  How they interact

  • A timeout is the minimum; without it nothing else works.
  • Retries inside a breaker inside a bulkhead is the usual nesting.
  • Everything needs metrics: breaker state, rejected calls, queue depth, timeout counts. Alert on them.
  • Test it: inject latency and failures (chaos experiments) to confirm the patterns work before production does it for you.

  Common mistakes

  Retries with no backoff; timeouts longer than the caller's; unbounded queues; a shared thread pool for everything; breakers that never close; fallbacks that call the same failing service; and testing only the happy path.

  Real-world reading

  Related reading

  • Timeouts, retries, and backoff with jitterAmazon Builders' Library · Amazon Web ServicesBlogs
  • Using load shedding to avoid overloadAmazon Builders' Library · Amazon Web ServicesBlogs
  • Circuit BreakerMartin FowlerBlogs
  • Circuit Breaker patternMicrosoft · Microsoft AzureDocumentation

  Show 6 moreShow fewer

  • Google SRE Book: Addressing Cascading FailuresGoogle SRE team · GoogleBlogs
  • Chaos Monkey (source)Netflix · GitHubGitHub
  • Bulkheads: Partition and Tolerate Failure in One PartSkife engineers · SkifeCase Studies
  • Protector: Circuit Breaker for Time Series Databases at TrivagoTrivago engineers · TrivagoCase Studies
  • Autoscaling Jenkins at TrivagoTrivago engineers · TrivagoCase Studies
  • Troubleshooting a Connection Timeout Issue with tcp_tw_recycle Enabled at eBayeBay engineers · eBayCase Studies

  Open this lesson with its quiz and exercise