Design a payment system

The problem

Let customers pay for orders, and make sure the right amount moves, exactly once, and every cent can be accounted for.

Requirements

• Correctness above all: no double charges, no lost payments, no negative balances.
• Reliability with partial failures: providers time out, webhooks arrive late.
• Security: protect card data, prevent fraud.
• Auditability: complete, immutable records.
• Support several payment methods and providers.

Components

100%

Figure SDA-RWS-07-1 · Payment system components · © forcsguy.com

• Checkout collects payment details using the provider's hosted fields or SDK, so raw card numbers never touch your servers (you get a token).
• Payment service owns the payment lifecycle and talks to providers.
• Payments database stores each payment, its state and its idempotency key.
• Ledger records money movements as balanced entries (see below).
• Event bus tells other services (orders, emails) that a payment succeeded or failed.
• Webhook handler receives the provider's asynchronous updates.
• Reconciliation compares your records with the provider's.

The payment flow

1. The order service asks the payment service to create a payment with an idempotency key (usually the order id).
2. The payment is saved as created, then submitted to the provider and moved to pending.
3. The customer may complete an extra check (3-D Secure) at their bank.
4. The provider reports the result, often asynchronously by webhook. The service verifies the webhook's signature, updates the state (authorised, captured, failed) and publishes an event.
5. The order service reacts to the event (ship, or release stock).

State machine

Make states and allowed transitions explicit. A payment cannot jump from failed to captured or be captured twice. Reject invalid transitions and make every handler idempotent, because webhooks and retries can repeat.

Idempotency

Every request that creates or changes money carries an idempotency key. The service stores the key with the outcome; a repeat returns the stored outcome without acting. This protects against:

• a customer double-clicking Pay,
• a client retrying after a timeout,
• the provider sending the same webhook twice.

100%

Figure SDA-RWS-07-2 · Retrying without paying twice · © forcsguy.com

Image: P199 · Public domain · Source · Wikimedia Commons: Bank vault 1901

The ledger

Record every movement as double-entry lines: each transaction debits one account and credits another by the same amount (customer funds, merchant balance, fees, refunds). Entries are append-only and amounts are integers in minor units (cents) with a currency. Balances are derived from entries, and the books always sum to zero, which makes errors visible. Write the ledger entries in the same database transaction as the payment state change, so they cannot disagree.

Example: a purchase in the ledger

A customer pays 200 for an order. The ledger records one transaction with two balanced lines:

Account	Debit	Credit
Customer funds	200
Merchant balance		200

Debits equal credits, so the books stay balanced. A later refund is a new transaction that reverses these lines. The original entries are never edited.

Handling uncertainty

If a call to the provider times out you do not know what happened. Do not retry blindly and do not mark it failed:

1. Keep the payment pending.
2. Query the provider with the same key, or wait for the webhook.
3. A reconciliation job later compares all pending and recent payments with the provider's report and settles any that are stuck.

100%

Figure SDA-RWS-07-3 · Authorise, clear, settle: three layers behind one payment · © forcsguy.com

Reconciliation: the safety net

Even with idempotency and a ledger, records in your system, the provider and the bank drift apart. Reconciliation compares them and flags the differences. Typical pains and answers:

• Different formats. Dates, amounts and ids look different in each system, so normalise them into one shape first.
• Volume. Nightly batch jobs are enough for most firms; a streaming job gives near-real-time matching.
• Cut-off time. A payment stamped 23:59:55 by you may be stamped just after midnight by the provider. Mark such mismatches as temporary breaks and re-match them against the next day's file; only unresolved ones need a human.
• Chained ids. Carry the order id through to the provider payment id and on to the payout id, so one purchase can be traced across all its events (purchase, fee, payout, and a currency conversion where one applies).

Reconciliation is still needed when you believe you have exactly-once behaviour, because something outside your code can always go wrong.

Security and fraud

• Tokenise card data, encrypt secrets, and never log sensitive fields.
• Verify webhook signatures and restrict who can call internal APIs.
• Risk checks: velocity limits, device and address signals, rules and models; send risky payments for review.
• Protect against replay by rejecting old timestamps and reused identifiers.

Threats and the defence for each

Threat	Typical defence
Eavesdropping on requests and responses	HTTPS everywhere
Tampering with data in transit	Encryption plus integrity checks and signatures
Impersonating a server (man in the middle)	Certificates and certificate validation
Losing payment data	Replication across zones and regular backups
Floods of traffic (DDoS)	Rate limiting, firewalls and edge protection
Stolen card numbers	Tokenisation: store and use tokens, not real card numbers
Compliance obligations	PCI DSS controls for anyone handling branded cards
Fraud	Address and card-code checks, device and behaviour analysis, risk scoring

Scale

Payments are write-heavy but not huge compared with social apps. Prioritise consistency and durability: use a strongly consistent database, a single writer for each payment, and a queue for the slow side effects. Shard by payment or merchant id if needed.

Real-world reading