Payment Transaction Management in Microservices at Mercari

Ideas you will meet

Microservices

Decompose systems into independently deployable services.

Payments

Idempotency, ledgers and reliable money movement.

What we already know about this

Our own lessons on these ideas. Read them here before you start, so it makes sense.

• Real-World Systems · 11 minHow microservices work togetherSynchronous calls, events, sagas and discovery: how separate services cooperate without becoming one tangled system.Read hereClose

  What a microservice architecture is

  Instead of one large application, the system is split into small services, each responsible for one business capability (orders, payments, inventory, users). Services are deployed and scaled independently, often by different teams, and talk over the network.

  The benefits are independent deployment, team autonomy and targeted scaling. The cost is that everything that used to be a function call is now a network call that can be slow or fail, and data is spread across services.

  Ways services talk

  Style	How	Good for	Cost
  Synchronous request and response (REST, gRPC)	Caller waits for the answer	Queries and steps the user waits for	Coupling in time; failures propagate
  Asynchronous messages and events (queues, streams)	Caller publishes and moves on	Reactions, workflows, decoupling	Eventual consistency; harder to trace

  100%

  Figure SDA-RWS-13-1 · Services: sync calls, events and owned data · © forcsguy.com

  100%

  Figure SDA-RWS-13-2 · Two ways to run a multi-step process · © forcsguy.com

  Before: a chain of synchronous calls

  100%

  After: events between services

  100%

  What changed

  • A slow service no longer blocks the order
  • New services can listen without changing the sender
  • The result is eventually consistent, so design for that

  Data ownership

  Each service owns its own data and exposes it only through its API or events. No other service reads its tables. This keeps boundaries real, but it means you cannot join across services in SQL: you call APIs or keep read models built from events.

  Keeping things consistent: sagas

  A business action often spans services (take payment, reserve stock, create shipment). A single database transaction cannot cover them. A saga is a sequence of local transactions, each with a compensating action:

  1. Create the order (pending).
  2. Reserve stock. Compensation: release stock.
  3. Charge the payment. Compensation: refund.
  4. Confirm the order. If any step fails, run the compensations for earlier steps in reverse.

  Two ways to run a saga:

  • Choreography: services react to each other's events. Simple and decoupled, but the overall flow is spread out and hard to see.
  • Orchestration: a coordinator (an orchestrator service or workflow engine) tells each service what to do next and handles failures. The flow is explicit and easier to reason about.

  The outbox pattern

  A service must update its database and publish an event, and one may succeed without the other. Write the event to an outbox table in the same transaction, and let a separate process publish it. Consumers must be idempotent, because delivery is at least once.

  Finding and reaching services

  • Service discovery (via the platform, DNS or a registry) tells callers where healthy instances are, since addresses change constantly.
  • An API gateway is the front door for external clients: routing, authentication, rate limiting. A backend for frontend shapes responses for a specific client.
  • A service mesh adds a sidecar proxy next to each service to provide mutual TLS, retries, timeouts and metrics without changing application code.

  Resilience between services

  Reuse the reliability tools: timeouts, retries with backoff and jitter, circuit breakers and bulkheads. Prefer asynchronous communication where a user does not need an immediate answer.

  Operating many services

  • Distributed tracing with a correlation id shows a request's path across services.
  • Contract testing and careful API versioning let teams change without breaking each other.
  • Beware the distributed monolith: services so entangled they must be deployed together, which gives you all the costs and none of the benefits. Split along business boundaries, not technical layers.
  • Start with a well-structured monolith if the team is small. Split when a real need appears.

  More on this site

  Related reading

  • Pattern: Transactional outboxChris Richardson · microservices.ioDocumentation
  • Pattern: SagaChris Richardson · microservices.ioDocumentation
  • Awesome MicroservicesMehdi Fornos · mfornosGitHub
  • Microservices PatternsChris Richardson · ManningBooks

  Show 6 moreShow fewer

  • Tech Dummies: System Design Case StudiesNarendra L · Tech DummiesYouTube
  • Payment Transaction Management in Microservices at MercariMercari engineers · MercariCase Studies
  • Circuit BreakerMartin FowlerBlogs
  • Avoiding insurmountable queue backlogsAmazon Builders' Library · Amazon Web ServicesBlogs
  • Microservice Architecture at MediumMedium engineers · MediumCase Studies
  • Back-end at FlickrFlickr engineers · FlickrCase Studies

  Open this lesson with its quiz and exercise

• Core Concepts: How Things Actually Work · 8 minWhere online payments are headingInstant payments, open banking, wallets and tokens, stronger authentication and new money types: what is changing and what it means for system design.Read hereClose

  Why this matters to a system designer

  Payment systems are changing from card-centric, batch and slow, to real-time, account-based and programmable. Each trend changes your architecture, so it helps to know the direction. The items below describe trends in the industry, not predictions about any one product.

  Trend 1: instant, always-on payments

  Many countries now run instant payment schemes: UPI (India), Pix (Brazil), Faster Payments (UK), SEPA Instant (Europe), FedNow (United States). Funds move between bank accounts in seconds, at any hour, including weekends.

  Design consequences:

  • Settlement is final and fast, so fraud checks must happen in real time, before sending.
  • Your system must run 24/7 with no overnight batch window.
  • Confirmation arrives asynchronously, so you need pending states, webhooks and reconciliation (see the payment design lesson).

  Trend 2: open banking and account-to-account payments

  Regulations such as the EU's PSD2 and similar rules elsewhere require banks to offer APIs so that, with the customer's consent, a regulated third party can see account data or initiate a payment directly from the bank account. This enables card-free checkout at lower cost.

  Design consequences: you integrate with consent flows and bank-specific quirks, handle redirects and expiring consents, and treat the bank as an unreliable dependency.

  Trend 3: wallets, network tokens and tokenisation

  Apple Pay, Google Pay and similar wallets store a token instead of the card number. Card networks also issue network tokens for saved cards that can be refreshed when a card is reissued. A stolen token is of little use elsewhere.

  Design consequences: store tokens, not card numbers; handle lifecycle updates (card replaced, token suspended); and rely on the provider's PCI-compliant fields.

  Trend 4: stronger authentication and less fraud

  Strong customer authentication (for example 3-D Secure 2) adds a risk-based extra check only when needed. Passkeys and device biometrics replace passwords and one-time codes. Machine learning scores transactions in milliseconds using device, behaviour and network signals.

  Design consequences: payment flows need extra steps and failure handling, and a risk engine on the critical path with tight latency budgets.

  Trend 5: embedded finance and flexible credit

  Payments are built into other products (marketplaces, software platforms, ride apps), and pay later options split a purchase into instalments. Platforms handle onboarding of sellers, split payouts and compliance.

  Design consequences: multi-party ledgers, payouts, KYC and risk for sub-merchants, and refunds that touch several accounts.

  Trend 6: richer messages and faster cross-border

  The ISO 20022 standard brings structured, data-rich payment messages to banks worldwide, improving reconciliation and screening. Cross-border networks are working to shorten the chain of intermediaries and improve tracking (see the cross-border lesson).

  Trend 7: new forms of money

  Stablecoins and central bank digital currencies (CBDCs) are being explored, piloted or regulated in different places. Treat them as additional payment rails with their own settlement, custody and regulatory models, not a replacement for the fundamentals.

  A ledger-based wallet on a blockchain keeps balances as shared, append-only records that many parties verify, which can shorten settlement. The trade-offs are throughput, finality rules, regulation and the loss of simple reversal, so treat it as another rail with its own failure modes, not a replacement for the ledger discipline above.

  The payment rails side by side

  100%

  Figure SDA-CORE-21-1 · Several rails into one payment service · © forcsguy.com

  What stays the same

  Whatever the rail, the engineering core is stable: idempotency, a double-entry ledger, a clear state machine, asynchronous confirmation, reconciliation, strong security, and observability. Build those well, and adding a new payment method becomes an integration, not a rewrite.

  Real-world reading

  Related reading

  • Stripe's payments APIs: the first 10 yearsStripeCase Studies
  • Stripe Engineering BlogStripe engineers · StripeBlogs
  • Square BlogBlock (Square) engineers · Block (Square)Blogs
  • Klarna BlogKlarna engineers · KlarnaBlogs

  Show 6 moreShow fewer

  • Coinbase BlogCoinbase engineers · CoinbaseBlogs
  • Nubank BlogNubank engineers · NubankBlogs
  • Kraken BlogKraken engineers · KrakenBlogs
  • Capital One BlogCapital One engineers · Capital OneBlogs
  • Automated Fake Account Detection at LinkedInLinkedIn engineers · LinkedInCase Studies
  • Squarespace Engineering BlogSquarespace engineers · SquarespaceBlogs

  Open this lesson with its quiz and exercise

• Core Concepts: How Things Actually Work · 9 minHow online payments workThe parties involved, why idempotency is essential, and how a ledger and reconciliation keep money correct.Read hereClose

  The players

  A card payment involves several parties:

  • Customer and their issuing bank (which holds the account).
  • Merchant and their acquiring bank.
  • A card network (such as Visa or Mastercard) connecting banks.
  • A payment service provider (PSP) that handles the complexity for merchants.

  Most businesses do not touch card numbers: they use a PSP's hosted form or tokens, which shrinks security and compliance scope (PCI DSS).

  Image: Syced · CC0 · Source · Wikimedia Commons: LaCuCa card reader - payment terminal at supermarket register

  A typical flow

  100%

  Figure SDA-CORE-15-1 · A typical flow · © forcsguy.com

  Steps are usually authorise (reserve funds) then capture (take the money), then settlement between banks days later.

  Idempotency

  Networks time out, users double-click, services retry. Without protection, a retry can charge twice. Solve it with an idempotency key: a unique value per intended payment, sent with the request. The server stores the result under that key, and any repeat returns the stored result instead of acting again.

  Model the money as a ledger

  Never just update a balance field. Record every movement as entries in an append-only ledger using double-entry bookkeeping: each transaction has a debit and an equal credit.

  • Balances are computed from entries, and always sum to zero across the system, so errors stand out.
  • You get a full audit trail.
  • Use integer minor units (cents), never floating point, for amounts, and store the currency.

  State machines and asynchronous reality

  A payment moves through states: created, pending, authorised, captured, failed, refunded. Make transitions explicit and allow only valid ones. Many results arrive asynchronously by webhook, so your system must accept webhooks idempotently, verify their signatures, and tolerate them arriving late or out of order.

  Reconciliation

  Your records, the provider's and the bank's will occasionally disagree (a timeout hid a success, a webhook was lost). A scheduled reconciliation job compares them, flags mismatches and drives them to a resolution. Treat unexplained differences as incidents.

  Reliability and security

  • Keep the write path simple and durable; do slow work (emails, analytics) asynchronously.
  • Use strong consistency for balances and payment state.
  • Protect against fraud with limits, velocity checks and risk scoring.
  • Never log card data or secrets; use tokenisation and encryption.

  Real-world reading

  Related reading

  • Designing robust and predictable APIs with idempotencyBrandur Leach · StripeBlogs
  • 10 Tips for Building Resilient Payment SystemsShopify Engineering · ShopifyCase Studies
  • Stripe Engineering BlogStripe engineers · StripeBlogs
  • Making retries safe with idempotent APIsAmazon Builders' Library · Amazon Web ServicesBlogs

  Show 6 moreShow fewer

  • Stripe's payments APIs: the first 10 yearsStripeCase Studies
  • Klarna BlogKlarna engineers · KlarnaBlogs
  • Square BlogBlock (Square) engineers · Block (Square)Blogs
  • Coinbase BlogCoinbase engineers · CoinbaseBlogs
  • Asana Engineering BlogAsana engineers · AsanaBlogs
  • Auth0 Engineering BlogAuth0 engineers · Auth0Blogs

  Open this lesson with its quiz and exercise