Exactly-once Semantics Is Possible: Here's How Apache Kafka Does It

Ideas you will meet

Kafka

Distributed commit log for event streaming at scale.

Idempotency

Making retries safe so duplicates do no harm.

What we already know about this

Our own lessons on these ideas. Read them here before you start, so it makes sense.

• Advanced Distributed Systems · 13 minKafka internals: segments, replication and consumer groupsWhat is inside a topic: partitions as append-only logs split into segments, how followers keep copies, and how consumer groups share the work.Read hereClose

  A topic is a set of logs

  A Kafka topic is split into partitions. Each partition is an append-only log: new records go on the end and get an increasing offset. Order is guaranteed only within a partition; records with the same key go to the same partition, so per-key order holds. Consumers track how far they have read by offset. (Why this design is fast is covered in the "Why is Kafka fast" lesson.)

  On disk: segments

  A partition's log is stored as a series of segment files. The newest segment is the one being written; older segments are immutable. Each segment has companion index files that map offsets (and timestamps) to positions in the file, so a consumer can jump to an offset quickly.

  • Retention is applied by deleting whole old segments (by age or total size), which is cheap.
  • Compaction (for some topics) keeps only the latest record per key, turning the log into a changelog of current state.

  100%

  Figure SDA-ADS-03-1 · Partitions, segments and replicas · © forcsguy.com

  Replication and durability

  Each partition has a replication factor (commonly 3): one leader and followers. Followers fetch from the leader. The in-sync replicas (ISR) are the followers that are caught up.

  A producer chooses an acknowledgement level:

  Setting	Meaning	Trade-off
  acks=0	Do not wait	Fastest, may lose data
  acks=1	Leader has written it	Loses data if the leader fails before replicating
  acks=all	All in-sync replicas have it	Safest, slower

  With acks=all and a minimum in-sync replicas setting, a write succeeds only if enough copies exist, trading availability for durability. If the leader fails, a new leader is chosen from the in-sync replicas, so acknowledged writes survive. (Older versions used ZooKeeper for cluster metadata; newer ones use a built-in Raft-based controller; the idea of consensus on metadata is the same as in the consensus lesson.)

  Consumer groups and rebalancing

  Consumers sharing a group id split a topic's partitions: each partition is read by one member at a time, so parallelism is limited by the number of partitions. Different groups each read the full topic independently.

  100%

  Figure SDA-ADS-03-2 · Consumer group rebalance · © forcsguy.com

  Offsets are committed back to Kafka. Commit after processing for at-least-once delivery (duplicates possible, so make consumers idempotent). Rebalances happen when members join, leave, crash or are too slow (missing heartbeats or taking too long to poll). Frequent rebalances hurt throughput, so keep processing per batch short and group membership stable.

  Practical rules

  • Partition count is a design decision: it caps consumer parallelism, and increasing it later changes the key-to-partition mapping.
  • Key choice controls ordering and balance: pick a key with many distinct values.
  • Monitor consumer lag per partition, under-replicated partitions and request latency.
  • Idempotent producers and transactions exist for stronger guarantees (exactly-once within Kafka); end-to-end exactly-once still needs idempotent consumers or transactional sinks.
  • Stream processing (Kafka Streams, Flink) builds on the same consumer mechanics; see the big-data platforms course.

  Real-world reading

  Related reading

  • Exactly-once Semantics Is Possible: Here's How Apache Kafka Does ItNeha Narkhede · ConfluentBlogs
  • Kafka Source & DocsApache Software Foundation · apacheGitHub
  • Confluent Developer: Apache Kafka 101ConfluentCourses
  • Confluent: Apache Kafka TalksConfluentYouTube

  Show 6 moreShow fewer

  • Jordan Has No Life: Systems Design Concepts SeriesJordan · Jordan Has No LifeYouTube
  • Apache Kafka DocumentationApache Kafka · Apache Software FoundationDocumentation
  • Apache CassandraApache Software Foundation · GitHubGitHub
  • Cassandra architecture overviewApache Cassandra · Apache Software FoundationDocumentation
  • REA Group Engineering BlogREA Group engineers · REA GroupBlogs
  • Repo-Topix: Topic Extraction Framework at GithubGithub engineers · GithubCase Studies

  Open this lesson with its quiz and exercise

• Real-World Systems · 13 minDesign a payment systemTaking a customer's payment safely: the payment service, the ledger, retries and idempotency, asynchronous results and reconciliation.Read hereClose

  The problem

  Let customers pay for orders, and make sure the right amount moves, exactly once, and every cent can be accounted for.

  Requirements

  • Correctness above all: no double charges, no lost payments, no negative balances.
  • Reliability with partial failures: providers time out, webhooks arrive late.
  • Security: protect card data, prevent fraud.
  • Auditability: complete, immutable records.
  • Support several payment methods and providers.

  Components

  100%

  Figure SDA-RWS-07-1 · Payment system components · © forcsguy.com

  • Checkout collects payment details using the provider's hosted fields or SDK, so raw card numbers never touch your servers (you get a token).
  • Payment service owns the payment lifecycle and talks to providers.
  • Payments database stores each payment, its state and its idempotency key.
  • Ledger records money movements as balanced entries (see below).
  • Event bus tells other services (orders, emails) that a payment succeeded or failed.
  • Webhook handler receives the provider's asynchronous updates.
  • Reconciliation compares your records with the provider's.

  The payment flow

  1. The order service asks the payment service to create a payment with an idempotency key (usually the order id).
  2. The payment is saved as created, then submitted to the provider and moved to pending.
  3. The customer may complete an extra check (3-D Secure) at their bank.
  4. The provider reports the result, often asynchronously by webhook. The service verifies the webhook's signature, updates the state (authorised, captured, failed) and publishes an event.
  5. The order service reacts to the event (ship, or release stock).

  State machine

  Make states and allowed transitions explicit. A payment cannot jump from failed to captured or be captured twice. Reject invalid transitions and make every handler idempotent, because webhooks and retries can repeat.

  Idempotency

  Every request that creates or changes money carries an idempotency key. The service stores the key with the outcome; a repeat returns the stored outcome without acting. This protects against:

  • a customer double-clicking Pay,
  • a client retrying after a timeout,
  • the provider sending the same webhook twice.

  100%

  Figure SDA-RWS-07-2 · Retrying without paying twice · © forcsguy.com

  Image: P199 · Public domain · Source · Wikimedia Commons: Bank vault 1901

  Before: a retry can charge twice

  100%

  After: an idempotency key makes retries safe

  100%

  What changed

  • A repeated request returns the stored result
  • Network timeouts no longer risk a double charge
  • The key and result are stored with the payment

  The ledger

  Record every movement as double-entry lines: each transaction debits one account and credits another by the same amount (customer funds, merchant balance, fees, refunds). Entries are append-only and amounts are integers in minor units (cents) with a currency. Balances are derived from entries, and the books always sum to zero, which makes errors visible. Write the ledger entries in the same database transaction as the payment state change, so they cannot disagree.

  Example: a purchase in the ledger

  A customer pays 200 for an order. The ledger records one transaction with two balanced lines:

  Account	Debit	Credit
  Customer funds	200
  Merchant balance		200

  Debits equal credits, so the books stay balanced. A later refund is a new transaction that reverses these lines. The original entries are never edited.

  Handling uncertainty

  If a call to the provider times out you do not know what happened. Do not retry blindly and do not mark it failed:

  1. Keep the payment pending.
  2. Query the provider with the same key, or wait for the webhook.
  3. A reconciliation job later compares all pending and recent payments with the provider's report and settles any that are stuck.

  100%

  Figure SDA-RWS-07-3 · Authorise, clear, settle: three layers behind one payment · © forcsguy.com

  Reconciliation: the safety net

  Even with idempotency and a ledger, records in your system, the provider and the bank drift apart. Reconciliation compares them and flags the differences. Typical pains and answers:

  • Different formats. Dates, amounts and ids look different in each system, so normalise them into one shape first.
  • Volume. Nightly batch jobs are enough for most firms; a streaming job gives near-real-time matching.
  • Cut-off time. A payment stamped 23:59:55 by you may be stamped just after midnight by the provider. Mark such mismatches as temporary breaks and re-match them against the next day's file; only unresolved ones need a human.
  • Chained ids. Carry the order id through to the provider payment id and on to the payout id, so one purchase can be traced across all its events (purchase, fee, payout, and a currency conversion where one applies).

  Reconciliation is still needed when you believe you have exactly-once behaviour, because something outside your code can always go wrong.

  Security and fraud

  • Tokenise card data, encrypt secrets, and never log sensitive fields.
  • Verify webhook signatures and restrict who can call internal APIs.
  • Risk checks: velocity limits, device and address signals, rules and models; send risky payments for review.
  • Protect against replay by rejecting old timestamps and reused identifiers.

  Threats and the defence for each

  Threat	Typical defence
  Eavesdropping on requests and responses	HTTPS everywhere
  Tampering with data in transit	Encryption plus integrity checks and signatures
  Impersonating a server (man in the middle)	Certificates and certificate validation
  Losing payment data	Replication across zones and regular backups
  Floods of traffic (DDoS)	Rate limiting, firewalls and edge protection
  Stolen card numbers	Tokenisation: store and use tokens, not real card numbers
  Compliance obligations	PCI DSS controls for anyone handling branded cards
  Fraud	Address and card-code checks, device and behaviour analysis, risk scoring

  Scale

  Payments are write-heavy but not huge compared with social apps. Prioritise consistency and durability: use a strongly consistent database, a single writer for each payment, and a queue for the slow side effects. Shard by payment or merchant id if needed.

  Real-world reading

  Related reading

  • Designing robust and predictable APIs with idempotencyBrandur Leach · StripeBlogs
  • 10 Tips for Building Resilient Payment SystemsShopify Engineering · ShopifyCase Studies
  • Stripe Engineering BlogStripe engineers · StripeBlogs
  • Making retries safe with idempotent APIsAmazon Builders' Library · Amazon Web ServicesBlogs

  Show 6 moreShow fewer

  • Stripe's payments APIs: the first 10 yearsStripeCase Studies
  • Exactly-once Semantics Is Possible: Here's How Apache Kafka Does ItNeha Narkhede · ConfluentBlogs
  • Klarna BlogKlarna engineers · KlarnaBlogs
  • Square BlogBlock (Square) engineers · Block (Square)Blogs
  • Stripe Engineering BlogStripe engineers · StripeBlogs
  • Grofers Engineering BlogGrofers engineers · GrofersBlogs

  Open this lesson with its quiz and exercise

• Messaging and Microservice Patterns · 12 minMessage brokers comparedQueues versus logs, and how Kafka, RabbitMQ, ActiveMQ, SQS, Pub/Sub and Pulsar differ, so you can pick one for a job.Read hereClose

  Why brokers

  A message broker sits between producers and consumers so they do not need to be up at the same time, can run at different speeds and can scale separately. It gives you buffering (absorb bursts), decoupling, retries and fan-out. (The idea is introduced in the queues and async lesson of the foundations course.)

  Two families

  	Queue-style broker	Log-style broker
  Model	Messages are routed to queues, delivered to a consumer, acknowledged and removed	Messages are appended to a retained log; consumers track their own position
  Replay	Not normally	Yes: re-read from any retained offset
  Consumers	Competing workers share one queue	Several groups read the same log independently
  Ordering	Per queue, loosened by competing workers	Per partition
  Strengths	Rich routing, per-message acks, priorities, delays	Huge throughput, history, stream processing
  Examples	RabbitMQ, ActiveMQ, Amazon SQS	Kafka, Pulsar (also queue mode), Kinesis

  100%

  Figure SDA-MSV-01-1 · Queue versus log · © forcsguy.com

  The common brokers

  • Kafka: distributed log; partitions give parallelism and per-key order; very high throughput; consumer groups; the foundation of many event pipelines and stream processors. More to operate (or use a managed service). Internals in the Kafka lesson of the advanced distributed systems course.
  • RabbitMQ (and ActiveMQ): classic brokers speaking AMQP and other protocols. Exchanges route messages to queues by rules (direct, topic, fanout, headers). Per-message acknowledgements, priorities, delays and dead letter handling. Excellent for task queues and flexible routing at moderate scale.
  • Amazon SQS: fully managed queue. Standard queues scale enormously with at-least-once delivery and best-effort ordering; FIFO queues keep order and deduplicate, at lower throughput. Visibility timeout and dead letter queues are built in. Pairs with SNS for fan-out.
  • Google Pub/Sub (and similar cloud services): managed, global pub/sub with subscriptions, acknowledgement deadlines and optional ordering keys.
  • Apache Pulsar: log-style with separated compute (brokers) and storage (BookKeeper), multi-tenancy, geo-replication, and both streaming and queue-style subscriptions.

  Delivery guarantees

  • At most once: may lose, never duplicates (acknowledge before processing). Fine for metrics.
  • At least once: no loss, possible duplicates (acknowledge after processing); the practical default. Consumers must be idempotent.
  • Exactly once: really means at-least-once delivery plus deduplication or transactions, available within some systems (for example Kafka transactions), but end-to-end it requires idempotent consumers or transactional sinks.

  Choosing

  100%

  Figure SDA-MSV-01-2 · Which broker family · © forcsguy.com

  Need	Lean towards
  Simple background jobs, no ops	SQS or a cloud queue
  Flexible routing, priorities, delays	RabbitMQ
  High-volume event streaming, replay	Kafka or Pulsar
  Fan-out to many consumers in a cloud	Pub/Sub or SNS plus queues
  Stream processing and joins	Kafka with Flink or Kafka Streams

  Questions that decide it

  How many messages per second, and how large? Do you need ordering (and by what key)? Do you need replay? How many independent consumers? What latency? Who will operate it? Start with the simplest tool that meets the needs; moving to a streaming platform when you truly need replay and scale is easier than operating one you did not need.

  Real-world reading

  Related reading

  • Exactly-once Semantics Is Possible: Here's How Apache Kafka Does ItNeha Narkhede · ConfluentBlogs
  • RabbitMQ TutorialsRabbitMQ · Broadcom / VMwareDocumentation
  • Confluent Developer: Apache Kafka 101ConfluentCourses
  • Confluent: Apache Kafka TalksConfluentYouTube

  Show 6 moreShow fewer

  • Apache Kafka DocumentationApache Kafka · Apache Software FoundationDocumentation
  • Delaying Asynchronous Message Processing with RabbitMQ at IndeedIndeed engineers · IndeedCase Studies
  • Confluent BlogConfluent engineers · ConfluentBlogs
  • Kafka Source & DocsApache Software Foundation · apacheGitHub
  • Deduplication at SegmentSegment engineers · SegmentCase Studies
  • Kafka at LinkedInLinkedIn engineers · LinkedInCase Studies

  Open this lesson with its quiz and exercise