Reliable messaging: retries, dead letters, ordering and the outbox

The honest baseline

Most brokers give at-least-once delivery: nothing is lost, but a message can arrive more than once and, with competing consumers, out of order. A correct system is built on that, not on wishes.

Idempotent consumers

An operation is idempotent if doing it twice has the same effect as once. Ways to get there:

• Natural idempotency: "set status to PAID" is safe to repeat; "add 10 to balance" is not.
• Deduplication table: record the message id (or an idempotency key) in the same transaction as the work; skip if already seen.
• Conditional updates: update only if the version or state matches.

async function handle(msg: { id: string; orderId: string }) {
  await db.tx(async (t) => {
    const fresh = await t.insertIfAbsent("processed_messages", { id: msg.id });  // unique key on id
    if (!fresh) return;                                // duplicate: do nothing
    await t.markOrderShipped(msg.orderId);             // the real work, in the same transaction
  });
  // acknowledge only after the transaction commits
}

Acknowledge after the work commits; if the process dies in between, the broker redelivers and the dedup record makes it safe.

Retries and dead letter queues

100%

Figure SDA-MSV-02-1 · Retry and dead-letter flow · © forcsguy.com

• Separate transient failures (a dependency is down: retry later) from permanent ones (invalid payload: send to the DLQ straight away).
• Use exponential backoff with jitter, and cap retries.
• A poison message retried forever blocks a partition or hogs a worker; the DLQ prevents it.
• Alert on DLQ depth and age; build a safe replay tool.
• In Kafka, which has no per-message acknowledgement, implement retry and DLQ with extra topics ("retry-5s", "dlq").

Ordering

Total order across a system is expensive. Aim for ordering only where needed, usually per entity.

• Partition by key (all events for one order go to one partition and stay ordered).
• A queue with competing consumers can reorder; use FIFO queues with message groups, or one consumer per key.
• Make handlers tolerate late or out-of-order events with versions or timestamps ("ignore if older than what I have").
• Retries break order: a failed message retried later arrives after its successors. Decide whether to block the key or tolerate reordering.

The dual-write problem and the outbox

A service must often change its database and publish an event. Doing both separately is unsafe: if the database commit succeeds and publishing fails, the event is lost; if the order is reversed, an event may announce something that did not happen.

100%

Figure SDA-MSV-02-2 · Transactional outbox · © forcsguy.com

The inbox side mirrors it: consumers record processed message ids in their own database transaction.

Putting the guarantees together

• Outbox gives "event published if and only if the data changed" (at least once).
• Idempotent consumers make duplicates harmless.
• DLQ and monitoring make failures visible.
• Per-key ordering where order matters.

Together these give effectively exactly-once results without relying on a mythical exactly-once network.

Other practical points

• Message design: include an id, a type, a version and a timestamp; keep payloads small or reference large data; evolve schemas compatibly (add optional fields; use a schema registry).
• Backpressure: bounded queues and consumer concurrency limits (see the resilience lesson).
• Visibility timeouts and acknowledgement deadlines must exceed processing time, or the broker redelivers work still in progress.
• Monitoring: queue depth, age of the oldest message, consumer lag, DLQ size, processing time.

Real-world reading