Kafka Delivery Semantics

When a producer sends a message, how sure can it be that the message was stored? When a consumer reads a message, how do you prevent duplicates or losses? These are Kafka's delivery semantics -- and getting them right is one of the hardest problems in distributed messaging.

Think first

When a producer sends a message, there is a window between sending and confirmation. What could go wrong during that window? Think about what happens if the leader crashes after writing the message but before sending the acknowledgment -- did the message get stored or not?

Producer delivery guarantees

The producer controls durability via the acks (acknowledgment) setting:

Setting	What happens	Durability	Performance
acks=0 (fire-and-forget)	Producer sends and doesn't wait for any confirmation	Lowest -- message could be lost	Fastest -- network speed
acks=1 (leader only)	Producer waits for the leader to confirm	Medium -- lost if leader crashes before replication	Fast
acks=all (leader + ISR)	Producer waits for the leader and all ISRs to confirm	Highest -- survives any single broker failure	Slowest

The trade-off

acks=0 is for fire-and-forget telemetry. acks=1 is the common default. acks=all is for data you absolutely cannot lose (financial transactions, order events). Always match the ack level to the business value of the data.

Think first

On the consumer side, there are two operations: processing a message and committing the offset. The order in which you do these two things determines whether you risk losing messages or processing them twice. Before reading on, think through what happens if you commit first vs. process first, and what happens if the consumer crashes between the two steps.

Consumer delivery guarantees

The consumer side is about the relationship between processing a message and committing the offset:

Guarantee	How it works	Risk
At-most-once	Commit offset before processing. If the consumer crashes mid-processing, the message is skipped.	Message loss
At-least-once	Commit offset after processing. If the consumer crashes after processing but before committing, the message is reprocessed.	Duplicate processing
Exactly-once	Processing and offset commit happen in a single transaction. If the consumer crashes, the transaction rolls back and the message is reprocessed cleanly.	None (but lower throughput)

Exactly-once: how it actually works

Exactly-once requires three mechanisms working together:

1. Idempotent producers: Each message gets a sequence number. The broker deduplicates based on (producer ID, sequence number).
2. Transactional writes: Producers can atomically write to multiple partitions and commit offsets in a single transaction.
3. Transactional consumers: Consumers configured to read only committed messages (isolation.level=read_committed).

Interview angle

"How does Kafka achieve exactly-once delivery?" is a common interview question. The answer: idempotent producers (sequence number dedup on the broker), transactions (atomic writes across partitions), and read_committed consumers. Note that exactly-once is end-to-end only if the consumer's side effects are also idempotent or part of the transaction.

Quiz

A payment processing consumer uses at-least-once delivery (commit after processing). The consumer successfully charges a customer's credit card but crashes before committing the offset. What happens when it restarts?

The message is skipped because the broker knows it was already processed.

The consumer re-reads the message and charges the customer again, resulting in a duplicate charge -- unless the payment logic is idempotent.

Kafka's exactly-once semantics automatically prevent the duplicate charge.

The message is moved to a dead-letter queue for manual review.