A booking event triggers a confirmation email. The consumer sends it, then crashes before recording its progress. When processing resumes, the same event may be delivered again. The broker has preserved the work; the application now has to decide whether repeating that work is safe.
This is an illustrative workflow, not an incident report. It captures a useful design question for any service that consumes events: what happens if this handler runs twice?
Separate delivery from the business effect
In an at-least-once processing design, committing an offset after successful processing avoids acknowledging unfinished work. It also leaves a failure window between completing a side effect and committing that offset. Moving the commit earlier changes the risk: a crash can leave acknowledged work unfinished.
Producer idempotence addresses duplicate writes caused by producer retries. It does not automatically deduplicate an external email, payment, or database operation performed by your consumer.
Give the operation a stable identity
Use a durable event ID and define uniqueness at the level of the business operation. An event ID identifies a delivery; a booking ID plus notification type may better identify a confirmation. Choose deliberately, because legitimate updates must still be able to happen.
BEGIN;
INSERT INTO processed_events (consumer, event_id)
VALUES ('booking-projection', :event_id)
ON CONFLICT DO NOTHING;
-- If inserted: update the booking projection.
-- Keep both changes in this database transaction.
COMMIT;
-- Then acknowledge processing to Kafka.The uniqueness constraint and projection update must share a transaction. Otherwise a crash could record an event as processed without applying its change. On replay, a conflicting insert tells the handler to skip the already completed database operation.
External effects need another boundary
A database transaction cannot atomically commit a request to an unrelated email API. Write a notification intent to an outbox in the same transaction as the business change. A separate worker delivers that intent and records its result. If the destination supports idempotency keys, reuse the stable operation ID on every retry.
An outbox alone does not guarantee that an email is sent only once: the delivery worker can also crash after sending. If the provider cannot deduplicate, document that remaining duplicate window rather than promising exactly-once behavior.
Make failure visible
- Retry transient failures with a bounded policy; keep permanently invalid events inspectable.
- Preserve the event ID and failure reason when routing a failed event for later investigation.
- Monitor lag, retry volume, and duplicate suppression. A healthy process can still be falling behind.
The important guarantee is the one the business effect actually provides.
Test the crash boundaries: before the write, after the transaction, and before acknowledgement. A replay test is more convincing than a handler that only passes the happy path.