The outbox pattern

The booking service now does two things for every booking: it saves the booking to Postgres and publishes BookingCreated to Kafka. Those are two different systems, and no transaction covers both. Somewhere in a million bookings, the process will die between the two writes. This lesson makes sure that when it does, the event still goes out.

The dual-write problem

Writing to two systems that don't share a transaction is called a dual write, and whichever order you choose, a crash in the middle leaves them disagreeing.

Save, then publish. The booking commits. The process crashes, or Kafka is briefly unreachable, before the publish. The booking exists, but the email, seat map and analytics services never hear about it. The fan has a booking with no ticket email, and nothing will ever retry, because nothing remembers that an event is owed.

Publish, then save. The event goes out, then the database insert fails on a constraint or the process dies. Now email sends a confirmation for a booking that doesn't exist, and analytics counts a sale that never happened.

Retrying doesn't save you: the crash can happen after the first write and before the retry loop records anything. Wrapping both in a distributed transaction is two-phase commit again, which Kafka and Postgres don't do together, and which you rejected two lessons ago for good reasons.

Write the event to your own database

The fix is to make it one write again. The transactional outbox pattern adds an outbox table to the booking service's own database. In the same local transaction that inserts the booking, you insert a row describing the event.

BEGIN;
INSERT INTO bookings (id, show_id, seats, amount, status)
  VALUES ('b_8812', 'show_17', '{A12,A13}', 1800, 'CONFIRMED');
INSERT INTO outbox (event_id, aggregate_id, type, payload, created_at)
  VALUES ('evt_7f3a', 'b_8812', 'BookingCreated', '{"showId":"show_17", ...}', now());
COMMIT;

Now the booking and its event are atomic, because they're in one database. Either both rows exist or neither does. A crash before COMMIT loses both, which is fine: the fan sees an error and tries again. A crash after COMMIT leaves the event safely on disk, waiting to be sent.

A separate process, the relay, reads new outbox rows and publishes them to Kafka. If the relay is down for ten minutes, events wait in the table and go out when it returns. Booking never talks to Kafka at all, so a Kafka outage no longer fails bookings either.

Two ways to run the relay

Polling. The relay runs a query every few hundred milliseconds: fetch unpublished rows in event_id or insertion order, publish them, mark them published. It's simple and works with any database. The costs are a steady trickle of queries, latency up to the polling interval, and care around concurrency: two relay instances polling the same rows will publish duplicates and scramble order, so either run one active relay with a lock, or split rows between relays by aggregate id.

Change data capture (CDC). The database already writes every committed change to its own log: the write-ahead log in Postgres, the binlog in MySQL. A CDC tool such as Debezium reads that log like a replica would, and turns each new outbox row into a Kafka message. There's no polling load and latency is milliseconds, and the order matches the commit order. The cost is another piece of infrastructure to run and monitor, with access to your database's replication stream.

Gatepass starts with polling, which is enough at a few hundred bookings a second, and moves to CDC when the polling query starts to show up in the database's load.

Duplicates, order and housekeeping

The outbox guarantees an event is never lost. It does not guarantee it is published once. The relay can publish a row and crash before marking it published; on restart it publishes it again. So the outbox gives you at-least-once delivery, and everything from the delivery-guarantees lesson applies: every event carries a stable event_id from the outbox row, and consumers deduplicate on it. Some teams mirror the pattern on the consumer side with an inbox table: record the incoming event_id in the same transaction as the work it triggers.

For order, publish each event with the aggregate id (the booking id) as its Kafka key, so all of a booking's events land in one partition in the order the relay read them.

Finally, the outbox is a table that only grows. Delete or archive rows once they're published and a safe margin has passed, say a day, and put a number on a dashboard: the age of the oldest unpublished row. If it climbs past a minute, the relay is stuck, and every downstream service is quietly falling behind.

Check yourself

With the outbox and a polling relay, what can still go wrong, and who handles it?

AA booking can be saved without its event. The relay must retry harder.BAn event can be published twice. Consumers deduplicate by event id.CAn event can be published for a booking that was never saved. Consumers must check the booking exists.DNothing. The outbox gives exactly-once delivery.

Key takeaways

• Writing to a database and a broker separately is a dual write. A crash between them loses events or invents them.
• The outbox pattern inserts the event into an outbox table in the same local transaction as the data change.
• A relay ships outbox rows to the broker, by polling or by change data capture from the database log.
• The result is at-least-once: give each event a stable id, key by aggregate id for order, and make consumers idempotent.
• Clean up published rows and alert on the age of the oldest unpublished one.