Idempotency

A worker sends Priya's ticket by SMS, the provider accepts it, and then the worker's network drops before the reply arrives. The worker can't tell whether the SMS went out, so the job times out and runs again. Priya gets two texts. Annoying, but harmless. Now replace “send SMS” with “charge ₹2,400” and it's a support ticket, a refund and a lost customer.

Why “exactly once” isn't on the menu

Every call over a network has three possible outcomes: it worked, it failed, or you don't know. A timeout is the third one. The request might never have arrived, might have been processed with the reply lost, or might still be running.

When you don't know, you have two choices. Don't retry, and some work is silently lost (at-most-once). Retry, and some work happens twice (at-least-once). Queues, as you saw, choose at-least-once: a worker that crashes before acknowledging gets its job redelivered. No queue, protocol or library can turn an unknown outcome into a known one.

So you don't get exactly-once delivery. What you can build is exactly-once effect: the work may be attempted several times, but the result is the same as doing it once. An operation with that property is idempotent.

Some operations are idempotent already

Look at what the operation does, not how often it's called:

• SET seat C14 status = 'sold' is idempotent. Running it twice leaves the seat sold.
• DELETE booking 9812 is idempotent. The second delete finds nothing to delete.
• PUT /users/81/phone with the full new value is idempotent.
• UPDATE events SET sold = sold + 1 is not. Run it twice and you've counted two tickets.
• POST /bookings that inserts a new row is not. Run it twice and the fan has two bookings.
• “Send an SMS” and “charge a card” are not. They have effects outside your database.

The rule of thumb: operations that set a state are naturally idempotent; operations that add, increment or trigger something are not. Where you can, design the operation as a set. Instead of “add one ticket to the count”, compute the count from the tickets table, or record the booking id that was counted.

Idempotency keys

For operations that can't be made naturally idempotent, give each logical operation a unique id, and remember which ids you've already done. This is an idempotency key.

For Gatepass's booking API, the app generates a random key when the fan taps “Pay” and sends it with the request: Idempotency-Key: 5f1c…. If the request times out, the app retries with the same key. The server does this:

1. Insert the key into an idempotency_keys table with a unique constraint, status in_progress.
2. If the insert fails because the key exists: if the stored status is done, return the saved response. If it's still in_progress, return 409 Conflict and let the client retry shortly.
3. Otherwise, do the work (reserve the seat, charge the card) and save the response with the key, status done, in the same transaction as the booking.

Two details make or break this:

• The key identifies the intent, not the request. Generate it once per tap of “Pay”, before the first try, and reuse it on every retry. A new key per retry protects nothing.
• Pass the key downstream. Gatepass sends its own key (for example booking-9812) to the payment provider, which also supports idempotency keys. If Gatepass retries the charge, the provider replays its earlier answer instead of charging again.

Workers: dedupe on what the job means

Background jobs work the same way. A send-ticket job carries bookingId: 9812. Before sending, the worker records what it's about to do, keyed by meaning:

INSERT INTO deliveries (booking_id, channel) VALUES (9812, 'sms')
ON CONFLICT DO NOTHING
RETURNING id;

If the insert returns nothing, this booking's SMS was already handled, so the worker acknowledges the job and stops. If it returns a row, the worker sends the SMS and marks the row sent.

There's still a gap. The worker can crash after sending and before marking. That's the unknown outcome again, now at the provider. You shrink the gap by passing the delivery id to the provider as its idempotency key, if it accepts one. If it doesn't, you choose which failure is acceptable: an occasional duplicate text (retry) or an occasional missing text (don't). For an SMS, a duplicate is the lesser evil. For a payment, you only use providers that support keys.

Race conditions: check-then-act isn't enough

A common first attempt is “check whether it's done, and if not, do it”:

if not already_sent(9812): send(9812); mark_sent(9812)

Two workers holding duplicate copies of the job can both run the check before either marks it, and both send. The check and the claim must be one atomic step: a unique constraint, an INSERT … ON CONFLICT, a conditional update (UPDATE … SET status='sending' WHERE status='pending', then checking that one row changed), or Redis SET key value NX. Let the database decide who wins.

Check yourself

Which of these Gatepass operations is naturally idempotent?

AIncrement the event's tickets_sold counter by 1BInsert a new row into bookingsCSet seat C14's status to 'sold' for booking 9812DSend a confirmation email

Your turn

An organiser cancels a show and Gatepass must refund 3,000 tickets via a background job per ticket. The job may run twice. How do you make sure nobody is refunded twice?

01

02

03

Stuck? Think about what uniquely identifies a refund, where the atomic claim happens, and what you pass to the payment provider.

Compare with an expertWrite at least 2 first.

Key takeaways

• A timeout means “unknown”, so retries are necessary and duplicates are inevitable. Aim for exactly-once effect.
• Setting a state is naturally idempotent; adding, incrementing and triggering are not.
• Idempotency keys identify one user intent. Store them with the result, replay the result on repeats, and pass keys to downstream providers.
• Claim work with an atomic step (unique constraint, conditional update), never check-then-act.
• Where a gap remains, decide consciously whether a duplicate or a miss is the smaller harm.