Idempotency and Retries

Every network call can be retried; make sure the second one is harmless.

The idea

A network call that times out has an unknown outcome. It may have failed before arriving, or succeeded with the response lost on the way back. The client cannot tell the difference, so it retries — and if the operation was "charge this card", the customer is charged twice.

This is not an edge case. It is the normal behaviour of every distributed system, and it is why at-least-once delivery plus idempotent handlers is the standard design rather than chasing exactly-once semantics that do not exist.

Idempotent means performing the operation twice has the same effect as once. Some operations already are: setting a value, deleting a record, marking an order shipped. Others are not: incrementing a counter, appending to a list, charging a card.

For those, the standard mechanism is an idempotency key. The client generates a unique id for the intent and sends it with the request. The server records the key with the result of the first execution; a repeat with the same key returns the stored result without doing the work again. Two details decide whether it works: the key must be generated by the client before the first attempt — a server-generated one is different on every retry, which defeats it — and storing the key must be atomic with performing the operation, or a crash in between produces exactly the double-charge you were preventing.

Retries need discipline of their own. Exponential backoff spreads attempts out; jitter is what stops every client retrying at the same instant and turning a brief blip into a thundering herd. And retries need a stopping condition: a circuit breaker that stops calling a dependency which is clearly down, so you stop adding load to something already struggling.

One rule worth stating plainly: only retry idempotent operations automatically. Retrying a non-idempotent call is how a single timeout becomes a duplicate charge.

Walkthrough

No walkthrough for this topic yet — the guided exercises below run the same way.

When to reach for it

Payments, orders, and anything that moves money or allocates a limited resource.
Any consumer of an at-least-once queue, which will redeliver eventually.
Public APIs where clients you do not control will retry however they like.
Webhook receivers — providers retry aggressively and duplicates are guaranteed.

Rather than the obvious alternative

Exactly-once delivery

It does not exist across a network in the general case. At-least-once with idempotent handlers is the achievable version of the same guarantee, and simpler than pretending otherwise.

Distributed transactions

Two-phase commit gives atomicity across services by holding locks across the network, which harms availability. A saga of idempotent steps with compensating actions is the usual trade.

Never retrying

Avoids duplicates by turning every transient blip into a user-visible failure. Retries are how a system survives an unreliable network; idempotency is what makes them safe.

Key terms

Idempotency: Repeating an operation has the same effect as performing it once.
Metrics, logs, and traces: Aggregates, events, and per-request causal chains.
SLIs, SLOs, and error budgets: Measured indicators, targets over them, and the failure allowance that follows.
Tail latency: The slow end of the distribution — p95, p99 — rather than the mean.

How to spot it

The operation charges, books, or allocates something finite.
The design includes a queue, which will redeliver.
The interviewer asks what happens if the request times out.
You are receiving webhooks from a third party.

Where it goes wrong

Server-generated idempotency keys

A key created on the server differs on every retry, so nothing is deduplicated. The client must generate it before the first attempt.

Storing the key separately from the effect

If the operation succeeds and recording the key fails, the retry runs it again. They must commit together.

Retrying without jitter

Synchronised retries turn a brief outage into a self-inflicted denial of service the moment the service recovers.

Retrying non-idempotent operations automatically

This is the double-charge, and it is usually a client-library default rather than a deliberate decision.