Engineering
Designing an Idempotent Payments API
Why every write endpoint needs an idempotency key, and how to design retries so a customer is never charged twice.
Marco Silva (sample author)
5 min read
Networks fail in the least convenient places. A request times out after the charge was created but before the response reached the client. A mobile app loses signal mid-checkout and the user taps "Pay" again. A queue worker crashes and redelivers the same job. In each case, the client cannot know whether the first attempt succeeded, so it does the only reasonable thing: it retries. Without idempotency, that retry can become a second charge. This note explains how we approach idempotency across write endpoints in the PAXORA API layer, and the trade-offs behind each decision.
Why retries are the default, not the exception
At low volume, duplicate requests look like edge cases. At scale they are a constant background rate. Load balancers retry on connection resets, SDKs retry on 5xx responses, and customers retry when a spinner runs longer than they expect. Treating every retry as a potential double-write is the only safe assumption.
The goal is simple to state: a client should be able to send the same logical request any number of times and get exactly one side effect, plus the same response every time. Achieving it requires agreement between client and server on what "the same request" means.
The idempotency key contract
Every write endpoint accepts an Idempotency-Key header. The client generates the key — a UUID is fine — once per logical operation, not once per HTTP attempt. The server stores the key alongside a fingerprint of the request and the final response.
If the same key arrives again with the same parameters, the server returns the stored response without executing anything. If it arrives with different parameters, the server rejects it with a conflict error, because reusing a key for a different operation is almost always a client bug worth surfacing loudly.
What the server actually stores
A naive implementation checks for the key, executes the request, then saves the result. That leaves a race: two concurrent retries can both pass the check before either saves. The fix is to make the key reservation itself atomic.
Reserve first. Insert the key with a status of
in_progressusing a unique constraint. If the insert fails, another attempt owns the key.Fingerprint the request. Hash the method, path and normalised body so a reused key with a different payload is detected.
Record the outcome, not just success. Store the full response, including deterministic errors such as a declined card. A retry should see the same decline, not a fresh attempt.
Expire deliberately. Keep keys long enough to cover realistic retry windows — we use 24 hours in the sample configuration — then let them age out.
When a retry finds a key still in_progress, the server returns a retryable status rather than waiting indefinitely. The client backs off and tries again; by then the original attempt has usually finished and its stored response is returned.
An idempotency key is a promise about intent. The client says what it meant to do once; the server guarantees it happens once.
Idempotency across the stack
The API layer is only the first boundary. A payment request fans out to routing, processing and settlement, and each hop can fail independently. If the API is idempotent but the call to a downstream processor is not, the guarantee leaks.
Propagate, don't regenerate
We derive downstream keys from the original one, so a retried request produces the same identifiers all the way through. When the processing layer retries a call to an acquirer, it reuses the derived key, and the acquirer's own deduplication does the rest.
Ledger entries are the final check
Every movement of money is written to the ledger with a reference to the originating key. A unique constraint on that reference means that even if every other safeguard failed, the ledger would refuse a duplicate posting. It is the last line of defence, and the one finance teams care about most.
Errors that should and should not be stored
Not every failure belongs in the idempotency record. The rule we follow is whether the error would be the same on retry.
Validation errors and declines are deterministic, so they are stored and replayed.
Timeouts to downstream systems, rate limits and internal errors are transient, so the key is released and the next attempt runs normally.
Authentication failures are never stored, because they say nothing about the operation itself.
Getting this split wrong in either direction is costly. Store a transient error and a customer is locked out of a payment that would have succeeded. Release a deterministic one and a retry might succeed on a different route after the customer was already told it failed.
Designing clients that retry well
Server-side guarantees only work if clients use them consistently. Generate the key before the first attempt and persist it with the order, so a process restart does not lose it. Retry with exponential backoff and jitter. Treat a conflict response as a bug, not a reason to generate a new key. And log the key with every attempt — when support asks what happened to a payment, it is the single most useful identifier you can give them.
Idempotency rarely shows up on a feature list, but it is the property that lets everything else be retried safely. Build it once, at every layer, and most double-charge incidents simply stop happening.
Sample engineering note for a website template — fictional API, illustrative values.