Field Notes · Payments
The Retry That Charged You Twice
The network is honest about very little, but it is honest about one thing: it will lie to you about whether your request arrived. A client fires a POST /charge, the money moves, and then the response evaporates on the way back. The client, seeing nothing, does the only reasonable thing available to it: it tries again.
Now you have charged someone twice for one coffee, and no amount of apologizing will make that feel like good engineering.
The fix is boring, which is why it works
The client sends an idempotency key, a unique string it generates once per logical operation, on the first attempt and every retry after it:
POST /charge
Idempotency-Key: 6f1e2c9a-...
On the server, the rule is simple:
- If you have never seen this key, do the work and store the result against it.
- If you have seen it, skip the work and return the stored result.
That is the whole idea. The retry becomes a lookup instead of a second charge.
The part people get wrong
The trap is storing the key after the work instead of around it. Two retries can arrive close enough that both check the store, both find nothing, and both proceed. You need the key reserved atomically, with a unique constraint in the database or a SET NX in Redis, so that the second request loses the race and waits for the first one's result.
Reliability isn't a feature you add. It's a bill you pay in small, careful decisions, and this is one of them.
Keep the keys around long enough to outlive a client's retry window, then let them expire. You are not building an archive; you are building a short memory for a forgetful network.
Also in the notebook
Building a Tunnel I Didn't Need
I use ngrok and I like it. I built my own tunnel anyway, to find out what it was doing underneath, and most of what I learned came from the parts that broke.
Do You Use Kafka, or Just Pay for It?
Most teams reach for a distributed log and then use it as a work queue. Here are the questions that tell you which one you actually have, before the invoice does.