Idempotency in Distributed Systems: Why Is It So Easy to Process the Same Request Twice?
A timeout does not necessarily mean an operation failed. That uncertainty is often the real challenge behind designing safe retries.
Some of the most difficult problems in distributed systems do not come from complex algorithms. A brief network interruption can turn an apparently simple operation into a consistency problem.
Consider a client sending a request to a server. The server receives it, performs the operation, and commits the result to the database. Just before the response reaches the client, the connection fails.
From the client’s perspective, the operation has failed. From the server’s perspective, it has completed.
What happens when the client sends the same request again?
This is where idempotency becomes important.
A failed response is not a failed operation
An HTTP timeout does not prove that the remote operation was never executed.
It only tells the client that the expected response was not received within the configured time.
Consider a payment, the creation of a work order, or a command sent to a remote device.
If the operation succeeds but its response is lost, an automatic retry may execute the same business action a second time.
There is an important distinction here:
Sending a request more than once should not necessarily mean performing the business operation more than once.
A well-designed system may receive the same logical request repeatedly while preventing unintended duplicate effects.
What does idempotency actually guarantee?
In its simplest form, an idempotent operation produces the same resulting effect whether it is applied once or multiple times.
In practice, however, the scope of that effect matters.
Setting a record’s status to Active may be naturally idempotent. Writing the same state again does not change the intended outcome.
Subtracting 100 units from a balance or creating a new order is different. Repeating either operation may produce another business effect.
Even when an HTTP method such as PUT is considered idempotent, that does not automatically make every application-level side effect safe. Notifications, audit events, or external service calls may still need separate handling.
Instead of asking only whether an endpoint is idempotent, we should ask:
Which business effect are we protecting from duplication, within what scope, and for how long?
An idempotency key is not a complete solution
A common approach is to let clients generate a unique identifier for each logical operation.
POST /api/payments
Idempotency-Key: 8d9b7c41-...If the client retries the same payment after a network failure, it reuses the same key.
The server checks whether an operation with that key has already been started or completed. Reusing the key with a different request payload should also be treated as a conflict rather than silently accepted.
The implementation sounds straightforward: check the key, execute the operation, store the result, and return that result for subsequent requests.
The difficult part is what can happen between those steps.
What if two requests arrive concurrently?
Two requests carrying the same key may arrive almost simultaneously.
Both might observe that no record exists yet and attempt to start the operation.
A simple read-before-write check cannot prevent this race condition.
Ownership of the logical operation must be established atomically. Database uniqueness constraints, conditional inserts, or suitable transactional mechanisms can help.
Even then, successfully claiming an operation does not prove that its effect in an external system has completed safely.
The operation succeeds, but recording the result fails
Imagine a payment provider successfully processes a transaction.
Immediately afterward, the application crashes before persisting the payment result in its own database.
After recovery, the application has a transaction whose outcome appears uncertain.
Blindly retrying may risk another charge. Never retrying may leave the workflow unresolved.
This is why reconciling an uncertain operation is just as important as recording its idempotency key.
When a payment provider supports idempotency keys, reusing the original key may offer an important safeguard. Otherwise, querying the transaction status, reconciling records, or introducing compensation procedures may be necessary.
We can atomically update multiple records inside one local database transaction. We cannot make a separate payment provider and our own database participate in that same local transaction simply by adding an idempotency key.
An additional dimension in industrial systems
One distinction I consider particularly important in industrial communication systems is the difference between successful communication and successful operation. Receiving a response to a command matters, but so does verifying the device’s actual state afterward. When communication is interrupted or delayed, I don’t believe retry decisions should be based solely on application-level timeouts.
Consider a gateway sending a command to a remote device. If communication fails before a response is received, it may be unclear whether the command was executed.
The type of command matters. Setting a parameter to a specific value is not equivalent to triggering an action. They do not necessarily have the same retry semantics.
Furthermore, command acceptance, command execution, and verification of the resulting state are distinct stages.
For this reason, retry policies in industrial systems should not be determined by timeout values or retry counts alone. Command semantics, current device state, and available verification mechanisms must be considered together.
Sometimes the appropriate next step is not to resend the command, but to query the device’s current state first.
Message consumers face the same problem
This uncertainty is not limited to HTTP requests.
A message consumer may receive a message and update business data successfully, but lose its connection before acknowledging the message to the broker.
The broker may deliver the same message again.
This is expected behavior in systems using at-least-once delivery.
One approach is to persist the processed message identifier within the same transaction that updates the business data. When the same message arrives again, the identifier can prevent another update.
But a database transaction does not automatically undo an email already sent, an external HTTP request, or a physical command executed outside that transaction.
Preventing duplicate message processing and ensuring that every external side effect happens only once are different problems.
Does every operation need the same strategy?
No.
Synchronizing a record to its latest known state does not carry the same risks as transferring money.
When evaluating an idempotency strategy, I prefer to begin with a few questions:
- What business effects would a repeated operation produce?
- How can the client verify the original outcome?
- What happens when concurrent requests use the same key?
- How does the system recover from a crash midway through execution?
- What happens when the key’s retention period expires?
- If an external system or physical device is involved, how can the resulting effect be verified?
The answers help determine the appropriate mechanism.
Some operations can be designed to be naturally idempotent. Others can be protected with unique business identifiers and transactions. More complex workflows may require persistent operation states, reconciliation, and sometimes human intervention.
Conclusion: Retries are not merely an error-handling mechanism
Retries are often treated as a technical reliability feature.
But when the retried operation has business consequences, the retry policy becomes part of the business logic itself.
We need to define not only when to retry, but what we are willing to execute again.
Idempotency is therefore more than an HTTP header, a database record, or middleware configuration.
It is an architectural decision involving operation identity, duplication boundaries, failure behavior, and recovery.
The goal is not to guarantee that every request arrives exactly once. The goal is to ensure that repeated requests do not produce unintended duplicate business effects.
Hands-on experiment: Same message, different business outcome
I also explored this problem through a runnable experiment in the Distributed Reliability Lab.
In E02 — Duplicate Delivery, messages representing four work orders are processed under normal delivery and controlled crash-and-redelivery scenarios.
Two consumers receive the same logical messages. The naïve consumer applies each delivery directly, while the idempotent consumer records processed message identifiers within the same database transaction as the reservation and inventory updates.
At the end of the experiment, the idempotent consumer preserves the expected inventory balance. The naïve consumer creates two additional reservations for the same four work orders.
The experiment demonstrates why idempotency is not merely a technical duplicate check. It protects the correctness of business state.