Idempotency is the property that performing an operation multiple times produces the same result as performing it once. In distributed systems, where network failures and retries are common, idempotency prevents duplicate effects that could corrupt data or cause incorrect behavior.
Consider a payment API that charges a customer. If the network fails after the charge succeeds but before the response reaches the client, the client might retry. Without idempotency, the customer gets charged twice. With idempotency, the retry is recognized as a duplicate and returns the original result.
Why Idempotency Matters
Distributed systems face unique challenges that make idempotency essential. Networks are unreliable; requests and responses can be lost. Processes crash and restart. Load balancers retry failed requests. Message queues deliver messages multiple times.
The reality is that "exactly once" delivery is nearly impossible in distributed systems. Instead, we design for "at least once" delivery and make operations idempotent so that multiple deliveries have the same effect as one.
Without idempotency, you face difficult choices. Aggressive retries risk duplicate operations. Conservative retries risk lost operations. Neither is acceptable for critical operations like payments or inventory changes.
Idempotency Keys
Idempotency keys uniquely identify operations. Clients generate a key for each operation and include it with requests. Servers track completed keys and return cached results for duplicates.
class PaymentController extends Controller
{
public function charge(Request $request): JsonResponse
{
$idempotencyKey = $request->header('Idempotency-Key');
if (!$idempotencyKey) {
return response()->json(['error' => 'Idempotency key required'], 400);
}
// Check for existing result
$existing = IdempotencyRecord::where('key', $idempotencyKey)->first();
if ($existing) {
return response()->json(
json_decode($existing->response_body, true),
$existing->response_status
);
}
// Process the request
try {
$result = $this->paymentService->charge(
$request->input('customer_id'),
$request->input('amount')
);
$response = response()->json($result, 200);
// Store result for future duplicates
IdempotencyRecord::create([
'key' => $idempotencyKey,
'response_body' => json_encode($result),
'response_status' => 200,
'expires_at' => now()->addHours(24),
]);
return $response;
} catch (Exception $e) {
// Don't store errors - allow retry
throw $e;
}
}
}
Key generation should ensure uniqueness. UUIDs are common. Combining user ID with timestamp and random component also works. The key should be deterministic for the same logical operation.
// Client-side idempotency key generation
const idempotencyKey = `${userId}-${orderId}-${Date.now()}-${crypto.randomUUID()}`;
await fetch('/api/payments', {
method: 'POST',
headers: {
'Content-Type': 'application/json',
'Idempotency-Key': idempotencyKey,
},
body: JSON.stringify({ orderId, amount }),
});
Request Fingerprinting
Request fingerprinting derives idempotency keys from request content. Instead of clients providing keys, the server computes a hash of relevant request fields.
class IdempotencyMiddleware
{
public function handle(Request $request, Closure $next): Response
{
// Only apply to POST/PUT requests
if (!in_array($request->method(), ['POST', 'PUT', 'PATCH'])) {
return $next($request);
}
$fingerprint = $this->computeFingerprint($request);
$existing = IdempotencyRecord::where('fingerprint', $fingerprint)
->where('expires_at', '>', now())
->first();
if ($existing && $existing->status === 'completed') {
return response(
$existing->response_body,
$existing->response_status
);
}
// Mark as processing to handle concurrent duplicates
$record = IdempotencyRecord::firstOrCreate(
['fingerprint' => $fingerprint],
['status' => 'processing', 'expires_at' => now()->addHours(24)]
);
if ($record->status === 'processing' && !$record->wasRecentlyCreated) {
// Another request is processing this - wait or reject
return response()->json(['error' => 'Request in progress'], 409);
}
$response = $next($request);
$record->update([
'status' => 'completed',
'response_body' => $response->getContent(),
'response_status' => $response->getStatusCode(),
]);
return $response;
}
private function computeFingerprint(Request $request): string
{
$data = [
'method' => $request->method(),
'path' => $request->path(),
'user' => $request->user()?->id,
'body' => $request->all(),
];
return hash('sha256', json_encode($data));
}
}
Fingerprinting removes client responsibility but requires careful field selection. Include fields that define the operation's identity, exclude timestamps and other varying fields.
Database-Level Idempotency
Some operations can be made idempotent at the database level through constraints and upserts.
Unique constraints prevent duplicate records. If you try to insert a duplicate, the database rejects it.
// Unique constraint on order_id ensures one payment per order
Schema::create('payments', function (Blueprint $table) {
$table->id();
$table->foreignId('order_id')->unique();
$table->decimal('amount', 10, 2);
$table->timestamps();
});
// Insert or ignore pattern
public function recordPayment(int $orderId, Money $amount): Payment
{
return Payment::firstOrCreate(
['order_id' => $orderId],
['amount' => $amount->cents()]
);
}
Upserts update if exists, insert if not. Combined with version checks, they enable idempotent updates.
public function updateInventory(string $sku, int $quantity, string $operationId): void
{
// Idempotent inventory update using operation ID
DB::statement("
INSERT INTO inventory_operations (operation_id, sku, quantity_change)
VALUES (?, ?, ?)
ON CONFLICT (operation_id) DO NOTHING
", [$operationId, $sku, $quantity]);
// Only update if operation was new
if (DB::affectedRows() > 0) {
DB::table('inventory')
->where('sku', $sku)
->decrement('quantity', $quantity);
}
}
Idempotent Consumers
Message queue consumers must handle duplicate messages. Message brokers provide at-least-once delivery; exactly-once is not guaranteed.
class OrderProcessingJob implements ShouldQueue
{
public function handle(ProcessOrderMessage $message): void
{
$orderId = $message->orderId;
$messageId = $message->messageId;
// Check if already processed
if ($this->isProcessed($messageId)) {
Log::info("Duplicate message, skipping", ['message_id' => $messageId]);
return;
}
DB::transaction(function () use ($orderId, $messageId) {
// Mark as processed first (within transaction)
ProcessedMessage::create(['message_id' => $messageId]);
// Then do the work
$order = Order::findOrFail($orderId);
$this->fulfillmentService->process($order);
});
}
private function isProcessed(string $messageId): bool
{
return ProcessedMessage::where('message_id', $messageId)->exists();
}
}
Handling Partial Failures
Complex operations might fail partway through. If step 2 of 3 fails, what happens on retry? Without careful design, you might repeat step 1.
Break operations into idempotent steps. Each step should be safe to repeat.
class OrderFulfillmentService
{
public function fulfill(Order $order): void
{
// Each step is idempotent
$this->reserveInventory($order); // Idempotent: no-op if already reserved
$this->chargePayment($order); // Idempotent: no-op if already charged
$this->createShipment($order); // Idempotent: no-op if shipment exists
$this->notifyCustomer($order); // Idempotent: tracks notification sent
}
private function chargePayment(Order $order): void
{
if ($order->payment_captured_at) {
return; // Already charged
}
$charge = $this->paymentGateway->capture(
$order->payment_authorization_id,
$order->total,
idempotencyKey: "capture-{$order->id}"
);
$order->update([
'payment_captured_at' => now(),
'payment_capture_id' => $charge->id,
]);
}
}
Conclusion
Idempotency protects against the duplicate operations that distributed systems make inevitable. Idempotency keys track completed operations. Database constraints prevent duplicate records. Idempotent consumers handle message redelivery safely.
Design for "at least once" delivery with idempotent operations. This combination provides reliability without the impossible guarantee of exactly-once delivery. Every API that modifies state should consider idempotency; the alternative is data corruption when things go wrong.
