Background Job Deduplication: Idempotency Keys and Locking

Queue workers run jobs more than once. They have to — at-least-once delivery is the only practical guarantee, and exactly-once is a mirage. The implication is that every job needs to be safe to run twice, three times, more.

In trivial cases this is easy. In complex ones — multi-step jobs, jobs that mutate external systems, jobs that produce side effects — it requires deliberate design. This post is the deduplication patterns that work in production Laravel queues and the failures they prevent.

Why a Job Runs Twice

Three common scenarios:

The worker died after the work but before recording success. Network blip, OOM, deploy restart. The queue thinks the job timed out and requeues it. The work was done; it gets done again.

The job timed out at the framework boundary but completed at the external service. Your payment API call took 31 seconds; your 30-second timeout fired; Stripe still charged the card. The job is requeued, and a retry would charge the card again.

The same job was dispatched twice by mistake. A user double-clicked, a webhook was delivered twice, a cron job overlapped. Two jobs, identical work.

Every queued job will hit one of these eventually. The question is whether your code handles it.

Idempotency Keys at the Boundary

The first line of defense: pass an idempotency key to every external service that supports them.

class ChargePayment implements ShouldQueue
{
    public function __construct(public Order $order) {}

    public function handle(): void
    {
        $idempotencyKey = "order:{$this->order->id}:charge";

        $charge = Http::withHeaders([
            'Idempotency-Key' => $idempotencyKey,
        ])->post('https://api.stripe.com/v1/charges', [
            'amount' => $this->order->total_cents,
            'currency' => 'usd',
            'customer' => $this->order->customer_id,
        ])->json();

        $this->order->update(['stripe_charge_id' => $charge['id']]);
    }
}

Stripe's API recognizes the idempotency key. If the same key arrives twice, Stripe returns the same charge object without charging again. Most modern payment, messaging, and SMS APIs support this pattern.

The key needs to be deterministic — same job, same key. Using a random UUID per attempt defeats the point.

Idempotency at the Database Layer

For local database writes, the equivalent is checking the post-state before writing.

public function handle(): void
{
    if ($this->order->fresh()->status === 'paid') {
        return; // already done
    }

    DB::transaction(function () {
        $this->order->markPaid();
        $this->order->save();
    });
}

This is a "check then act" with a race window — two workers can both see status !== 'paid' and both proceed. For high-concurrency cases, combine with row-level locking:

DB::transaction(function () {
    $order = Order::where('id', $this->order->id)
        ->lockForUpdate()
        ->first();

    if ($order->status === 'paid') {
        return;
    }

    $order->markPaid();
    $order->save();
});

The lock ensures only one worker can process the order at a time. The check inside the lock catches the race.

Job Fingerprinting

For jobs where the "is this duplicate" question is about the inputs rather than the result, store a fingerprint per job.

public function handle(): void
{
    $fingerprint = hash('xxh128', json_encode([
        'order_id' => $this->order->id,
        'action' => 'charge',
        'amount' => $this->order->total_cents,
    ]));

    if (ProcessedJob::where('fingerprint', $fingerprint)->exists()) {
        return; // already processed
    }

    DB::transaction(function () use ($fingerprint) {
        $this->charge();
        ProcessedJob::create(['fingerprint' => $fingerprint]);
    });
}

The fingerprint table is a lightweight log of "we did this work." Prune entries older than the queue's longest possible redelivery window (usually a few hours to a few days).

This pattern works for jobs that do not have a natural "is this done?" state on the entity itself.

Laravel's WithoutOverlapping

For preventing concurrent execution of jobs with the same logical target, Laravel's WithoutOverlapping middleware uses a cache-based lock.

class SyncCustomer implements ShouldQueue
{
    public function __construct(public Customer $customer) {}

    public function middleware(): array
    {
        return [
            (new WithoutOverlapping($this->customer->id))->expireAfter(300),
        ];
    }
}

Two jobs with the same customer ID cannot run at the same time. The lock expires after 300 seconds in case a worker dies.

This is concurrency control, not idempotency. The first job completes; the second job runs after the first finishes — and if both will produce the same effect, the second is still a duplicate. Combine with one of the other patterns.

The Dispatching Side

The cheapest deduplication is preventing the duplicate dispatch in the first place.

// Bad: dispatched on every webhook, no dedupe
class HandleStripeWebhook
{
    public function __invoke(Request $request)
    {
        $event = $request->json('id');
        ProcessStripeEvent::dispatch($event);
    }
}

// Better: check first
class HandleStripeWebhook
{
    public function __invoke(Request $request)
    {
        $eventId = $request->json('id');

        $created = ProcessedStripeEvent::firstOrCreate(['id' => $eventId]);
        if (!$created->wasRecentlyCreated) {
            return response('Already processed', 200);
        }

        ProcessStripeEvent::dispatch($eventId);
    }
}

firstOrCreate is atomic at the database level. The first webhook delivery wins; the second sees the existing record and returns without dispatching.

This pattern is essential for webhook receivers. Webhook senders retry; a non-deduplicating receiver processes the same event many times.

Compensating for Non-Idempotent Operations

Some operations cannot be made idempotent. Sending an email is the classic example — most email APIs do not deduplicate. The fix is to dedupe before the send.

public function handle(): void
{
    $sentKey = "email:welcome:{$this->user->id}";

    if (Cache::has($sentKey)) {
        return;
    }

    Mail::to($this->user)->send(new WelcomeEmail($this->user));
    Cache::put($sentKey, true, now()->addDays(30));
}

The cache key marks the send. A retry hits the cache and skips. The 30-day TTL prevents the cache from growing indefinitely.

For higher-stakes sends (financial notifications, security alerts), store the send record in the database with a unique constraint:

DB::transaction(function () {
    SentEmail::create([
        'user_id' => $this->user->id,
        'template' => 'welcome',
    ]); // throws on duplicate, prevents send
    Mail::to($this->user)->send(new WelcomeEmail($this->user));
});

The database constraint plus transaction ensures at-most-once delivery. If the send fails after the row is inserted, the user does not get the email and you have to retry manually — usually the right tradeoff for high-stakes communications.

A Test for Idempotency

The best test for a job's idempotency is to run it twice in a row in a test and assert the world looks the same as if it ran once.

test('charge order is idempotent', function () {
    $order = Order::factory()->create();

    Bus::fake();
    (new ChargePayment($order))->handle();
    (new ChargePayment($order))->handle(); // run twice

    expect($order->fresh()->status)->toBe('paid');
    expect(StripeCharge::where('order_id', $order->id)->count())->toBe(1);
});

This catches the easy idempotency bugs at test time. Real idempotency violations involve race conditions that are harder to test, but the basic check is essential before considering a job production-ready.

Common Mistakes

• Assuming the queue gives you exactly-once. No queue does. Always design for at-least-once.
• Using random_int() for idempotency keys. The key must be deterministic from the job inputs.
• Caching idempotency state in process memory. Workers restart; in-memory state is lost.
• Forgetting to handle the dedupe state failure mode. If the dedupe lookup fails (Redis down), what does the job do? Usually proceeding without dedupe is worse than failing.
• No expiration on dedupe records. A table that grows forever becomes a performance problem.

A Checklist Before Shipping

Before any queued job goes to production:

• External calls use idempotency keys
• Database mutations check post-state or use locking
• Non-idempotent operations (emails, notifications) have explicit dedupe
• Dispatch points dedupe at the source where possible
• At least one test exercises the "run twice" path
• Dedupe state has a clear expiration policy

Skip these and the failure mode is silent — the job runs, mostly works, and occasionally double-charges someone. Idempotency is not optional; it is the price of using a queue.

Reviewing a queue-driven system that has occasional, hard-to-diagnose duplicate-operation incidents? We help teams audit job idempotency and dedupe-at-the-edges. scopeforged.com