Background Jobs Guide | Queues, Workers, and Reliability Patterns

Background job processing moves time-consuming tasks out of the request cycle, improving user experience and application reliability. This guide covers patterns for implementing robust background processing in Laravel applications.

Why Background Jobs?

The Request Cycle Problem

User Request → Processing → Response
              (blocking)
              2-30 seconds

User: "Is this thing frozen?"

Moving work to background:

User Request → Queue Job → Response (immediate)
                    ↓
              Worker Process
              (async, no waiting)

What to Process in Background

Email sending
PDF generation
Image processing
API calls to external services
Report generation
Data imports/exports
Notifications
Search indexing
Cleanup tasks

Laravel Queue Basics

Creating Jobs

php artisan make:job ProcessOrderPayment

// app/Jobs/ProcessOrderPayment.php
namespace App\Jobs;

use App\Models\Order;
use Illuminate\Bus\Queueable;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Bus\Dispatchable;
use Illuminate\Queue\InteractsWithQueue;
use Illuminate\Queue\SerializesModels;

class ProcessOrderPayment implements ShouldQueue
{
    use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;

    public function __construct(
        public Order $order
    ) {}

    public function handle(PaymentGateway $gateway): void
    {
        $result = $gateway->charge(
            $this->order->customer,
            $this->order->total
        );

        $this->order->update([
            'payment_status' => $result->success ? 'paid' : 'failed',
            'transaction_id' => $result->transactionId,
        ]);
    }
}

Dispatching Jobs

// Dispatch immediately to queue
ProcessOrderPayment::dispatch($order);

// Dispatch with delay
ProcessOrderPayment::dispatch($order)
    ->delay(now()->addMinutes(5));

// Dispatch to specific queue
ProcessOrderPayment::dispatch($order)
    ->onQueue('payments');

// Dispatch to specific connection
ProcessOrderPayment::dispatch($order)
    ->onConnection('redis');

// Conditional dispatch
ProcessOrderPayment::dispatchIf($order->needsPayment(), $order);
ProcessOrderPayment::dispatchUnless($order->isPaid(), $order);

Job Configuration

Retries and Backoff

class ProcessOrderPayment implements ShouldQueue
{
    public int $tries = 3;            // Maximum attempts
    public int $maxExceptions = 2;    // Max exceptions before failing
    public int $timeout = 60;         // Seconds before timeout

    // Exponential backoff between retries
    public function backoff(): array
    {
        return [10, 60, 300]; // 10s, 1m, 5m
    }

    // Or calculate dynamically
    public function backoff(): int
    {
        return $this->attempts() * 60;
    }
}

Unique Jobs

Prevent duplicate jobs:

use Illuminate\Contracts\Queue\ShouldBeUnique;

class SendWelcomeEmail implements ShouldQueue, ShouldBeUnique
{
    public function __construct(
        public User $user
    ) {}

    // Unique based on user ID
    public function uniqueId(): string
    {
        return $this->user->id;
    }

    // Unique lock duration
    public int $uniqueFor = 3600; // 1 hour
}

Rate Limiting

use Illuminate\Queue\Middleware\RateLimited;

class SendNotification implements ShouldQueue
{
    public function middleware(): array
    {
        return [new RateLimited('notifications')];
    }
}

// In AppServiceProvider
RateLimiter::for('notifications', function ($job) {
    return Limit::perMinute(100);
});

Error Handling

Failed Jobs

class ProcessOrderPayment implements ShouldQueue
{
    public function handle(PaymentGateway $gateway): void
    {
        $result = $gateway->charge($this->order);

        if (!$result->success) {
            // Explicit failure - won't retry
            $this->fail(new PaymentFailedException($result->error));
        }
    }

    public function failed(?Throwable $exception): void
    {
        // Called when job fails permanently
        Log::error('Payment processing failed', [
            'order_id' => $this->order->id,
            'error' => $exception?->getMessage(),
        ]);

        // Notify admin
        AdminNotification::send(
            "Payment failed for order {$this->order->id}"
        );

        // Update order status
        $this->order->update(['status' => 'payment_failed']);
    }
}

Retry Specific Exceptions

class CallExternalApi implements ShouldQueue
{
    // Only retry these exceptions
    public function retryUntil(): DateTime
    {
        return now()->addHours(1);
    }

    public function handle(): void
    {
        try {
            Http::timeout(30)->get('https://api.example.com');
        } catch (ConnectionException $e) {
            // Will retry
            throw $e;
        } catch (ClientException $e) {
            // 4xx errors - don't retry
            $this->fail($e);
        }
    }
}

Job Chaining

Sequential Jobs

use Illuminate\Support\Facades\Bus;

// Jobs run in sequence
Bus::chain([
    new ProcessOrder($order),
    new ChargePayment($order),
    new SendConfirmation($order),
    new NotifyWarehouse($order),
])->dispatch();

// With error handling
Bus::chain([
    new ProcessOrder($order),
    new ChargePayment($order),
    new SendConfirmation($order),
])->catch(function (Throwable $e) use ($order) {
    Log::error('Order chain failed', ['order' => $order->id]);
    $order->update(['status' => 'failed']);
})->dispatch();

Job Batches

use Illuminate\Bus\Batch;
use Illuminate\Support\Facades\Bus;

$batch = Bus::batch([
    new ProcessImage($image1),
    new ProcessImage($image2),
    new ProcessImage($image3),
])
->then(function (Batch $batch) {
    // All jobs completed successfully
    Log::info('Batch completed', ['id' => $batch->id]);
})
->catch(function (Batch $batch, Throwable $e) {
    // First job failure
    Log::error('Batch failed', ['error' => $e->getMessage()]);
})
->finally(function (Batch $batch) {
    // Batch finished (success or failure)
    Notification::send('Batch processing complete');
})
->allowFailures()  // Continue even if some jobs fail
->dispatch();

// Check batch status
$batch = Bus::findBatch($batchId);
echo $batch->progress();      // 75
echo $batch->finished();      // false
echo $batch->failedJobs;      // 1

Queue Workers

Running Workers

# Basic worker
php artisan queue:work

# Specific queue
php artisan queue:work --queue=high,default,low

# Memory limit
php artisan queue:work --memory=512

# Process single job
php artisan queue:work --once

# Timeout per job
php artisan queue:work --timeout=60

# Sleep when no jobs
php artisan queue:work --sleep=3

# Stop when queue is empty
php artisan queue:work --stop-when-empty

Production Worker Management

# /etc/supervisor/conf.d/laravel-worker.conf
[program:laravel-worker]
process_name=%(program_name)s_%(process_num)02d
command=php /var/www/html/artisan queue:work redis --sleep=3 --tries=3 --max-time=3600
autostart=true
autorestart=true
stopasgroup=true
killasgroup=true
user=www-data
numprocs=4
redirect_stderr=true
stdout_logfile=/var/www/html/storage/logs/worker.log
stopwaitsecs=3600

Laravel Horizon

Redis queue dashboard and worker manager:

// config/horizon.php
return [
    'environments' => [
        'production' => [
            'supervisor-1' => [
                'connection' => 'redis',
                'queue' => ['high', 'default', 'low'],
                'balance' => 'auto',
                'minProcesses' => 1,
                'maxProcesses' => 10,
                'balanceMaxShift' => 1,
                'balanceCooldown' => 3,
            ],
        ],
    ],
];

Queue Patterns

Priority Queues

// Define queue priorities
class SendNotification implements ShouldQueue
{
    public function __construct(
        public Notification $notification
    ) {}

    public function queue(): string
    {
        return match ($this->notification->priority) {
            'urgent' => 'high',
            'normal' => 'default',
            'low' => 'low',
        };
    }
}

// Worker processes high first
php artisan queue:work --queue=high,default,low

Job Middleware

// Prevent overlapping jobs
use Illuminate\Queue\Middleware\WithoutOverlapping;

class SyncInventory implements ShouldQueue
{
    public function middleware(): array
    {
        return [
            new WithoutOverlapping($this->product->id)
                ->expireAfter(300)
                ->dontRelease(),
        ];
    }
}

// Skip if duplicate in progress
use Illuminate\Queue\Middleware\Skip;

class ProcessReport implements ShouldQueue
{
    public function middleware(): array
    {
        return [
            Skip::when($this->reportAlreadyProcessing()),
        ];
    }
}

Throttling External APIs

use Illuminate\Queue\Middleware\ThrottlesExceptions;

class CallExternalApi implements ShouldQueue
{
    public function middleware(): array
    {
        return [
            (new ThrottlesExceptions(10, 5))  // 10 exceptions per 5 minutes
                ->backoff(5),                  // Wait 5 minutes after throttle
        ];
    }
}

Testing Jobs

Faking Queues

use Illuminate\Support\Facades\Queue;

public function test_order_dispatches_payment_job(): void
{
    Queue::fake();

    $order = Order::factory()->create();
    $order->process();

    Queue::assertPushed(ProcessOrderPayment::class, function ($job) use ($order) {
        return $job->order->id === $order->id;
    });
}

public function test_batch_jobs_dispatched(): void
{
    Queue::fake();

    $this->service->processImages($images);

    Queue::assertBatched(function ($batch) use ($images) {
        return $batch->jobs->count() === count($images);
    });
}

Testing Job Logic

public function test_payment_job_charges_customer(): void
{
    $gateway = $this->mock(PaymentGateway::class);
    $gateway->expects('charge')
        ->once()
        ->andReturn(new PaymentResult(success: true));

    $order = Order::factory()->create();
    $job = new ProcessOrderPayment($order);

    $job->handle($gateway);

    $this->assertEquals('paid', $order->fresh()->payment_status);
}

public function test_failed_payment_notifies_admin(): void
{
    Notification::fake();

    $order = Order::factory()->create();
    $job = new ProcessOrderPayment($order);

    $job->failed(new PaymentFailedException('Card declined'));

    Notification::assertSentTo(
        Admin::first(),
        PaymentFailedNotification::class
    );
}

Monitoring

Queue Metrics

// Track job duration
class ProcessOrder implements ShouldQueue
{
    public function handle(): void
    {
        $start = microtime(true);

        // Process...

        $duration = microtime(true) - $start;
        Log::info('Job completed', [
            'job' => class_basename($this),
            'duration_ms' => round($duration * 1000),
        ]);
    }
}

// Dashboard metrics
$metrics = [
    'pending' => Queue::size('default'),
    'failed' => DB::table('failed_jobs')->count(),
    'processed_today' => Cache::get('jobs_processed_today', 0),
];

Alerting

// Alert on queue backup
if (Queue::size('default') > 1000) {
    Alert::send('Queue backup detected');
}

// Alert on failed jobs
Event::listen(JobFailed::class, function ($event) {
    if ($this->isHighPriorityJob($event->job)) {
        Alert::send("Critical job failed: {$event->job->displayName()}");
    }
});

Best Practices

Keep jobs small - Single responsibility, one task per job
Make jobs idempotent - Safe to run multiple times
Serialize minimally - Pass IDs, not full models when possible
Handle failures gracefully - Always implement failed() method
Set appropriate timeouts - Match expected job duration
Use queue priorities - Separate urgent from batch work
Monitor queue health - Track size, failures, processing time
Test job logic - Unit test the handle() method

Conclusion

Background job processing is essential for responsive applications. Laravel's queue system provides everything needed: job creation, retry logic, batching, and monitoring. Start with simple jobs, add proper error handling, implement monitoring, and scale workers as needed. The key is moving any work that doesn't need to complete during the request cycle into the background.

Background Job Processing Patterns