Message Queues and Async Processing Patterns

Message queues decouple system components, enabling asynchronous processing that improves reliability and scalability. Instead of processing everything synchronously, you can offload work to background workers, smooth out traffic spikes, and build more resilient architectures.

Why Message Queues?

The Problems They Solve

Consider what happens in a typical synchronous order flow. The user waits while every operation completes sequentially.

Synchronous bottlenecks:

// User waits for all this to complete
$order = Order::create($data);
$this->sendConfirmationEmail($order);  // 500ms
$this->notifyWarehouse($order);        // 300ms
$this->updateAnalytics($order);        // 200ms
// Total: 1000ms+ response time

Now compare that to an asynchronous approach. The user gets an immediate response while background workers handle the rest.

With a queue:

// User gets immediate response
$order = Order::create($data);
OrderCreated::dispatch($order);  // <1ms
// Background workers handle the rest

The difference in user experience is dramatic. A one-second response becomes nearly instantaneous.

Benefits

• Decoupling: Services don't need to know about each other
• Resilience: Failed processing can be retried
• Scalability: Add more workers as load increases
• Traffic smoothing: Handle bursts without overload
• Guaranteed delivery: Messages persist until processed

Queue Technologies

Redis Queues

Fast, simple, great for most applications. If you are already using Redis for caching, adding queues requires minimal additional infrastructure.

// Laravel Redis queue
// config/queue.php
'redis' => [
    'driver' => 'redis',
    'connection' => 'default',
    'queue' => 'default',
    'retry_after' => 90,
],

Pros: Fast, simple setup, good Laravel integration Cons: Messages lost if Redis restarts (unless persistence enabled)

RabbitMQ

Advanced message broker with routing and persistence. Choose RabbitMQ when you need complex routing patterns or guaranteed delivery across restarts.

// composer require php-amqplib/php-amqplib
use PhpAmqpLib\Connection\AMQPStreamConnection;
use PhpAmqpLib\Message\AMQPMessage;

$connection = new AMQPStreamConnection('localhost', 5672, 'guest', 'guest');
$channel = $connection->channel();

$channel->queue_declare('orders', false, true, false, false);

$msg = new AMQPMessage(json_encode($data), ['delivery_mode' => 2]);
$channel->basic_publish($msg, '', 'orders');

The delivery_mode => 2 setting makes messages persistent, ensuring they survive broker restarts.

Pros: Persistent, advanced routing, multiple protocols Cons: More complex setup and operations

Amazon SQS

Fully managed queue service. SQS eliminates operational overhead at the cost of some flexibility and higher latency.

// Laravel SQS queue
// config/queue.php
'sqs' => [
    'driver' => 'sqs',
    'key' => env('AWS_ACCESS_KEY_ID'),
    'secret' => env('AWS_SECRET_ACCESS_KEY'),
    'prefix' => env('SQS_PREFIX', 'https://sqs.us-east-1.amazonaws.com/your-account'),
    'queue' => env('SQS_QUEUE', 'default'),
    'region' => env('AWS_DEFAULT_REGION', 'us-east-1'),
],

Pros: No infrastructure to manage, auto-scaling, pay per use Cons: Higher latency, AWS lock-in, less control

Laravel Queues

Creating Jobs

Jobs encapsulate the work you want to perform asynchronously. The ShouldQueue interface tells Laravel to dispatch this job to the queue rather than running it immediately.

// app/Jobs/ProcessOrder.php
class ProcessOrder implements ShouldQueue
{
    use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;

    public function __construct(
        public Order $order
    ) {}

    public function handle(): void
    {
        // Send confirmation email
        Mail::to($this->order->email)->send(new OrderConfirmation($this->order));

        // Notify warehouse
        $this->order->warehouse->notify(new NewOrderNotification($this->order));

        // Update analytics
        Analytics::track('order_placed', [
            'order_id' => $this->order->id,
            'total' => $this->order->total,
        ]);
    }

    public function failed(Throwable $exception): void
    {
        // Handle failure - alert team, log, etc.
        Log::error('Order processing failed', [
            'order_id' => $this->order->id,
            'error' => $exception->getMessage(),
        ]);
    }
}

The SerializesModels trait stores the model's ID rather than the entire model, then re-fetches it when the job runs. This prevents stale data issues when jobs are delayed.

Dispatching Jobs

Laravel provides several ways to dispatch jobs depending on your needs.

// Basic dispatch
ProcessOrder::dispatch($order);

// Delayed dispatch
ProcessOrder::dispatch($order)->delay(now()->addMinutes(10));

// Specific queue
ProcessOrder::dispatch($order)->onQueue('orders');

// Chained jobs
Bus::chain([
    new ProcessPayment($order),
    new SendConfirmation($order),
    new NotifyWarehouse($order),
])->dispatch();

Chained jobs are powerful for workflows where each step depends on the previous one completing successfully. If any job in the chain fails, subsequent jobs are not executed.

Job Configuration

You can configure retry behavior, timeouts, and rate limiting directly on the job class.

class ProcessOrder implements ShouldQueue
{
    // Maximum attempts before failing
    public $tries = 3;

    // Timeout in seconds
    public $timeout = 120;

    // Unique job (prevent duplicates)
    public $uniqueFor = 3600;

    // Specific queue
    public $queue = 'orders';

    // Custom retry delays
    public function backoff(): array
    {
        return [30, 60, 120]; // Seconds between retries
    }

    public function uniqueId(): string
    {
        return $this->order->id;
    }

    // Rate limiting
    public function middleware(): array
    {
        return [
            new RateLimited('orders'),
            new WithoutOverlapping($this->order->id),
        ];
    }
}

The uniqueFor and uniqueId combination prevents duplicate jobs for the same order from being queued within the specified time window. This is essential for payment processing and similar operations where duplicate execution would be problematic.

Running Workers

Workers are long-running processes that pull jobs from the queue and execute them.

# Start single worker
php artisan queue:work

# Specific queue
php artisan queue:work --queue=orders,default

# Production settings
php artisan queue:work --sleep=3 --tries=3 --max-time=3600

The --max-time flag tells the worker to exit gracefully after an hour, allowing it to pick up code changes when restarted by Supervisor.

Supervisor Configuration

In production, use Supervisor to keep workers running and restart them if they crash.

; /etc/supervisor/conf.d/laravel-worker.conf
[program:laravel-worker]
process_name=%(program_name)s_%(process_num)02d
command=php /var/www/app/artisan queue:work --sleep=3 --tries=3 --max-time=3600
autostart=true
autorestart=true
stopasgroup=true
killasgroup=true
user=www-data
numprocs=8
redirect_stderr=true
stdout_logfile=/var/www/app/storage/logs/worker.log
stopwaitsecs=3600

The numprocs=8 setting runs 8 worker processes in parallel. Adjust this based on your server resources and job characteristics. CPU-bound jobs need fewer workers; I/O-bound jobs can handle more.

Queue Patterns

Fan-Out

One message triggers multiple consumers. Laravel's event system makes this pattern natural.

// Publish event
event(new OrderPlaced($order));

// Multiple listeners handle it
class SendOrderConfirmation { }
class NotifyWarehouse { }
class UpdateInventory { }
class RecordAnalytics { }

Each listener runs as a separate queued job, allowing them to process in parallel and fail independently.

Work Queue

Distribute work across multiple workers. This is useful for batch operations like sending newsletters.

// Dispatch many jobs
foreach ($users as $user) {
    SendNewsletter::dispatch($user);
}

// Multiple workers process in parallel

With eight workers running, eight newsletters are sent concurrently, dramatically reducing total processing time.

Priority Queues

Not all jobs are equally urgent. Use separate queues with different priorities.

// High priority
ProcessPayment::dispatch($order)->onQueue('high');

// Low priority
UpdateAnalytics::dispatch($data)->onQueue('low');

// Worker processes high first
// php artisan queue:work --queue=high,default,low

The queue list order matters. Workers process all jobs from the first queue before moving to the next, ensuring urgent jobs are never starved.

Dead Letter Queue

Handle persistently failing jobs instead of losing them forever.

// config/queue.php for SQS
'sqs' => [
    'driver' => 'sqs',
    // ... other config
    'after_commit' => true,
],

// Job that moves to DLQ after max retries
class ProcessOrder implements ShouldQueue
{
    public $tries = 3;

    public function failed(Throwable $e): void
    {
        FailedJob::create([
            'job_class' => static::class,
            'payload' => serialize($this->order),
            'exception' => $e->getMessage(),
        ]);
    }
}

This pattern lets you investigate failures later without blocking the main queue. You can build an admin interface to retry or discard failed jobs manually.

Error Handling

Retry Strategies

Different errors require different handling. Temporary failures should be retried; permanent failures should not.

class ProcessOrder implements ShouldQueue
{
    public function handle(): void
    {
        try {
            $this->process();
        } catch (TemporaryException $e) {
            // Retry later
            $this->release(60);
        } catch (PermanentException $e) {
            // Don't retry
            $this->fail($e);
        }
    }

    // Exponential backoff
    public function backoff(): array
    {
        return [1, 5, 10]; // 1s, 5s, 10s delays
    }
}

The release(60) method puts the job back on the queue with a 60-second delay. This is useful when you hit rate limits or encounter temporary network issues.

Monitoring Failed Jobs

Laravel provides built-in commands to manage failed jobs.

# List failed jobs
php artisan queue:failed

# Retry specific job
php artisan queue:retry 5

# Retry all
php artisan queue:retry all

# Clear failed jobs
php artisan queue:flush

Job Batching

Process groups of jobs together and track overall progress.

// Create batch
$batch = Bus::batch([
    new ProcessOrderItem($item1),
    new ProcessOrderItem($item2),
    new ProcessOrderItem($item3),
])->then(function (Batch $batch) {
    // All jobs completed
    SendOrderComplete::dispatch($batch->name);
})->catch(function (Batch $batch, Throwable $e) {
    // First failure
    Log::error('Batch failed', ['batch' => $batch->id]);
})->finally(function (Batch $batch) {
    // Batch finished (success or fail)
})->dispatch();

// Check batch status
$batch = Bus::findBatch($batchId);
$batch->progress();     // Percentage complete
$batch->finished();     // Boolean
$batch->cancelled();    // Boolean

Batching is perfect for import operations where you want to process many records but need to know when the entire import completes.

Idempotency

Jobs may run multiple times. Design for idempotency. Network glitches can cause jobs to be delivered twice, or a job might crash after completing work but before acknowledging completion.

class ProcessPayment implements ShouldQueue
{
    public function handle(): void
    {
        // Check if already processed
        if ($this->order->payment_processed_at) {
            return;
        }

        DB::transaction(function () {
            // Process payment
            $this->chargeCard();

            // Mark as processed
            $this->order->update([
                'payment_processed_at' => now(),
            ]);
        });
    }
}

The check-then-set pattern inside a transaction ensures the payment is only processed once, even if the job runs multiple times.

Or use unique jobs to prevent duplicates from being queued in the first place.

class ProcessPayment implements ShouldQueue, ShouldBeUnique
{
    public function uniqueId(): string
    {
        return $this->order->id;
    }

    public $uniqueFor = 3600;
}

This approach prevents duplicates at the queue level rather than in your job logic.

Testing Queues

Laravel's queue faking makes testing asynchronous code straightforward.

use Illuminate\Support\Facades\Queue;

public function test_order_dispatches_job()
{
    Queue::fake();

    // Perform action
    $order = Order::create($data);

    // Assert job was dispatched
    Queue::assertPushed(ProcessOrder::class, function ($job) use ($order) {
        return $job->order->id === $order->id;
    });

    // Assert job on specific queue
    Queue::assertPushedOn('orders', ProcessOrder::class);

    // Assert nothing pushed
    Queue::assertNothingPushed();
}

Remember that faking the queue prevents actual job execution. If you need to test the job logic itself, test the job class directly.

Monitoring

Horizon (Laravel)

Horizon provides a beautiful dashboard for monitoring Redis queues.

composer require laravel/horizon
php artisan horizon:install
php artisan horizon

Dashboard at /horizon shows:

• Job throughput
• Failed jobs
• Wait times
• Worker status

Custom Metrics

For deeper insights, instrument your jobs with custom metrics.

class ProcessOrder implements ShouldQueue
{
    public function handle(): void
    {
        $start = microtime(true);

        $this->process();

        Metrics::timing('job.process_order', microtime(true) - $start);
        Metrics::increment('jobs.processed');
    }
}

Track job duration to identify slow jobs that need optimization. Alert when job durations exceed expectations.

Conclusion

Message queues transform how you build applications;moving from synchronous request-response to asynchronous, resilient processing. Start with Laravel's built-in queues and Redis, add Horizon for monitoring, and evolve to RabbitMQ or SQS as your needs grow. Design jobs to be idempotent, implement proper retry strategies, and monitor queue health to build reliable systems.