Message queues decouple system components, enabling asynchronous processing that improves reliability and scalability. Instead of processing everything synchronously, you can offload work to background workers, smooth out traffic spikes, and build more resilient architectures.
Why Message Queues?
The Problems They Solve
Consider what happens in a typical synchronous order flow. The user waits while every operation completes sequentially, and any failure in the chain breaks the entire process.
Synchronous bottlenecks:
// User waits for all this to complete
$order = Order::create($data);
$this->sendConfirmationEmail($order); // 500ms
$this->notifyWarehouse($order); // 300ms
$this->updateAnalytics($order); // 200ms
// Total: 1000ms+ response time
Now compare that to an asynchronous approach. The user gets an immediate response while background workers handle the rest independently.
With a queue:
// User gets immediate response
$order = Order::create($data);
OrderCreated::dispatch($order); // <1ms
// Background workers handle the rest
The difference in user experience is dramatic. A one-second response becomes nearly instantaneous, and if the email server is slow, the user is unaffected.
Benefits
- Decoupling: Services don't need to know about each other
- Resilience: Failed processing can be retried
- Scalability: Add more workers as load increases
- Traffic smoothing: Handle bursts without overload
- Guaranteed delivery: Messages persist until processed
Queue Technologies
Redis Queues
Fast, simple, great for most applications. If you are already using Redis for caching, adding queues requires minimal additional infrastructure.
// Laravel Redis queue
// config/queue.php
'redis' => [
'driver' => 'redis',
'connection' => 'default',
'queue' => 'default',
'retry_after' => 90,
],
Pros: Fast, simple setup, good Laravel integration Cons: Messages lost if Redis restarts (unless persistence enabled)
RabbitMQ
Advanced message broker with routing and persistence. Choose RabbitMQ when you need complex routing patterns or guaranteed delivery across restarts.
// composer require php-amqplib/php-amqplib
use PhpAmqpLib\Connection\AMQPStreamConnection;
use PhpAmqpLib\Message\AMQPMessage;
$connection = new AMQPStreamConnection('localhost', 5672, 'guest', 'guest');
$channel = $connection->channel();
$channel->queue_declare('orders', false, true, false, false);
$msg = new AMQPMessage(json_encode($data), ['delivery_mode' => 2]);
$channel->basic_publish($msg, '', 'orders');
The delivery_mode => 2 setting makes messages persistent, ensuring they survive broker restarts. This is essential for production workloads where message loss is unacceptable.
Pros: Persistent, advanced routing, multiple protocols Cons: More complex setup and operations
Amazon SQS
Fully managed queue service. SQS eliminates operational overhead at the cost of some flexibility and higher latency.
// Laravel SQS queue
// config/queue.php
'sqs' => [
'driver' => 'sqs',
'key' => env('AWS_ACCESS_KEY_ID'),
'secret' => env('AWS_SECRET_ACCESS_KEY'),
'prefix' => env('SQS_PREFIX', 'https://sqs.us-east-1.amazonaws.com/your-account'),
'queue' => env('SQS_QUEUE', 'default'),
'region' => env('AWS_DEFAULT_REGION', 'us-east-1'),
],
Pros: No infrastructure to manage, auto-scaling, pay per use Cons: Higher latency, AWS lock-in, less control
Laravel Queues
Creating Jobs
Jobs encapsulate the work you want to perform asynchronously. The ShouldQueue interface tells Laravel to dispatch this job to the queue rather than running it immediately. This separation makes your controllers fast and your processing reliable.
// app/Jobs/ProcessOrder.php
class ProcessOrder implements ShouldQueue
{
use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;
public function __construct(
public Order $order
) {}
public function handle(): void
{
// Send confirmation email
Mail::to($this->order->email)->send(new OrderConfirmation($this->order));
// Notify warehouse
$this->order->warehouse->notify(new NewOrderNotification($this->order));
// Update analytics
Analytics::track('order_placed', [
'order_id' => $this->order->id,
'total' => $this->order->total,
]);
}
public function failed(Throwable $exception): void
{
// Handle failure - alert team, log, etc.
Log::error('Order processing failed', [
'order_id' => $this->order->id,
'error' => $exception->getMessage(),
]);
}
}
The SerializesModels trait stores the model's ID rather than the entire model, then re-fetches it when the job runs. This prevents stale data issues when jobs are delayed. The failed method gives you a hook to handle permanent failures gracefully.
Dispatching Jobs
Laravel provides several ways to dispatch jobs depending on your needs. Choose the approach that best matches your use case.
// Basic dispatch
ProcessOrder::dispatch($order);
// Delayed dispatch
ProcessOrder::dispatch($order)->delay(now()->addMinutes(10));
// Specific queue
ProcessOrder::dispatch($order)->onQueue('orders');
// Chained jobs
Bus::chain([
new ProcessPayment($order),
new SendConfirmation($order),
new NotifyWarehouse($order),
])->dispatch();
Chained jobs are powerful for workflows where each step depends on the previous one completing successfully. If any job in the chain fails, subsequent jobs are not executed, maintaining data consistency.
Job Configuration
You can configure retry behavior, timeouts, and rate limiting directly on the job class. These settings give you fine-grained control over how jobs execute.
class ProcessOrder implements ShouldQueue
{
// Maximum attempts before failing
public $tries = 3;
// Timeout in seconds
public $timeout = 120;
// Unique job (prevent duplicates)
public $uniqueFor = 3600;
// Specific queue
public $queue = 'orders';
// Custom retry delays
public function backoff(): array
{
return [30, 60, 120]; // Seconds between retries
}
public function uniqueId(): string
{
return $this->order->id;
}
// Rate limiting
public function middleware(): array
{
return [
new RateLimited('orders'),
new WithoutOverlapping($this->order->id),
];
}
}
The uniqueFor and uniqueId combination prevents duplicate jobs for the same order from being queued within the specified time window. This is essential for payment processing and similar operations where duplicate execution would be problematic. The backoff array provides exponential delays between retries.
Running Workers
Workers are long-running processes that pull jobs from the queue and execute them. Understanding worker options helps you tune performance for your workload.
# Start single worker
php artisan queue:work
# Specific queue
php artisan queue:work --queue=orders,default
# Production settings
php artisan queue:work --sleep=3 --tries=3 --max-time=3600
The --max-time flag tells the worker to exit gracefully after an hour, allowing it to pick up code changes when restarted by Supervisor. This ensures deployments take effect without manual intervention.
Supervisor Configuration
In production, use Supervisor to keep workers running and restart them if they crash. This configuration ensures your queue processing is always available.
; /etc/supervisor/conf.d/laravel-worker.conf
[program:laravel-worker]
process_name=%(program_name)s_%(process_num)02d
command=php /var/www/app/artisan queue:work --sleep=3 --tries=3 --max-time=3600
autostart=true
autorestart=true
stopasgroup=true
killasgroup=true
user=www-data
numprocs=8
redirect_stderr=true
stdout_logfile=/var/www/app/storage/logs/worker.log
stopwaitsecs=3600
The numprocs=8 setting runs 8 worker processes in parallel. Adjust this based on your server resources and job characteristics. CPU-bound jobs need fewer workers; I/O-bound jobs can handle more. The stopwaitsecs matches the max-time to allow graceful shutdown.
Queue Patterns
Fan-Out
One message triggers multiple consumers. Laravel's event system makes this pattern natural and keeps your code organized.
// Publish event
event(new OrderPlaced($order));
// Multiple listeners handle it
class SendOrderConfirmation { }
class NotifyWarehouse { }
class UpdateInventory { }
class RecordAnalytics { }
Each listener runs as a separate queued job, allowing them to process in parallel and fail independently. One slow listener does not block the others.
Work Queue
Distribute work across multiple workers. This is useful for batch operations like sending newsletters where you have many similar tasks to perform.
// Dispatch many jobs
foreach ($users as $user) {
SendNewsletter::dispatch($user);
}
// Multiple workers process in parallel
With eight workers running, eight newsletters are sent concurrently, dramatically reducing total processing time. The queue acts as a buffer between job creation and execution.
Priority Queues
Not all jobs are equally urgent. Use separate queues with different priorities to ensure critical work is processed first.
// High priority
ProcessPayment::dispatch($order)->onQueue('high');
// Low priority
UpdateAnalytics::dispatch($data)->onQueue('low');
// Worker processes high first
// php artisan queue:work --queue=high,default,low
The queue list order matters. Workers process all jobs from the first queue before moving to the next, ensuring urgent jobs are never starved by a backlog of low-priority work.
Dead Letter Queue
Handle persistently failing jobs instead of losing them forever. This pattern gives you visibility into problems and the ability to recover.
// config/queue.php for SQS
'sqs' => [
'driver' => 'sqs',
// ... other config
'after_commit' => true,
],
// Job that moves to DLQ after max retries
class ProcessOrder implements ShouldQueue
{
public $tries = 3;
public function failed(Throwable $e): void
{
FailedJob::create([
'job_class' => static::class,
'payload' => serialize($this->order),
'exception' => $e->getMessage(),
]);
}
}
This pattern lets you investigate failures later without blocking the main queue. You can build an admin interface to retry or discard failed jobs manually, giving operators control over error recovery.
Error Handling
Retry Strategies
Different errors require different handling. Temporary failures should be retried; permanent failures should not. Distinguishing between them prevents wasted effort.
class ProcessOrder implements ShouldQueue
{
public function handle(): void
{
try {
$this->process();
} catch (TemporaryException $e) {
// Retry later
$this->release(60);
} catch (PermanentException $e) {
// Don't retry
$this->fail($e);
}
}
// Exponential backoff
public function backoff(): array
{
return [1, 5, 10]; // 1s, 5s, 10s delays
}
}
The release(60) method puts the job back on the queue with a 60-second delay. This is useful when you hit rate limits or encounter temporary network issues. The job will be picked up again after the delay.
Monitoring Failed Jobs
Laravel provides built-in commands to manage failed jobs. Make these part of your operational toolkit.
# List failed jobs
php artisan queue:failed
# Retry specific job
php artisan queue:retry 5
# Retry all
php artisan queue:retry all
# Clear failed jobs
php artisan queue:flush
Job Batching
Process groups of jobs together and track overall progress. Batching is perfect for import operations where you want to process many records but need to know when the entire import completes.
// Create batch
$batch = Bus::batch([
new ProcessOrderItem($item1),
new ProcessOrderItem($item2),
new ProcessOrderItem($item3),
])->then(function (Batch $batch) {
// All jobs completed
SendOrderComplete::dispatch($batch->name);
})->catch(function (Batch $batch, Throwable $e) {
// First failure
Log::error('Batch failed', ['batch' => $batch->id]);
})->finally(function (Batch $batch) {
// Batch finished (success or fail)
})->dispatch();
// Check batch status
$batch = Bus::findBatch($batchId);
$batch->progress(); // Percentage complete
$batch->finished(); // Boolean
$batch->cancelled(); // Boolean
The callbacks give you hooks for success, failure, and completion regardless of outcome. You can use these to send notifications or update UI elements showing progress.
Idempotency
Jobs may run multiple times. Design for idempotency. Network glitches can cause jobs to be delivered twice, or a job might crash after completing work but before acknowledging completion.
class ProcessPayment implements ShouldQueue
{
public function handle(): void
{
// Check if already processed
if ($this->order->payment_processed_at) {
return;
}
DB::transaction(function () {
// Process payment
$this->chargeCard();
// Mark as processed
$this->order->update([
'payment_processed_at' => now(),
]);
});
}
}
The check-then-set pattern inside a transaction ensures the payment is only processed once, even if the job runs multiple times. The timestamp serves as a processing flag and provides audit information.
Or use unique jobs to prevent duplicates from being queued in the first place. This approach is more efficient when you can identify duplicates early.
class ProcessPayment implements ShouldQueue, ShouldBeUnique
{
public function uniqueId(): string
{
return $this->order->id;
}
public $uniqueFor = 3600;
}
This approach prevents duplicates at the queue level rather than in your job logic, reducing unnecessary work.
Testing Queues
Laravel's queue faking makes testing asynchronous code straightforward. You can verify jobs were dispatched without actually running them.
use Illuminate\Support\Facades\Queue;
public function test_order_dispatches_job()
{
Queue::fake();
// Perform action
$order = Order::create($data);
// Assert job was dispatched
Queue::assertPushed(ProcessOrder::class, function ($job) use ($order) {
return $job->order->id === $order->id;
});
// Assert job on specific queue
Queue::assertPushedOn('orders', ProcessOrder::class);
// Assert nothing pushed
Queue::assertNothingPushed();
}
Remember that faking the queue prevents actual job execution. If you need to test the job logic itself, test the job class directly by calling its handle method.
Monitoring
Horizon (Laravel)
Horizon provides a beautiful dashboard for monitoring Redis queues. It is the standard choice for Laravel applications using Redis.
composer require laravel/horizon
php artisan horizon:install
php artisan horizon
Dashboard at /horizon shows:
- Job throughput
- Failed jobs
- Wait times
- Worker status
Custom Metrics
For deeper insights, instrument your jobs with custom metrics. This data helps you identify bottlenecks and optimize performance.
class ProcessOrder implements ShouldQueue
{
public function handle(): void
{
$start = microtime(true);
$this->process();
Metrics::timing('job.process_order', microtime(true) - $start);
Metrics::increment('jobs.processed');
}
}
Track job duration to identify slow jobs that need optimization. Alert when job durations exceed expectations to catch problems before they impact users.
Conclusion
Message queues transform how you build applications;moving from synchronous request-response to asynchronous, resilient processing. Start with Laravel's built-in queues and Redis, add Horizon for monitoring, and evolve to RabbitMQ or SQS as your needs grow. Design jobs to be idempotent, implement proper retry strategies, and monitor queue health to build reliable systems.
