Message queues decouple services and enable asynchronous processing. Understanding queue patterns helps you build resilient systems that handle failures gracefully and scale independently.
Why Message Queues?
Synchronous Problems
In a synchronous architecture, every service in the chain must respond before the user gets a response. This creates tight coupling and cascading failures.
This diagram shows the fundamental problem with synchronous architectures. Every service in the chain must complete before the user gets their response, creating a fragile dependency chain.
User Request → API → Payment Service → Inventory Service → Email Service → Response
Problems:
- User waits for all services (slow)
- One service failure breaks everything
- Can't scale services independently
- Retry logic scattered everywhere
Asynchronous Solution
With message queues, you respond to the user immediately and process work in the background. Each service can scale and fail independently.
Queues decouple your request handling from your processing. You can acknowledge the user's request immediately and handle the actual work asynchronously.
User Request → API → Queue → Response (immediate)
↓
Worker processes:
- Payment
- Inventory update
- Email notification
The user gets a fast response, and your system handles the complexity behind the scenes.
Core Patterns
Point-to-Point
The simplest pattern: one producer sends a message, one consumer processes it. Each message is processed exactly once by a single consumer.
Point-to-point queues are ideal when you need to distribute tasks among workers. Each message goes to exactly one consumer.
Producer → Queue → Consumer
One message, one consumer.
This pattern is ideal for task distribution where you don't want duplicate processing.
Here's how you'd implement point-to-point in Laravel. The controller queues the job, and a worker processes it asynchronously.
// Producer
class OrderService
{
public function createOrder(array $data): Order
{
$order = Order::create($data);
// Queue for processing
ProcessOrder::dispatch($order);
return $order;
}
}
// Consumer (Job)
class ProcessOrder implements ShouldQueue
{
public function handle(): void
{
$this->processPayment();
$this->updateInventory();
$this->sendConfirmation();
}
}
The order is created immediately, and the heavy processing happens asynchronously. Users don't wait for payment processing to complete.
Publish/Subscribe
When multiple consumers need to react to the same event, publish/subscribe delivers the message to all subscribers.
Use pub/sub when the same event needs to trigger multiple independent actions. Each subscriber receives a copy of every message.
Publisher → Topic → Subscriber A
→ Subscriber B
→ Subscriber C
One message, multiple consumers.
This pattern is perfect for event-driven architectures where different services need to respond to the same business event.
In Laravel, you can implement pub/sub using events and listeners. Each listener handles its own concern independently.
// Publisher
class OrderCreated
{
public function __construct(public Order $order) {}
}
event(new OrderCreated($order));
// Subscribers (Listeners)
class SendOrderConfirmation
{
public function handle(OrderCreated $event): void
{
Mail::to($event->order->customer)->send(new OrderConfirmationMail($event->order));
}
}
class UpdateInventory
{
public function handle(OrderCreated $event): void
{
foreach ($event->order->items as $item) {
Inventory::decrement($item->product_id, $item->quantity);
}
}
}
class NotifyWarehouse
{
public function handle(OrderCreated $event): void
{
$this->warehouseApi->queueForPicking($event->order);
}
}
Each listener handles its own concern independently. Adding a new listener doesn't require modifying the publisher.
Request/Reply
Sometimes you need a response from an asynchronous operation. The request/reply pattern uses correlation IDs to match responses with requests.
Request/reply bridges synchronous needs with asynchronous processing. The correlation ID links requests to their responses.
Requester → Request Queue → Responder
← Reply Queue ←
This pattern is useful when you need asynchronous processing but must wait for the result.
This implementation shows how to send a request and wait for the correlated response. The correlation ID ensures you receive the correct reply.
// Requester
class PriceCalculator
{
public function getPrice(Product $product, Customer $customer): float
{
$correlationId = Str::uuid();
// Send request
Queue::push(new CalculatePriceRequest(
product: $product,
customer: $customer,
correlationId: $correlationId,
replyTo: 'price-responses'
));
// Wait for response (with timeout)
return $this->waitForResponse($correlationId, timeout: 5);
}
}
// Responder
class PriceCalculatorWorker implements ShouldQueue
{
public function handle(CalculatePriceRequest $request): void
{
$price = $this->calculatePrice($request->product, $request->customer);
Queue::connection('rabbitmq')
->pushRaw(json_encode([
'correlationId' => $request->correlationId,
'price' => $price,
]), $request->replyTo);
}
}
Be careful with timeouts. If the responder is slow or down, you need graceful degradation strategies.
Dead Letter Queues
Handling Failed Messages
Messages that can't be processed after multiple retries need a place to go. Dead letter queues capture these failures for later investigation.
Configure your jobs with retry logic and exponential backoff. When all retries are exhausted, the failed method captures the message for review.
// config/queue.php
'connections' => [
'redis' => [
'driver' => 'redis',
'connection' => 'default',
'queue' => 'default',
'retry_after' => 90,
'block_for' => null,
],
],
// Job with retry configuration
class ProcessPayment implements ShouldQueue
{
public $tries = 3;
public $maxExceptions = 3;
public $backoff = [60, 300, 900]; // 1min, 5min, 15min
public function handle(): void
{
// Process payment
}
public function failed(Throwable $exception): void
{
// Move to dead letter queue for manual review
DeadLetterJob::dispatch([
'job' => self::class,
'payload' => $this->order->toArray(),
'exception' => $exception->getMessage(),
'failed_at' => now(),
]);
// Alert team
Notification::send(
User::admins()->get(),
new PaymentFailedNotification($this->order, $exception)
);
}
}
The exponential backoff array gives transient failures time to resolve before the final retry. The failed method ensures nothing falls through the cracks.
Dead Letter Processing
Dead letters aren't the end of the road. Build tooling to review, retry, or discard failed messages systematically.
This processor provides the operations you need for managing dead letters: review pending failures, retry fixable issues, and discard unfixable ones with documentation.
// Review and reprocess dead letters
class DeadLetterProcessor
{
public function review(): Collection
{
return DeadLetterJob::where('status', 'pending')
->orderBy('failed_at')
->get();
}
public function retry(DeadLetterJob $deadLetter): void
{
$jobClass = $deadLetter->job;
$payload = $deadLetter->payload;
// Recreate and dispatch original job
$job = new $jobClass(...$payload);
dispatch($job);
$deadLetter->update(['status' => 'retried', 'retried_at' => now()]);
}
public function discard(DeadLetterJob $deadLetter, string $reason): void
{
$deadLetter->update([
'status' => 'discarded',
'discard_reason' => $reason,
'discarded_at' => now(),
]);
}
}
Regular dead letter review should be part of your operations process. Patterns in failures often reveal systemic issues.
Idempotency
The Problem
Network failures can cause duplicate message delivery. Without idempotency, you risk processing the same message multiple times.
This timeline shows how duplicate delivery happens. The acknowledgment is lost, causing the message to be redelivered and processed again.
Message sent → Consumer processes → Ack lost → Message redelivered → Processed again!
Result: Double charge, duplicate email, wrong inventory count
Solutions
Use idempotency keys to detect and skip duplicate processing. The key should be unique to the logical operation, not the message delivery.
This implementation uses a cache-based idempotency key to prevent duplicate processing. The check happens before any side effects occur.
class ProcessPayment implements ShouldQueue
{
public function handle(): void
{
// Idempotency key based on order
$idempotencyKey = "payment:{$this->order->id}";
// Check if already processed
if (Cache::has($idempotencyKey)) {
Log::info('Payment already processed', ['order_id' => $this->order->id]);
return;
}
// Process payment
$result = $this->paymentGateway->charge($this->order);
// Mark as processed (with TTL longer than retry window)
Cache::put($idempotencyKey, $result->transaction_id, now()->addDays(7));
}
}
The cache TTL must be longer than your maximum retry period, or you could still get duplicates.
Database-Based Idempotency
For stronger guarantees, use database constraints. This approach survives cache failures and provides an audit trail.
This database-based approach uses a unique constraint to guarantee exactly-once processing even under concurrent execution.
class ProcessOrder implements ShouldQueue
{
public function handle(): void
{
// Use database transaction with unique constraint
DB::transaction(function () {
// Will fail if already exists
ProcessedMessage::create([
'message_id' => $this->messageId,
'processed_at' => now(),
]);
$this->processOrder();
});
}
}
The unique constraint on message_id ensures only one attempt succeeds, even with concurrent processing.
Ordering Guarantees
When Order Matters
Some operations must happen in sequence. Processing a shipment notification before a payment confirmation could result in invalid state.
This sequence shows events that must be processed in order. Processing them out of sequence would create an invalid system state.
// Events must be processed in order
OrderCreated { order_id: 123 }
PaymentReceived { order_id: 123 }
OrderShipped { order_id: 123 }
// If processed out of order:
OrderShipped before PaymentReceived = Invalid state!
Partition Keys
Use partition keys to ensure related messages go to the same consumer in order. All events for a specific entity follow the same path.
Both SQS FIFO queues and Kafka support partition keys. Use them to ensure all events for a specific entity are processed in order.
// SQS FIFO Queue
class OrderEvent implements ShouldQueue
{
public $queue = 'orders.fifo';
public function getMessageGroupId(): string
{
// All events for same order go to same partition
return "order:{$this->order->id}";
}
public function getMessageDeduplicationId(): string
{
return $this->eventId;
}
}
// Kafka
$producer->produce(
topic: 'orders',
key: $order->id, // Partition key ensures ordering per order
value: json_encode($event)
);
With partitioning, you get ordering within a partition while still allowing parallel processing across partitions.
Sequential Processing
When you can't use queue-level ordering, validate sequence at the application level.
This processor validates event sequences at the application level. Out-of-order events are requeued for later processing.
class OrderEventProcessor
{
public function handle(OrderEvent $event): void
{
// Verify event sequence
$lastSequence = Cache::get("order:{$event->orderId}:sequence", 0);
if ($event->sequence <= $lastSequence) {
Log::warning('Duplicate or out-of-order event', [
'expected' => $lastSequence + 1,
'received' => $event->sequence,
]);
return;
}
if ($event->sequence > $lastSequence + 1) {
// Missing events - requeue for later
throw new OutOfOrderException($event);
}
// Process in order
$this->processEvent($event);
Cache::put("order:{$event->orderId}:sequence", $event->sequence);
}
}
This approach handles out-of-order delivery by requeuing messages that arrive too early.
Delivery Guarantees
At-Most-Once
Acknowledge before processing for maximum speed, but accept that messages may be lost if processing fails.
At-most-once delivery prioritizes speed over reliability. Use it when occasional message loss is acceptable.
// Acknowledge before processing
// Fast but may lose messages
$message = $queue->receive();
$queue->acknowledge($message); // Ack first
$this->process($message); // May fail after ack
Use this when losing occasional messages is acceptable, like analytics or metrics.
At-Least-Once
Acknowledge after processing to ensure messages aren't lost. This requires idempotent consumers to handle redelivery.
At-least-once delivery ensures no messages are lost, but requires idempotent consumers to handle potential duplicates.
// Acknowledge after processing
// May duplicate but won't lose
$message = $queue->receive();
$this->process($message); // Process first
$queue->acknowledge($message); // Ack after success
// Requires idempotent consumers!
This is the most common choice for business-critical operations.
Exactly-Once (Transactional)
True exactly-once delivery requires the transactional outbox pattern, where message publishing and business logic share the same database transaction.
The transactional outbox pattern stores events in the same database transaction as your business logic. A separate publisher process then reliably delivers those events.
// Use transactional outbox pattern
class OrderService
{
public function createOrder(array $data): Order
{
return DB::transaction(function () use ($data) {
$order = Order::create($data);
// Store event in same transaction
OutboxMessage::create([
'aggregate_type' => 'Order',
'aggregate_id' => $order->id,
'event_type' => 'OrderCreated',
'payload' => json_encode($order->toArray()),
]);
return $order;
});
}
}
// Separate process publishes outbox messages
class OutboxPublisher
{
public function publish(): void
{
$messages = OutboxMessage::where('published_at', null)
->orderBy('created_at')
->limit(100)
->get();
foreach ($messages as $message) {
$this->queue->push($message->payload, $message->event_type);
$message->update(['published_at' => now()]);
}
}
}
The outbox pattern guarantees that if the order is created, the message will eventually be published. The publisher runs separately and handles delivery.
Backpressure
Handling Overload
When producers outpace consumers, queues grow unbounded. Implement backpressure to prevent system overload.
This rate-limited queue processor caps concurrent processing to prevent overwhelming downstream systems. It maintains a steady processing rate regardless of queue depth.
class RateLimitedQueue
{
private int $maxConcurrent = 10;
private int $currentProcessing = 0;
public function process(): void
{
while (true) {
if ($this->currentProcessing >= $this->maxConcurrent) {
usleep(100000); // Wait 100ms
continue;
}
$message = $this->queue->receive();
if (!$message) {
usleep(100000);
continue;
}
$this->currentProcessing++;
// Process async
go(function () use ($message) {
try {
$this->handleMessage($message);
$this->queue->acknowledge($message);
} finally {
$this->currentProcessing--;
}
});
}
}
}
The concurrent limit prevents workers from taking on more than they can handle.
Queue Depth Monitoring
Monitor queue depth to detect problems before they cascade. Auto-scaling workers based on queue depth keeps processing times consistent.
This monitoring class tracks queue depth and automatically scales workers. High queue depth triggers scale-up, while low depth triggers scale-down.
class QueueMonitor
{
public function check(): void
{
$depth = Queue::size('default');
if ($depth > 10000) {
Log::critical('Queue depth critical', ['depth' => $depth]);
$this->scaleWorkers(up: true);
} elseif ($depth > 1000) {
Log::warning('Queue depth high', ['depth' => $depth]);
} elseif ($depth < 100) {
$this->scaleWorkers(down: true);
}
}
}
Consider implementing circuit breakers on producers when queue depth gets critical, preventing the system from accepting more work than it can process.
Laravel Queue Configuration
Multiple Queues
Separate queues for different priorities ensure critical jobs don't wait behind bulk operations.
Configure multiple queue connections for different priority levels. Critical operations like payments get their own dedicated queue.
// config/queue.php
'connections' => [
'high-priority' => [
'driver' => 'redis',
'connection' => 'default',
'queue' => 'high',
],
'default' => [
'driver' => 'redis',
'connection' => 'default',
'queue' => 'default',
],
'low-priority' => [
'driver' => 'redis',
'connection' => 'default',
'queue' => 'low',
],
],
// Dispatch to specific queue
ProcessPayment::dispatch($order)->onQueue('high');
SendNewsletter::dispatch($users)->onQueue('low');
// Worker processes multiple queues with priority
// php artisan queue:work --queue=high,default,low
The queue order in the worker command determines priority. Jobs on high are always processed before default.
Horizon Configuration
Laravel Horizon provides a dashboard and sophisticated worker management. Configure supervisors per queue for fine-grained control.
This Horizon configuration creates dedicated worker pools for different queue types. Each supervisor can have its own process count and retry settings.
// config/horizon.php
'environments' => [
'production' => [
'supervisor-payments' => [
'connection' => 'redis',
'queue' => ['payments'],
'processes' => 10,
'tries' => 3,
],
'supervisor-emails' => [
'connection' => 'redis',
'queue' => ['emails'],
'processes' => 5,
'tries' => 3,
],
'supervisor-default' => [
'connection' => 'redis',
'queue' => ['default'],
'processes' => 3,
'tries' => 3,
],
],
],
Dedicated process pools for different queues let you allocate resources based on each queue's importance and throughput requirements.
Conclusion
Message queues enable resilient, scalable architectures by decoupling services and handling failures gracefully. Choose the right pattern for your use case: point-to-point for task distribution, pub/sub for event broadcasting. Implement dead letter queues for failed message handling, ensure idempotency for at-least-once delivery, and use partition keys when ordering matters. Monitor queue depth and implement backpressure to prevent system overload. The right queue architecture makes your system both more reliable and easier to scale.