On a ship, a bulkhead is a wall that divides the hull into watertight compartments. If one compartment floods, the others stay dry, and the ship stays afloat. The bulkhead pattern in software does the same thing: it isolates resources so that when one part of the system gets into trouble, it cannot drown the rest.
The pattern shows up most often in distributed systems, where the failure mode you fear is not a clean crash but a slow leak — a backend dependency that gets slow, takes longer to respond, and eventually consumes every thread, connection, or worker your application has.
The Failure Mode
A typical Laravel application running on PHP-FPM has, say, 50 worker processes. Each can handle one request at a time. If 50 requests come in for a normal endpoint that returns in 100 ms, you serve them all in 100 ms.
Now imagine one of your endpoints calls an external recommendation service that is suddenly slow — taking 5 seconds per request instead of 100 ms. Traffic on that endpoint pins workers for 5 seconds each. Within seconds, all 50 workers are stuck waiting on the recommendation service. Every other endpoint — including the homepage and the login form — starts queuing because there are no workers free.
One slow dependency took down the entire application. There was no bug in your code. Your code just shared the same worker pool with the slow dependency.
The Pattern
The bulkhead solution is to give the troublesome dependency its own pool of resources, capped at a size that protects the rest of the system. If the recommendation service is slow, only the bounded number of workers assigned to it can be stuck — the rest stay free.
Without bulkhead:
[worker pool: 50] → all endpoints share, one slow dep takes all 50
With bulkhead:
[worker pool: 40] → fast endpoints
[worker pool: 10] → recommendation service (capped)
If the recommendation pool fills up, new requests to it fail fast (or queue with a short timeout). The other 40 workers keep serving traffic.
Implementations
The pattern can be applied at several layers, and most production systems use it at more than one.
Separate thread or worker pools. In a language with explicit thread pools (Java, .NET), you can create a dedicated executor for each downstream dependency. In PHP, the equivalent is running separate processes — for example, a dedicated PHP-FPM pool for endpoints that hit slow dependencies.
; /etc/php/8.2/fpm/pool.d/recommendations.conf
[recommendations]
listen = /run/php/recommendations.sock
pm = static
pm.max_children = 10
Nginx routes recommendation endpoints to that socket; everything else goes to the main pool. The recommendation pool can saturate without affecting general traffic.
Connection pool partitioning. Database and HTTP client connection pools should be capped per dependency, not pooled globally. If you have a 100-connection database pool and one slow query pattern takes 30 connections to run, you do not want it to consume the other 70.
// Guzzle client with per-service connection limits
$slowService = new Client([
'base_uri' => 'https://slow-api.example.com',
'timeout' => 2.0,
RequestOptions::CONNECT_TIMEOUT => 1.0,
RequestOptions::POOL_LIMIT => 10, // bulkhead
]);
$fastService = new Client([
'base_uri' => 'https://fast-api.example.com',
RequestOptions::POOL_LIMIT => 50,
]);
Queue partitioning. If you process background jobs through a queue, give slow or unreliable job types their own queues with their own workers. A single misbehaving job class should not back up the rest of the queue.
// Horizon configuration
'environments' => [
'production' => [
'general' => ['connection' => 'redis', 'queue' => ['default'], 'processes' => 20],
'webhooks' => ['connection' => 'redis', 'queue' => ['webhooks'], 'processes' => 5],
'reports' => ['connection' => 'redis', 'queue' => ['reports'], 'processes' => 3],
],
],
Webhooks to flaky third parties get 5 dedicated workers. If those workers all stall on slow webhook receivers, the 20 general workers keep processing everything else.
Bulkheads and Circuit Breakers
The bulkhead pattern caps how much harm a slow dependency can do. A circuit breaker stops sending traffic to a dependency that is clearly failing. They solve different problems and work well together.
Without a circuit breaker, every request to a dead service still takes up a bulkheaded worker until it times out. With a circuit breaker, the worker fails immediately and is free to serve a different request.
$circuit = new CircuitBreaker(
failureThreshold: 5,
timeout: 30, // seconds
);
if ($circuit->isOpen()) {
return $this->fallback();
}
try {
$result = $this->slowClient->get('/recommendations');
$circuit->recordSuccess();
return $result;
} catch (Throwable $e) {
$circuit->recordFailure();
return $this->fallback();
}
Bulkhead first, circuit breaker on top. The bulkhead bounds the worst case, the circuit breaker reduces wasted work when the worst case is happening.
Choosing the Cap
Picking the size of the bulkheaded pool is the hard part. Too small and the dependency is a bottleneck even when healthy. Too large and the bulkhead is doing no work.
The starting point is the throughput the dependency normally needs, plus headroom for spikes. If the recommendation service averages 30 requests per second with 100 ms latency, that is 3 in-flight requests on average. A pool of 8–10 gives 3x headroom without putting the dependency in a position to consume everything.
Watch the pool's saturation as a first-class metric. If it stays under 30% utilized, you can shrink. If it spikes to 100% during normal traffic, the dependency is your problem and a bigger pool will not save you — the bulkhead is just hiding it.
When to Apply
The pattern is worth the effort when:
- You have one or more external dependencies whose latency you do not control
- A single slow dependency can plausibly take down your application
- You can identify clear traffic classes that should be isolated from each other
If your application talks only to its own database, bulkheads at the database connection level are usually enough. If you call ten external APIs of varying reliability, treat each one as a separate compartment.
Looking at an outage that started with a slow dependency and ended with everything down? We help teams design boundaries that contain blast radius. scopeforged.com
