The Bulkhead pattern isolates critical resources into separate pools to prevent cascade failures. Named after ship bulkheads that contain flooding to one section, this pattern ensures that failure in one part of the system doesn’t bring down the entire application.
Bulkhead Is About Isolation, Not Performance
The bulkhead pattern takes its name from ship design — compartmentalized hulls prevent a single breach from sinking the entire vessel. In software, bulkheads isolate resources (thread pools, connections, memory) so failure in one part of the system doesn’t cascade. The most common implementation is separate thread pools for different services or endpoints — if Service A’s pool is exhausted, Service B and C remain unaffected.
The key design question is how many bulkheads to create. Too many wastes resources; too few provides poor isolation. A typical split assigns one pool per downstream dependency, or one pool per endpoint category (read vs write, critical vs non-critical). Thread pool sizes should be based on each dependency’s latency profile — a fast API needs fewer threads than a slow one.
Bulkhead complements the circuit breaker: the breaker prevents calls to a failing service, while the bulkhead ensures those calls don’t starve other resources while waiting. Real-world examples include Hystrix thread-pools, Istio connection pools, and Kubernetes resource quotas (CPU/memory limits per container).
Understanding Bulkhead Pattern
The Problem Without Bulkheads
The bulkhead pattern is best understood by first examining the failure mode it prevents. The diagram below shows an application with a single thread pool shared by every request. All work — database queries, cache lookups, external API calls — flows through the same 100 threads. This setup is simple and cheap, which is why it is so common, but it has a critical weakness: the pool is a single point of contention. If any one downstream dependency becomes slow, the threads it occupies cannot be reused, and the entire pool can be consumed waiting on one service.
In the scenario illustrated, a slow external API takes thirty seconds or more to respond. Each request that touches that API occupies a thread for the full thirty seconds. At sufficient request volume, all one hundred threads end up parked on the slow API, and every new request is rejected — not because the application is overloaded in any real sense, but because a single dependency starved the shared pool. The consequence is total failure: the fast, healthy dependencies are just as unavailable as the slow one. This is the cascade failure that bulkheads are designed to prevent.
This single-pool design fails the fundamental isolation test. The blast radius of one slow dependency is the entire application. Users who never touch the slow API at all are still blocked, because their requests cannot acquire a thread. Monitoring makes the situation worse, because a single pool hides which dependency is at fault — the queue backs up globally and the operator has no visibility into the breakdown. The fix is not to make the slow API faster, but to give each dependency a private pool, so that a slow dependency can only consume its own allocation.
┌─────────────────────────────────────────────────────────────────┐
│ Single Resource Pool (No Isolation) │
│ │
│ ┌─────────────────────────────────────────────────────────┐ │
│ │ Application Thread Pool │ │
│ │ (100 threads total) │ │
│ │ │ │
│ │ ┌─────────┐ ┌─────────┐ ┌─────────┐ ┌─────────┐ │ │
│ │ │ Request │ │ Request │ │ Request │ │ Request │ ... │ │
│ │ │ 1 │ │ 2 │ │ 3 │ │ 4 │ │ │
│ │ └────┬────┘ └────┬────┘ └────┬────┘ └────┬────┘ │ │
│ │ │ │ │ │ │ │
│ │ └──────────┴──────────┴──────────┘ │ │
│ │ │ │ │
│ │ ▼ │ │
│ │ ┌────────────────────────────┐ │ │
│ │ │ Slow External API │ │ │
│ │ │ (takes 30+ seconds) │ │ │
│ │ └────────────────────────────┘ │ │
│ │ │ │ │
│ │ ◄────────────┴────────────┘ │ │
│ │ │ │ │
│ │ All 100 threads waiting! │ │
│ │ │ │
│ │ ✗ New requests rejected (pool exhausted) │ │
│ │ ✗ App becomes completely unresponsive │ │
│ │ ✗ Everything fails, not just the slow API calls │ │
│ └─────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────┘
With Bulkhead Isolation
The opposite design divides the application’s concurrency budget into separate pools, one per dependency. In the diagram, the API gets 20 threads, the database 30, and the cache 10. Each pool is an independent bulkhead — a watertight compartment in the ship analogy. When the external API slows down, only its 20-thread pool fills up. Those 20 threads are dedicated to API work and cannot be used by other dependencies anyway, so their exhaustion costs nothing except the API calls themselves. The database and cache pools, with 80 threads between them, continue to serve normally.
This isolation changes the failure profile fundamentally. The application as a whole remains responsive: requests that depend only on the database or cache succeed at full speed, and even API-dependent requests can fail fast with a clear “bulkhead full” rejection rather than waiting indefinitely. The blast radius of the slow API shrinks from “the whole application” to “the API calls alone.” Equally important, the failure is now visible and attributable — the API pool’s rejection counter climbs while every other pool stays flat, instantly identifying the culprit on any dashboard.
There is no free lunch: the total capacity is the same, and under true overload the isolated design rejects slightly more requests than the shared pool, because threads cannot be borrowed across bulkheads. That trade-off is the entire point. Bulkheads buy predictability — a bounded, attributable, graceful degradation — in exchange for a small efficiency loss. For most systems, especially those with one slow or flaky third-party dependency, that exchange is overwhelmingly favorable. The rest of this article shows how to implement bulkheads at three levels of isolation: threads, connections, and processes.
┌─────────────────────────────────────────────────────────────────┐
│ Bulkhead Pattern - Resource Isolation │
│ │
│ ┌──────────────┬──────────────┬──────────────┐ │
│ │ Thread Pool │ Thread Pool │ Thread Pool │ │
│ │ for API │ for DB │ for Cache │ │
│ │ (20 threads)│ (30 threads) │ (10 threads) │ │
│ └──────┬───────┴──────┬───────┴──────┬───────┘ │
│ │ │ │ │
│ ▼ ▼ ▼ │
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
│ │ External │ │ Database │ │ Redis │ │
│ │ API │ │ Connection │ │ Cache │ │
│ └──────────────┘ └──────────────┘ └──────────────┘ │
│ │ │
│ ▼ │
│ ┌──────────────┐ │
│ │ Slow API │ ← Only 20 threads affected │
│ │ (30+ secs) │ (20% of capacity) │
│ └──────────────┘ │
│ │ │
│ ▼ │
│ ✓ 80 threads still available │
│ ✓ DB and Cache operations continue │
│ ✓ App remains responsive │
└─────────────────────────────────────────────────────────────────┘
The two diagrams contrast the essence of the pattern: shared resources propagate failure, partitioned resources contain it. The numbers in the second diagram — 20 threads affected out of 100 — make the benefit concrete: only 20 percent of capacity is tied up in the failing dependency, and the remaining 80 percent keeps serving. This is the property to preserve when you design your own bulkheads: never let one dependency’s concurrency appetite exceed its own compartment. With that mental model in place, we can turn to the actual implementations.
Types of Bulkheads
Thread Pool Bulkhead
Thread pool bulkheads are the most common implementation, and the first code block shows a complete one. ThreadPoolBulkhead wraps a concurrent.futures.ThreadPoolExecutor with a bounded queue and a metrics object.
The queue is the crucial addition: it gives the bulkhead a defined capacity beyond the executor’s thread count, and — more importantly — a way to reject work cleanly when that capacity is exceeded. submit checks whether the queue is full before handing work to the executor; if it is, the method increments a rejection counter and raises BulkheadRejectedError instead of silently queueing work that will never run.
The design choices here reward attention.
The max_size parameter caps concurrent threads, while queue_size caps how many extra jobs can wait; together they form the bulkhead’s full admission boundary.
Using a Queue with a max size means rejection is proactive rather than reactive — the caller learns immediately that the bulkhead is at capacity instead of blocking or timing out later.
The name parameter threads through everything, from the thread name prefix (which shows up in profiling output and thread dumps) to the metrics object, making each bulkhead individually observable in production.
The class supports both synchronous submission through submit and async callers through submit_async, which offloads blocking work to the executor using run_in_executor.
This duality matters because a bulkhead is most valuable exactly where async code meets blocking dependencies — an asyncio event loop that runs blocking calls directly would freeze, so the bulkhead provides both isolation and a bridge to a thread pool.
The BulkheadMetrics class that follows uses a lock to keep its counters accurate under concurrent access, since the executor’s worker threads and the main thread all touch the same counters.
import concurrent.futures
from queue import Queue
import threading
class ThreadPoolBulkhead:
def __init__(
self,
name: str,
core_size: int = 10,
max_size: int = 20,
queue_size: int = 100,
keep_alive_seconds: int = 60
):
self.name = name
self.executor = ThreadPoolExecutor(
max_workers=max_size,
thread_name_prefix=f"bulkhead-{name}",
keep_alive_time=keep_alive_seconds
)
self.queue = Queue(maxsize=queue_size)
self._metrics = BulkheadMetrics(name)
def submit(self, fn, *args, **kwargs):
if self.queue.full():
self._metrics.increment_rejected()
raise BulkheadRejectedError(
f"Bulkhead {self.name} is at capacity"
)
future = self.executor.submit(fn, *args, **kwargs)
self._metrics.increment_submitted()
future.add_done_callback(
lambda f: self._metrics.increment_completed()
)
return future
async def submit_async(self, fn, *args, **kwargs):
loop = asyncio.get_event_loop()
if self.queue.full():
self._metrics.increment_rejected()
raise BulkheadRejectedError(
f"Bulkhead {self.name} is at capacity"
)
future = loop.run_in_executor(
self.executor,
lambda: fn(*args, **kwargs)
)
self._metrics.increment_submitted()
return await future
def get_metrics(self) -> dict:
return {
"name": self.name,
"submitted": self._metrics.submitted,
"completed": self._metrics.completed,
"rejected": self._metrics.rejected,
"active": self._metrics.active,
"queue_size": self.queue.qsize()
}
class BulkheadMetrics:
def __init__(self, name: str):
self.name = name
self.submitted = 0
self.completed = 0
self.rejected = 0
self.active = 0
self._lock = threading.Lock()
def increment_submitted(self):
with self._lock:
self.submitted += 1
self.active += 1
def increment_completed(self):
with self._lock:
self.completed += 1
self.active = max(0, self.active - 1)
def increment_rejected(self):
with self._lock:
self.rejected += 1
The rejection behavior is the feature that makes this a bulkhead rather than just a thread pool.
A raw executor will happily grow its queue unboundedly, and under a slow dependency that queue becomes a memory bomb that turns a 30-second outage into an OOM crash.
The bounded queue plus explicit BulkheadRejectedError converts that failure mode into a fast, catchable signal that the caller can handle — by failing fast, returning degraded results, or triggering a fallback.
That single design decision is why the metrics object records rejected as a first-class counter.
Connection Pool Bulkhead
Connections are a scarcer and more fragile resource than threads — a database can only hold a limited number of connections at once, and each one consumes memory and a server-side session.
The ConnectionPoolBulkhead class manages a pool of database or cache connections with a strict cap on how many can exist simultaneously.
The cap is enforced through an asyncio.Semaphore, initialized to max_connections, which gates _create_connection so that the pool can never overshoot its bound even when many coroutines request connections at once.
The acquire path shows the pool’s admission policy.
When no connection is free, the caller waits on a _waiters queue rather than spinning or failing immediately — up to acquire_timeout seconds.
If the wait exceeds the timeout, ConnectionPoolExhaustedError is raised, giving the caller a clear, catchable signal that this bulkhead is saturated.
On release, healthy connections are returned to the pool while unhealthy ones are discarded and rebuilt, and a waiter is woken to consume the freed slot.
This subtle detail — replacing bad connections instead of recycling them — is what keeps the pool from accumulating poisoned connections during an incident.
The ConnectionPoolManager that follows introduces the second half of the pattern’s value: organization.
Rather than scattering pools through the codebase, the manager owns a registry keyed by name, so db.primary, db.read, and cache.sessions are each first-class, inspectable pools.
Creating a pool is declarative — pass a name and a config dict — which makes the topology of isolation visible in one place.
This registry pattern also becomes the natural point to attach metrics, alerting, and per-pool health checks, as shown in the monitoring section later in this article.
import asyncio
from dataclasses import dataclass, field
from typing import Optional
@dataclass
class ConnectionPoolBulkhead:
host: str
port: int
min_connections: int = 5
max_connections: int = 50
max_waiters: int = 100
acquire_timeout: float = 30.0
idle_timeout: float = 300.0
_connections: list = field(default_factory=list, init=False)
_waiters: asyncio.Queue = field(default_factory=asyncio.Queue, init=False)
_lock: asyncio.Lock = field(default_factory=asyncio.Lock, init=False)
_semaphore: asyncio.Semaphore = field(init=False)
def __post_init__(self):
self._semaphore = asyncio.Semaphore(self.max_connections)
async def acquire(self) -> 'Connection':
if not self._connections:
await self._create_connection()
try:
async with asyncio.timeout(self.acquire_timeout):
while not self._connections:
await self._waiters.get()
conn = self._connections.pop()
return conn
except asyncio.TimeoutError:
raise ConnectionPoolExhaustedError(
f"Could not acquire connection within {self.acquire_timeout}s"
)
async def release(self, conn: 'Connection'):
if conn.is_healthy():
self._connections.append(conn)
else:
await self._create_connection()
if not self._waiters.empty():
self._waiters.put_nowait(True)
async def _create_connection(self):
async with self._semaphore:
conn = Connection(self.host, self.port)
await conn.connect()
return conn
async def close_all(self):
for conn in self._connections:
await conn.close()
self._connections.clear()
class ConnectionPoolManager:
def __init__(self):
self.pools: dict[str, ConnectionPoolBulkhead] = {}
def get_pool(self, name: str) -> ConnectionPoolBulkhead:
return self.pools.get(name)
def create_pool(
self,
name: str,
config: dict
) -> ConnectionPoolBulkhead:
pool = ConnectionPoolBulkhead(**config)
self.pools[name] = pool
return pool
The connection pool bulkhead demonstrates that isolation applies to any finite resource, not just threads.
The same skeleton — a semaphore for the hard limit, a wait queue for admission, a timeout to avoid unbounded waiting, and a health check on return — generalizes to HTTP client connection pools, message-broker sessions, and even file handles.
The important operational property is that each named pool fails independently: if the analytics database exhausts its pool, the checkout-database pool keeps working, and acquire_timeout guarantees the failure is bounded in time rather than a hang.
Process Isolation Bulkhead
Threads and connections isolate work at the application level, but a corrupted state, a segfault, or an unhandled native-library crash can still take down the whole process.
For genuinely dangerous work — running untrusted code, invoking third-party binaries, processing data that might trigger native crashes — the strongest form of isolation is a separate operating-system process. ProcessBulkhead implements this with multiprocessing.
The worker runs in its own memory space, so if it crashes, the main process and every other worker remain untouched.
The design mirrors the thread-pool bulkhead, but with process semantics. submit checks the count of running processes against max_processes and rejects with ProcessBulkheadExhaustedError when the cap is reached.
Accepted tasks are pushed onto a shared _work_queue; each worker pulls tasks in a loop, executes them, and pushes results onto _result_queue, with a None sentinel signaling shutdown.
The message-passing design is deliberate: workers communicate through queues rather than shared memory, which keeps them robustly isolated and avoids the corruption that motivates process isolation in the first place.
The trade-off here is the most severe of the three bulkhead types.
Process creation is orders of magnitude more expensive than thread creation, so max_processes is typically small — often 2 to 8.
Inter-process communication via queues adds serialization overhead and latency.
And each worker carries a full Python interpreter, inflating memory usage.
Process bulkheads are therefore reserved for the highest-risk work, where the isolation benefits outweigh the cost.
When you cannot trust the code, or the code can crash the runtime, a process boundary is the only isolation that is truly airtight.
import subprocess
import multiprocessing
from dataclasses import dataclass
@dataclass
class ProcessBulkhead:
name: str
max_processes: int = 4
worker_script: str = None
_processes: list = field(default_factory=list, init=False)
_work_queue: multiprocessing.Queue = field(init=False)
_result_queue: multiprocessing.Queue = field(init=False)
_lock: multiprocessing.Lock = field(init=False)
def __post_init__(self):
self._work_queue = multiprocessing.Queue()
self._result_queue = multiprocessing.Queue()
self._lock = multiprocessing.Lock()
def submit(self, task: dict) -> multiprocessing.Process:
if len(self._processes) >= self.max_processes:
raise ProcessBulkheadExhaustedError(
f"All {self.max_processes} processes busy"
)
process = multiprocessing.Process(
target=self._worker,
args=(self._work_queue, self._result_queue)
)
process.start()
self._processes.append(process)
self._work_queue.put(task)
return process
def _worker(self, work_queue, result_queue):
while True:
task = work_queue.get()
if task is None:
break
try:
result = self._execute_task(task)
result_queue.put({"success": True, "result": result})
except Exception as e:
result_queue.put({"success": False, "error": str(e)})
def _execute_task(self, task: dict) -> any:
# Execute task in isolated process
pass
Note the _execute_task stub — in a real system this is where the dangerous work happens, and the isolation guarantees apply regardless of what it does.
A crashed worker leaves a dead process in _processes that the parent must reap, so production implementations add liveness monitoring and automatic respawn, and they push a timeout per task so a hung worker can be killed rather than occupying a process slot forever.
Process bulkheads are the heavy artillery of the pattern: expensive, but the only option when corruption or a crash in the workload itself is the threat model.
Implementation Examples
Python Bulkhead with Resilience4j-style API
Production resilience libraries — most famously Netflix’s Hystrix and the JVM’s Resilience4j — expose bulkheads through a compact API: create a bulkhead with a max-concurrent-calls limit, then wrap a function so that every invocation goes through it.
The code below reproduces that API shape in Python.
The Bulkhead class centers on an asyncio.Semaphore initialized to max_concurrent_calls, which is the entire admission mechanism: when the semaphore is free, the call executes under its context manager; when it is fully locked, the call is rejected immediately with BulkheadFullError.
No waiting, no queue — the caller learns instantly that this compartment is full.
The design decision worth noting is the rejection policy: max_waiting_threads is declared as a parameter but the implementation rejects rather than queues.
This is a deliberate simplification of the Resilience4j semantics, where maxWaitDuration allows bounded waiting.
Rejecting immediately has a clear virtue — it bounds the time-to-failure at zero, which makes the bulkhead extremely predictable under load — but it means every rejection is an error the caller must handle.
The permitted_number_of_calls and sliding_window_size parameters anticipate a more advanced variant that tracks calls over a rolling window to smooth rejection decisions.
The bulkhead decorator makes the API ergonomic.
It detects at decoration time whether the target is a coroutine and wraps accordingly, returning an async wrapper for async def functions and a sync wrapper that pumps an event loop for plain functions.
The usage example at the bottom shows the payoff: two bulkheads, one for the external API and one for the database, declared once and applied with a single annotation per method.
From here on, every call to call_api or query passes through its bulkhead, and the concurrency limits become a documented, enforced property of the class rather than an assumption.
import time
import asyncio
from functools import wraps
from typing import Callable, TypeVar, ParamSpec
P = ParamSpec('P')
T = TypeVar('T')
class Bulkhead:
def __init__(
self,
max_concurrent_calls: int = 100,
max_waiting_threads: int = 50,
permitted_number_of_calls: int = 1000,
sliding_window_size: int = 100
):
self.max_concurrent_calls = max_concurrent_calls
self.max_waiting_threads = max_waiting_threads
self.semaphore = asyncio.Semaphore(max_concurrent_calls)
self.metrics = BulkheadMetrics(sliding_window_size)
async def execute(self, fn: Callable[P, T], *args: P.args, **kwargs: P.kwargs) -> T:
if not self.semaphore.locked():
async with self.semaphore:
self.metrics.record_start()
start_time = time.time()
try:
result = await fn(*args, **kwargs)
self.metrics.record_success(time.time() - start_time)
return result
except Exception as e:
self.metrics.record_failure(e)
raise
else:
self.metrics.record_rejected()
raise BulkheadFullError(
f"Bulkhead full: {self.max_concurrent_calls} concurrent calls"
)
def bulkhead(bulkhead_instance: Bulkhead):
def decorator(fn: Callable) -> Callable:
@wraps(fn)
async def async_wrapper(*args, **kwargs):
return await bulkhead_instance.execute(fn, *args, **kwargs)
@wraps(fn)
def sync_wrapper(*args, **kwargs):
loop = asyncio.get_event_loop()
return loop.run_until_complete(
bulkhead_instance.execute(fn, *args, **kwargs)
)
if asyncio.iscoroutinefunction(fn):
return async_wrapper
return sync_wrapper
return decorator
# Usage
api_bulkhead = Bulkhead(max_concurrent_calls=50)
db_bulkhead = Bulkhead(max_concurrent_calls=30)
class ExternalAPIClient:
@bulkhead(api_bulkhead)
async def call_api(self, endpoint: str):
async with aiohttp.ClientSession() as session:
async with session.get(endpoint) as resp:
return await resp.json()
class DatabaseClient:
@bulkhead(db_bulkhead)
async def query(self, sql: str):
async with pool.acquire() as conn:
return await conn.fetch(sql)
The key takeaway is how small the bulkhead’s core really is: a semaphore, a rejection branch, and a metrics hook.
Everything else — decorators, config parameters, window tracking — is ergonomics on top of that core.
This is why bulkheads compose so well with other patterns: the same Bulkhead object can sit underneath a circuit breaker, a retry layer, or a timeout, because it only ever answers one question: “is there room to run this call right now?” When you design your own bulkhead, keep that single responsibility in mind — it is the difference between a tool and a framework.
Bulkhead with Fallback
A bulkhead that rejects calls when full is only half the story; the other half is what happens to the rejected call.
Fail fast is correct, but failing with a usable result is better. BulkheadWithFallback wraps the base bulkhead and intercepts its rejections.
When the bulkhead raises BulkheadFullError, the wrapper invokes a fallback callable instead of propagating the error.
The fallback runs outside the bulkhead’s admission gate — it is deliberately not subject to the same concurrency cap, because a fallback is supposed to be cheap and near-guaranteed to run.
The example at the bottom shows the pattern’s practical value. get_user_with_fallback attempts to fetch a user from the database through its bulkhead; if the database bulkhead is full, it falls back to reading from Redis, the cache.
This is graceful degradation in the classic sense: the user still gets a response, only from a faster, more available source, and the code logs that the fallback was used so operators can see the degradation happening.
The fallback signature even accepts the error object, so the cache path can log why it was invoked.
Two design rules make fallbacks safe. First, the fallback must be dramatically cheaper and more available than the primary — if the fallback is as expensive as the failing call, the bulkhead is just relocating the load. Second, the fallback must not require the resource that is exhausted; falling back from a full database pool to a full cache pool buys nothing. When those rules hold, fallbacks turn a hard failure into a soft one, and the metrics from the earlier sections reveal which bulkhead is degrading, enabling proactive fixing rather than reactive incident response.
class BulkheadWithFallback:
def __init__(self, bulkhead: Bulkhead):
self.bulkhead = bulkhead
async def execute_with_fallback(
self,
fn: Callable,
fallback: Callable,
*args, **kwargs
):
try:
return await self.bulkhead.execute(fn, *args, **kwargs)
except BulkheadFullError:
if fallback:
return await fallback(*args, **kwargs)
raise
except Exception as e:
if fallback:
return await fallback(*args, **kwargs, error=e)
raise
# Example usage with fallback
async def get_user_with_fallback(user_id: str):
bulkhead_exec = BulkheadWithFallback(user_bulkhead)
async def fetch_from_db():
return await db.fetch("SELECT * FROM users WHERE id = ?", user_id)
async def fetch_from_cache(error=None):
cache_key = f"user:{user_id}"
cached = await redis.get(cache_key)
if cached:
logger.warning(f"Fallback used for user {user_id}")
return json.loads(cached)
return None
return await bulkhead_exec.execute_with_fallback(
fetch_from_db,
fetch_from_cache
)
Notice the symmetry with the circuit breaker fallback shown later in the Java example: every resilience pattern that rejects work should also offer a degraded path, and the fallback hierarchy should be monotonic — cache before database, degraded-but-real response before an error. The logging in this example is not incidental; it is the observability hook that tells you the fallback path is actually being used, which is the first sign that a bulkhead needs resizing. If rejections and fallbacks become routine rather than exceptional, the correct response is to right-size the pool, not to accept degradation as normal.
Spring Boot Bulkhead (Java)
On the JVM, Resilience4j is the de facto standard for resilience patterns, and this example shows how a bulkhead composes with a circuit breaker through annotations. ExternalApiService.callExternalApi carries two annotations: @Bulkhead limits concurrent calls to the external API, and @CircuitBreaker stops calls entirely once the failure threshold is reached.
Each annotation names its own fallback method.
When the bulkhead rejects a call, fallback runs; when the circuit is open, circuitFallback runs.
The method signatures matter: a fallback must accept the original request plus an exception parameter, which Resilience4j injects by type.
The @Configuration block is where the bulkhead is sized. BulkheadConfigCustom declares maxConcurrentCalls(50) and maxWaitDuration(500ms) — the JVM version supports bounded waiting, so callers can spend up to half a second waiting for a slot before being rejected.
Setting this to a small value keeps time-to-failure predictable; setting it high lets short bursts ride out a brief spike.
The fallbacks return a DEGRADED or UNAVAILABLE response rather than throwing, which is a deliberate API contract: downstream consumers can handle a degraded-but-valid response far more gracefully than an exception.
This example is instructive for any language because it shows the composition pattern in its final form. The request passes through the circuit breaker guard first (is this dependency worth calling at all?), then the bulkhead (is there capacity for this call right now?), and only then the actual call, with fallbacks available at each layer. The ordering is not arbitrary — fail-fast guards sit outside capacity checks so the cheap rejections happen first. The same layered composition appears in the Python implementations in this article, confirming that the pattern is language-independent.
import io.github.resilience4j.bulkhead.annotation.Bulkhead;
import io.github.resilience4j.circuitbreaker.annotation.CircuitBreaker;
@Service
public class ExternalApiService {
@Bulkhead(name = "externalApiBulkhead", fallbackMethod = "fallback")
@CircuitBreaker(name = "externalApiCircuitBreaker", fallbackMethod = "circuitFallback")
public ExternalResponse callExternalApi(Request request) {
return externalApiClient.call(request);
}
private ExternalResponse fallback(Request request, BulkheadFullException ex) {
log.warn("Bulkhead full for API call, returning fallback");
return ExternalResponse.builder()
.status("DEGRADED")
.message("Service temporarily busy")
.fallback(true)
.build();
}
private ExternalResponse circuitFallback(Request request, Exception ex) {
log.warn("Circuit breaker open, returning circuit fallback");
return ExternalResponse.builder()
.status("UNAVAILABLE")
.message("Service unavailable")
.fallback(true)
.build();
}
}
// Configuration
@Configuration
public class BulkheadConfig {
@Bean
public BulkheadRegistry bulkheadRegistry() {
return BulkheadConfigCustom.of(
Map.of(
"externalApiBulkhead", BulkheadConfig.custom()
.maxConcurrentCalls(50)
.maxWaitDuration(Duration.ofMillis(500))
.fallbackDisabled(false)
.build()
)
);
}
}
The annotation style trades explicitness for concision: the behavior is declared at the method, which keeps call sites readable, but the actual limits live in configuration, so teams must keep the two in sync. Resilience4j exposes per-bulkhead metrics out of the box (available calls, max available, wait duration), which plugs directly into the monitoring section below. Whether you implement bulkheads with annotations or with explicit wrappers, the Java example’s most transferable lesson is the layering: breaker outside, bulkhead inside, fallback at the end.
Monitoring Bulkheads
Metrics Collection
None of the isolation guarantees are useful if you cannot see them working, and a bulkhead that is always full — or always empty — is a sign of misconfiguration that monitoring will expose instantly. BulkheadMonitor pushes per-bulkhead metrics to a StatsD-compatible client: gauges for available slots and active calls, counters for total calls and rejections.
The per-bulkhead keying is the entire point; the metric name embeds the bulkhead name, so bulkhead.db.available and bulkhead.api.available are distinct series that can be compared side by side.
The Prometheus definitions that follow encode the same observability in the native Prometheus vocabulary. bulkhead_calls_total is a Counter labeled by bulkhead and status, which supports rate queries like rejections per minute. bulkhead_available_slots is a Gauge, the instantaneous headroom of each pool. bulkhead_wait_seconds is a Histogram with buckets from 10 milliseconds to 5 seconds, capturing how long callers wait for a slot — a histogram, not a gauge, because wait times need percentiles (p50, p95, p99) to be meaningful.
Together these three metric types cover level, rate, and distribution.
The choice of what to measure is itself a design decision. Availability gauges tell you the current pressure; call and rejection counters tell you the rate of change; wait-time histograms tell you whether admission is becoming the bottleneck. The rejections ratio — rejections divided by calls — is the single most useful derived metric, because it normalizes for traffic and immediately flags a bulkhead that is sized too small. Most teams alert on this ratio crossing a few percent, which indicates either a dependency regression or a pool that needs resizing, before the pool fully saturates.
class BulkheadMonitor:
def __init__(self, statsd_client: StatsD):
self.statsd = statsd_client
self.bulkheads: dict[str, Bulkhead] = {}
def register_bulkhead(self, name: str, bulkhead: Bulkhead):
self.bulkheads[name] = bulkhead
async def collect_metrics(self):
for name, bulkhead in self.bulkheads.items():
metrics = bulkhead.get_metrics()
self.statsd.gauge(
f"bulkhead.{name}.available",
metrics["available"]
)
self.statsd.gauge(
f"bulkhead.{name}.active",
metrics["active"]
)
self.statsd.increment(
f"bulkhead.{name}.calls",
metrics["calls"]
)
self.statsd.increment(
f"bulkhead.{name}.rejections",
metrics["rejections"]
)
# Prometheus metrics
from prometheus_client import Counter, Gauge, Histogram
BULKHEAD_CALLS = Counter(
'bulkhead_calls_total',
'Total bulkhead calls',
['bulkhead_name', 'status']
)
BULKHEAD_AVAILABLE = Gauge(
'bulkhead_available_slots',
'Available bulkhead slots',
['bulkhead_name']
)
BULKHEAD_WAIT_TIME = Histogram(
'bulkhead_wait_seconds',
'Time waiting for bulkhead',
['bulkhead_name'],
buckets=[0.01, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0]
)
A useful refinement is to enrich each metric with the downstream dependency’s health — for example, tagging API-call metrics with the upstream service’s circuit-breaker state. This correlation lets you distinguish “the bulkhead is full because we under-provisioned” from “the bulkhead is full because the downstream is failing,” which have very different remediation. Whatever the sink — StatsD, Prometheus, or a cloud metrics service — the discipline is identical: every bulkhead emits its own series, rejections are counted as first-class events, and wait times are captured as distributions rather than averages.
Alerting Rules
Metrics only help if they trigger action, and the Prometheus alerting rules in this block convert the metrics into actionable pages.
The first alert, BulkheadHighRejectionRate, fires when the rejections-per-call ratio exceeds 10 percent over a five-minute window, sustained for two minutes.
This is the early-warning alert: the bulkhead is not yet exhausted, but the trend says it is heading there.
A ten-percent threshold is deliberately conservative — it catches problems while there is still headroom to react, rather than paging everyone only when the pool is already full.
The second alert, BulkheadAlmostFull, is the critical escalation: it fires when available slots drop below ten percent of capacity, indicating that the pool is nearly exhausted and rejection is imminent.
The thresholds are separated by severity — warning for the rate-based alert, critical for the capacity alert — so the on-call rotation sees a graduated signal instead of a single binary page.
The alert expressions reference the metric names established in the previous section, which is why consistent naming between collection and alerting is a hard requirement rather than a nicety.
Both rules embed two good alerting practices.
First, they aggregate with by (bulkhead_name), preserving the per-bulkhead attribution that makes the alert actionable — the page names the exact dependency at fault.
Second, they require sustained conditions (for: 2m, for: 1m) before firing, which filters out transient blips that would otherwise page people for noise.
A bulkhead that occasionally dips under load is normal; a bulkhead that stays exhausted is an incident.
The for clause is what distinguishes those two cases.
# Prometheus alerting rules
groups:
- name: bulkhead
rules:
- alert: BulkheadHighRejectionRate
expr: |
sum(rate(bulkhead_rejections_total[5m])) by (bulkhead_name)
/ sum(rate(bulkhead_calls_total[5m])) by (bulkhead_name) > 0.1
for: 2m
labels:
severity: warning
annotations:
summary: "High rejection rate on {{ $labels.bulkhead_name }}"
description: "Bulkhead {{ $labels.bulkhead_name }} rejecting >10% of calls"
- alert: BulkheadAlmostFull
expr: |
bulkhead_available_slots / bulkhead_max_slots < 0.1
for: 1m
labels:
severity: critical
annotations:
summary: "Bulkhead {{ $labels.bulkhead_name }} almost exhausted"
When you deploy these rules, tune the thresholds to your traffic shape rather than copying them verbatim.
A bursty service may legitimately saturate a bulkhead for seconds at a time, so the rejection-rate alert’s for window should exceed the longest normal burst.
Conversely, a steady-state service should never see double-digit rejection rates, so a 10 percent threshold may be too loose.
The ultimate goal of the alert set is to fire before users are affected — rejection alerts that page after the pool is exhausted have missed the point of monitoring.
Best Practices
Good Patterns
The final section distills everything into a checklist of good and bad patterns, and the first block captures the good side. The central rule is right-sizing each bulkhead to its resource type. The comment block gives the canonical guidance: API calls are I/O-bound and can tolerate 20-50 threads because they spend most of their time waiting on the network; database pools are limited by server-side connections, so 10-30 is typical; cache connections sit in the 20-50 range; file I/O is often disk-bound and needs only 5-10 threads. The sizing logic is the same in every case: estimate the dependency’s latency and throughput, then set the pool so that the dependency, not the pool, is the bottleneck.
The second good pattern is per-bulkhead monitoring.
Every bulkhead tracks available slots, rejection counts, and wait times as its own series, and alerts are scoped to individual pools rather than the application as a whole.
This is what makes the bulkhead pattern observable: when a dependency degrades, the dashboard lights up in exactly one column.
The third pattern — combining with a circuit breaker — closes the loop described throughout this article.
The progression in the comment (bulkhead.full -> circuit.partial_open -> circuit.open) reflects the escalation from momentary saturation to sustained failure: the bulkhead contains the spike, and the breaker stops the bleeding when the spike turns into an outage.
These three patterns are mutually reinforcing. Right-sizing reduces the false rejections that would otherwise make the bulkhead feel like a failure. Monitoring makes the sizing decisions data-driven rather than guessed. The circuit breaker partnership ensures that even a correctly sized bulkhead is not left to absorb a truly dead dependency. If you implement nothing else from this article, implement these three — they form a complete, minimal bulkhead posture that will survive most incident scenarios.
GOOD_PATTERNS = {
"size_bulkheads_appropriately": """
# Right-size bulkheads based on resource needs
✅ Good:
- API calls: 20-50 threads (I/O bound, can handle more)
- Database: 10-30 connections (connection limited)
- Cache: 20-50 connections
- File I/O: 5-10 threads (often disk bound)
❌ Bad:
- Same pool size for all resources
- Over-provisioning (wastes resources)
- Under-provisioning (too many rejections)
""",
"monitor_per_bulkhead": """
# Monitor each bulkhead individually
✅ Good:
- Track available slots per bulkhead
- Alert on rejection thresholds
- Monitor wait times per resource type
- Correlate with downstream service health
❌ Bad:
- Only monitoring total app threads
- No visibility into which resource is failing
- Same alert for all bulkheads
""",
"combine_with_circuit_breaker": """
# Use bulkhead with circuit breaker
✅ Good:
# If bulkhead is full for extended time
# Open circuit to fail fast
bulkhead.full → circuit.partial_open → circuit.open
❌ Bad:
# Bulkhead fills up but circuit stays closed
# Continues accepting requests that will be rejected
"""
}
Notice that the good patterns are stated as invariants to hold, not recipes to follow: “track available slots per bulkhead,” “alert on rejection thresholds,” “open the circuit when the bulkhead stays full.” Each one is checkable in a code review and each maps to a concrete metric from the monitoring section. A good test of whether your bulkhead implementation follows these patterns is to simulate a slow dependency in staging and ask: does only the affected pool fill up, does an alert fire before the pool exhausts, and does the circuit breaker eventually take over? If the answer to any question is no, the pattern is only partially implemented.
Bad Patterns
The bad patterns block shows the failure modes that the good patterns exist to prevent.
The first, single_pool_for_all, is the exact anti-pattern from the opening diagram: one ThreadPoolExecutor shared by every dependency.
The illustration shows why it is insidious — a request that calls both the database and the API submits both to the same pool, so when the API is slow, the database queries line up behind it.
The correction is the three-pool design with dedicated pools for API, database, and cache, so the database work is never hostage to the API’s latency.
The second bad pattern is the absence of a fallback. When a bulkhead rejects a call and the caller has no fallback, the client receives a hard error at the exact moment the system is under the most stress. The good version routes the rejection to a degraded source — in the example, reading from cache when the primary source is full. The distinction is between “graceful degradation” and “propagated failure”; both are honest, but only one keeps users served during an incident. The third bad pattern is creating bulkheads and never monitoring them, which is arguably worse than not creating them at all — it breeds false confidence while the pools silently saturate.
All three bad patterns share a common root cause: treating the bulkhead as a point solution rather than a system. A pool without a fallback is a door that slams shut with no alternative route. A pool without monitoring is a fire alarm with the sound muted. A shared pool is no pool at all. The good-versus-bad framing in this block is designed to be auditable — each bad example shows the naive code, explains why it fails, and presents the corrected alternative, so a code reviewer can hold implementations to the standard directly.
BAD_PATTERNS = {
"single_pool_for_all": """
❌ Bad:
# One thread pool for entire application
pool = ThreadPoolExecutor(max_workers=100)
async def handle_request(req):
db_result = await pool.submit(db_query, req) # Blocks
api_result = await pool.submit(call_api, req) # Also blocks
# If API is slow, DB queries also wait!
✅ Good:
# Separate pools for different resource types
api_pool = ThreadPoolExecutor(max_workers=20)
db_pool = ThreadPoolExecutor(max_workers=30)
cache_pool = ThreadPoolExecutor(max_workers=10)
async def handle_request(req):
db_result = await db_pool.submit(db_query, req)
api_result = await api_pool.submit(call_api, req)
""",
"no_fallback_strategy": """
❌ Bad:
# No fallback when bulkhead rejects
async def get_data(req):
return await bulkhead.execute(expensive_call)
# Client gets 500 error on rejection
✅ Good:
# Fallback to degraded mode
async def get_data(req):
try:
return await bulkhead.execute(expensive_call)
except BulkheadFullError:
return await get_from_cache(req) # Graceful degradation
""",
"ignoring_bulkhead_metrics": """
❌ Bad:
# Create bulkheads but don't monitor them
api_pool = ThreadPoolExecutor(max_workers=50)
# No metrics collection
✅ Good:
# Monitor and alert on bulkhead health
async def monitor_pools():
for name, pool in pools.items():
queue_size = pool._work_queue.qsize()
active = pool._active_count
if queue_size > threshold:
alert(f"{name} queue growing: {queue_size}")
"""
}
The most reliable way to catch these bad patterns in your own codebase is to review where ThreadPoolExecutor, asyncio.Semaphore, or connection pools are created: every one of those is a potential bulkhead boundary.
Ask whether it is shared across dependencies, whether its exhaustion has a catchable consequence, and whether it is emitting metrics.
If a pool exists for a single dependency, has a bounded queue, and has a rejection path that the caller handles, you have a bulkhead.
If any of those is missing, you have a shared resource waiting to become the next cascade failure.
Comparing Bulkhead with Other Patterns
Bulkheads do not operate in isolation — they are one member of a family of resilience patterns that each answer a different question about a failing dependency. The comparison table in the diagram makes the division of labor explicit. A timeout answers “how long is too long?” and kills calls that exceed a bound. A bulkhead answers “how much concurrency is too much?” and rejects calls once a pool is full. A circuit breaker answers “when is this dependency simply broken?” and stops calling it after a failure threshold. Each pattern bounds a different resource: time, concurrency, and failures respectively.
The diagram’s bottom half shows how the three compose on a single request path. The request first hits the timeout guard — the cheapest check, purely time-based. It then passes through the bulkhead, which enforces the concurrency limit. Finally, the circuit breaker decides whether the dependency is healthy enough to be worth calling at all. The ordering reflects cost and intent: cheap guards run first, and the fail-fast decisions are made before the expensive work of actually issuing the call. When a dependency degrades, all three engage in sequence — the timeout fires early for stragglers, the bulkhead rejects when the pool fills, and the breaker stops the traffic entirely once the failure rate confirms the dependency is down.
The practical lesson is that these patterns should be deployed together rather than chosen among. A timeout without a bulkhead still allows a slow dependency to consume unlimited concurrency. A bulkhead without a timeout lets rejected calls hang while waiters accumulate. A circuit breaker without the other two flips open and closed in a noisy loop. The resilient reference stack — timeout, bulkhead, circuit breaker, plus the retry-with-backoff pattern from its companion article — is what turns each individual guard into a coherent defense. Which one dominates in your system depends on your failure mode, but the full stack is what withstands the realistic ones.
┌─────────────────────────────────────────────────────────────────┐
│ Bulkhead vs Circuit Breaker vs Timeout │
├─────────────────────────────────────────────────────────────────┤
│ │
│ Pattern │ Purpose │ Trigger │
│ ─────────────────┼──────────────────────┼─────────────────────│
│ Timeout │ Limit wait time │ Time exceeded │
│ Bulkhead │ Limit concurrent │ Pool full │
│ Circuit Breaker │ Stop calling failing │ Failure threshold │
│ │ service │ exceeded │
│ │
│ ┌──────────────────────────────────────────────────────────┐ │
│ │ Often Used Together │ │
│ │ │ │
│ │ Request ──► [Timeout] ──► [Bulkhead] ──► [Circuit] │ │
│ │ │ │ │ │ │
│ │ Kill if too Limit Stop if │ │
│ │ long concurrent broken │ │
│ │ calls service │ │
│ └──────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────┘
Related Articles
Summary
The Bulkhead pattern provides resource isolation to prevent cascade failures:
- Thread Pool Bulkhead - Limits concurrent executions for different operations
- Connection Pool Bulkhead - Isolates database/cache connections
- Process Bulkhead - Complete process isolation for dangerous operations
Key benefits:
- Prevents one slow service from consuming all resources
- Provides graceful degradation when limits are reached
- Enables targeted monitoring and alerting
- Works well with Circuit Breaker and Timeout patterns
Choose isolation granularity based on:
- Resource characteristics (I/O bound vs CPU bound)
- Failure blast radius (what should stay available)
- Operational overhead (more pools = more complexity)
- Cost of unavailability (critical paths need isolation)
Comments