The Retry Pattern is a fundamental resilience pattern that handles transient failures by automatically retrying failed operations. When combined with exponential backoff and jitter, it becomes a powerful tool for building robust distributed systems.
When to Retry and When to Fail
Not every failure should be retried. The key distinction hinges on whether the failure is transient or permanent. Transient failures — such as network timeouts, database deadlocks, 503 Service Unavailable, or 429 Rate Limited — are retry-worthy because the underlying condition may resolve on its own. Permanent failures — including 400 Bad Request, 401 Unauthorized, 403 Forbidden, 404 Not Found, and 422 Validation Error — should never be retried, as the same request will fail identically.
Exponential backoff is the standard approach: first retry after 100ms, then 200ms, 400ms, 800ms, and so on. Adding jitter prevents the thundering herd problem, where all retrying clients synchronize and hit the server simultaneously. Implement a retry budget that limits both total retry time (e.g., max 30 seconds) and the number of retries (e.g., 3-5 attempts). Integrate with a circuit breaker: if retries keep failing, open the circuit and fail fast. The most dangerous pattern is infinite retries with no backoff — this is a self-inflicted DDoS. Real-world SDKs such as the AWS SDKs implement exponential backoff with jitter by default.
Understanding the Retry Pattern
Why Retries Matter
Understanding why retries work begins with recognizing how often failures are actually transient. Studies across cloud providers consistently report that well over half of all failed requests stem from conditions that clear up within seconds: a dropped TCP packet, a server being rolled for a deploy, a database lock that another transaction is about to release. These failures are not bugs in your code, and they will not be fixed by changing your request. The only sensible reaction is to issue the call again, after a short pause, and let the system settle. Without a retry mechanism, every one of these recoverable blips becomes a hard user-visible error, inflating your error budget and degrading the experience for everyone.
A second reason retries matter is operational. An on-call engineer paged about “500 errors” often discovers the spike was caused by a single dependency that recovered before anyone could react. Retries with backoff smooth over these brief windows automatically, converting a potential incident into a silent recovery. They also protect your own infrastructure: by waiting progressively longer between attempts, you give the struggling service room to drain its queue, clear its backlog, and return to a healthy state. This is why the retry pattern is a pillar of the resilience patterns family alongside circuit breakers, timeouts, and bulkheads.
One statistic often quoted in reliability literature is that the vast majority of transient failures are resolved within three attempts. That observation drives the default configuration of most retry libraries: three to five attempts, an initial delay in the hundreds of milliseconds, and an exponential growth factor of two. It is a pragmatic sweet spot — enough attempts to ride out short-lived storms, few enough that a genuinely unhealthy service is not hammered endlessly. The diagrams below contrast the outcome of a transient timeout with and without this machinery, showing how a single retry turns a lost request into a successful response.
┌─────────────────────────────────────────────────────────────────┐
│ Transient Failures Are Common │
│ │
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
│ │ Network │ │ Service │ │ Database │ │
│ │ Timeout │ │ Restart │ │ Lock Wait │ │
│ └──────────────┘ └──────────────┘ └──────────────┘ │
│ │ │ │ │
│ ▼ ▼ ▼ │
│ Usually recover quickly with retry! │
│ │
│ Statistics: │
│ - 60% of failures are transient │
│ - 90% succeed on retry within 3 attempts │
│ - Exponential backoff reduces load by 99% │
└─────────────────────────────────────────────────────────────────┘
Without Retry Pattern
The first scenario shows what happens when an application has no retry logic at all. A request reaches out to a downstream dependency, hits a connection timeout, and surfaces an error immediately to the caller. From the user’s perspective the operation simply failed, even though the underlying condition — a brief network blip or a momentarily busy server — was about to clear. The diagram captures the full cost of this approach: the user sees an error, the error propagates into monitoring as a real failure, and no attempt is made to recover the lost work.
The absence of retries is most damaging precisely where transient failures are most common: long-running background jobs, batch workers, and integration pipelines. A single timeout mid-batch can abort the entire run, forcing a restart from scratch and duplicating whatever partial results were already persisted. In interactive paths the damage is more subtle but still real — retries are one of the cheapest reliability wins available, yet their absence forces users to manually retry, which often makes the problem worse as dozens of annoyed users re-hit the same endpoint at once.
Note that “no retry” is a legitimate design decision in specific cases. Idempotent, cheap, read-only operations with tight latency budgets sometimes prefer to fail fast rather than add uncertainty to the response time. The point of this section is not that every call must be retried, but that the decision must be deliberate. Most systems, however, land on the side of retrying, because the asymmetry is stark: the cost of a retry is a fraction of a second of latency, while the cost of a spurious failure is a lost transaction, a failed job, or a paged engineer.
┌─────────────────────────────────────────────────────────────────┐
│ Immediate Failure (No Retry) │
│ │
│ Request ──► ✗ Connection timeout │
│ │ │
│ ▼ │
│ ┌─────────────┐ │
│ │ ERROR! │ User sees failure immediately │
│ │ User upset │ │
│ └─────────────┘ │
│ │
│ ✗ Wasted opportunity │
│ ✗ Poor user experience │
│ ✗ No recovery attempt │
└─────────────────────────────────────────────────────────────────┘
With Retry Pattern
With a retry mechanism in place, the same timeout plays out very differently. The request fails, the caller waits a short backoff interval, then tries again. The first retry may also fail if the server is still recovering, so the delay grows — 100 milliseconds, then 200, then 400 — until the service returns a healthy response. The diagram shows the exponential progression: each successive wait is longer than the last, which keeps the total number of attempts small while maximizing the chance that one of them lands after the outage clears.
The exponential schedule matters for two reasons. First, it balances responsiveness against load: the early attempts come quickly, when recovery is most likely, while later attempts space themselves out so a failing service is not bombarded. Second, the growing gaps give the downstream dependency real time to recover. A 400-millisecond pause, and then an 800-millisecond pause, buys a struggling server meaningful seconds to drain its queue. This is the key difference between exponential backoff and naive fixed-interval retries: exponential backoff adapts its pressure to the persistence of the failure.
The final element visible in this diagram is the success path — the moment the retry succeeds, the caller returns a normal response as if nothing went wrong. From the caller’s perspective, resilience is invisible: the user just sees a slightly slower request. That transparency is the goal of the pattern. It is also why retry configuration belongs in a shared, well-tested utility rather than scattered across call sites; every developer should be able to trust that retries are applied consistently, with the same budgets and the same backoff policy.
┌─────────────────────────────────────────────────────────────────┐
│ Retry with Exponential Backoff │
│ │
│ Request ──► ✗ Connection timeout │
│ │ │
│ ▼ │
│ Wait 100ms ──► ✗ Still timeout │
│ │ │
│ ▼ │
│ Wait 200ms ──► ✗ Still timeout │
│ │ │
│ ▼ │
│ Wait 400ms ──► ✓ Success! │
│ │ │
│ ▼ │
│ ┌──────────┐│
│ │ SUCCESS! ││
│ └──────────┘│
│ │
│ ✓ Automatic recovery │
│ ✓ Better user experience │
│ ✓ Reduced load from retry storms │
└─────────────────────────────────────────────────────────────────┘
Implementation
Basic Retry with Exponential Backoff
This section moves from theory to working code.
The implementation below is a self-contained retry utility written in Python, chosen because it demonstrates the core mechanics without the distraction of framework specifics.
The design centers on a RetryConfig object that captures every tunable parameter in one place — maximum attempts, the base delay, an upper ceiling on delay, the exponential base, and a flag for jitter.
Centralizing this configuration is a deliberate decision: it makes retry behavior uniform across the codebase, testable in isolation, and easy to adjust in production without hunting through dozens of call sites.
The heart of the utility is calculate_delay, which computes the wait for a given attempt number.
It applies the exponential formula base * (base ** attempt), caps the result at max_delay so a long series of failures never produces absurd waits, and then applies optional jitter.
The cap is critical — without it, a 30-second ceiling is meaningless, and attempt ten of a runaway failure could sleep for minutes.
Jitter, implemented as a random multiplier between 0.5 and 1.0, de-synchronizes retrying clients, preventing the thundering herd problem discussed earlier.
The public API is deliberately small. retry_async is the imperative function: it loops up to max_attempts, returns on the first success, and raises the last exception when all attempts are exhausted. retry_decorator wraps the same logic for functions, transparently supporting both async and sync callables by detecting coroutine functions at decoration time.
Both entry points accept an optional tuple of exception types to retry on, defaulting to all exceptions — a reasonable default for a library, but a dangerous one in application code, which is why the next sections build in proper error classification.
Notice what this implementation does not do. It does not take over the entire request lifecycle, it does not know about HTTP status codes or database error codes, and it does not attempt to distinguish transient from permanent failures. Keeping these concerns separate is intentional. The retry loop is a generic skeleton; the intelligence about which failures deserve a retry lives in the error-classification layer presented later in this article. That separation of concerns is what allows the same retry machinery to serve HTTP clients, database drivers, and custom business logic alike.
import asyncio
import time
from functools import wraps
from typing import Callable, Type, Tuple
class RetryConfig:
def __init__(
self,
max_attempts: int = 3,
base_delay: float = 0.1,
max_delay: float = 30.0,
exponential_base: float = 2.0,
jitter: bool = True
):
self.max_attempts = max_attempts
self.base_delay = base_delay
self.max_delay = max_delay
self.exponential_base = exponential_base
self.jitter = jitter
def calculate_delay(attempt: int, config: RetryConfig) -> float:
delay = config.base_delay * (config.exponential_base ** attempt)
delay = min(delay, config.max_delay)
if config.jitter:
import random
delay = delay * (0.5 + random.random())
return delay
async def retry_async(
func: Callable,
*args,
config: RetryConfig = None,
exceptions: Tuple[Type[Exception], ...] = (Exception,),
**kwargs
):
config = config or RetryConfig()
last_exception = None
for attempt in range(config.max_attempts):
try:
return await func(*args, **kwargs)
except exceptions as e:
last_exception = e
if attempt < config.max_attempts - 1:
delay = calculate_delay(attempt, config)
await asyncio.sleep(delay)
else:
raise last_exception
raise last_exception
def retry_decorator(config: RetryConfig = None):
def decorator(func: Callable):
@wraps(func)
async def async_wrapper(*args, **kwargs):
return await retry_async(func, *args, config=config, **kwargs)
@wraps(func)
def sync_wrapper(*args, **kwargs):
config = config or RetryConfig()
last_exception = None
for attempt in range(config.max_attempts):
try:
return func(*args, **kwargs)
except Exception as e:
last_exception = e
if attempt < config.max_attempts - 1:
delay = calculate_delay(attempt, config)
time.sleep(delay)
raise last_exception
if asyncio.iscoroutinefunction(func):
return async_wrapper
return sync_wrapper
return decorator
A few properties of this implementation are worth carrying forward.
The RetryConfig object is immutable after construction, which makes it safe to share across many concurrent call sites.
The loop always sleeps before the next attempt, never after the last one, so a failed final attempt fails fast rather than adding an unnecessary delay.
And because the wrapper captures the last exception and re-raises it verbatim, the original stack trace is preserved for debugging.
The trade-off of this simple design is that the delay curve is fixed at construction time — which is exactly the limitation the next section addresses.
Configurable Retry Strategies
The basic implementation hard-codes one delay curve. Real systems, however, operate under different constraints: a fast internal API might tolerate aggressive retries, while a slow external service needs gentler spacing. This section refactors the delay calculation behind a strategy interface, so each caller can choose the curve that fits its dependency without touching the retry loop itself. This is the strategy pattern applied to backoff: the loop asks the strategy “how long should I wait for attempt N?”, and the strategy answers according to its own rules.
Four strategies are provided. ExponentialBackoff reproduces the classic curve from the previous section. LinearBackoff grows the delay by a fixed increment, useful when a dependency degrades gracefully and you want to limit the maximum wait tightly. ConstantBackoff never varies, appropriate for idempotent operations where a uniform cadence is safe. FibonacciBackoff uses the Fibonacci sequence, whose property of growing from small to large values quickly is prized in protocols like TCP’s retransmission because it provides an aggressive early schedule that still backs off meaningfully.
Each strategy shares the same get_delay contract, so they are interchangeable at runtime.
The design trade-off here is flexibility versus predictability. A pluggable strategy makes the retry policy a first-class configuration concern, testable in isolation and swappable per dependency. The cost is that each strategy must be thoroughly unit-tested, and operators must understand the shape of the curve they are choosing — an exponential curve with a base of 2.0 and a linear curve with an increment of 0.1 produce wildly different behavior at attempt ten. Whichever strategy you choose, keep the choice visible: logging which backoff policy a call site uses makes runtime behavior diagnosable and prevents accidental policy drift between services.
class RetryStrategy:
def get_delay(self, attempt: int) -> float:
raise NotImplementedError
class ExponentialBackoff(RetryStrategy):
def __init__(
self,
base: float = 0.1,
max_delay: float = 30.0,
multiplier: float = 2.0
):
self.base = base
self.max_delay = max_delay
self.multiplier = multiplier
def get_delay(self, attempt: int) -> float:
delay = self.base * (self.multiplier ** attempt)
return min(delay, self.max_delay)
class LinearBackoff(RetryStrategy):
def __init__(self, base: float = 0.1, increment: float = 0.1):
self.base = base
self.increment = increment
def get_delay(self, attempt: int) -> float:
return self.base + (attempt * self.increment)
class ConstantBackoff(RetryStrategy):
def __init__(self, delay: float = 1.0):
self.delay = delay
def get_delay(self, attempt: int) -> float:
return self.delay
class FibonacciBackoff(RetryStrategy):
def __init__(self, multiplier: float = 1.0):
self.multiplier = multiplier
self._cache = {0: 1, 1: 1}
def _fib(self, n: int) -> float:
if n in self._cache:
return self._cache[n]
self._cache[n] = self._fib(n-1) + self._fib(n-2)
return self._cache[n]
def get_delay(self, attempt: int) -> float:
return self._fib(attempt) * self.multiplier
All four strategies share an important property: they are pure functions of the attempt number, deterministic and side-effect free.
That makes them trivially unit-testable — you can assert that attempt 3 of the exponential strategy returns exactly base * 8, capped at max_delay.
It also means the strategies are safe to use concurrently, since they hold no mutable state between calls.
The Fibonacci implementation does use an internal cache, but that cache is pure memoization of the sequence, so it changes nothing about the result.
The next section adds the element that deliberately breaks determinism: jitter.
Jitter Strategies
Pure exponential backoff has a well-known flaw. When many clients observe the same failure at the same time — a database restart, a deploy window, a DNS outage — they all retry in lockstep, arriving at the server in synchronized waves. The technical term is the thundering herd problem, and it can turn a minor incident into a self-inflicted DDoS. Jitter breaks the synchronization by adding randomness to each delay, so retries scatter across the timeline instead of clustering. Almost every serious retry library ships jitter enabled by default for exactly this reason.
The code presents four jitter flavors. no_jitter is the baseline that leaves the delay untouched. full_jitter randomizes the delay across its full range, from zero up to the computed backoff, producing the widest spread and the best herd protection. equal_jitter keeps half the delay fixed and randomizes only the remaining half, trading some protection for a guaranteed minimum wait — useful when a brief pause is functionally required, such as letting a rate-limit window reset. decorrelated_jitter is the most sophisticated: instead of randomizing each delay independently, it jitters relative to the previous delay, producing a smoother, more natural progression that is widely recommended for high-load production systems.
The choice among these is a tuning decision, not a correctness decision. Full jitter maximizes server protection but occasionally returns a near-zero delay, which can be wasteful when a dependency genuinely needs time. Equal jitter is a good default for most HTTP services. Decorrelated jitter shines in systems with high concurrency and long recovery windows, since it naturally avoids both the thundering herd and the alternating “burst then silence” pattern that independent randomization can produce. Whichever you pick, the guiding rule is the same: some randomness is always better than none, and the randomness should be applied on top of the exponential schedule rather than instead of it.
import random
import math
class Jitter:
@staticmethod
def no_jitter(delay: float) -> float:
return delay
@staticmethod
def full_jitter(delay: float) -> float:
return delay * random.random()
@staticmethod
def equal_jitter(delay: float) -> float:
return delay / 2 + (delay / 2) * random.random()
@staticmethod
def decorrelated_jitter(delay: float, last_delay: float = None) -> float:
if last_delay is None:
last_delay = delay
new_delay = last_delay * random.uniform(1.3, 2.0)
return min(new_delay, 30.0)
class ExponentialBackoffWithJitter(ExponentialBackoff):
def __init__(self, base: float = 0.1, max_delay: float = 30.0, jitter_type: str = "full"):
super().__init__(base, max_delay)
self.jitter_type = jitter_type
self.last_delay = base
def get_delay(self, attempt: int) -> float:
delay = self.base * (2 ** attempt)
delay = min(delay, self.max_delay)
if self.jitter_type == "full":
delay = Jitter.full_jitter(delay)
elif self.jitter_type == "equal":
delay = Jitter.equal_jitter(delay)
elif self.jitter_type == "decorrelated":
delay = Jitter.decorrelated_jitter(delay, self.last_delay)
self.last_delay = delay
elif self.jitter_type == "none":
delay = Jitter.no_jitter(delay)
return delay
The jitter strategies are deliberately small and composable — each is a single function that takes a delay and returns a modified delay.
This composition is what makes the ExponentialBackoffWithJitter class in the same block possible: it layers a chosen jitter function on top of the base exponential curve, and for decorrelated jitter it tracks the previous delay as internal state.
The one caveat is thread safety: last_delay is mutable instance state, so a single instance should not be shared across concurrent threads without synchronization.
In practice this class is usually instantiated once per call site, which sidesteps the issue entirely.
Handling Different Failure Types
Transient vs Permanent Errors
Up to this point, the retry loop has been blindly retrying every exception. That is safe in a toy example and dangerous in production, because it means a 400 Bad Request from an API will be retried three times before being surfaced — and it will fail identically all three times, wasting latency and load. The distinction that every resilient system must encode is between transient failures, which may succeed on a second attempt, and permanent failures, which are guaranteed to fail again. This section builds a small exception hierarchy that makes that distinction explicit and machine-readable.
The hierarchy is deliberately shallow. RetryableError and NonRetryableError form the two branches, and concrete exception types — ServiceUnavailableError, TimeoutError, RateLimitError on one side; ValidationError, AuthenticationError, NotFoundError on the other — hang off the appropriate branch.
The choice to use exceptions rather than error codes or status flags is what makes the classification enforceable: a function simply raises the correct type, and the retry layer can decide based on the class hierarchy alone.
The RateLimitError variant carries a retry_after value, because for rate limiting the server explicitly tells you when to come back.
SmartRetryHandler ties the pieces together.
Its should_retry method encodes the full decision tree: if attempts are exhausted, never retry; if the error is non-retryable, never retry; if the error is a rate limit, honor the server’s retry_after; otherwise fall through to the standard backoff calculation.
The return value is a tuple of “should I retry” and “how long to wait”, which cleanly separates policy from mechanics.
This handler can be dropped into the generic retry loop from earlier, and every call site instantly gains correct, per-error-type behavior — the separation of concerns promised at the end of the implementation section.
class RetryableError(Exception):
"""Transient errors that should be retried."""
pass
class NonRetryableError(Exception):
"""Permanent errors that should not be retried."""
pass
class ServiceUnavailableError(RetryableError):
pass
class TimeoutError(RetryableError):
pass
class RateLimitError(RetryableError):
def __init__(self, retry_after: float = 60.0):
self.retry_after = retry_after
super().__init__(f"Rate limited, retry after {retry_after}s")
class ValidationError(NonRetryableError):
pass
class AuthenticationError(NonRetryableError):
pass
class NotFoundError(NonRetryableError):
pass
class SmartRetryHandler:
def __init__(self, config: RetryConfig):
self.config = config
def should_retry(self, exception: Exception, attempt: int) -> tuple[bool, float]:
if attempt >= self.config.max_attempts:
return False, 0
if isinstance(exception, NonRetryableError):
return False, 0
if isinstance(exception, RateLimitError):
return True, exception.retry_after
if isinstance(exception, RetryableError):
delay = calculate_delay(attempt, self.config)
return True, delay
if isinstance(exception, (TimeoutError, ConnectionError)):
delay = calculate_delay(attempt, self.config)
return True, delay
return True, calculate_delay(attempt, self.config)
The lesson of this section is that error classification is the highest-leverage part of a retry system. It is cheap to build, and it is what prevents the most common retry anti-pattern: retrying requests that will never succeed, or worse, retrying requests with side effects (payments, email sends, state changes) when the failure is ambiguous. If a permanent failure is retried even once, the service pays latency and load for nothing. If a transient failure is never retried, the system forfeits the easy wins that make the pattern worthwhile. Get the hierarchy right, and everything downstream — HTTP logic, database logic, circuit breakers — becomes simpler.
HTTP-Specific Retry Logic
HTTP is where retry logic meets the real world, because the failure signal is not an exception but a status code.
This section presents HTTPRetryConfig, which encodes the retryable statuses as a configurable tuple defaulting to the classic transient set: 429 (rate limited), 500, 502, 503, and 504.
Each of those codes carries a different meaning — 429 means “you are going too fast”, 503 means “the server is overloaded or down for maintenance”, 504 means “the upstream timed out” — but all of them are conditions that may clear, making them candidates for a retry. Status codes like 400, 401, 403, and 422 are deliberately excluded, and the earlier exception hierarchy is mirrored here for the same reason.
The fetch_with_retry function wraps an aiohttp session in a retry loop.
Three failure modes are handled distinctly.
If the response carries a retryable status, the loop applies backoff — and, crucially, honors the Retry-After header when present, because the server’s explicit instruction always wins over a client-side guess.
If a timeout occurs, the loop sleeps and retries, subject to retry_on_timeout.
If an aiohttp.ClientError occurs — a connection reset, a DNS failure, a protocol error — the loop treats it as transient and retries.
The one failure mode that is not retried is a non-retryable status, which is returned to the caller immediately.
This layered handling mirrors how real HTTP clients fail: a request can die before a response exists (a connection error), produce a response that signals temporary trouble (a retryable status), or time out mid-flight (a timeout).
Each requires a different reaction, and the code keeps them in separate except branches so the retry decision for each is explicit and auditable.
A common refinement not shown here is to record which failure mode triggered each retry, so dashboards can distinguish “retries caused by 503s” from “retries caused by timeouts” — useful signal when deciding whether to scale a dependency or fix a client.
import aiohttp
class HTTPRetryConfig:
def __init__(
self,
max_attempts: int = 3,
retry_on_status: Tuple[int, ...] = (429, 500, 502, 503, 504),
retry_on_timeout: bool = True,
**backoff_kwargs
):
self.max_attempts = max_attempts
self.retry_on_status = retry_on_status
self.retry_on_timeout = retry_on_timeout
self.backoff = RetryConfig(max_attempts=max_attempts, **backoff_kwargs)
async def fetch_with_retry(
session: aiohttp.ClientSession,
url: str,
config: HTTPRetryConfig = None,
**kwargs
) -> aiohttp.ClientResponse:
config = config or HTTPRetryConfig()
last_exception = None
for attempt in range(config.max_attempts):
try:
async with session.get(url, **kwargs) as response:
if response.status in config.retry_on_status:
if attempt < config.max_attempts - 1:
delay = calculate_delay(attempt, config.backoff)
if "Retry-After" in response.headers:
delay = float(response.headers["Retry-After"])
await asyncio.sleep(delay)
continue
return response
except asyncio.TimeoutError as e:
last_exception = e
if not config.retry_on_timeout or attempt >= config.max_attempts - 1:
raise
await asyncio.sleep(calculate_delay(attempt, config.backoff))
except aiohttp.ClientError as e:
last_exception = e
if attempt < config.max_attempts - 1:
await asyncio.sleep(calculate_delay(attempt, config.backoff))
else:
raise
raise last_exception
Two details in this implementation reward attention.
First, when a 429 or 503 includes a Retry-After header, the code trusts it over the backoff calculation — this is the correct hierarchy, since the server knows its own recovery state.
Second, the loop returns the response as soon as a non-retryable status is seen, which means the caller always receives the authoritative server response rather than a synthetic exception, preserving headers and body for logging.
The main limitation is that this wrapper retries only GET-style requests; retrying POST requests that are not idempotent can duplicate side effects, which is why many systems add an idempotency-key header for retried writes.
Database Retry Logic
Databases add two complications that HTTP clients do not have.
The first is that a failed connection may leave you holding a broken handle — retrying the query on the same dead connection is pointless, so the retry path must first reconnect.
The second is that not all database errors deserve a retry: a connection failure does, a serialization conflict does (the transaction is aborted and can simply be re-run), but a constraint violation or a syntax error never will. DatabaseRetryHandler encodes exactly this distinction, retrying only ConnectionFailureError and SerializationError while letting every other exception propagate immediately.
execute_with_retry iterates the same loop shape seen throughout this article.
On a connection failure, it sleeps according to the backoff config and then calls _reconnect, which closes the dead connection and opens a fresh one with the same host, port, database, and user — swallowing the close error because there is nothing productive to do with a failed cleanup.
On a serialization error — the classic PostgreSQL deadlock or serialization-failure case — it sleeps and re-runs the query against the existing connection, since the transaction was already aborted server-side and the connection remains usable.
Any other exception is re-raised immediately, matching the “never retry permanent failures” rule.
The reconnect helper is the subtle part.
It rebuilds a connection from the old connection’s attributes, which keeps the helper decoupled from connection-configuration plumbing — at the cost of reaching into private fields (_host, _port), a known fragility.
In a production codebase you would pass a connection factory or a connection string instead, so reconnects are reconstructed from source configuration rather than from a potentially half-closed object.
The retry semantics, however, are the point of this section: sleeping before retrying, classifying which errors are worth another attempt, and re-establishing the prerequisite (a live connection) before the retry fires.
import asyncpg
class DatabaseRetryHandler:
def __init__(self, config: RetryConfig):
self.config = config
async def execute_with_retry(
self,
conn: asyncpg.Connection,
query: str,
*args
):
last_exception = None
for attempt in range(self.config.max_attempts):
try:
return await conn.fetch(query, *args)
except asyncpg.exceptions.ConnectionFailureError as e:
last_exception = e
if attempt < self.config.max_attempts - 1:
await asyncio.sleep(calculate_delay(attempt, self.config))
conn = await self._reconnect(conn)
else:
raise
except asyncpg.exceptions.SerializationError as e:
last_exception = e
if attempt < self.config.max_attempts - 1:
await asyncio.sleep(calculate_delay(attempt, self.config))
else:
raise
except Exception as e:
raise
raise last_exception
async def _reconnect(self, old_conn):
try:
await old_conn.close()
except:
pass
return await asyncpg.connect(
host=old_conn._host,
port=old_conn._port,
database=old_conn._database,
user=old_conn._user
)
The database example reinforces the pattern’s central idea: the retry loop is generic, and the domain knowledge lives in the error classification and the reconnect logic. It also demonstrates that retries must sometimes recreate the environment of the failed call, not just repeat the call itself. One operational note: re-running a transaction that failed mid-way can have side effects if the first attempt partially committed before failing — which is why production database retry layers pair with transaction boundaries, such as retrying at the statement level only for idempotent statements or retrying the whole transaction with proper rollback handling.
Circuit Breaker Integration
Retries and circuit breakers are complementary resilience patterns that are often discussed together but serve different purposes. A retry assumes the failure is momentary and worth repeating. A circuit breaker assumes the failure is systemic and worth stopping — when a service has been failing repeatedly, continuing to retry is wasted effort that only adds load to an already struggling dependency. The two work in sequence: retries handle the transient blips, and the breaker opens when the blips turn into a sustained outage, causing the system to fail fast instead of hammering a dead service.
The CircuitBreaker class tracks a simple three-state machine: closed (normal operation), open (failing fast, no calls allowed), and half-open (a probe call to test recovery).
The transition rules are straightforward.
Failures increment a counter; when the counter crosses failure_threshold, the breaker opens.
While open, can_execute returns false — until recovery_timeout seconds have passed, at which point the breaker transitions to half-open and permits a single trial call.
A success in half-open closes the breaker and resets the counter; another failure reopens it.
This is the minimal but complete state machine that production circuit breakers implement.
RetryWithCircuitBreaker composes the two patterns.
Before any attempt, it consults the breaker and refuses to run if the circuit is open — this is the fail-fast path.
When an attempt fails, it records the failure in the breaker, and if attempts remain, it sleeps with backoff and tries again.
When an attempt succeeds, it records success and returns.
The interaction is what matters: the breaker adds an outer guard so that the retry loop stops being invoked at all once the dependency is judged to be down.
Without this layer, the retry loop itself would keep generating load even during a sustained outage, converting a slow burn into a fast one.
import asyncio
class CircuitBreaker:
def __init__(
self,
failure_threshold: int = 5,
recovery_timeout: int = 60,
expected_exception: Type[Exception] = Exception
):
self.failure_threshold = failure_threshold
self.recovery_timeout = recovery_timeout
self.expected_exception = expected_exception
self.failure_count = 0
self.last_failure_time = None
self.state = "closed" # closed, open, half-open
def can_execute(self) -> bool:
if self.state == "closed":
return True
if self.state == "open":
if time.time() - self.last_failure_time >= self.recovery_timeout:
self.state = "half-open"
return True
return False
return True
def record_success(self):
self.failure_count = 0
self.state = "closed"
def record_failure(self):
self.failure_count += .last_failure_time =1
self time.time()
if self.failure_count >= self.failure_threshold:
self.state = "open"
class RetryWithCircuitBreaker:
def __init__(self, retry_config: RetryConfig, circuit_breaker: CircuitBreaker):
self.retry_config = retry_config
self.circuit_breaker = circuit_breaker
async def execute(self, func: Callable, *args, **kwargs):
if not self.circuit_breaker.can_execute():
raise Exception("Circuit breaker is open")
last_exception = None
for attempt in range(self.retry_config.max_attempts):
try:
result = await func(*args, **kwargs)
self.circuit_breaker.record_success()
return result
except self.circuit_breaker.expected_exception as e:
last_exception = e
self.circuit_breaker.record_failure()
if attempt < self.retry_config.max_attempts - 1:
delay = calculate_delay(attempt, self.retry_config)
await asyncio.sleep(delay)
else:
raise
raise last_exception
A practical deployment detail is that the breaker and the retry loop should share their configuration philosophy: the breaker’s failure_threshold and the retry’s max_attempts should be set so that the breaker opens only after the retry budget is genuinely exhausted, otherwise the breaker may open while retries still have value to offer.
Similarly, recovery_timeout should be long enough that the half-open probe happens only after the dependency has had a real chance to recover.
Many teams start with a 5-failure threshold and a 60-second recovery window, then tune based on observed failure patterns.
Note that a single breaker instance should be shared across all call sites for one dependency — per-call-site breakers defeat the pattern’s purpose.
Monitoring and Observability
Retries improve resilience, but they also hide information: when a system silently retries and recovers, the underlying instability is invisible unless it is measured.
Monitoring is what turns retries from a blunt tool into an observable, tunable one.
This section shows a RetryMetrics dataclass that accumulates the counters operators actually need — attempts, successes, failures, total retries, and total accumulated delay — and a RetryWithMetrics wrapper that updates those counters as it executes.
Metrics like “how many requests needed 2 retries before succeeding” are exactly the signal that distinguishes a healthy system from one that is one nudge away from failing.
The wrapper logs at three distinct levels, which is itself a design decision worth copying. A warning is emitted for each intermediate failure, with the computed delay, so a short burst of warnings paints the picture of a struggling dependency without paging anyone. An info line is emitted when an attempt eventually succeeds after retries, marking the resolution. An error is logged only when all attempts are exhausted — the signal that a real problem exists and operators should look. This graduated logging means the log volume is proportional to actual instability, not to request volume.
The total_delay and total_retries counters deserve particular attention, because they measure the cost of retries.
A request that succeeds on the fifth attempt has consumed far more wall-clock time and produced far more load than one that succeeds immediately, even though both are “successes” from the caller’s perspective.
Tracking these costs is what lets you set alerting and budgets on retry overhead, and it feeds directly into the tuning decisions described throughout this article: if total_retries is high, raise the base delay or lower max_attempts; if failures dominate, the problem is no longer retryable and the circuit breaker should be involved.
import logging
from dataclasses import dataclass, field
@dataclass
class RetryMetrics:
attempts: int = 0
successes: int = 0
failures: int = 0
total_retries: int = 0
total_delay: float = 0.0
class RetryWithMetrics:
def __init__(self, config: RetryConfig, logger: logging.Logger = None):
self.config = config
self.logger = logger
self.metrics = RetryMetrics()
async def execute(self, func: Callable, *args, **kwargs):
self.metrics.attempts += 1
for attempt in range(self.config.max_attempts):
try:
result = await func(*args, **kwargs)
self.metrics.successes += 1
self.metrics.total_retries += attempt
if attempt > 0:
self.logger.info(
f"Succeeded after {attempt + 1} attempts"
)
return result
except Exception as e:
if attempt < self.config.max_attempts - 1:
delay = calculate_delay(attempt, self.config)
self.metrics.total_delay += delay
self.logger.warning(
f"Attempt {attempt + 1} failed: {e}. "
f"Retrying in {delay:.2f}s"
)
await asyncio.sleep(delay)
else:
self.metrics.failures += 1
self.logger.error(
f"All {self.config.max_attempts} attempts failed: {e}"
)
raise
raise
The metric names here map naturally onto real monitoring systems — Prometheus counters for attempts, successes, and failures; a histogram for delays. The important discipline is to export them per dependency, so a dashboard can compare “retries against the payment API” versus “retries against the search API” rather than lumping everything together. It is also worth exporting a ratio, such as retries-per-request, which normalizes for traffic and is far more useful than a raw count. With these signals in place, retry configuration stops being guesswork and becomes a tuning loop guided by data.
Best Practices
The code in this article embodies a set of recurring principles, and the final section makes them explicit as concrete good and bad patterns. The first rule is to always use exponential backoff rather than a fixed or linear delay — exponential spacing prevents the thundering herd while still recovering quickly when a dependency clears. The second is to always apply jitter on top of the exponential curve, because even exponential retries cluster when many clients share the same failure signal. The third is to distinguish error types, retrying only transient failures and letting permanent ones fail immediately.
The negative patterns are equally instructive. Retrying everything, including authentication failures, multiplies load without any chance of success. Omitting a max-attempts cap produces an infinite retry loop that is effectively a self-inflicted DDoS against your own dependency. Ignoring the circuit breaker means that even a well-tuned retry layer will keep hammering a service that has been down for minutes. Each bad pattern in the code block shows the naive version side by side with the corrected version, which makes the contrast easy to audit in code review.
A useful mental model is that retries should be configured at three levels, each with its own budget. First, the per-attempt backoff schedule, which controls spacing. Second, the total retry budget — both a maximum number of attempts and a maximum wall-clock time — which bounds the damage of a persistent failure. Third, the interaction with the circuit breaker, which stops retries altogether when a dependency is judged down. Getting all three right, and monitoring them, is what separates a resilient system from one that merely looks resilient in a diagram.
GOOD_PATTERNS = {
"use_exponential_backoff": """
# Exponential backoff prevents thundering herd
✅ Good:
delay = base * (2 ** attempt) # 0.1s, 0.2s, 0.4s, 0.8s...
delay = min(delay, max_delay)
❌ Bad:
delay = base * attempt # 0.1s, 0.2s, 0.3s...
# Still causes stampede at higher loads
""",
"add_jitter": """
# Jitter randomizes retry times to reduce collisions
✅ Good:
delay = delay * (0.5 + random.random())
❌ Bad:
# No jitter = synchronized retries
# All clients retry at exactly same time
""",
"distinguish_error_types": """
# Only retry transient errors
✅ Good:
if isinstance(e, ValidationError):
raise immediately
if isinstance(e, TimeoutError):
retry with backoff
❌ Bad:
retry(Exception) # Never retry everything!
"""
}
BAD_PATTERNS = {
"retry_everything": """
❌ Bad:
# Retry authentication errors?
try:
return await make_request()
except Exception:
return await retry() # Wrong!
# Authentication failures won't succeed on retry
✅ Good:
async def make_request():
try:
return await http.request()
except TimeoutError:
return await retry()
except ValidationError:
raise # Don't retry validation
""",
"no_max_attempts": """
❌ Bad:
while True:
try:
return await request()
except:
await asyncio.sleep(1)
# Infinite retry loop!
✅ Good:
for attempt in range(max_attempts):
try:
return await request()
except:
if attempt == max_attempts - 1:
raise
await sleep(delay)
""",
"ignore_circuit_breaker": """
❌ Bad:
# Retry forever even when service is down
for i in range(1000):
try:
await failing_service()
except:
await sleep(backoff)
# Hammering a dead service!
✅ Good:
# Use circuit breaker
breaker = CircuitBreaker(failure_threshold=5)
async def call():
if not breaker.can_execute():
raise ServiceUnavailable()
try:
return await failing_service()
except Exception as e:
breaker.record_failure()
raise
"""
}
When you review a retry implementation, run it through these checks: is there a jitter on every delay, is there a hard cap on attempts, are permanent errors excluded from the retry set, and is there a companion circuit breaker or timeout guard? If any answer is no, the system has a gap that will show up under load. The good patterns in this block are deliberately short — they are meant to be copied as a checklist, not memorized. Combined with the monitoring from the previous section, they form a complete, defensible retry posture for any service.
Related Articles
Summary
The Retry Pattern with Exponential Backoff is essential for building resilient systems:
- Exponential Backoff - Increase delay exponentially between retries (prevents overload)
- Jitter - Add randomness to prevent synchronized retry storms
- Error Classification - Distinguish between retryable and non-retryable errors
- Circuit Breaker - Stop retrying when service is clearly down
- Monitoring - Track retry success rates and delays
Key configuration tips:
- Base delay: 100-500ms
- Max attempts: 3-5
- Max delay: 30-60 seconds
- Jitter: Always enabled for production systems
The combination of retries, backoff, jitter, and circuit breakers provides defense in depth for distributed systems.
Comments