Building systems that withstand failures requires redundancy, automated failover, and tested recovery procedures.
Availability Levels
Uptime Tiers
Availability is fundamentally a numbers game. Every architecture decision, from the number of copies of your data to the speed of your failover automation, ultimately translates into a percentage that tells your customers how often they can expect your service to be up. Understanding the tiers is the first step toward designing a system that meets a concrete, measurable target rather than a vague aspiration of being “highly available.”
The table below is the standard reference used across the industry. It maps each availability percentage to its equivalent annual downtime, which is far easier to reason about than an abstract percentage. Notice how each additional nine reduces allowed downtime by roughly an order of magnitude: going from 99% to 99.9% shrinks permissible downtime from nearly an hour per year to under nine minutes. The jump from 99.99% to 99.999% (the so-called “five nines”) is particularly brutal, because five nines leaves only about five seconds of downtime per year, which is essentially impossible to achieve without multi-region active-active deployments, automated health checks, and extensive redundancy at every layer of the stack.
99% (52.6 min downtime/year) - Single server, basic backups
99.9% (8.8 min downtime/year) - Multi-AZ, failover
99.99% (52.6 sec downtime/year) - Multi-region, active-active
99.999% (5.26 sec downtime/year) - Massive redundancy (5 nines)
Calculating Availability
Real systems are composed of many components, and their combined availability depends on how those components are wired together. The two fundamental topologies are series and parallel. In a series topology, every component in the chain must be operational for the system to work, so availability is the product of the individual availabilities. Because multiplying numbers less than one always shrinks the result, adding more components in series always reduces total availability. In a parallel topology, the system remains available as long as at least one redundant component works, so availability increases with each redundant copy.
This distinction drives almost every real design trade-off. A single database behind a single API server is a series system; if either fails, the service is down. Deploying two API servers behind a load balancer creates a parallel arrangement for the application tier, but the database beneath them is still a series point of failure unless you also replicate it. The calculator below implements both formulas and converts the resulting percentages into annual downtime, so you can quickly quantify the impact of each architectural choice in minutes per year rather than in abstract probabilities.
class AvailabilityCalculator:
"""Calculate system availability from component reliability"""
def series_availability(self, components: list[float]) -> float:
"""
Series: All components must work
A_total = A1 × A2 × A3
"""
result = 1.0
for availability in components:
result *= availability
return result
def parallel_availability(self, components: list[float]) -> float:
"""
Parallel: At least one must work
A_total = 1 - (1-A1) × (1-A2)
"""
unavailability = 1.0
for availability in components:
unavailability *= (1 - availability)
return 1 - unavailability
def calculate_downtime_minutes(self, uptime_percentage: float) -> float:
"""Minutes per year"""
return (100 - uptime_percentage) / 100 * 525600 # minutes/year
## Example: API with database and cache
calc = AvailabilityCalculator()
## Series: All needed
api_server = 0.9999
database = 0.99999
cache = 0.99
series_avail = calc.series_availability([api_server, database, cache])
print(f"Series (all needed): {series_avail:.6f} (99.99%)")
print(f"Downtime: {calc.calculate_downtime_minutes(series_avail * 100):.1f} min/year")
## Parallel: Fallback possible
availability = calc.parallel_availability([0.95, 0.95])
print(f"Parallel (redundant): {availability:.4f} (99.75%)")
The example at the bottom of the script illustrates the two opposing forces at work. Wiring an API server, a database, and a cache in series multiplies three high percentages together and still lands at roughly 99.88% availability, which translates into over ten hours of downtime per year once the weaker cache component drags the whole chain down. By contrast, two independently unreliable servers with 95% availability each produce a combined 99.75% when placed in parallel, beating the series chain despite every component being weaker. This is the core intuition of high availability: redundancy in parallel is far more valuable than marginal reliability improvements on individual machines.
Multi-AZ Architecture
Synchronized Replication
Once you understand series and parallel availability, the next question is where to apply redundancy first. Within a single cloud region, the highest-value target is the database tier, because it is the classic single point of failure that sits beneath every other component. A multi-Availability-Zone (multi-AZ) deployment places a primary database and one or more standby replicas in physically separate data centers inside the same region, so that a power outage, network partition, or hardware failure in one facility does not take down the entire system.
Synchronous replication is the mechanism that makes these standbys useful for durability. With synchronous replication, the primary does not acknowledge a write until at least one standby has durably recorded it, which means the standbys are always a faithful copy of the primary and can be promoted without losing committed transactions. The trade-off is latency: every write now has to travel to another data center and wait for its acknowledgment before returning to the client, which can noticeably slow write-heavy workloads. Asynchronous replication avoids that latency penalty but risks losing recent writes during a crash, so the choice between the two is fundamentally a business decision about how much data loss is acceptable.
The Kubernetes manifest below shows a PostgreSQL StatefulSet configured for two replicas with
synchronous streaming replication. Several details are worth noting because they are easy to get
wrong. The headless Service with clusterIP: None gives each pod a stable DNS name, which the
replication protocol relies on to find peers. The liveness probe using pg_isready ensures the
orchestrator can detect a hung database and restart it, while the environment variable activates the
streaming replication mode that keeps the standby in sync with the primary.
## AWS RDS with Multi-AZ failover
apiVersion: v1
kind: Service
metadata:
name: database
spec:
ports:
- port: 5432
name: postgres
clusterIP: None
selector:
app: postgres
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: postgres
spec:
serviceName: postgres
replicas: 2 # Primary + standby
selector:
matchLabels:
app: postgres
template:
metadata:
labels:
app: postgres
spec:
containers:
- name: postgres
image: postgres:15
ports:
- containerPort: 5432
env:
- name: POSTGRES_REPLICATION_MODE
value: "streaming" # Synchronous replication
volumeMounts:
- name: data
mountPath: /var/lib/postgresql
livenessProbe:
exec:
command: ["pg_isready"]
initialDelaySeconds: 30
periodSeconds: 10
A replicated database is only half the solution; you also need a way to promote a standby when the primary fails. Modern database operators handle promotion automatically, but the sequencing matters. The orchestrator must first detect that the primary has stopped responding, then verify that the standby is caught up, and only then promote it and repoint the application layer. Any step done out of order can produce split-brain scenarios where two nodes believe they are the primary, leading to conflicting writes. That is why failover logic is normally pushed into a well-tested operator rather than left to ad-hoc scripts.
Automatic Failover
MongoDB replica sets are a particularly instructive example of automatic failover because the election protocol is built into the database itself. In a replica set, every node votes to elect a primary, and clients talk to whichever node wins the election. When the primary becomes unreachable, the remaining members hold an election and promote one of the secondaries automatically, so the application does not need to know which physical node is currently the primary. To benefit from this behavior, the client must connect using a seed list of all nodes rather than a single address, and should configure its write concern so that writes are acknowledged only after reaching a majority of the set.
The HighAvailabilityCluster class below demonstrates the client-side experience of failover.
Notice the connection string lists all three nodes, and the write concern w='majority' guarantees
that acknowledged writes survive any single node’s failure. The retry loop with exponential backoff
is the second half of the story: even with automatic elections, there is a brief window where no
primary exists, and well-behaved clients must retry rather than fail. Each retry doubles the wait
time, which lets the replica set converge on a new primary without hammering the cluster with
requests during the transition.
import asyncio
from pymongo import MongoClient
from pymongo.errors import ServerSelectionTimeoutError
class HighAvailabilityCluster:
"""MongoDB replica set with automatic failover"""
def __init__(self):
# Connection string with multiple nodes
self.client = MongoClient(
'mongodb://node1:27017,node2:27017,node3:27017',
replicaSet='rs0',
retryWrites=True,
w='majority' # Wait for majority replication
)
self.db = self.client['app_database']
async def write_with_failover(self, collection: str, document: dict) -> bool:
"""Write with automatic failover to secondary"""
max_retries = 3
retry_count = 0
while retry_count < max_retries:
try:
# Write with majority acknowledgment
result = self.db[collection].insert_one(document)
print(f"Written to {self.client.primary}: {result.inserted_id}")
return True
except ServerSelectionTimeoutError:
retry_count += 1
print(f"Primary unavailable, retrying... ({retry_count}/{max_retries})")
await asyncio.sleep(2 ** retry_count) # Exponential backoff
# Trigger failover
self.client.close()
self.client = MongoClient(
'mongodb://node1:27017,node2:27017,node3:27017',
replicaSet='rs0'
)
return False
## Usage
cluster = HighAvailabilityCluster()
success = asyncio.run(cluster.write_with_failover(
'transactions',
{'amount': 100, 'status': 'pending'}
))
This example captures the two-layer division of responsibility that makes failover reliable. The replica set handles the hard part: detecting failure, electing a new primary, and re-establishing majority. The application handles the part only it can: retrying writes that arrive during the transition and giving up gracefully if the cluster never recovers. The takeaway for any HA design is that failover is a collaboration between infrastructure and application code, and both sides must be tested together, not in isolation.
Multi-Region Architecture
Active-Active Setup
A single region, no matter how well engineered, remains vulnerable to events that take out an entire region: severe weather, undersea cable cuts, or cloud provider outages that span every availability zone. Multi-region architecture moves redundancy up one more level by running the service in two or more geographic regions and routing traffic between them. This provides protection against region-level disasters and, for latency-sensitive global audiences, lets users connect to a nearby region instead of one halfway around the world.
The most demanding configuration is active-active, where every region serves traffic simultaneously rather than sitting idle as a standby. An active-active setup improves utilization and cost efficiency, but it introduces hard problems: writes must be replicated across regions with low enough lag to keep both copies consistent, and the global load balancer must route each user to a healthy region. The diagram below shows the canonical topology: a global load balancer such as Route 53 or Cloudflare distributes requests across three regional clusters, each with its own API and database, while the databases replicate among themselves.
┌─────────────────────────────────────────────┐
│ Global Load Balancer │
│ (Route53, Cloudflare) │
└────────────────┬────────────────────────────┘
│
┌──────────────┼──────────────┐
│ │ │
US-East EU-West Asia-Pacific
┌────────┐ ┌────────┐ ┌────────┐
│ API │ │ API │ │ API │
├────────┤ ├────────┤ ├────────┤
│ DB │◄──►│ DB │◄──►│ DB │
└────────┘ └────────┘ └────────┘
Replication across regions
Cross-region replication is where the architecture either succeeds or fails. Strongly consistent replication across continents is physically impossible because the speed of light imposes latency, so every active-active design must decide how to reconcile conflicting writes that arrive at different regions. Common strategies include single-leader replication with reads from local replicas, or conflict-free replicated data types for the rare workloads that truly need multi-leader writes. The diagram intentionally hides this complexity, but you should never implement the topology shown here without explicitly answering the question of what happens when the same key is updated in two regions simultaneously.
Multi-Region Deployment
Deploying and operating a multi-region system by hand is error-prone, which is why the tooling is
usually codified as infrastructure as code. The MultiRegionHA class below automates the two
essential steps: launching identical instances in each region and wiring up DNS-based failover.
Using the same AMI across all regions ensures that code and configuration are identical everywhere,
so failover does not require deploying new software in the middle of an incident.
The second method demonstrates the critical role of DNS and health checks in global failover. Route
53 health checks probe each regional endpoint over HTTPS on a fixed interval; when a region stops
responding to enough consecutive probes, its health check flips to unhealthy, and Route 53 stops
returning that region’s record in DNS responses. This is why the TTL is set to 60 seconds rather
than hours: the lower the TTL, the faster clients notice a changed DNS answer, but the higher the
load on the authoritative name servers. The first region is marked PRIMARY and acts as the
preferred destination, while the others are SECONDARY fallbacks, giving you explicit control over
where traffic normally flows.
import boto3
from typing import List
class MultiRegionHA:
"""Deploy and manage multi-region infrastructure"""
def __init__(self, regions: List[str]):
self.regions = regions
self.clients = {region: boto3.client('ec2', region_name=region)
for region in regions}
def deploy_across_regions(self, ami_id: str):
"""Deploy instances to all regions"""
deployments = {}
for region in self.regions:
ec2 = self.clients[region]
# Launch instances
response = ec2.run_instances(
ImageId=ami_id,
MinCount=2,
MaxCount=2,
InstanceType='t3.medium',
Placement={'AvailabilityZone': f'{region}a'},
)
instance_ids = [i['InstanceId'] for i in response['Instances']]
deployments[region] = instance_ids
return deployments
def setup_global_failover(self):
"""Configure Route53 health checks and failover"""
route53 = boto3.client('route53')
# Create health checks for each region
health_checks = {}
for region in self.regions:
health_check = route53.create_health_check(
HealthCheckConfig={
'Type': 'HTTPS',
'ResourcePath': '/health',
'FullyQualifiedDomainName': f'api.{region}.example.com',
'Port': 443,
'RequestInterval': 30,
'FailureThreshold': 3
}
)
health_checks[region] = health_check['HealthCheck']['Id']
# Create failover routing records
for idx, region in enumerate(self.regions):
route53.change_resource_record_sets(
HostedZoneId='Z123456',
ChangeBatch={
'Changes': [{
'Action': 'UPSERT',
'ResourceRecordSet': {
'Name': 'api.example.com',
'Type': 'A',
'TTL': 60,
'Failover': 'PRIMARY' if idx == 0 else 'SECONDARY',
'SetIdentifier': region,
'HealthCheckId': health_checks[region],
'AliasTarget': {
'HostedZoneId': 'Z456',
'DNSName': f'elb.{region}.amazonaws.com',
'EvaluateTargetHealth': True
}
}
}]
}
)
## Deploy to 3 regions with automatic failover
ha = MultiRegionHA(['us-east-1', 'eu-west-1', 'ap-southeast-1'])
ha.deploy_across_regions(ami_id='ami-0123456789')
ha.setup_global_failover()
DNS-based failover has one important limitation that this code makes visible: it only helps new DNS lookups. Clients that have already cached the address of a region that just went down will keep hitting it until their cached record expires, which is exactly why the TTL is kept low. For real traffic you typically pair DNS routing with client-side retry logic and, for HTTP traffic, with anycast or load balancer integration so that a dead regional endpoint can be abandoned mid-connection. The takeaway is that multi-region routing is a stack of mechanisms, and each layer must be tuned with failover speed in mind.
RTO & RPO Strategies
Recovery Time Objective (RTO)
Availability percentages describe how much downtime a system tolerates over a year, but they say nothing about how long a single outage may last or how much data may be lost during it. That is the job of two disaster recovery metrics: Recovery Time Objective (RTO) and Recovery Point Objective (RPO). RTO answers “how fast must we be back?” and is measured in time, while RPO answers “how much data may we lose?” and is measured in data age. Together they define what a disaster recovery plan has to deliver, and almost every architecture decision flows from them.
RTO is a business decision masquerading as a technical one. A four-hour RTO may be perfectly acceptable for an internal analytics tool, while a payment gateway might need sub-minute RTO to keep regulators and merchants happy. The trade-off is cost: the more aggressive the RTO, the more infrastructure must be kept warm and waiting, and the more automation is required to switch over without human intervention. The table below maps RTO targets to the typical strategies that can meet them, and it is worth noting that the progression is roughly exponential in cost.
RTO = Time to restore service after failure
Strategies by RTO:
- 4 hours : Daily snapshots + restore (low cost)
- 1 hour : Hot standby + manual failover
- 15 min : Automated failover + warm standby
- 1 min : Active-active replication
- < 30 sec : Instant failover (5 nines)
Recovery Point Objective (RPO)
Where RTO governs the outage itself, RPO governs the aftermath. Every time data is written, there is a window of vulnerability between the write reaching the database and the write being durably copied to a backup or replica. If a failure strikes inside that window, the lost writes are gone forever. RPO is the maximum acceptable size of that window, and it is expressed as a duration: a one-hour RPO means the business can tolerate losing at most the last hour of data.
The strategies below mirror the RTO ladder in cost and complexity. Nightly backups give you a 24-hour RPO at minimal cost, but they also guarantee that up to a full day of transactions can vanish. Moving to continuous replication or write-ahead log shipping narrows the window toward seconds, but requires the network bandwidth and storage to keep a live copy continuously in sync. Note the fundamental tension between RPO and RTO: aggressive values for both simultaneously are the most expensive to engineer, so mature organizations deliberately choose modest targets that match the business’s actual tolerance rather than chasing the best available numbers.
RPO = Data loss acceptable after failure
Strategies by RPO:
- 24 hours : Nightly backups (acceptable data loss)
- 1 hour : Hourly snapshots
- 15 min : Continuous replication (sync)
- < 1 sec : Real-time sync + write-ahead logs
RTO/RPO Calculator
Picking RTO and RPO targets is a business decision, but it should be an informed one. The calculator below helps make the financial trade-off explicit by comparing the annual cost of prevention against the expected cost of failure. On one side sits the fixed annual cost of each strategy, from a modest $5,000 for daily backups to a quarter-million dollars for instant failover infrastructure. On the other side sits the expected loss: the number of hours the service is expected to be down per year multiplied by the business impact per hour, plus the cost of losing the records that fall inside the RPO window.
This style of analysis is valuable because it frequently overturns intuition. A business losing $50,000 per hour might assume it needs the most aggressive strategy, but the math may show that a one-hour RTO with hourly snapshots produces acceptable expected risk at a fraction of the cost, freeing budget for other improvements. Conversely, a company with modest traffic but extremely high-value data might find that data loss cost dominates, pushing it toward continuous replication even though its RTO is relaxed. The code below iterates over three candidate strategies and prints the annual risk of each, giving decision-makers numbers rather than gut feel.
class DisasterRecoveryPlanning:
"""Plan RTO/RPO strategy based on business needs"""
def __init__(self, business_impact_per_hour: float):
self.impact_per_hour = business_impact_per_hour
def analyze_strategy(self, rto_hours: float, rpo_hours: float):
"""Analyze cost vs business impact"""
recovery_cost = {
'daily_backup': 5000, # Annual cost
'hourly_snapshot': 15000,
'continuous_replication': 50000,
'active_active': 150000,
'instant_failover': 250000
}
# Calculate failure cost
annual_failure_hours = (365 * 24) / (24/rto_hours) # Simplified
annual_failure_cost = annual_failure_hours * self.impact_per_hour
# Data loss cost
avg_data_loss_records = (rpo_hours * 1000) # 1k records/hour
data_loss_cost = avg_data_loss_records * 50 # $50 per record
return {
'rto_hours': rto_hours,
'rpo_hours': rpo_hours,
'annual_failure_cost': annual_failure_cost,
'data_loss_cost': data_loss_cost,
'total_risk': annual_failure_cost + data_loss_cost
}
## Example: E-commerce losing $50k per hour of downtime
dr = DisasterRecoveryPlanning(business_impact_per_hour=50000)
for strategy, cost in [
('Daily backups (24h RTO)', 24),
('Hourly snapshots (1h RTO)', 1),
('Replication (15min RTO)', 0.25),
]:
result = dr.analyze_strategy(rto_hours=cost, rpo_hours=cost/2)
print(f"{strategy}: ${result['total_risk']:,.0f} annual risk")
Two simplifications in this model are worth flagging so you do not mistake it for a precise predictor. The failure frequency is treated as uniform across the year even though real outages cluster around specific risk periods, and the cost of lost data is linear when in practice one catastrophic data loss can dwarf many small ones. Use this calculator to compare strategies relative to one another and to frame the conversation with stakeholders, then validate the assumptions with real incident history before committing to a plan.
Backup Strategies
Incremental Backups
Redundancy and replication protect against hardware failure, but they do not protect against accidental deletion, malicious corruption, or a bug that quietly rewrites production data. For those threats you need backups, and backups must be treated as a separate discipline with its own testing cadence. The defining constraint of backup design is the trade-off between the frequency of backups and their cost: taking a full backup of every file every time is simple but wastes storage and bandwidth on data that has not changed.
Incremental backups solve this by tracking which files actually changed since the last backup and copying only those. The class below implements the pattern with a hash-based change detector. Each file’s SHA-256 digest is recorded in a manifest, and on the next run any file whose digest differs is considered changed and copied. Hashing is superior to timestamp comparison because it catches modifications that preserve modification times, such as some editor saves and in-place updates. The manifest doubles as a record of exactly what was captured, which makes restore verification and audit compliance far easier.
import hashlib
from datetime import datetime
from pathlib import Path
class IncrementalBackup:
"""Efficient incremental backup with change tracking"""
def __init__(self, backup_dir: str):
self.backup_dir = Path(backup_dir)
self.manifest = {} # Track file hashes
def calculate_file_hash(self, filepath: str) -> str:
"""Calculate SHA256 of file"""
sha256 = hashlib.sha256()
with open(filepath, 'rb') as f:
for chunk in iter(lambda: f.read(4096), b''):
sha256.update(chunk)
return sha256.hexdigest()
def backup_incrementally(self, data_dir: str) -> dict:
"""Only backup changed files"""
backup_meta = {
'timestamp': datetime.utcnow().isoformat(),
'files': {}
}
for filepath in Path(data_dir).rglob('*'):
if filepath.is_file():
file_hash = self.calculate_file_hash(str(filepath))
# Check if file changed since last backup
prev_hash = self.manifest.get(str(filepath))
if prev_hash != file_hash:
# Only copy changed files
backup_path = self.backup_dir / filepath.name
backup_path.write_bytes(filepath.read_bytes())
backup_meta['files'][str(filepath)] = {
'hash': file_hash,
'size': filepath.stat().st_size
}
# Update manifest
self.manifest[str(filepath)] = file_hash
return backup_meta
def restore_from_backup(self, target_dir: str):
"""Restore from backup"""
for backup_file in self.backup_dir.glob('*'):
restore_path = Path(target_dir) / backup_file.name
restore_path.write_bytes(backup_file.read_bytes())
print(f"Restored {backup_file.name}")
## Usage
backup = IncrementalBackup('/backups')
meta = backup.backup_incrementally('/data')
print(f"Backed up {len(meta['files'])} changed files")
This implementation also highlights why incremental backups are never sufficient on their own. If the manifest itself is lost or corrupted, you cannot reconstruct the full backup set because the earlier full and incremental generations are only meaningful together. Production systems therefore keep the manifest alongside the backups in the same location, retain full backups on a rolling schedule, and store everything in a different facility from the primary data. Whatever strategy you adopt, the single most important rule is that a backup you have never restored from does not exist.
Cross-Region Backup
Once backups exist, they must be protected from the same disasters that motivated them. A backup stored next to the primary database fails exactly when you need it: the region-level outage that destroyed the database also destroys the backup. Cross-region backup replication solves this by automatically copying every backup artifact to an object store in another region, so the recovery path no longer depends on the source region being healthy.
The manifest below shows a CloudNativePG cluster configured with two Barman object stores. The primary store sits in the same region as the database for fast local backup, while a second, external store in another region receives replicated copies. Both stores are addressed with the same credentials pattern, and the cluster operator handles the orchestration of moving backup data between them. This is a common and pragmatic pattern: keep one copy local for fast restores, and a second copy remote for surviving a regional disaster, without forcing every recovery to travel over a wide-area network.
## AWS S3 with cross-region replication
apiVersion: storage.cnpg.io/v1
kind: Cluster
metadata:
name: postgres-multi-region
spec:
instances: 3
# Primary backup
backup:
barmanObjectStore:
destinationPath: s3://backups-us-east
s3Credentials:
accessKeyId:
name: aws-creds
key: access_key
secretAccessKey:
name: aws-creds
key: secret_key
# Secondary region backup
externalClusters:
- name: backup-eu
barmanObjectStore:
destinationPath: s3://backups-eu-west
s3Credentials:
accessKeyId:
name: aws-creds
key: access_key
secretAccessKey:
name: aws-creds
key: secret_key
Two operational cautions apply to any cross-region backup setup. First, credentials for both stores must be kept in a secret manager and rotated on a schedule, since a leaked backup key undermines the security of the entire recovery chain. Second, you must verify that the replicated copy is actually restorable from the target region, because misconfigured replication policies sometimes produce silent failures where artifacts appear to copy but cannot be read back. Testing the restore from the remote region is the only way to prove the setup works.
Testing Failover
Chaos Engineering
The most common reason disaster recovery fails in production is not missing infrastructure but untested procedure. Manual failover runbooks are valuable, but humans are slow, inconsistent, and unavailable at 3 a.m. during an actual incident. Chaos engineering addresses this by deliberately injecting failures into a running system on a regular schedule, so that failover mechanisms are exercised continuously rather than discovered to be broken during the one outage that matters. The philosophy is simple: if your system cannot survive a controlled, planned failure in a test environment, it will certainly not survive an unplanned one in production.
The FailoverTesting class below outlines the three layers of failure you should practice. Instance
failure kills a single compute node and verifies that the service recovers within its RTO budget.
Database failure kills the primary and asserts that a secondary is promoted with no data loss,
checking the two properties that matter most: role transition and data integrity. Region failure is
the most dramatic, partitioning an entire region out of the network and confirming that DNS and
routing redirect traffic elsewhere. Each test measures the time to recover, which feeds directly
back into validating or invalidating your declared RTO targets.
from chaos_monkey import ChaosMonkey
from datetime import datetime
class FailoverTesting:
"""Regularly test failover capabilities"""
def __init__(self):
self.chaos = ChaosMonkey()
def test_instance_failure(self):
"""Kill random instance and verify failover"""
print(f"[{datetime.now()}] Starting instance failure test")
# Kill instance
victim = self.chaos.kill_random_instance()
print(f"Killed instance: {victim}")
# Monitor failover
start_time = datetime.now()
while not self.is_healthy():
elapsed = (datetime.now() - start_time).total_seconds()
if elapsed > 300: # 5 minute timeout
raise TimeoutError("Failover took too long")
time.sleep(5)
failover_time = (datetime.now() - start_time).total_seconds()
print(f"Service recovered in {failover_time:.1f} seconds (RTO: 5min)")
def test_database_failure(self):
"""Fail primary database and verify promotion"""
print("Starting database failover test")
# Fail primary
self.chaos.kill_db_primary()
# Verify secondary promoted
assert self.get_db_role() == 'primary'
print("Secondary promoted to primary ✓")
# Verify no data loss
assert self.verify_data_integrity()
print("All data intact ✓")
def test_region_failure(self):
"""Simulate entire region failure"""
print("Starting region failure test")
# Disable entire region
self.chaos.partition_network_segment('us-east-1')
# Verify traffic routed to other regions
start = datetime.now()
while not self.traffic_routed_to('eu-west-1'):
if (datetime.now() - start).seconds > 60:
raise TimeoutError("DNS failover delayed")
time.sleep(1)
print("Traffic failed over to eu-west-1 ✓")
def is_healthy(self) -> bool:
"""Check system health"""
# Verify API responds
# Check database is writable
# Validate data consistency
pass
Two design choices in this code deserve emphasis. Every test is bounded by an explicit timeout, because a chaos test that hangs is itself an incident; if recovery has not completed within the budgeted window, the test fails loudly rather than masking the problem. And the health check is a composite: the API must respond, the database must accept writes, and data must be consistent. Checking only that a process is alive produces false confidence, since a replica set can be healthy by process count while refusing writes. Run these tests on a regular schedule and after every significant architecture change, and treat each regression as a bug in your availability story, not merely a test failure.
Glossary
- RTO: Recovery Time Objective - time to restore after failure
- RPO: Recovery Point Objective - acceptable data loss
- Multi-AZ: Multiple Availability Zones in same region
- Multi-region: Active deployment across geographic regions
- Failover: Automatic switch to backup resource
- MTBF: Mean Time Between Failures
- MTTR: Mean Time To Repair
Conclusion
High availability and disaster recovery require deliberate architecture, not afterthoughts. Design for failure from day one: redundant infrastructure, automated failover, and regularly tested recovery procedures. The key metric is not whether failures happen but how quickly and completely you recover when they do.
Resources
- AWS Well-Architected Framework: Reliability
- Google Cloud High Availability
- Azure Disaster Recovery
- Chaos Engineering Handbook
Comments