Database replication distributes data across multiple nodes for high availability, read scaling, and disaster recovery. This guide covers replication strategies and implementations.
Replication Types
replication_types:
- name: "Primary-Replica (Leader-Follower)"
description: "One primary, multiple replicas"
pros: "Simple, consistent writes"
cons: "Single point of failure for writes"
- name: "Multi-Master"
description: "Multiple primaries accept writes"
pros: "No write bottleneck"
cons: "Conflict resolution complex"
- name: "Leaderless"
description: "No designated leader"
pros: "High availability"
cons: "Eventual consistency"
Primary-Replica Replication
Implementation
# PostgreSQL Primary-Replica setup
# PRIMARY (write to master)
import psycopg2
class PrimaryDatabase:
def __init__(self, connection_string):
self.conn = psycopg2.connect(connection_string)
def write(self, query, params):
with self.conn.cursor() as cur:
cur.execute(query, params)
self.conn.commit()
def get_replication_lag(self):
with self.conn.cursor() as cur:
cur.execute("""
SELECT now() - pg_last_xact_replay_timestamp()
AS lag
""")
return cur.fetchone()[0]
# Read from replica (load balancing)
class ReplicaPool:
def __init__(self, replica_hosts):
self.hosts = replica_hosts
self.current = 0
def read(self, query, params=None):
host = self.hosts[self.current]
self.current = (self.current + 1) % len(self.hosts)
conn = psycopg2.connect(host)
try:
with conn.cursor() as cur:
cur.execute(query, params or ())
return cur.fetchall()
finally:
conn.close()
def is_replica_synchronized(self, max_lag=1):
"""Check if replica is caught up"""
for host in self.hosts:
conn = psycopg2.connect(host)
try:
with conn.cursor() as cur:
cur.execute("SELECT pg_is_in_recovery()")
is_recovering = cur.fetchone()[0]
if is_recovering:
return False
finally:
conn.close()
return True
Streaming Replication
# PostgreSQL streaming replication config
# postgresql.conf (primary)
wal_level = replica
max_wal_senders = 3
wal_keep_size = 1GB
# postgresql.conf (replica)
primary_conninfo = 'host=primary port=5432 user=replication'
# Create replication slot
# On replica:
# pg_create_physical_replication_slot('replica_slot')
Multi-Master Replication
Conflict Resolution
class MultiMasterDB:
"""Simple last-writer-wins conflict resolution"""
def __init__(self, nodes):
self.nodes = nodes
def write(self, key, value, timestamp=None):
if timestamp is None:
timestamp = time.time()
# Write to all nodes
for node in self.nodes:
node.put(key, value, timestamp)
def read(self, key):
# Read from all and pick latest
latest = None
latest_value = None
for node in self.nodes:
value, ts = node.get(key)
if ts and (latest is None or ts > latest):
latest = ts
latest_value = value
return latest_value
# More sophisticated: CRDTs (Conflict-free Replicated Data Types)
from collections import Counter
class GCounter:
"""Grow-only counter CRDT"""
def __init__(self):
self.counts = {} # node_id -> count
def increment(self, node_id):
self.counts[node_id] = self.counts.get(node_id, 0) + 1
def value(self):
return sum(self.counts.values())
def merge(self, other):
for node_id, count in other.counts.items():
self.counts[node_id] = max(
self.counts.get(node_id, 0),
count
)
Leaderless Replication
Dynamo-Style Quorum
class LeaderlessDB:
def __init__(self, nodes, r=2, w=2):
self.nodes = nodes
self.r = r # Read quorum
self.w = w # Write quorum
def write(self, key, value):
# Write to N nodes
success_count = 0
for node in self.nodes:
if node.put(key, value):
success_count += 1
return success_count >= self.w
def read(self, key):
# Read from R nodes
values = []
for node in self.nodes[:self.r]:
value = node.get(key)
if value:
values.append(value)
# Return most recent (simplified)
return max(values, key=lambda v: v.timestamp) if values else None
Monitoring Replication
# Monitor replication health
def check_replication_health(conn, replica_conns):
health = {
"primary": {"healthy": True},
"replicas": []
}
# Check primary
with conn.cursor() as cur:
cur.execute("SELECT pg_is_in_recovery()")
health["primary"]["healthy"] = not cur.fetchone()[0]
# Check each replica
for replica in replica_conns:
try:
with replica.cursor() as cur:
# Check if receiving
cur.execute("SELECT pg_is_in_recovery()")
is_replica = cur.fetchone()[0]
# Get lag
if is_replica:
cur.execute("""
SELECT now() - pg_last_xact_replay_timestamp()
AS lag
""")
lag = cur.fetchone()[0]
health["replicas"].append({
"healthy": True,
"lag_seconds": lag.total_seconds()
})
except Exception as e:
health["replicas"].append({
"healthy": False,
"error": str(e)
})
return health
Synchronous vs Asynchronous Replication
Consistency vs Latency Trade-off
| Aspect | Synchronous | Asynchronous |
|---|---|---|
| Durability | Confirms to replica before commit | Confirms to primary only |
| Latency | Higher (wait for replica) | Lower |
| Data loss on failover | Zero (if quorum met) | Possible (unreplicated tail) |
| Availability | Lower (replica must respond) | Higher |
| Use case | Financial transactions, critical data | Read scaling, analytics |
Semi-Synchronous (Best of Both)
# PostgreSQL synchronous replication
synchronous_commit = on
synchronous_standby_names = 'replica1, replica2'
Semi-synchronous mode confirms writes to at least one replica before acknowledging the client. It balances durability and latency. For maximum data safety in a three-node cluster, use synchronous with quorum:
synchronous_commit = on
synchronous_standby_names = 'ANY 2 (replica1, replica2, replica3)'
Replication Topologies
Star Topology
┌── Replica A
│
Primary ──┼── Replica B
│
└── Replica C
Simple, single-hop. Good for most read-scaling needs.
Cascade Topology
Primary → Replica A → Replica B
↓
Replica C
Reduces load on primary. Each downstream replica lags one more hop. Used when primary cannot handle many WAL senders.
Ring Topology (Multi-Master)
Node A ↔ Node B
↕ ↕
Node D ↔ Node C
All nodes accept writes. Complex conflict resolution. Used in geographically distributed multi-master setups.
Conflict Resolution Strategies
| Strategy | Description | Use Case |
|---|---|---|
| Last-Writer-Wins (LWW) | Highest timestamp wins | Simple, most common |
| Merge functions | Combine conflicting values | CRDTs, counters |
| Application-level resolution | App decides on conflict | Business-specific rules |
| Version vectors | Track causality | Distributed systems |
| Conflict-free CRDTs | Mathematically mergeable | Counters, sets, maps |
CRDT Types
| CRDT | Operation | Merge Rule |
|---|---|---|
| G-Counter | Increment only | Max of each node count |
| PN-Counter | Increment + decrement | Sum of G and N counters |
| G-Set | Add only | Union |
| OR-Set | Add + remove | Union with tombstones |
| LWW-Register | Set value | Highest timestamp wins |
Read/Write Consistency Levels
Quorum-Based Systems
N = total nodes, R = read quorum, W = write quorum
Strong consistency: R + W > N
Eventual consistency: R + W <= N
Consistency Configuration Examples
| Configuration | R | W | N | Property |
|---|---|---|---|---|
| Strong | 3 | 3 | 3 | R+W=6>3, highest latency |
| Typical | 2 | 2 | 3 | R+W=4>3, good balance |
| Write-heavy | 1 | 3 | 3 | Fast reads, strong writes |
| Read-heavy | 3 | 1 | 3 | Fast writes, consistent reads |
| Eventual | 1 | 1 | 3 | Lowest latency, may read stale |
Failover and High Availability
Automated Failover
# PostgreSQL + Patroni
patroni:
bootstrap:
dcs:
ttl: 30
loop_wait: 10
retry_timeout: 10
maximum_lag_on_failover: 1048576
postgresql:
use_pg_rewind: true
parameters:
wal_level: replica
hot_standby: "on"
max_connections: 100
Failover Best Practices
| Practice | Why |
|---|---|
| Test failover regularly | Verify it works before you need it |
| Automate promotion | Reduce human error and RTO |
| Use health checks | Detect primary failure quickly |
| Set proper max lag | Avoid promoting a stale replica |
| Prefer quorum-based | Avoid split-brain |
| Document runbooks | Ensure operators know the procedure |
Split-Brain Problem
Split-brain occurs when replicas cannot communicate and multiple nodes believe they are primary. Mitigations:
- Quorum/consensus (etcd, ZooKeeper, Patroni DCS): only majority node can be primary
- Fencing tokens: old primary rejects writes after failover
- Network partitions: lease-based leadership with expiry
Replication Lag Management
Causes of Lag
| Cause | Symptom | Fix |
|---|---|---|
| Large write bursts | Lag spikes | Throttle writes, increase replicas |
| Slow replica hardware | Sustained lag | Right-size replicas |
| Long transactions | Blocked apply | Split transactions |
| Index rebuilds | Stalled replication | Schedule off-peak |
| Network latency | Consistent small lag | Co-locate replicas |
Monitoring Replication Lag
# Monitor replication health
def check_replication_health(conn, replica_conns):
health = {
"primary": {"healthy": True},
"replicas": []
}
# Check primary
with conn.cursor() as cur:
cur.execute("SELECT pg_is_in_recovery()")
health["primary"]["healthy"] = not cur.fetchone()[0]
# Check each replica
for replica in replica_conns:
try:
with replica.cursor() as cur:
# Check if receiving
cur.execute("SELECT pg_is_in_recovery()")
is_replica = cur.fetchone()[0]
# Get lag
if is_replica:
cur.execute("""
SELECT now() - pg_last_xact_replay_timestamp()
AS lag
""")
lag = cur.fetchone()[0]
health["replicas"].append({
"healthy": True,
"lag_seconds": lag.total_seconds()
})
except Exception as e:
health["replicas"].append({
"healthy": False,
"error": str(e)
})
return health
Lag Alerting Thresholds
| Metric | Warning | Critical | Action |
|---|---|---|---|
| Replication lag | > 5 seconds | > 60 seconds | Investigate, throttle writes |
| Replica down | 1 replica | All replicas | Failover readiness |
| Disk usage | > 70% | > 85% | Clean up WAL, scale storage |
| WAL senders | > 80% used | > 95% used | Add senders, cascade |
Replication Implementation by Database
PostgreSQL
# Streaming replication
wal_level = replica
max_wal_senders = 3
hot_standby = on
synchronous_commit = on
MySQL
# MySQL source-replica replication
# my.cnf (source)
server_id = 1
log_bin = mysql-bin
binlog_format = ROW
# my.cnf (replica)
server_id = 2
read_only = ON
relay_log = mysql-relay-bin
MongoDB
# MongoDB replica set (primary + 2 secondaries)
replication:
replSetName: rs0
Redis
# Redis master-replica
replicaof 127.0.0.1 6379
Cassandra (Leaderless)
# Cassandra replication (network topology strategy)
replication:
class: NetworkTopologyStrategy
datacenter1: 3
datacenter2: 3
Replication Read/Write Splitting
A common pattern is to route reads to replicas and writes to primary:
class ReadWriteRouter:
"""Route reads to replicas, writes to primary."""
def __init__(self, primary, replicas):
self.primary = primary
self.replicas = replicas
self.replica_idx = 0
def execute(self, query, params=None, is_write=False):
if is_write or not self._is_read_only(query):
return self.primary.execute(query, params)
return self._read_from_replica(query, params)
def _is_read_only(self, query):
return query.strip().upper().startswith(
('SELECT', 'SHOW', 'EXPLAIN')
)
def _read_from_replica(self, query, params):
replica = self.replicas[self.replica_idx % len(self.replicas)]
self.replica_idx += 1
return replica.execute(query, params)
Read Replica Staleness Handling
Applications must handle the possibility of reading stale data from replicas:
| Pattern | Approach | Trade-off |
|---|---|---|
| Read-your-writes | Route recent writes to primary | Extra primary load |
| Max lag tolerance | Accept lag < threshold | Possible stale reads |
| Version check | Compare data version | Extra query |
| Session affinity | Pin session to replica | Uneven replica load |
Replication and Backups
Replication complements but does not replace backups:
| Concern | Replication | Backup |
|---|---|---|
| Recovery from corruption | No | Yes |
| Recovery from deletion | No (propagates) | Yes (PITR) |
| Recovery from errors | No | Yes |
| High availability | Yes | No |
| Read scaling | Yes | No |
| Disaster recovery | Yes | Yes |
Best practice: Use both. Replication for availability and read scaling; backups with point-in-time recovery for data protection. Never rely on replication alone — a bad UPDATE replicates to all nodes.
Multi-Region Replication
Regional vs Global Deployment
| Aspect | Single Region | Multi-Region |
|---|---|---|
| Write latency | Low | High (cross-region) |
| Read latency | Region-local | Global, low |
| Disaster recovery | Zone-based | Region-based |
| Consistency | Strong | Eventual (or sync) |
| Cost | Lower | Higher (cross-region transfer) |
| Compliance | Local | Data residency options |
Active-Active Multi-Region
Region A (primary) ←→ Region B (primary)
↓ ↓
Replica A1 Replica B1
Replica A2 Replica B2
Both regions accept writes. Conflict resolution via timestamps or CRDTs. Used by global SaaS to place writes near users.
Replication Testing Checklist
- Verify lag returns to zero after catch-up
- Test failover with a stale replica (high lag)
- Test primary failure detection timing
- Verify automatic promotion works
- Confirm old primary doesn’t resume writes (split-brain)
- Test read traffic routing after failover
- Verify monitoring alerts fire correctly
- Test replica addition and removal
- Verify backups work alongside replication
- Test recovery after network partition heals
Common Replication Failures
| Failure | Symptom | Fix |
|---|---|---|
| Replica stuck | Lag grows indefinitely | Restart apply, check WAL |
| Broken replication | “replication slot is active” | Drop and recreate slot |
| WAL disk full | Primary fails writes | Archive WAL, increase disk |
| Replica OOM | Memory pressure | Right-size replica, reduce connections |
| Config drift | Replicas out of sync | Verify config, restart |
| Split-brain | Two primaries | Restore quorum, demote loser |
| Cross-region latency | High write latency | Async replication, local writes |
Frequently Asked Questions
Q: How many replicas should I have? A: Minimum 3 total nodes (1 primary + 2 replicas) for failover safety. More replicas for read scaling and regional coverage. Balance cost against availability requirements.
Q: What’s an acceptable replication lag? A: Depends on use case. Analytics can tolerate minutes; user-facing reads need sub-second lag. Alert on sustained lag > 60 seconds regardless.
Q: Does replication protect against data corruption? A: No. Corruption and logical errors replicate to all nodes. You need backups with point-in-time recovery for protection against bad data.
Q: Can I use the same replica for reads and analytics? A: Yes, but heavy analytics queries can impact read latency. Consider a dedicated analytics replica or a read replica with different resource allocation.
Q: What happens if a replica fails? A: Reads fail over to other replicas automatically. The failed replica can be rebuilt from the primary or another replica. Monitor and replace promptly.
Best Practices
# Replication best practices
configuration:
- "Use async for lower latency"
- "Sync for critical data"
- "3 replicas minimum"
- "Spread across AZs"
- "Enable semi-sync for durability"
- "Configure proper quorum"
monitoring:
- "Track replication lag"
- "Alert on high lag"
- "Monitor disk usage"
- "Watch replica health"
- "Track WAL sender usage"
failover:
- "Automatic failover"
- "Test failover regularly"
- "Promote carefully"
- "Use quorum to avoid split-brain"
- "Verify lag before promotion"
Replication Strategy Decision Guide
| Requirement | Recommended Strategy |
|---|---|
| Read scaling, single region | Primary-replica (async) |
| Zero data loss | Synchronous or semi-synchronous |
| Global writes, low latency | Multi-master (regional) |
| Highest availability | Leaderless (quorum) |
| Simple setup | Primary-replica |
| Analytics offloading | Primary + analytics replica |
| Compliance (multi-region) | Primary-replica with sync replicas |
Cost Considerations
| Factor | Primary-Replica | Multi-Master | Leaderless |
|---|---|---|---|
| Write latency | Low (async) / High (sync) | High (cross-region) | Medium |
| Infrastructure | N nodes | 2N+ nodes | N nodes |
| Operational complexity | Low | High | Medium |
| Conflict handling | None | Complex | Via quorum |
| Best for | Most apps | Global distribution | Large-scale, fault-tolerant |
Conclusion
Choose replication strategy:
- Primary-replica: Simple, read scaling, most common
- Multi-master: Write scaling, needs conflict resolution
- Leaderless: Highest availability, eventual consistency
Key principles:
- Match consistency to needs — synchronous for critical data, async for reads
- Monitor lag continuously — lag is the leading indicator of replication problems
- Test failover regularly — an untested failover procedure will fail when needed
- Use quorum wisely — R + W > N for consistency, with latency trade-offs
- Plan for split-brain — use consensus or fencing to prevent multi-primary
Always monitor replication lag and test failover procedures.
Replication Strategy Comparison Summary
| Strategy | Writes | Reads | Consistency | Latency | Complexity | Use Case |
|---|---|---|---|---|---|---|
| Primary-Replica (async) | Primary only | Any replica | Eventual | Low | Low | Read scaling |
| Primary-Replica (sync) | Primary only | Any replica | Strong | Higher | Medium | Zero data loss |
| Semi-sync | Primary only | Any replica | Near-strong | Moderate | Medium | Balanced |
| Multi-Master | Any master | Any node | Eventual (LWW) | Varies | High | Global writes |
| Leaderless (quorum) | Any node | Any node | Quorum-based | Medium | Medium | High availability |
| Cascade | Primary → relay | Any node | Eventual | High lag | Medium | Reduce primary load |
Replication Implementation Patterns
Pattern: Read-Heavy Web Application
Writes → Primary
Reads → Replica 1, Replica 2 (round-robin)
Analytics → Dedicated Replica 3
Failover → Patroni manages promotion
Pattern: Global SaaS
US Region: Primary-US + 2 replicas
EU Region: Primary-EU + 2 replicas
APAC Region: Primary-APAC + 2 replicas
Cross-region: Async bidirectional replication
Conflict: LWW with application-level merge
Pattern: High-Durability Financial
Primary + 2 sync replicas (quorum)
synchronous_commit = on
synchronous_standby_names = 'ANY 2 (...)'
Async replicas for analytics/backup
Backups with PITR every 5 minutes
Replication Resource Requirements
| Node Type | CPU | Memory | Disk | Network |
|---|---|---|---|---|
| Primary | High (writes) | High (buffer pool) | Fast SSD | High (WAL senders) |
| Sync replica | Medium | High (buffer pool) | Fast SSD | High (must confirm) |
| Async replica | Medium | Medium | SSD | Medium |
| Analytics replica | High (queries) | High | Large | Medium |
Summary
Replication is a cornerstone of database reliability and performance. Choose the strategy that matches your consistency, latency, and availability requirements:
- Start with primary-replica for most applications
- Add synchronous commit for critical data
- Scale to multi-master for global distribution
- Use leaderless when availability trumps consistency
- Monitor lag continuously — it’s the health signal
- Test failover regularly — untested failover fails
Related Articles
- ACID vs BASE: CAP Theorem
- NewSQL Databases
- Database Failover: High Availability Strategies
- Database Connection Pooling
- Database Replication: Cassandra
Comments