The Outbox Pattern solves a fundamental problem in microservices: how to reliably publish events when updating a database, without requiring distributed transactions. It ensures that database changes and event publishing happen atomically.
The Dual-Write Problem
The Outbox Pattern solves the dual-write problem: the challenge of atomically updating a database and publishing an event. Without the outbox, you face an impossible dilemma. Write to the database first, and the event may be lost if the publish fails. Publish the event first, and the consumer may act on stale data if the subsequent database write fails.
The pattern works in three steps: (1) write the event to an outbox table within the same database transaction as the business data, (2) a separate process reads the outbox table and publishes events to the message broker, and (3) events are deleted or marked as published after successful delivery. This guarantees at-least-once delivery, meaning consumers must be idempotent.
Two implementation approaches exist. The transactional outbox uses the same database transaction for strong consistency. Change data capture (CDC) with tools like Debezium and Kafka Connect streams database changes directly, offering looser coupling at the cost of higher latency. The outbox table itself should be lightweight: event_id, aggregate_id, event_type, payload (JSON), and created_at. A common pitfall is publishing events without removing them from the outbox, causing infinite replays. Always monitor outbox table size — sustained growth indicates a publishing failure.
When to Use the Outbox Pattern
The outbox pattern is the right choice when your microservices need to maintain consistency between database state and event emissions — the classic dual-write scenario. You should reach for it when:
- Database changes must trigger events that downstream services depend on. If an order status change must reliably produce an
OrderPaidEvent, the outbox guarantees that relationship. - You cannot use distributed transactions. Two-phase commit across a database and a message broker is slow, fragile, and unsupported by most brokers. The outbox achieves the same atomicity without it.
- At-least-once delivery is acceptable. If your consumers can handle duplicates via idempotency, the outbox is a clean fit.
Conversely, the outbox is overkill when you don’t need strict consistency, or when event delivery can tolerate loss. For fire-and-forget analytics events or non-critical notifications, a simpler direct-publish approach with retries is often sufficient.
Trade-offs at a Glance
| Approach | Consistency | Latency | Complexity | Best For |
|---|---|---|---|---|
| Direct publish (no outbox) | None | Lowest | Lowest | Non-critical, loss-tolerant events |
| Transactional outbox + polling | Strong | Poll interval | Medium | Most business-critical events |
| Transactional outbox + CDC | Strong | Near real-time | High | High-volume, low-latency needs |
The Problem
Without Outbox Pattern
┌─────────────────────────────────────────────────────────────────┐
│ Race Condition Without Outbox Pattern │
│ │
│ ┌──────────────────┐ ┌──────────────────┐ │
│ │ Order Service │ │ Message Queue │ │
│ │ │ │ │ │
│ │ 1. UPDATE order │ │ │ │
│ │ status='paid'│ │ │ │
│ │ │ │ │ │
│ │ 2. PUBLISH │ ────▶│ OrderPaidEvent │ │
│ │ event │ │ │ │
│ └──────────────────┘ └──────────────────┘ │
│ │ │
│ ▼ │
│ Problems: │
│ ✗ If step 2 fails: DB updated but no event │
│ ✗ If step 2 crashes: Inconsistent state │
│ ✗ If step 2 times out: Unknown if event published │
│ ✗ No atomicity between DB and message queue │
└─────────────────────────────────────────────────────────────────┘
With Outbox Pattern
┌─────────────────────────────────────────────────────────────────┐
│ Outbox Pattern Solution │
│ │
│ ┌───────────────────────────────────────────────────────────┐ │
│ │ Order Service (Single DB) │ │
│ │ │ │
│ │ ┌─────────────────────────────────────────────────┐ │ │
│ │ │ orders table │ │ │
│ │ │ ───────────────────────────────────────────── │ │ │
│ │ │ id: 123 │ │ │
│ │ │ status: 'paid' │ │ │
│ │ │ ... │ │ │
│ │ └─────────────────────────────────────────────────┘ │ │
│ │ │ │ │
│ │ ┌─────────────────────────────────────────────────┐ │ │
│ │ │ outbox table (same transaction!) │ │ │
│ │ │ ───────────────────────────────────────────── │ │ │
│ │ │ id: 1 │ │ │
│ │ │ aggregate_type: 'Order' │ │ │
│ │ │ aggregate_id: '123' │ │ │
│ │ │ event_type: 'OrderPaidEvent' │ │ │
│ │ │ payload: {...} │ │ │
│ │ │ created_at: '2026-02-28T10:00:00Z' │ │ │
│ │ └─────────────────────────────────────────────────┘ │ │
│ └───────────────────────────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ ┌───────────────────────────────────────────────────────────┐ │
│ │ Outbox Relay (Background Worker) │ │
│ │ │ │
│ │ 1. SELECT * FROM outbox WHERE processed = false │ │
│ │ 2. PUBLISH each event to message broker │ │
│ │ 3. UPDATE outbox SET processed = true │ │
│ └───────────────────────────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ ┌──────────────────┐ ┌──────────────────┐ │
│ │ Inventory │ │ Notification │ │
│ │ Service │ │ Service │ │
│ └──────────────────┘ └──────────────────┘ │
│ │
│ ✓ Atomic: Both DB update and outbox write in one transaction │
│ ✓ Reliable: Events guaranteed to be published │
│ ✓ Simple: No distributed transactions needed │
└─────────────────────────────────────────────────────────────────┘
Implementation
Database Schema
The outbox table is the heart of the pattern. It stores events in the same database as the business data, so the two writes happen atomically. Let’s examine the key design decisions:
The payload column holds the event data as JSON. Using JSONB in PostgreSQL (or JSON in MySQL) rather than a plain TEXT column lets you query payload fields directly and provides faster serialization. The metadata column stores routing information like trace IDs or correlation IDs that aren’t part of the event payload itself.
The processed boolean flag tracks whether an event has been published. Combined with retry_count and last_error, it enables the retry logic we’ll build later — the relay skips events that have already failed too many times.
The most important indexing decision is the partial index on created_at WHERE processed = FALSE. This is what makes polling efficient: the relay queries only unprocessed events ordered by creation time, and the index keeps that lookup fast even as the table grows to millions of rows. Without it, every poll would trigger a full table scan.
-- PostgreSQL outbox table
CREATE TABLE outbox (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
aggregate_type VARCHAR(100) NOT NULL,
aggregate_id VARCHAR(100) NOT NULL,
event_type VARCHAR(100) NOT NULL,
payload JSONB NOT NULL,
metadata JSONB DEFAULT '{}',
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
processed_at TIMESTAMPTZ,
retry_count INT DEFAULT 0,
last_error TEXT,
processed BOOLEAN DEFAULT FALSE
);
CREATE INDEX idx_outbox_unprocessed ON outbox(created_at)
WHERE processed = FALSE;
CREATE INDEX idx_outbox_aggregate ON outbox(aggregate_type, aggregate_id);
-- MySQL outbox table
CREATE TABLE outbox (
id CHAR(36) PRIMARY KEY,
aggregate_type VARCHAR(100) NOT NULL,
aggregate_id VARCHAR(100) NOT NULL,
event_type VARCHAR(100) NOT NULL,
payload JSON NOT NULL,
metadata JSON DEFAULT ('{}'),
created_at TIMESTAMP(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3),
processed_at TIMESTAMP(3),
retry_count INT DEFAULT 0,
last_error TEXT,
processed BOOLEAN DEFAULT FALSE,
INDEX idx_outbox_unprocessed (created_at)
) ENGINE=InnoDB;
Event Publisher Service
The event publisher service has two responsibilities: writing events to the outbox table (atomically with business data) and reading unprocessed events for delivery. The OutboxRepository class handles both sides.
The create_event method inserts a new row into the outbox. Crucially, this is designed to be called inside an existing database transaction alongside the business write — that’s what gives you atomicity. The get_unprocessed_events method reads events that haven’t been delivered yet, ordered by creation time so consumers see events in the order they occurred. The mark_processed and mark_failed methods update event state after delivery attempts.
The MessageBrokerPublisher wraps the message broker client. It publishes each event as a persistent message with structured headers (event-type, aggregate-id) so consumers can filter without parsing the payload. Using delivery_mode=PERSISTENT ensures the message survives broker restarts.
import asyncio
import json
import logging
from dataclasses import dataclass
from datetime import datetime
from typing import Optional
import aio_pika
import asyncpg
logger = logging.getLogger(__name__)
@dataclass
class OutboxEvent:
id: str
aggregate_type: str
aggregate_id: str
event_type: str
payload: dict
metadata: dict
created_at: datetime
class OutboxRepository:
def __init__(self, db_pool: asyncpg.Pool):
self.pool = db_pool
async def create_event(
self,
aggregate_type: str,
aggregate_id: str,
event_type: str,
payload: dict,
metadata: Optional[dict] = None
) -> OutboxEvent:
async with self.pool.acquire() as conn:
row = await conn.fetchrow("""
INSERT INTO outbox
(aggregate_type, aggregate_id, event_type, payload, metadata)
VALUES ($1, $2, $3, $4, $5)
RETURNING *
""", aggregate_type, aggregate_id, event_type,
json.dumps(payload), json.dumps(metadata or {}))
return self._row_to_event(row)
async def get_unprocessed_events(
self,
limit: int = 100
) -> list[OutboxEvent]:
async with self.pool.acquire() as conn:
rows = await conn.fetch("""
SELECT * FROM outbox
WHERE processed = FALSE
AND retry_count < 5
ORDER BY created_at ASC
LIMIT $1
""", limit)
return [self._row_to_event(row) for row in rows]
async def mark_processed(self, event_id: str):
async with self.pool.acquire() as conn:
await conn.execute("""
UPDATE outbox
SET processed = TRUE,
processed_at = NOW()
WHERE id = $1
""", event_id)
async def mark_failed(
self,
event_id: str,
error: str,
retry_count: int
):
async with self.pool.acquire() as conn:
await conn.execute("""
UPDATE outbox
SET last_error = $2,
retry_count = $3
WHERE id = $1
""", event_id, error, retry_count)
def _row_to_event(self, row) -> OutboxEvent:
return OutboxEvent(
id=str(row["id"]),
aggregate_type=row["aggregate_type"],
aggregate_id=str(row["aggregate_id"]),
event_type=row["event_type"],
payload=json.loads(row["payload"]),
metadata=json.loads(row["metadata"]),
created_at=row["created_at"]
)
class MessageBrokerPublisher:
def __init__(self, rabbitmq_url: str):
self.rabbitmq_url = rabbitmq_url
self.connection: Optional[aio_pika.Connection] = None
self.channel: Optional[aio_pika.Channel] = None
async def connect(self):
self.connection = await aio_pika.connect_robust(self.rabbitmq_url)
self.channel = await self.connection.channel()
async def publish(
self,
exchange: str,
routing_key: str,
event: OutboxEvent
):
message = aio_pika.Message(
body=json.dumps({
"event_id": event.id,
"event_type": event.event_type,
"aggregate_type": event.aggregate_type,
"aggregate_id": event.aggregate_id,
"payload": event.payload,
"metadata": event.metadata,
"timestamp": event.created_at.isoformat()
}).encode(),
content_type="application/json",
delivery_mode=aio_pika.DeliveryMode.PERSISTENT,
headers={
"event-type": event.event_type,
"aggregate-id": event.aggregate_id
}
)
await self.channel.default_exchange.publish(
message,
routing_key=routing_key
)
async def close(self):
if self.connection:
await self.connection.close()
Outbox Relay Worker
The relay is the background process that bridges the outbox table and the message broker. It runs in a continuous loop, polling for unprocessed events and publishing them.
The start method runs an infinite loop with a configurable poll interval. Each iteration calls _process_events, which fetches a batch of unprocessed events, publishes each to the broker, and marks it processed on success. On failure, it records the error and increments the retry counter instead of crashing — the event will be retried on the next poll.
The _get_exchange and _get_routing_key methods implement the routing strategy. Events are routed to per-aggregate exchanges (orders.events, payments.events) so consumers can subscribe to only the event types they care about. This keeps the outbox decoupled from consumer topology.
class OutboxRelay:
def __init__(
self,
outbox_repo: OutboxRepository,
publisher: MessageBrokerPublisher,
batch_size: int = 100,
poll_interval: float = 1.0
):
self.outbox = outbox_repo
self.publisher = publisher
self.batch_size = batch_size
self.poll_interval = poll_interval
self._running = False
async def start(self):
self._running = True
logger.info("Outbox relay started")
while self._running:
try:
await self._process_events()
except Exception as e:
logger.error(f"Error processing outbox: {e}")
await asyncio.sleep(self.poll_interval)
async def stop(self):
self._running = False
logger.info("Outbox relay stopped")
async def _process_events(self):
events = await self.outbox.get_unprocessed_events(self.batch_size)
if not events:
return
for event in events:
try:
exchange = self._get_exchange(event.aggregate_type)
routing_key = self._get_routing_key(event.event_type)
await self.publisher.publish(
exchange=exchange,
routing_key=routing_key,
event=event
)
await self.outbox.mark_processed(event.id)
logger.info(f"Published event {event.id}: {event.event_type}")
except Exception as e:
logger.error(f"Failed to publish event {event.id}: {e}")
await self.outbox.mark_failed(
event.id,
str(e),
retry_count=1
)
def _get_exchange(self, aggregate_type: str) -> str:
exchange_map = {
"Order": "orders.events",
"Payment": "payments.events",
"User": "users.events",
"Product": "products.events"
}
return exchange_map.get(aggregate_type, "default.events")
def _get_routing_key(self, event_type: str) -> str:
return event_type.lower().replace("event", "")
class OutboxRelayRunner:
def __init__(self, config: dict):
self.db_pool = asyncpg.create_pool(config["database_url"])
self.publisher = MessageBrokerPublisher(config["rabbitmq_url"])
self.relay = OutboxRelay(
OutboxRepository(self.db_pool),
self.publisher
)
async def run(self):
await self.publisher.connect()
await self.relay.start()
async def shutdown(self):
await self.relay.stop()
await self.publisher.close()
await self.db_pool.close()
Transactional Outbox in Service
The OrderService demonstrates how the outbox integrates with business logic. The key is that both the business write and the outbox insert happen inside the same database transaction.
In create_order, the order row and the outbox event are written together within async with conn.transaction(). If either fails, the entire transaction rolls back — guaranteeing that you never have an order without its event, or an event without its order. This is the core guarantee that eliminates the dual-write problem.
The mark_order_paid method follows the same pattern: update the order status and write an OrderPaidEvent atomically. Because the event payload captures the state change (order_id, payment_id, paid_at), consumers can reconstruct the fact that happened even if they weren’t subscribed at the time.
class OrderService:
def __init__(self, db_pool: asyncpg.Pool, outbox_repo: OutboxRepository):
self.db = db_pool
self.outbox = outbox_repo
async def create_order(self, order_data: dict) -> Order:
async with self.db.acquire() as conn:
async with conn.transaction():
order = await conn.fetchrow("""
INSERT INTO orders (customer_id, items, total_amount, status)
VALUES ($1, $2, $3, 'pending')
RETURNING *
""", order_data["customer_id"],
json.dumps(order_data["items"]),
order_data["total_amount"])
await self.outbox.create_event(
aggregate_type="Order",
aggregate_id=str(order["id"]),
event_type="OrderCreatedEvent",
payload={
"order_id": str(order["id"]),
"customer_id": order["customer_id"],
"items": order_data["items"],
"total_amount": order_data["total_amount"]
}
)
return Order(**dict(order))
async def mark_order_paid(self, order_id: str, payment_id: str):
async with self.db.acquire() as conn:
async with conn.transaction():
await conn.execute("""
UPDATE orders
SET status = 'paid',
payment_id = $2,
paid_at = NOW()
WHERE id = $1
""", order_id, payment_id)
await self.outbox.create_event(
aggregate_type="Order",
aggregate_id=order_id,
event_type="OrderPaidEvent",
payload={
"order_id": order_id,
"payment_id": payment_id,
"paid_at": datetime.utcnow().isoformat()
}
)
Change Data Capture (CDC) Approach
The polling-based relay works well for low to medium volumes, but it has a fundamental limitation: polling latency. The relay only sees new events on its next poll, so delivery latency is bounded by the poll interval. Change Data Capture (CDC) eliminates this by streaming database changes in near-real-time.
CDC tools like Debezium read the database’s write-ahead log (WAL) and emit every committed change as a stream of events. When you combine CDC with the outbox table, you get the best of both worlds: atomic writes via the outbox, and low-latency delivery via CDC streaming.
Debezium Integration
Debezium runs as a Kafka Connect connector. The docker-compose setup below wires together PostgreSQL, Kafka, and Debezium. The connector watches the outbox table specifically (table.include.list: "public.outbox") so it only streams outbox events, not every table in the database.
# docker-compose.yml for Debezium
version: '3.8'
services:
postgres:
image: postgres:15
environment:
POSTGRES_USER: postgres
POSTGRES_PASSWORD: postgres
POSTGRES_DB: orders
ports:
- "5432:5432"
volumes:
- postgres_data:/var/lib/postgresql/data
debezium:
image: debezium/connect:2.4
environment:
BOOTSTRAP_SERVERS: kafka:9092
GROUP_ID: debezium-group
CONFIG_STORAGE_TOPIC: debezium_configs
OFFSET_STORAGE_TOPIC: debezium_offsets
STATUS_STORAGE_TOPIC: debezium_status
ports:
- "8083:8083"
kafka:
image: confluentinc/cp-kafka:7.5.0
environment:
KAFKA_BROKER_ID: 1
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092
ports:
- "9092:9092"
The connector configuration below registers the orders-connector with Kafka Connect. The transforms.unwrap section extracts just the new record state from Debezium’s change envelope, so consumers receive clean event payloads rather than the full CDC wrapper. The key.converter and value.converter settings specify JSON serialization for both the message key and value.
{
"name": "orders-connector",
"config": {
"connector.class": "io.debezium.connector.postgresql.PostgresConnector",
"database.hostname": "postgres",
"database.port": "5432",
"database.user": "postgres",
"database.password": "postgres",
"database.dbname": "orders",
"database.server.name": "orders",
"table.include.list": "public.outbox",
"transforms": "unwrap",
"transforms.unwrap.type": "io.debezium.transforms.ExtractNewRecordState",
"transforms.unwrap.drop.tombstones": "false",
"transforms.unwrap.delete.handling.mode": "rewrite",
"key.converter": "org.apache.kafka.connect.json.JsonConverter",
"value.converter": "org.apache.kafka.connect.json.JsonConverter"
}
}
Building Custom CDC
If you prefer not to adopt a full CDC framework, a lightweight polling-based alternative is straightforward to implement. The PollingCDC class tracks a last_sequence cursor and polls for rows inserted after it, publishing each to Kafka.
This approach is simpler than Debezium — no additional infrastructure beyond your existing database and Kafka — but it inherits the polling latency trade-off. Use it for workloads where sub-second delivery isn’t critical and you want to avoid running a Kafka Connect cluster.
class PollingCDC:
"""Simple CDC using polling of the outbox table."""
def __init__(
self,
db_pool: asyncpg.Pool,
kafka_producer: AIOKafkaProducer,
last_sequence: int = 0
):
self.db = db_pool
self.kafka = kafka_producer
self.last_sequence = last_sequence
async def poll_and_publish(self):
async with self.db.acquire() as conn:
rows = await conn.fetch("""
SELECT * FROM outbox
WHERE id > $1
AND processed = FALSE
ORDER BY id ASC
LIMIT 1000
""", str(self.last_sequence))
for row in rows:
await self._publish_to_kafka(row)
self.last_sequence = row["id"]
return len(rows)
async def _publish_to_kafka(self, row):
await self.kafka.send(
topic=f"{row['aggregate_type'].lower()}.events",
key=row["aggregate_id"].encode(),
value=json.dumps({
"event_type": row["event_type"],
"payload": json.loads(row["payload"]),
"metadata": json.loads(row["metadata"]),
"timestamp": row["created_at"].isoformat()
}).encode()
)
Failure Modes and Recovery
Real-world outbox deployments fail in several distinct ways, and understanding each failure mode helps you design the right recovery. Here are the most common scenarios:
The relay crashes mid-batch. If the relay publishes an event to the broker but crashes before marking it processed, the event will be republished on the next poll. This is the at-least-once behavior — the consumer must handle the duplicate. The alternative failure — the relay crashes before publishing — means the event simply remains unprocessed and is delivered on restart. In both cases, no event is ever lost, but duplicates are possible.
The broker is temporarily unavailable. When Kafka or RabbitMQ is down, every publish in a batch fails. The relay should continue polling (with backoff) rather than crash. Events accumulate in the outbox and are delivered once the broker recovers. This is a key advantage over direct publishing — the outbox acts as a buffer that survives broker outages.
A poisoned event blocks the queue. If one event consistently fails to publish (malformed payload, incompatible schema), an unbounded retry loop would stall all events behind it. The retry cap and dead letter queue prevent this. The DLQ isolates the poison event so healthy events continue flowing while operators investigate.
Outbox table grows unboundedly. If the relay stops processing (deployment gone wrong, permission issue) but the service keeps writing events, the outbox table grows without limit. Monitoring table size and alerting on sustained growth catches this early. Consider an archiving job that moves processed events to a history table after a retention period.
Operational Checklist
- Monitor outbox table size and alert on growth
- Alert on
retry_countapproaching the max - Track delivery lag (oldest unprocessed event age)
- Test broker failure and recovery behavior
- Run consumers idempotently (they must be, by design)
- Archive or purge processed events after retention
Idempotent Processing
Idempotent consumers must be able to process the same event twice without incorrect side effects. The IdempotentEventHandler uses Redis to track processed event IDs. On each event, it checks whether the ID has already been handled; if so, it skips processing. After successful handling, it stores the ID with a TTL (7 days in this example) so the check remains valid across retries.
This approach is simple and fast — a single Redis lookup per event. The trade-off is that the TTL window limits how long deduplication is active; events retried after the TTL expires could be processed twice. For most business events this is acceptable, but for financial transactions you may want permanent deduplication in your event store.
class IdempotentEventHandler:
def __init__(self, redis: Redis):
self.redis = redis
async def process_event(self, event: OutboxEvent) -> bool:
processed_key = f"event:processed:{event.id}"
already_processed = await self.redis.exists(processed_key)
if already_processed:
logger.info(f"Event {event.id} already processed, skipping")
return True
try:
await self._handle_event(event)
await self.redis.setex(
processed_key,
86400 * 7, # Keep for 7 days
"1"
)
return True
except Exception as e:
logger.error(f"Failed to handle event {event.id}: {e}")
raise
async def _handle_event(self, event: OutboxEvent):
pass # Implement actual handling
Retry with Dead Letter Queue
Not every delivery failure is transient. A malformed payload, a bug in the consumer, or a permanent broker misconfiguration won’t resolve with retries. The OutboxWithDLQ class implements the standard remedy: after max_retries failures, the event is moved to a dead letter queue (DLQ) instead of being retried forever.
The process_event method attempts publication. On success it marks the event processed. On failure it increments the retry counter; if the counter reaches the maximum, it sends the event to the DLQ with the original payload and the error details, then marks it processed so the relay stops retrying it. A separate DLQ consumer can then investigate and manually replay or fix the problematic event.
class OutboxWithDLQ:
def __init__(self, outbox_repo: OutboxRepository, dlq_topic: str):
self.outbox = outbox_repo
self.dlq_topic = dlq_topic
self.max_retries = 5
async def process_event(self, event: OutboxEvent) -> bool:
try:
await self._publish(event)
await self.outbox.mark_processed(event.id)
return True
except Exception as e:
new_retry_count = event.metadata.get("retry_count", 0) + 1
if new_retry_count >= self.max_retries:
await self._send_to_dlq(event, str(e))
await self.outbox.mark_processed(event.id)
logger.error(f"Event {event.id} sent to DLQ after {new_retry_count} retries")
else:
await self.outbox.mark_failed(
event.id,
str(e),
new_retry_count
)
return False
async def _send_to_dlq(self, event: OutboxEvent, error: str):
dlq_message = {
"original_event": {
"id": event.id,
"type": event.event_type,
"aggregate_id": event.aggregate_id,
"payload": event.payload
},
"error": error,
"failed_at": datetime.utcnow().isoformat()
}
await self.kafka.send(
topic=self.dlq_topic,
key=event.aggregate_id.encode(),
value=json.dumps(dlq_message).encode()
)
Best Practices
Applying the outbox pattern well requires attention to several design details. The code samples below capture both good patterns and anti-patterns to avoid.
Design Guidelines
Use JSONB for payloads. In PostgreSQL, JSONB gives you queryable, fast-serializing payloads that support schema evolution. Avoid plain TEXT — every read then requires manual parsing and you lose query capability entirely.
Keep payloads small. Store only the event-relevant data (IDs, status, timestamps), not the entire aggregate. Large payloads slow down replication, bloat the outbox table, and duplicate data that consumers can fetch on demand.
Index for polling. The partial index WHERE processed = FALSE is what keeps the relay fast. Without it, every poll scans the whole table, and the outbox degrades linearly with volume.
Preserve ordering. Always ORDER BY created_at ASC when reading unprocessed events. Out-of-order delivery can produce incorrect state in consumers that process dependent events.
Bound retries. Never retry failed events indefinitely — they can block the relay’s progress. Cap retries, then move failures to a dead letter queue for manual investigation.
Common Anti-Patterns
Publishing directly from the service. Writing to the database and publishing in separate calls is the exact dual-write problem the outbox solves. If the publish fails after the write succeeds, you’ve lost the event.
No ordering guarantee. SELECT * FROM outbox WHERE processed = FALSE without ordering can deliver events in any sequence, corrupting consumers that rely on event order.
Infinite retries. A poison event will be retried forever, stalling the relay and blocking all subsequent events. Always impose a retry ceiling.
GOOD_PATTERNS = {
"use_jsonb_postgres": """
# Use JSONB for payload flexibility
✅ Good:
payload JSONB NOT NULL
# Can query by payload fields
# Fast serialization
# Schema evolution support
❌ Bad:
payload TEXT NOT NULL
# Must parse on every read
# No query capability
""",
"keep_payload_small": """
# Don't store entire objects
✅ Good:
payload: {"order_id": "123", "status": "paid"}
❌ Bad:
payload: {"order": {...entire order object...}}
# Large payload = slow replication
# Data duplication
""",
"index_wisely": """
# Index for polling efficiency
✅ Good:
CREATE INDEX idx_outbox_unprocessed ON outbox(created_at)
WHERE processed = FALSE;
# Fast finding of next events to process
❌ Bad:
# No index, full table scan every poll
"""
}
BAD_PATTERNS = {
"publish_directly": """
❌ Bad:
async def create_order():
await db.execute("INSERT INTO orders...")
await message_queue.publish(event) # Not atomic!
# If publish fails, data is inconsistent
✅ Good:
async def create_order():
async with transaction():
await db.execute("INSERT INTO orders...")
await db.execute("INSERT INTO outbox...")
# Both succeed or both fail
""",
"no_ordering": """
❌ Bad:
# Process events out of order
SELECT * FROM outbox WHERE processed = FALSE
# Can cause wrong state if events have dependencies
✅ Good:
ORDER BY created_at ASC
# Preserve event ordering for consistency
""",
"infinite_retries": """
❌ Bad:
# Never give up on failed events
# Can block processing of other events
✅ Good:
retry_count < 5
# After max retries, move to DLQ
# Manual intervention for problematic events
"""
}
Related Articles
Summary
The Outbox Pattern provides reliable event publishing:
- Atomicity — Database changes and event creation happen in a single transaction
- Reliability — Events are guaranteed to be published (at-least-once)
- Simplicity — No distributed transactions or complex coordination needed
- Scalability — Outbox relay can scale horizontally
Two main approaches:
- Polling — Simple, works with any database, suitable for low-medium volume
- CDC (Debezium) — More complex but more scalable, real-time streaming
Decision Guide
Use the transactional outbox with polling when you need strong consistency, delivery latency in the range of seconds is acceptable, and you want minimal infrastructure. This is the right default for most business-critical events — order lifecycle, payment status, user account changes.
Use CDC with Debezium when you need near-real-time delivery at high volume, or when your outbox table is large enough that polling becomes inefficient. The added complexity of running Kafka Connect is justified when latency and throughput matter more than operational simplicity.
Skip the outbox entirely for non-critical, loss-tolerant events like analytics telemetry or internal notifications where a missed event has no business impact.
The pattern is essential for building reliable event-driven microservices without sacrificing data consistency.
Comments