Introduction
In distributed systems, the concept of a transaction becomes remarkably complex. When your application spans multiple services, databases, or even data centers, the traditional ACID properties that guarantee data consistency in single-node databases no longer apply directly. Yet business processes often require multiple operations to succeed atomically—think of an e-commerce order that involves inventory deduction, payment processing, and shipping notification all happening together.
This article explores the fundamental challenges of distributed transactions, the patterns that address them, and practical guidance for implementing these solutions in production systems.
The Challenge of Distributed Transactions
What Makes Transactions Distributed?
A transaction becomes distributed when it spans multiple resources. Consider an operation that debits one bank account and credits another, where each account lives in a different database. In a single database system, this would be a straightforward transaction. Across databases, we need a mechanism to ensure either both operations succeed or both fail—no partial states where money disappears into thin air.
The challenge intensifies with microservices. A single business operation might involve the order service, the inventory service, the payment service, and the notification service. Each service owns its data, and no service can directly control another service’s database. Yet the business process requires all these operations to coordinate toward a consistent outcome.
The CAP Theorem Revisited
The CAP theorem states that a distributed system can provide only two of three guarantees: Consistency, Availability, and Partition tolerance. In practice, network partitions are inevitable, forcing systems to choose between consistency and availability. Most distributed transaction patterns prioritize availability and tolerate eventual consistency, recognizing often consistency is that perfect too expensive to maintain.
Understanding this trade-off is fundamental. If your business process can tolerate temporary inconsistency—if it is acceptable for inventory to show slightly outdated information for a few seconds—you can build simpler, more available systems. If absolute consistency is required, you must accept limitations during network partitions.
Fundamental Patterns
Two-Phase Commit (2PC)
Two-Phase Commit is the classic approach to achieving atomicity across distributed resources. The protocol involves a coordinator that manages the transaction across all participants.
Phase One (Prepare): The coordinator asks each participant to prepare for commit. Each participant ensures it can commit the transaction, locks necessary resources, and votes yes or no. A “no” vote aborts the transaction.
Phase Two (Commit): If all participants vote yes, the coordinator requests commit. Each participant finalizes the transaction and releases locks. If any participant fails to respond or votes no, the coordinator requests rollback.
Two-Phase Commit provides strong consistency but has significant limitations. It is blocking—if the coordinator fails after participants prepare, those participants cannot finalize or abort until the coordinator recovers. The protocol also introduces latency, as all participants must wait for each other.
When to Use 2PC: When you need strong consistency and can tolerate higher latency, such as financial transactions where correctness outweighs performance. When participants are within the same data center, where network failures are rare.
When to Avoid 2PC: Across geographically distributed systems where network partitions are common. When latency is critical. When you need to scale horizontally.
Three-Phase Commit (3PC)
Three-Phase Commit extends 2PC with an extra phase to avoid blocking. The additional “pre-commit” phase ensures all participants know the transaction outcome before committing begins. However, 3PC assumes synchronous communication and can still fail during network partitions, making it rarely used in practice.
Saga Pattern
The Saga pattern approaches distributed transactions as a sequence of local transactions, with each service performing its part and publishing an event that triggers the next step. If one step fails, saga executors perform compensating transactions to undo the completed steps.
Choreography vs Orchestration
There are two primary ways to coordinate sagas:
Choreography: Services emit and listen to events. The order service publishes “order created,” triggering the inventory service to reserve items. If inventory fails, it publishes an event that the order service listens to and compensates by canceling the order.
Orchestration: A central orchestrator tells each service what to do. The orchestrator knows the business process and handles success and failure logic. This centralization simplifies debugging and reduces coupling between services.
Choreography works well for simple workflows with few services. Orchestration provides better control for complex processes with many steps.
Implementing Compensation
The key to sagas is designing compensating operations that undo completed work. If you reserve inventory and later discover payment failed, you must unreserve that inventory. These compensations must be idempotent—you should be able to run them multiple times without causing issues.
Example Saga:
- Order Service: Creates order in “pending” state, publishes “OrderCreated” event
- Payment Service: Processes payment, publishes “PaymentSucceeded” or “PaymentFailed”
- Inventory Service: Reserves items, publishes “InventoryReserved” or “InventoryFailed”
- Shipping Service: Creates shipment, publishes “ShipmentCreated”
On payment failure, the Order Service must cancel the order. On inventory failure, Payment Service refunds the payment and Order Service cancels the order.
Challenges with Sagas
Sagas do not provide isolation—while the saga executes, other transactions might see intermediate states. This requires designing your domain to handle visible pending states. Additionally, debugging saga failures can be difficult, as you must trace execution across multiple services.
Event Sourcing
Event Sourcing stores the state of an entity as a sequence of events rather than the current state. Instead of updating a record, you append events to an event store. To find the current state, you replay all events.
This approach naturally supports distributed transactions through event log replication. If all services write to a shared event log (like Kafka), they can coordinate through events. The event log becomes the source of truth, with each service maintaining a materialized view for performance.
Event sourcing provides complete audit trail—all state changes are recorded as events. This is valuable for financial systems, regulatory compliance, and debugging. However, it requires different thinking about data modeling and introduces complexity around event schema evolution.
Practical Implementation Strategies
Outbox Pattern
The outbox pattern solves a common problem: how to reliably publish events when your database and message broker are separate resources. If you update your database and then publish a message, and the message publish fails, you end up with inconsistent state.
The solution: write events to an “outbox” table in the same database transaction as your business data. A separate process reads from the outbox and publishes to the message broker. This ensures atomicity—you either write both business data and outbox entry, or neither.
Message Delivery Guarantees
At-least-once delivery is the practical reality for most distributed systems. Messages may be delivered multiple times, so your handlers must be idempotent. Design operations that can safely execute multiple times without causing issues.
For operations that absolutely must execute exactly once, you need exactly-once semantics, which is notoriously difficult. A common pattern: use idempotent operations with deduplication based on operation identifiers. If the same operation ID appears twice, skip the second execution.
Handling Failures
Design for failure from the start. Implement retry logic with exponential backoff and circuit breakers to prevent cascading failures. When retries fail, move to a dead letter queue for manual intervention.
Consider the business impact of different failure modes. If inventory reservation fails, should you cancel the order immediately or hold it for retry? Different business contexts require different approaches.
Consistency Models
Eventual Consistency
Eventual consistency accepts that reads might return stale data temporarily. After a write, the system converges to the correct state given enough time without new writes. This model enables high availability and partition tolerance.
Most web-scale systems adopt eventual consistency by default. Shopping cart counts might briefly show incorrect values. Inventory might briefly appear available when it is not. These inconsistencies are typically short-lived and acceptable.
Strong Eventual Consistency
Strong eventual consistency adds convergence guarantees beyond basic eventual consistency. With conflict-free replicated data types (CRDTs), you can guarantee that concurrent updates will eventually converge to the same state regardless of order. This enables highly available systems without coordination.
CRDTs work for specific data types: counters, sets, registers. They cannot express all data structures, limiting their applicability.
Linearizability
Linearizability provides the illusion that operations happen instantaneously at some point between their invocation and response. All nodes see operations in the same order, and each operation appears to take effect atomically. This is the strongest consistency model but requires coordination, limiting scalability.
Technology Choices
Message Brokers
Kafka provides durable, ordered event streams with exactly-once semantics. Its log-based architecture naturally supports event sourcing. RabbitMQ offers flexible routing and lower latency for simpler workflows. AWS SQS provides managed queueing with built-in dead letter handling.
Databases
CockroachDB and Spanner provide globally distributed SQL with strong consistency guarantees. MongoDB offers flexible schemas with tunable consistency. Cassandra prioritizes availability and partition tolerance, requiring careful data modeling for consistency.
Transaction Coordination
Service frameworks like Seata provide distributed transaction support with multiple modes. choreography-based saga, and TCC (Try-Confirm-Cancel). Kafka Streams enables stateful stream processing that can maintain consistency.
Best Practices
Design for compensation from the beginning. Every operation that might fail should have a corresponding undo operation. Test failure scenarios explicitly—simulate network partitions, service failures, and database outages.
Monitor sagas and distributed transactions comprehensively. Track in-progress transactions, measure latency, and alert on failures. Without observability, debugging distributed failures becomes a nightmare.
Accept that perfect consistency is impossible. Design your business processes to tolerate temporary inconsistency when appropriate. The question is not whether to accept eventual consistency, but where and for how long.
Conclusion
Distributed transactions remain one of the hardest problems in distributed systems. The CAP theorem forces fundamental trade-offs between consistency, availability, and partition tolerance. No single pattern solves all scenarios; the right approach depends on your consistency requirements, scale, and business context.
Two-Phase Commit provides strong consistency but at the cost of availability and latency. The Saga pattern embraces eventual consistency with compensation logic. Event sourcing provides auditability and natural integration with event-driven systems.
The key is understanding your requirements deeply. Can your business process tolerate eventual consistency? How long is acceptable for the system to converge? What happens during failure at each step? Answer these questions, and the right pattern becomes clearer.
Resources
- Distributed Sagas: A Stabilized Approach to Distributed Transactions by Caitie McCaffrey
- Enterprise Integration Patterns by Gregor Hohpe
- Kafka Documentation
- Seata Distributed Transaction Solution
Comments