Introduction
The Saga pattern manages distributed transactions across microservices without traditional ACID guarantees. This guide covers implementing sagas in Go using both choreography and orchestration approaches. See Go Installation Guide, Go Ecosystem Overview, Go Best Practices for more context.
Sagas break long-running transactions into smaller, manageable steps with compensating transactions for rollback, enabling reliable distributed operations.
When a single business operation — like placing an order — spans multiple services, each service has its own database and its own transaction boundaries, so a global BEGIN/COMMIT is impossible. A two-phase commit exists but is rarely viable because it holds locks across the network and couples every participant to a coordinator, which harms availability. Sagas solve the same problem differently: they accept that the operation cannot be atomic and instead guarantee eventual consistency by defining a compensation for every step. If step three fails, the saga reverses steps two and one, leaving the system in the state it would have been in had the operation never run.
Saga Fundamentals
Every saga, regardless of implementation style, is built from the same primitive: a saga step. A step represents one unit of work that participates in the larger transaction, such as creating an order, charging a payment, or arranging shipping. What makes a step saga-compatible is its ability to be rolled back. The interface below captures that contract: every step exposes an Execute method for the forward path and a Compensate method for the reverse path, plus a Name that is used for logging and observability.
Saga Step Model
package main
import (
"context"
"fmt"
"time"
)
// SagaStep represents a step in a saga
type SagaStep interface {
Execute(ctx context.Context) error
Compensate(ctx context.Context) error
Name() string
}
// BaseSagaStep provides common functionality
type BaseSagaStep struct {
name string
}
func (s *BaseSagaStep) Name() string {
return s.name
}
// OrderStep represents an order creation step
type OrderStep struct {
BaseSagaStep
orderID string
customerID string
total float64
}
// NewOrderStep creates a new order step
func NewOrderStep(orderID, customerID string, total float64) *OrderStep {
return &OrderStep{
BaseSagaStep: BaseSagaStep{name: "CreateOrder"},
orderID: orderID,
customerID: customerID,
total: total,
}
}
// Execute creates an order
func (s *OrderStep) Execute(ctx context.Context) error {
select {
case <-ctx.Done():
return ctx.Err()
default:
}
fmt.Printf("Creating order %s for customer %s\n", s.orderID, s.customerID)
// Call order service
return nil
}
// Compensate cancels the order
func (s *OrderStep) Compensate(ctx context.Context) error {
fmt.Printf("Cancelling order %s\n", s.orderID)
// Call order service to cancel
return nil
}
// PaymentStep represents a payment step
type PaymentStep struct {
BaseSagaStep
orderID string
amount float64
}
// NewPaymentStep creates a new payment step
func NewPaymentStep(orderID string, amount float64) *PaymentStep {
return &PaymentStep{
BaseSagaStep: BaseSagaStep{name: "ProcessPayment"},
orderID: orderID,
amount: amount,
}
}
// Execute processes payment
func (s *PaymentStep) Execute(ctx context.Context) error {
select {
case <-ctx.Done():
return ctx.Err()
default:
}
fmt.Printf("Processing payment of $%.2f for order %s\n", s.amount, s.orderID)
// Call payment service
return nil
}
// Compensate refunds the payment
func (s *PaymentStep) Compensate(ctx context.Context) error {
fmt.Printf("Refunding payment for order %s\n", s.orderID)
// Call payment service to refund
return nil
}
// ShippingStep represents a shipping step
type ShippingStep struct {
BaseSagaStep
orderID string
}
// NewShippingStep creates a new shipping step
func NewShippingStep(orderID string) *ShippingStep {
return &ShippingStep{
BaseSagaStep: BaseSagaStep{name: "ArrangeShipping"},
orderID: orderID,
}
}
// Execute arranges shipping
func (s *ShippingStep) Execute(ctx context.Context) error {
select {
case <-ctx.Done():
return ctx.Err()
default:
}
fmt.Printf("Arranging shipping for order %s\n", s.orderID)
// Call shipping service
return nil
}
// Compensate cancels shipping
func (s *ShippingStep) Compensate(ctx context.Context) error {
fmt.Printf("Cancelling shipping for order %s\n", s.orderID)
// Call shipping service to cancel
return nil
}
The step model encodes the two-phase reality of distributed transactions. Execute performs the actual business action against a downstream service, while Compensate undoes that action if a later step fails. The context.Context passed to both methods carries cancellation and deadline propagation, so a saga that exceeds its time budget can be aborted cleanly instead of hanging. The OrderStep, PaymentStep, and ShippingStep types are concrete implementations of this contract — each holds only the data its service call needs and delegates to the real service call in production code.
Defining steps as an interface gives the saga engine a uniform way to store, invoke, and reverse work without knowing anything about the business logic inside each step. This separation is what allows a single generic orchestrator or event bus to drive completely different workflows with the same machinery.
Orchestration-Based Saga
Orchestration centralizes workflow control in a single coordinator object — the SagaOrchestrator. The orchestrator knows the full sequence of steps ahead of time, invokes each step in order, and records which steps have completed so it can roll them back in reverse order if anything fails. This gives the team a single place to understand the entire business flow, which makes the workflow explicit, easy to trace, and simple to modify. New steps can be inserted or removed without touching the participating services.
package main
import (
"context"
"fmt"
"log"
"sync"
"time"
)
// SagaOrchestrator orchestrates saga execution
type SagaOrchestrator struct {
steps []SagaStep
mu sync.Mutex
}
// NewSagaOrchestrator creates a new orchestrator
func NewSagaOrchestrator() *SagaOrchestrator {
return &SagaOrchestrator{
steps: []SagaStep{},
}
}
// AddStep adds a step to the saga
func (so *SagaOrchestrator) AddStep(step SagaStep) {
so.mu.Lock()
defer so.mu.Unlock()
so.steps = append(so.steps, step)
}
// Execute executes the saga
func (so *SagaOrchestrator) Execute(ctx context.Context) error {
so.mu.Lock()
steps := make([]SagaStep, len(so.steps))
copy(steps, so.steps)
so.mu.Unlock()
executedSteps := []SagaStep{}
for _, step := range steps {
select {
case <-ctx.Done():
return so.compensate(ctx, executedSteps)
default:
}
fmt.Printf("Executing step: %s\n", step.Name())
if err := step.Execute(ctx); err != nil {
log.Printf("Step %s failed: %v", step.Name(), err)
return so.compensate(ctx, executedSteps)
}
executedSteps = append(executedSteps, step)
}
fmt.Println("Saga completed successfully")
return nil
}
// compensate compensates executed steps
func (so *SagaOrchestrator) compensate(ctx context.Context, executedSteps []SagaStep) error {
fmt.Println("Compensating saga...")
// Compensate in reverse order
for i := len(executedSteps) - 1; i >= 0; i-- {
step := executedSteps[i]
fmt.Printf("Compensating step: %s\n", step.Name())
if err := step.Compensate(ctx); err != nil {
log.Printf("Compensation of %s failed: %v", step.Name(), err)
// Continue compensating other steps
}
}
return fmt.Errorf("saga failed and was compensated")
}
// Example usage
func ExampleOrchestration() {
orchestrator := NewSagaOrchestrator()
// Add steps
orchestrator.AddStep(NewOrderStep("order-001", "customer-001", 99.99))
orchestrator.AddStep(NewPaymentStep("order-001", 99.99))
orchestrator.AddStep(NewShippingStep("order-001"))
// Execute saga
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
if err := orchestrator.Execute(ctx); err != nil {
log.Printf("Saga execution failed: %v", err)
}
}
Choreography-Based Saga
Choreography takes the opposite approach: there is no central coordinator. Each service subscribes to the events it cares about and publishes new events after completing its work. The flow emerges from these event exchanges — the order service publishes OrderCreated, the payment service reacts by charging the card and publishes PaymentProcessed, and the shipping service reacts to that. Services stay decoupled because none of them knows which service will handle an event next, which makes the model well suited to domains where ownership is split across teams.
package main
import (
"context"
"fmt"
"log"
"sync"
"time"
)
// SagaEvent represents a saga event
type SagaEvent interface {
EventType() string
OrderID() string
}
// OrderCreatedEvent represents order creation
type OrderCreatedEvent struct {
orderID string
customerID string
total float64
}
func (e *OrderCreatedEvent) EventType() string {
return "OrderCreated"
}
func (e *OrderCreatedEvent) OrderID() string {
return e.orderID
}
// PaymentProcessedEvent represents payment processing
type PaymentProcessedEvent struct {
orderID string
amount float64
}
func (e *PaymentProcessedEvent) EventType() string {
return "PaymentProcessed"
}
func (e *PaymentProcessedEvent) OrderID() string {
return e.orderID
}
// ShippingArrangedEvent represents shipping arrangement
type ShippingArrangedEvent struct {
orderID string
}
func (e *ShippingArrangedEvent) EventType() string {
return "ShippingArranged"
}
func (e *ShippingArrangedEvent) OrderID() string {
return e.orderID
}
// SagaEventBus manages saga events
type SagaEventBus struct {
handlers map[string][]func(ctx context.Context, event SagaEvent) error
mu sync.RWMutex
}
// NewSagaEventBus creates a new event bus
func NewSagaEventBus() *SagaEventBus {
return &SagaEventBus{
handlers: make(map[string][]func(ctx context.Context, event SagaEvent) error),
}
}
// Subscribe subscribes to events
func (seb *SagaEventBus) Subscribe(eventType string, handler func(ctx context.Context, event SagaEvent) error) {
seb.mu.Lock()
defer seb.mu.Unlock()
seb.handlers[eventType] = append(seb.handlers[eventType], handler)
}
// Publish publishes an event
func (seb *SagaEventBus) Publish(ctx context.Context, event SagaEvent) error {
seb.mu.RLock()
handlers, exists := seb.handlers[event.EventType()]
seb.mu.RUnlock()
if !exists {
return nil
}
for _, handler := range handlers {
go func(h func(ctx context.Context, event SagaEvent) error) {
if err := h(ctx, event); err != nil {
log.Printf("Error handling event: %v", err)
}
}(handler)
}
return nil
}
// OrderService handles order events
type OrderService struct {
eventBus *SagaEventBus
}
// NewOrderService creates a new order service
func NewOrderService(eventBus *SagaEventBus) *OrderService {
return &OrderService{eventBus: eventBus}
}
// CreateOrder creates an order
func (os *OrderService) CreateOrder(ctx context.Context, customerID string, total float64) error {
orderID := fmt.Sprintf("order-%d", time.Now().Unix())
fmt.Printf("Creating order %s\n", orderID)
event := &OrderCreatedEvent{
orderID: orderID,
customerID: customerID,
total: total,
}
return os.eventBus.Publish(ctx, event)
}
// PaymentService handles payment events
type PaymentService struct {
eventBus *SagaEventBus
}
// NewPaymentService creates a new payment service
func NewPaymentService(eventBus *SagaEventBus) *PaymentService {
ps := &PaymentService{eventBus: eventBus}
// Subscribe to order created events
eventBus.Subscribe("OrderCreated", ps.HandleOrderCreated)
return ps
}
// HandleOrderCreated handles order creation
func (ps *PaymentService) HandleOrderCreated(ctx context.Context, event SagaEvent) error {
orderEvent := event.(*OrderCreatedEvent)
fmt.Printf("Processing payment for order %s\n", orderEvent.orderID)
paymentEvent := &PaymentProcessedEvent{
orderID: orderEvent.orderID,
amount: orderEvent.total,
}
return ps.eventBus.Publish(ctx, paymentEvent)
}
// ShippingService handles shipping events
type ShippingService struct {
eventBus *SagaEventBus
}
// NewShippingService creates a new shipping service
func NewShippingService(eventBus *SagaEventBus) *ShippingService {
ss := &ShippingService{eventBus: eventBus}
// Subscribe to payment processed events
eventBus.Subscribe("PaymentProcessed", ss.HandlePaymentProcessed)
return ss
}
// HandlePaymentProcessed handles payment processing
func (ss *ShippingService) HandlePaymentProcessed(ctx context.Context, event SagaEvent) error {
paymentEvent := event.(*PaymentProcessedEvent)
fmt.Printf("Arranging shipping for order %s\n", paymentEvent.orderID)
shippingEvent := &ShippingArrangedEvent{
orderID: paymentEvent.orderID,
}
return ss.eventBus.Publish(ctx, shippingEvent)
}
// Example usage
func ExampleChoreography() {
eventBus := NewSagaEventBus()
// Create services
orderService := NewOrderService(eventBus)
_ = NewPaymentService(eventBus)
_ = NewShippingService(eventBus)
// Create order
ctx := context.Background()
if err := orderService.CreateOrder(ctx, "customer-001", 99.99); err != nil {
log.Printf("Failed to create order: %v", err)
}
// Wait for async processing
time.Sleep(2 * time.Second)
}
Saga State Machine
Long-running sagas are stateful, and their lifecycle is best modeled as a finite state machine. The saga moves through a small set of well-defined states — PENDING, EXECUTING, COMPLETED, COMPENSATING, and FAILED — and only certain transitions are legal. Encoding these transitions in code prevents bugs where a saga is marked completed while it is still rolling back, or where a compensation path runs twice against a saga that already failed.
package main
import (
"context"
"fmt"
)
// SagaState represents saga state
type SagaState string
const (
StatePending SagaState = "PENDING"
StateExecuting SagaState = "EXECUTING"
StateCompleted SagaState = "COMPLETED"
StateCompensating SagaState = "COMPENSATING"
StateFailed SagaState = "FAILED"
)
// SagaStateMachine manages saga state transitions
type SagaStateMachine struct {
state SagaState
}
// NewSagaStateMachine creates a new state machine
func NewSagaStateMachine() *SagaStateMachine {
return &SagaStateMachine{
state: StatePending,
}
}
// Transition transitions to a new state
func (ssm *SagaStateMachine) Transition(newState SagaState) error {
validTransitions := map[SagaState][]SagaState{
StatePending: {StateExecuting},
StateExecuting: {StateCompleted, StateCompensating},
StateCompensating: {StateFailed},
StateCompleted: {},
StateFailed: {},
}
if validStates, exists := validTransitions[ssm.state]; exists {
for _, valid := range validStates {
if valid == newState {
ssm.state = newState
return nil
}
}
}
return fmt.Errorf("invalid transition from %s to %s", ssm.state, newState)
}
// GetState returns current state
func (ssm *SagaStateMachine) GetState() SagaState {
return ssm.state
}
Best Practices
The following snippets highlight the operational concerns that separate a toy saga from a production one. Each addresses a failure mode that surfaces under real traffic and must be handled before the saga goes live.
1. Idempotent Steps
// Ensure steps can be retried safely
func (s *OrderStep) Execute(ctx context.Context) error {
// Check if already executed
// Execute only once
return nil
}
An idempotent step is one that produces the same result whether it runs once or ten times. This matters because network timeouts make it impossible to know whether a step succeeded before the connection dropped. If a step records a database write keyed by a unique operation ID, the orchestrator can retry it safely after a crash, knowing a duplicate attempt is a no-op.
2. Timeout Management
// Set appropriate timeouts
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
Every saga must carry a deadline. A single hanging service call can block a saga forever and leak the threads and connections it holds. Wrapping the whole saga context in a timeout — here 30 seconds — guarantees that compensation is triggered when the budget is exhausted rather than letting the workflow wait indefinitely.
3. Monitoring
// Monitor saga execution
type SagaMetrics struct {
SuccessCount int
FailureCount int
AverageDuration time.Duration
}
Distributed transactions fail in ways that are invisible without instrumentation. Tracking success and failure counts per saga type, plus average duration, lets you detect a compensating-heavy workflow that is burning resources on repeated rollbacks, and lets you set alerting on sagas that consistently exceed their expected runtime.
4. Compensation Logging
// Log all compensations for audit trail
func (so *SagaOrchestrator) compensate(ctx context.Context, executedSteps []SagaStep) error {
// Log compensation events
return nil
}
Every compensation should be recorded in an audit log. When a business dispute arises — a customer charged twice, or inventory decremented without a shipped order — the compensation log is the evidence trail that explains exactly which steps ran, in what order, and which were rolled back and why.
Choreography vs Orchestration: Choosing the Right Model
The choice between choreography and orchestration is the most consequential design decision in a saga-based system. Orchestration wins when the workflow is complex, has many conditional branches, and needs to be clearly traceable: the coordinator is the single source of truth for the flow, making it easy to debug and to add compensating logic in one place. The downside is that the orchestrator becomes a coupled component every service depends on, and it can become a bottleneck or a single point of failure under high throughput.
Choreography wins when services are independently owned and the flow is relatively simple, because it removes the coordinator dependency entirely and lets each team evolve its own piece of the flow. The cost is operational: the control flow is implicit in event wiring, so tracing a failed order across five services means replaying the event log, and without careful event schema management the system becomes hard to reason about as it grows.
A pragmatic middle ground is hybrid sagas: use orchestration for the critical transaction path and choreography for the secondary, best-effort side effects. Whatever the model, the step contract, idempotency, timeouts, and monitoring rules from this section apply unchanged.
Common Pitfalls
1. Non-Idempotent Steps
The most common saga bug is a step that mutates state twice when retried. If Execute decrements inventory or applies a coupon without checking whether it already ran, a single timeout and retry corrupts the data. Make every step idempotent by keying side effects on an operation ID — the same ID yields the same outcome no matter how many times the step executes.
2. No Timeout Management
A saga without a deadline can hang forever on one slow dependency, silently consuming a worker goroutine and its connection. Every saga needs an overall deadline, and every individual step needs its own context timeout, so a stuck service call fails fast and triggers compensation instead of blocking the pipeline.
3. Insufficient Logging
Sagas fail across process boundaries, and without a structured log of transitions — which step started, which completed, which compensated, and why — diagnosing a failed order requires cross-referencing half a dozen services manually. Log every state change and every step invocation with a correlation ID that links the whole saga together.
4. No Compensation Testing
Compensation code is the least exercised path in most systems because it only runs during failures, which are rare in happy-path testing. Teams that never simulate failures ship rollback logic that is broken. Introduce chaos tests that fail a service mid-saga and verify that compensation restores the system to a consistent state.
Resources
Summary
The Saga pattern enables reliable distributed transactions. Key takeaways:
- Use orchestration for complex workflows
- Use choreography for loosely coupled services
- Make all steps idempotent
- Implement proper compensation logic
- Monitor saga execution
- Log all transitions
- Test compensation paths thoroughly
By mastering sagas, you can build reliable distributed systems.
Comments