Skip to main content

Idempotency in Distributed Systems: Building Safe APIs

Published: March 12, 2026 Updated: May 8, 2026 Larry Qu 21 min read

Introduction

Network failures, timeouts, and client retries can cause the same operation to be executed multiple times. Without proper idempotency handling, this can lead to duplicate charges, duplicated records, or inconsistent state. Idempotency ensures that executing an operation multiple times produces the same result as executing it once.

Why Idempotency Matters in Production

In production systems, idempotency failures translate directly to revenue loss and data corruption. The canonical example is payment processing: a network timeout causes your API client to retry a charge request, and without idempotency, the customer gets charged twice. This exact scenario is why Stripe requires clients to send an Idempotency-Key header — Stripe deduplicates requests with the same key, guaranteeing at-most-once processing. Beyond payments, non-idempotent operations cause race conditions in inventory systems (deducting stock twice), duplicate user registrations (same email, two accounts), and inconsistent database state from retried writes. The critical design decision is distinguishing operations that are naturally idempotent — GET, DELETE, PUT with full payloads — from those requiring explicit idempotency keys, such as payment intents, order creation, and any operation with side effects you cannot safely replay.

Understanding Idempotency

Before diving into implementation, it is essential to build a precise mental model of what idempotency means at the HTTP layer and why some operations are naturally idempotent while others require explicit machinery. An operation is idempotent when repeating it with the same input produces the same observable state as executing it exactly once. This property is independent of the response returned: the first attempt may return a freshly created resource while a retry returns the same resource, but the system state remains identical either way.

In the HTTP specification, methods have well-defined idempotency semantics that map cleanly onto real-world use cases. GET requests are inherently safe and idempotent because they only read state. PUT is idempotent because it replaces a resource with a full, deterministic payload — replaying it converges to the same final state. DELETE is idempotent from a state perspective: once a resource is gone, deleting it again leaves the system unchanged, even if a second call returns a different status code. The problematic method is POST, which by definition creates a new resource each time it is invoked. This is precisely why payment, order, and transfer endpoints — all built on POST — require an explicit idempotency contract layered on top of the HTTP semantics.

The diagrams below illustrate both sides of the problem. The first summarizes the classification of idempotent versus non-idempotent operations. The second walks through the classic duplicate-payment failure mode, showing how a client timeout followed by a retry can cause a payment to be processed twice at the database layer. Together they motivate every technique covered in the rest of this article.

┌─────────────────────────────────────────────────────────────────┐
│                    Idempotency Concepts                          │
├─────────────────────────────────────────────────────────────────┤
│                                                                 │
│  Idempotent Operations:                                         │
│                                                                 │
│  GET /users/123     → Always returns user 123                   │
│  DELETE /users/123  → Deleting user 123 twice = same as once    │
│  PUT /users/123    → Setting name to "John" (same value)       │
│                                                                     │
│  Non-Idempotent Operations:                                       │
│                                                                 │
│  POST /orders     → Creating order (each call = new order)      │
│  POST /payments   → Each call = new payment                     │
│  DELETE /items/1  → First call deletes, second returns 404      │
│                                                                 │
└─────────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────────┐
│                 The Problem: Duplicate Payments                  │
├─────────────────────────────────────────────────────────────────┤
│                                                                 │
│  Client              Server              Database                │
│     │                  │                   │                    │
│     │ POST /pay        │                   │                    │
│     │──────────────────▶│                   │                    │
│     │                  │  (processing)      │                    │
│     │   (timeout)      │                   │                    │
│     │✗ ───────────────▶│                   │                    │
│     │                  │                   │                    │
│     │ (retry)          │                   │                    │
│     │ POST /pay        │                   │                    │
│     │──────────────────▶│                   │                    │
│     │                  │  Process payment │                    │
│     │                  │──────────────────▶│                   │
│     │                  │                   │                    │
│     │                  │  Process payment │                    │
│     │                  │──────────────────▶│ (DUPLICATE!)       │
│     │                  │                   │                    │
│     │◀─────────────────│                   │                    │
│     │   200 OK         │                   │                    │
│                                                                 │
│  Solution: Idempotency Key                                       │
│                                                                 │
└─────────────────────────────────────────────────────────────────┘

Idempotency Keys

The core mechanism for making POST endpoints safe under retries is the idempotency key: a client-generated unique identifier sent in a request header that lets the server recognize and deduplicate retries. When a client sends a request with an Idempotency-Key, the server records the key along with the response it produced. If the client retries — for example after a timeout or network blip — it sends the same key, and the server recognizes it, short-circuits the business logic, and returns the stored response instead of executing the operation a second time.

The design choices in the code below reflect production reality. First, the IdempotencyRecord stores not just a boolean “processed” flag but the full response status and body. This is critical because retries must return exactly what the first attempt would have returned — replaying the original response keeps client logic simple and preserves observable behavior. Second, records carry an expiry timestamp so the store can bound its memory footprint; a default TTL of 24 hours matches the practical window in which clients retry. Third, all access to the in-memory store is guarded by a reentrant lock, since HTTP servers handle many concurrent requests and a race here would reintroduce exactly the duplicate-processing bug we are trying to eliminate.

The IdempotentEndpoint decorator ties the pieces together at the HTTP boundary. It enforces that a key is present and well-formed (rejecting keys shorter than 16 characters early), checks for an existing record, and only executes the underlying handler when no prior result exists. Notice that failures are deliberately not cached: if the underlying function raises, the wrapper re-raises without storing anything, so a subsequent retry with the same key gets a fresh chance to succeed. The X-Idempotent-Replayed response header also gives clients a way to distinguish a genuine first execution from a replay, which is invaluable for debugging.

import uuid
import hashlib
import time
from dataclasses import dataclass
from typing import Optional
import threading

@dataclass
class IdempotencyRecord:
    """Record of an idempotent request."""
    key: str
    response_status: int
    response_body: dict
    created_at: float
    expires_at: float

class IdempotencyStore:
    """Store for idempotency records."""
    
    def __init__(self, ttl_seconds: int = 86400):
        self.records = {}
        self.lock = threading.RLock()
        self.ttl = ttl_seconds
    
    def get(self, key: str) -> Optional[IdempotencyRecord]:
        """Get idempotency record if exists and valid."""
        with self.lock:
            record = self.records.get(key)
            
            if record and time.time() < record.expires_at:
                return record
            
            # Clean up expired
            if record:
                del self.records[key]
            
            return None
    
    def set(self, key: str, status: int, body: dict):
        """Store idempotency record."""
        with self.lock:
            now = time.time()
            self.records[key] = IdempotencyRecord(
                key=key,
                response_status=status,
                response_body=body,
                created_at=now,
                expires_at=now + self.ttl
            )
    
    def delete(self, key: str):
        """Delete idempotency record."""
        with self.lock:
            self.records.pop(key, None)
    
    def cleanup_expired(self):
        """Clean up expired records."""
        with self.lock:
            now = time.time()
            expired = [
                k for k, v in self.records.items()
                if now >= v.expires_at
            ]
            for k in expired:
                del self.records[k]


class IdempotentEndpoint:
    """Decorator for idempotent endpoints."""
    
    def __init__(self, store: IdempotencyStore):
        self.store = store
    
    def __call__(self, func):
        """Decorator to make endpoint idempotent."""
        
        def wrapper(request, *args, **kwargs):
            # Get idempotency key from header
            idempotency_key = request.headers.get('Idempotency-Key')
            
            if not idempotency_key:
                return {"error": "Idempotency-Key required"}, 400
            
            # Validate key format
            if len(idempotency_key) < 16:
                return {"error": "Invalid idempotency key"}, 400
            
            # Check for existing request
            existing = self.store.get(idempotency_key)
            
            if existing:
                return (
                    existing.response_body,
                    existing.response_status,
                    {'X-Idempotent-Replayed': 'true'}
                )
            
            # Execute request
            try:
                result = func(request, *args, **kwargs)
                
                # Store successful result
                if isinstance(result, tuple):
                    body, status = result
                else:
                    body, status = result, 200
                
                self.store.set(idempotency_key, status, body)
                
                return body, status
            
            except Exception as e:
                # Don't store failed results
                raise
        
        return wrapper

This in-memory implementation is instructive but has important limitations in production. A single-process dictionary cannot survive restarts, cannot be shared across multiple application instances, and will lose all idempotency records if the server crashes mid-request. For that reason, real systems store idempotency records in the same durable store that holds the business data, or in a shared cache such as Redis that survives individual node failures. The persistence guarantees of the idempotency store define how strong your at-most-once guarantee is: a store that can lose records degrades to at-least-once behavior, which is why the database-backed approach covered next is the more common production pattern.

Database-Level Idempotency

Application-layer idempotency protects a single service, but the strongest guarantees come when the idempotency check and the business write share the same transactional boundary. If a payment is processed and then the process crashes before the idempotency record is written, a retry will process the payment a second time. The database-backed approach below eliminates that window by performing the lookup, the business operation, and the record insert inside the same connection and committing them together, so either all succeed or none do.

The execute_idempotent method implements this pattern: it first queries the idempotency_keys table for an existing result, and only if none exists runs the provided operation. Because everything happens on a single connection, the SELECT and the subsequent INSERT observe a consistent view of the database. A unique primary key on the key column provides a second line of defense: even if two concurrent requests race and both miss the initial lookup, the second INSERT fails with an IntegrityError, which the execute_with_unique_constraint method converts into a fetch of the already-created record.

This approach also demonstrates a clean separation of concerns. The IdempotentDatabase class owns the mechanics of idempotency — checking keys, storing results, handling races — while the PaymentRepository only knows how to construct payment-specific operations. The operation closure returned by do_payment captures the incoming request data, letting the generic idempotency machinery call arbitrary business logic without needing to understand it. The trade-off is subtle but important: because the result is serialized to JSON and stored in the same transaction, response bodies must be JSON-serializable, and large payloads can bloat the idempotency table over time unless records are pruned.

import sqlite3
from typing import Optional
import json
import uuid

class IdempotentDatabase:
    """Database operations with idempotency support."""
    
    def __init__(self, db_path: str):
        self.conn = sqlite3.connect(db_path)
        self._init_tables()
    
    def _init_tables(self):
        """Initialize idempotency table."""
        self.conn.execute("""
            CREATE TABLE IF NOT EXISTS idempotency_keys (
                key TEXT PRIMARY KEY,
                result TEXT,
                status_code INTEGER,
                created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
            )
        """)
        self.conn.commit()
    
    def execute_idempotent(self, key: str, operation: callable) -> dict:
        """Execute operation with idempotency check."""
        cursor = self.conn.cursor()
        
        # Check for existing result
        cursor.execute(
            "SELECT result, status_code FROM idempotency_keys WHERE key = ?",
            (key,)
        )
        row = cursor.fetchone()
        
        if row:
            return json.loads(row[0]), row[1]
        
        # Execute operation
        try:
            result = operation()
            self.conn.commit()
            
            # Store result
            self.conn.execute(
                "INSERT INTO idempotency_keys (key, result, status_code) VALUES (?, ?, ?)",
                (key, json.dumps(result[0]), result[1])
            )
            self.conn.commit()
            
            return result
        
        except Exception as e:
            self.conn.rollback()
            raise
    
    def execute_with_unique_constraint(self, table: str, 
                                      unique_fields: list[str],
                                      data: dict) -> int:
        """Execute insert with unique constraint handling."""
        unique_cols = ', '.join(unique_fields)
        placeholders = ', '.join(['?'] * len(unique_fields))
        values = tuple(data.get(f) for f in unique_fields)
        
        try:
            cursor = self.conn.cursor()
            
            # Try to insert
            cols = ', '.join(data.keys())
            val_placeholders = ', '.join(['?'] * len(data))
            
            cursor.execute(
                f"INSERT INTO {table} ({cols}) VALUES ({val_placeholders})",
                tuple(data.values())
            )
            self.conn.commit()
            
            return cursor.lastrowid
        
        except sqlite3.IntegrityError as e:
            # Duplicate - fetch existing record
            cursor.execute(
                f"SELECT id FROM {table} WHERE {unique_cols} = ?",
                values
            )
            row = cursor.fetchone()
            return row[0] if row else None


class PaymentRepository:
    """Payment repository with idempotency."""
    
    def __init__(self, db: IdempotentDatabase):
        self.db = db
    
    def create_payment(self, payment_data: dict) -> dict:
        """Create payment with idempotency."""
        key = payment_data.get("idempotency_key")
        
        def do_payment():
            return self._process_payment(payment_data)
        
        return self.db.execute_idempotent(key, do_payment)
    
    def _process_payment(self, data: dict) -> dict:
        """Actual payment processing logic."""
        # Payment processing logic here
        return {
            "payment_id": str(uuid.uuid4()),
            "status": "completed",
            "amount": data["amount"]
        }, 201

Both approaches so far solve the same problem in different places: application code in memory versus the database transaction. The in-memory store is fast and simple but non-durable; the database-backed store is durable and race-safe but couples idempotency records to the business database. A hybrid pattern — writing the idempotency record in Redis first with a short TTL and falling back to the database for the final commit — combines the best of both and is the architecture used by many real payment platforms. Whichever you choose, the invariant to preserve is that the idempotency record and the side effect it represents must be written atomically with respect to each other.

Optimistic Locking

Idempotency keys protect against duplicate submission, but they do not solve a related problem: concurrent or conflicting updates to the same resource. When two clients read the same row, both modify it, and both write back, the last writer silently overwrites the first writer’s changes. Optimistic locking addresses this by versioning each resource and refusing writes that are based on a stale version.

The pattern is simple to reason about. Each entity carries an integer version that increments on every successful update. When a client reads a resource, it receives the current version, typically surfaced as an HTTP ETag header. When the client attempts an update, it sends the version back in an If-Match header. The server compares the expected version against the current one: if they match, the update proceeds and the version advances; if they differ, the update is rejected with a 409 Conflict, signaling the client to re-read the resource and reconcile. This is exactly how many REST APIs implement last-writer-wins prevention without holding locks.

The code below shows both the optimistic-lock store and the REST-layer glue around it. The store-level update method compares the stored entity’s version with the caller’s expected version and raises OptimisticLockError on mismatch, while the VersionedAPI translates that error into the HTTP 409 response. The ETag format chosen — "<entity-id>-<version>" — is deliberate: it is stable, cheap to compute, and easily parsed back into a version integer on the server side. Note also the graceful 400 response when the If-Match header is entirely absent, which keeps the API honest about requiring optimistic concurrency rather than silently falling back to unsafe blind writes.

import threading
from dataclasses import dataclass
from typing import Optional

@dataclass
class VersionedEntity:
    id: str
    version: int
    data: dict

class OptimisticLockStore:
    """Store with optimistic locking."""
    
    def __init__(self):
        self.entities = {}
        self.lock = threading.RLock()
    
    def get(self, entity_id: str) -> Optional[VersionedEntity]:
        with self.lock:
            return self.entities.get(entity_id)
    
    def update(self, entity_id: str, expected_version: int, 
               new_data: dict) -> VersionedEntity:
        """Update with optimistic locking."""
        with self.lock:
            entity = self.entities.get(entity_id)
            
            if not entity:
                # Create new entity
                entity = VersionedEntity(
                    id=entity_id,
                    version=1,
                    data=new_data
                )
                self.entities[entity_id] = entity
                return entity
            
            # Check version
            if entity.version != expected_version:
                raise OptimisticLockError(
                    f"Version mismatch: expected {expected_version}, "
                    f"found {entity.version}"
                )
            
            # Update
            entity.version += 1
            entity.data = new_data
            
            return entity

class OptimisticLockError(Exception):
    """Exception for optimistic lock failures."""
    pass

# REST API Implementation
class VersionedAPI:
    """API with optimistic locking via ETag."""
    
    def __init__(self, store: OptimisticLockStore):
        self.store = store
    
    def get_with_etag(self, entity_id: str) -> tuple[dict, str, int]:
        """Get entity with ETag."""
        entity = self.store.get(entity_id)
        
        if not entity:
            return {"error": "Not found"}, 404, None
        
        # Generate ETag
        etag = f'"{entity.id}-{entity.version}"'
        
        return entity.data, 200, etag
    
    def update(self, entity_id: str, data: dict, 
              if_match: str) -> tuple[dict, int]:
        """Update with optimistic locking."""
        if not if_match:
            return {"error": "If-Match header required"}, 400
        
        # Parse ETag
        try:
            expected_version = int(if_match.strip('"').split('-')[1])
        except (ValueError, IndexError):
            return {"error": "Invalid ETag"}, 400
        
        try:
            entity = self.store.update(entity_id, expected_version, data)
            etag = f'"{entity.id}-{entity.version}"'
            
            return {
                "data": entity.data,
                "version": entity.version
            }, 200
        
        except OptimisticLockError as e:
            return {"error": str(e)}, 409


# Example: ETag in response headers
def handle_get(request, entity_id: str):
    api = VersionedAPI(store)
    data, status, etag = api.get_with_etag(entity_id)
    
    if status == 200:
        return data, 200, {'ETag': etag}
    return data, status


def handle_update(request, entity_id: str, data: dict):
    api = VersionedAPI(store)
    if_match = request.headers.get('If-Match')
    
    result, status = api.update(entity_id, data, if_match)
    
    if status == 200:
        result['ETag'] = f'"{entity_id}-{result["version"]}"'
    
    return result, status

Optimistic locking trades a small amount of client complexity for a large gain in correctness. The client must handle the 409 Conflict case and re-fetch, but in exchange the server never needs to hold locks, never risks deadlock, and stays trivially horizontally scalable because state is only touched at the moment of the write. When combined with idempotency keys, the two techniques compose cleanly: the key prevents duplicate side effects from retries, while the version guard prevents conflicting concurrent updates from silently overwriting each other. Together they form the foundation of a safe mutation API.

Client-Side Idempotency

So far every mechanism has lived on the server, but idempotency is a contract between both parties, and the client has just as much responsibility to get right. The IdempotentClient below shows what a well-behaved client looks like. It automatically attaches a unique Idempotency-Key header to every request, freeing callers from remembering to generate one, while still allowing an explicit key to be passed for operations that must be retry-safe, such as payments.

The retry loop is where the client earns its keep. On a timeout, the client cannot know whether the server actually processed the request before the connection died — the classic unknown-state problem in distributed systems. The safe answer is to retry with the same key, which the client does automatically with exponential backoff. A 409 status is treated specially: it signals that another in-flight request already holds the key, so the client backs off and retries rather than treating it as a hard error. This coordination prevents two concurrent attempts from double-charging a customer.

There is an important asymmetry to appreciate here. The client can retry aggressively with exponential backoff because the server guarantees deduplication; without that server-side contract, retrying would be dangerous. Conversely, the server relies on the client to keep the key stable across retries. The same key must be reused for every attempt of the same logical operation, and a different key must be used for a genuinely new operation. Many subtle production bugs trace back to clients regenerating a fresh key on each retry, silently breaking the very guarantee they are relying on.

import requests
import uuid
import time
from typing import Callable
import threading

class IdempotentClient:
    """HTTP client with automatic idempotency."""
    
    def __init__(self, base_url: str = "", retry_on_timeout: bool = True):
        self.base_url = base_url
        self.retry_on_timeout = retry_on_timeout
        self.pending_requests = {}
        self.lock = threading.Lock()
    
    def request(self, method: str, endpoint: str, 
               idempotency_key: str = None,
               **kwargs) -> requests.Response:
        """Make request with idempotency support."""
        url = f"{self.base_url}{endpoint}"
        
        # Generate idempotency key if not provided
        if not idempotency_key:
            idempotency_key = str(uuid.uuid4())
        
        headers = kwargs.pop('headers', {})
        headers['Idempotency-Key'] = idempotency_key
        
        max_retries = 3
        last_error = None
        
        for attempt in range(max_retries):
            try:
                response = requests.request(
                    method,
                    url,
                    headers=headers,
                    **kwargs
                )
                
                # Check if response indicates in-progress
                if response.status_code == 409:
                    # Another request in progress, wait and retry
                    time.sleep(0.5 * (attempt + 1))
                    continue
                
                return response
            
            except requests.Timeout as e:
                last_error = e
                
                if not self.retry_on_timeout:
                    raise
                
                # Check if request might have succeeded
                if attempt < max_retries - 1:
                    print(f"Request timeout, retrying ({attempt + 1}/{max_retries})")
                    time.sleep(2 ** attempt)  # Exponential backoff
                    continue
                
                raise last_error
        
        raise last_error
    
    def post(self, endpoint: str, data: dict, **kwargs) -> requests.Response:
        return self.request('POST', endpoint, json=data, **kwargs)
    
    def put(self, endpoint: str, data: dict, **kwargs) -> requests.Response:
        return self.request('PUT', endpoint, json=data, **kwargs)
    
    def delete(self, endpoint: str, **kwargs) -> requests.Response:
        return self.request('DELETE', endpoint, **kwargs)


# Usage
client = IdempotentClient("https://api.example.com")

# Payment request - use same key for retries
payment_data = {
    "amount": 100.00,
    "currency": "USD",
    "customer_id": "cust_123"
}

# First attempt
try:
    response = client.post(
        "/payments",
        payment_data,
        idempotency_key="payment_cust123_12345"
    )
except requests.Timeout:
    # Retry with same key - won't create duplicate
    response = client.post(
        "/payments",
        payment_data,
        idempotency_key="payment_cust123_12345"
    )

The client-side patterns here generalize well beyond payments. Any long-running, state-changing operation benefits from the same discipline: generate a stable key once, reuse it across retries, back off on transient failures, and treat unknown outcomes as retryable. When the server and client both honor the contract, the system as a whole behaves as at-most-once from the user’s perspective even though individual components are at-least-once. This layering of guarantees is a recurring theme in reliable distributed systems, and it reappears in the message-queue context covered next.

Idempotency in Message Queues

HTTP APIs are not the only place duplicate executions occur; message queues are arguably even more prone to them. Distributed message brokers such as Kafka, RabbitMQ, and SQS provide at-least-once delivery semantics by design: a consumer that crashes after processing a message but before acknowledging it will receive that message again on reconnection. The result is the same failure mode as a retried payment — a side effect executed twice — but at the level of background jobs, data pipelines, and event handlers.

The solution mirrors the idempotency-key approach, adapted to the message’s content. The MessageIdempotency class derives a deterministic fingerprint from the message’s stable fields — type, id, timestamp, and source — hashes it with SHA-256, and uses it as a key in Redis. Before processing, the consumer checks whether the key already exists; after processing, it records the key with a 24-hour TTL. If a duplicate message arrives, the check finds the key and the consumer skips re-executing the handler.

Three design decisions in this snippet are worth calling out. First, the fingerprint is built only from fields that are present in both the original and the redelivered copy — if the broker adds or mutates fields on redelivery, a naive hash of the whole message would break deduplication. Second, the idempotency marker is stored with a TTL, so the guarantee holds only within that window; messages redelivered after the key expires are processed again, which is acceptable for most pipelines but must be a conscious decision. Third, a crash between processing and recording the key still permits a duplicate, so for truly critical handlers the marker write must be atomic with the business write — the same transactional boundary concern from the database section above.

import hashlib
import json
from typing import Optional

class MessageIdempotency:
    """Idempotency for message processing."""
    
    def __init__(self, redis_client):
        self.redis = redis_client
    
    def process_message(self, message: dict, processor: callable) -> bool:
        """Process message with idempotency."""
        # Create unique key from message
        message_key = self._create_message_key(message)
        
        # Check if already processed
        if self.redis.exists(message_key):
            print(f"Message already processed: {message_key}")
            return True
        
        try:
            # Process message
            result = processor(message)
            
            # Mark as processed (with TTL)
            self.redis.setex(message_key, 86400, json.dumps(result))
            
            return result
        
        except Exception as e:
            print(f"Error processing message: {e}")
            raise
    
    def _create_message_key(self, message: dict) -> str:
        """Create unique key from message."""
        # Use deterministic fields
        key_data = json.dumps({
            "type": message.get("type"),
            "id": message.get("id"),
            "timestamp": message.get("timestamp"),
            "source": message.get("source")
        }, sort_keys=True)
        
        return f"message:processed:{hashlib.sha256(key_data).hexdigest()}"
    
    def cleanup_old_keys(self):
        """Clean up old processed message keys."""
        # Redis handles TTL automatically
        pass

Using Redis as the deduplication store brings welcome properties: it is fast, survives application restarts, and is shared across all consumer instances, so duplicate messages are caught even when different workers pick them up. The TTL on the marker bounds the store’s growth and naturally expires deduplication state for old messages. The main caveat is that Redis-based checks are not transactional with the consumer’s business database, so the strongest guarantees still require moving the marker into the same transactional system that owns the side effect — the same lesson that emerged from the database-level section.

Best Practices

The mechanisms are in place, but design discipline is what separates systems that are merely correct in theory from those that hold up in production. The final code snippet encodes the hard-won lessons of running idempotent systems at scale as a small reference library. should_be_idempotent encodes the method-and-resource classification discussed throughout the article — safe methods are always idempotent, while POST operations that create financial or capacity-sensitive resources must opt in.

The key-generation helper shows another subtle point: keys should be deterministic when they describe the same logical operation, but unique across operations. Hashing a set of natural identifiers, such as customer and payment identifiers, produces a compact, debuggable key that is the same on every retry yet distinct from other operations. The idempotency_key_requirements method finally documents the operational contract — minimum length to prevent trivial collisions, a bounded maximum to keep indexes and headers small, a restricted character set to avoid header-injection issues, and a TTL floor so clients can rely on retry protection for a known window. Encoding these rules as code keeps every service in the organization consistent instead of relying on tribal knowledge.

class IdempotencyBestPractices:
    """Guidelines for idempotent API design."""
    
    @staticmethod
    def should_be_idempotent(method: str, resource: str) -> bool:
        """Determine if operation should be idempotent."""
        idempotent_methods = ['GET', 'PUT', 'DELETE', 'HEAD', 'OPTIONS']
        
        # GET is always idempotent
        if method in idempotent_methods:
            return True
        
        # POST operations that create resources should use idempotency
        if method == 'POST':
            return resource in [
                '/payments', '/orders', '/subscriptions',
                '/transfers', '/bookings'
            ]
        
        return False
    
    @staticmethod
    def generate_idempotency_key(parts: list[str]) -> str:
        """Generate idempotency key from parts."""
        combined = '-'.join(str(p) for p in parts)
        return hashlib.sha256(combined.encode()).hexdigest()[:32]
    
    @staticmethod
    def idempotency_key_requirements() -> dict:
        """Requirements for idempotency keys."""
        return {
            "min_length": 16,
            "max_length": 256,
            "format": "alphanumeric with dashes/underscores",
            "uniqueness": "one per user per operation",
            "ttl": "24 hours minimum"
        }

A common thread runs through all of the techniques in this article: correctness is achieved by making the idempotency decision explicit and consistent at every layer. Pick one source of truth for idempotency state, record responses rather than mere acknowledgements, tie the record write to the business write, and document the contract for clients so they generate stable keys. The investment pays for itself the first time a network partition, a broken client, or an aggressive retry policy would otherwise double-charge a customer or corrupt a database.

Conclusion

Idempotency is crucial for building reliable distributed systems. By implementing proper idempotency handling, you can safely handle retries without worrying about duplicate operations.

Key takeaways:

  • Use idempotency keys for all state-changing POST operations
  • Implement optimistic locking for updates with ETags
  • Store idempotency records with appropriate TTL
  • Generate descriptive idempotency keys for debugging
  • Consider idempotency at all layers: API, database, and message queues

Resources

Comments

👍 Was this article helpful?