Introduction
Decentralized systems eliminate central points of failure and enable trustless data sharing. This article covers IPFS, distributed storage, and decentralized web technologies.
Key Statistics:
- IPFS nodes: 100K+ active
- Filecoin storage: 10+ EB secured
- DWeb market: Growing rapidly
- Decentralized storage: 80% cheaper than cloud
The statistics above hint at why the decentralized web has moved from research curiosity to production infrastructure, but they also conceal the complexity that comes with it. When you replace a centralized service with a peer-to-peer network, you trade operational simplicity for resilience and control. The articles and code examples that follow walk through the complete stack bottom-up: first the storage architecture, then the IPFS protocol that implements content addressing, then the Filecoin market that pays for persistence, then the database layer built on top, and finally the peer-to-peer protocols that make the whole system work. Each layer solves a different problem, and each introduces its own failure modes and design trade-offs that you need to understand before building on top of it.
This guide assumes you already know the basics of distributed systems, such as replication and consensus, and want the implementation-level detail that tutorials usually skip. We will spend most of our time inside working Python code, because the fastest way to internalize a protocol is to build a simplified version of it yourself. The code is deliberately simplified to highlight the core algorithm rather than the surrounding production scaffolding, but every simplification is called out so you know exactly what a real deployment would add. By the end, you should be able to reason about a decentralized system end-to-end, from the byte-level addressing scheme to the marketplace incentives that keep it alive.
Decentralized Architecture
Before diving into implementation, it is worth contrasting how centralized and decentralized architectures fundamentally differ in their failure modes and economics. A traditional architecture funnels every request through a single database or object store, which means a single hardware fault, cloud outage, or misconfiguration can take the entire service offline. It also creates vendor lock-in: migrating between cloud providers typically involves expensive egress fees, and pricing scales superlinearly once you exceed a modest data footprint. Decentralized architectures flip this model on its head by treating every node as equal and every copy of the data as a first-class citizen, so there is no single point of failure and no monopoly on storage.
The price of that resilience is a very different set of building blocks, each shown in the diagram below. Content addressing (CID) replaces location-based URLs with a cryptographic hash of the data itself, which means the address is stable regardless of which machine stores the bytes. The Merkle DAG layers a tree of linked, deduplicated blocks over that addressing scheme so large files can be verified and retrieved block by block. A Distributed Hash Table (DHT) provides the lookup service that maps a CID to the peers currently holding it, while the Bitswap protocol handles the actual block exchange. Finally, cryptographic verification ensures that whatever bytes arrive actually match the CID that was requested.
Storage tiering is another key design decision. Hot data that must be available immediately typically lives on IPFS or Swarm, where nodes actively serve and replicate it. Warm data moves to incentivized networks like Filecoin or Arweave, which pay miners to store it for a contractual duration. Cold archives end up on lower-cost networks such as Crust or Sia. Understanding where your data falls on this hot-to-cold spectrum determines which layer you build on, what it costs, and how quickly you can retrieve it.
Keep in mind that these layers are not interchangeable; they form a pipeline. Data enters the system through a content-addressing operation, is chunked into a Merkle DAG, and its blocks are located through a DHT and transferred over Bitswap. Persistence is then negotiated separately, usually with a Filecoin deal that pins the content on behalf of a paying client. If you think of each layer as an independent service with a narrow interface, you can mix and match providers—store hot data on one network, archive it on another—which is exactly how many production systems are actually assembled today.
┌─────────────────────────────────────────────────────────────────┐
│ Decentralized Storage Architecture │
├─────────────────────────────────────────────────────────────────┤
│ │
│ Traditional (Centralized) │
│ ├── Single point of failure │
│ ├── Vendor lock-in │
│ ├── Slow at scale │
│ └── Expensive at scale │
│ │
│ Decentralized │
│ ├── No single point of failure │
│ ├── Vendor neutral │
│ ├── Fast via caching nearby │
│ └── Cheaper at scale │
│ │
│ Key Technologies │
│ ├── Content addressing (CID) │
│ ├── Merkle DAG (Directed Acyclic Graph) │
│ ├── DHT (Distributed Hash Table) │
│ ├── P2P protocols (Bitswap) │
│ └── Cryptographic verification │
│ │
│ Storage Layers │
│ ├── Hot: IPFS, Swarm │
│ ├── Warm: Filecoin, Arweave │
│ └── Cold: Crust, Sia │
│ │
└─────────────────────────────────────────────────────────────────┘
The diagram also reveals an important operational reality: a decentralized storage network is only as good as the incentives that keep nodes alive. Each layer in the stack—addressing, lookup, transfer, and verification—has a corresponding failure mode, so production deployments typically combine multiple networks rather than betting on a single protocol. A common pattern is to store hot content on IPFS for speed, pin it via Filecoin for durability, and keep a centralized index as a safety net for discovery.
Two recurring questions should shape every architecture decision you make here. First, who guarantees that my data remains available tomorrow? The answer is whoever is paid or motivated to keep replicas: IPFS pinning services, Filecoin storage providers, or your own cluster of nodes. Second, what happens when a node lies or serves corrupted bytes? The Merkle DAG and CID verification give you the tooling to detect tampering, but you still need a policy for what to do when verification fails—typically fetch another replica and quarantine the offending peer.
IPFS Implementation
The most practical way to interact with IPFS from an application is through an HTTP client library such as ipfshttpclient, which speaks the same JSON-over-HTTP API exposed by the local IPFS daemon. The IPFSManager class below wraps that client and models the core operations you will perform in nearly every real application: adding a single file, adding a directory tree, reading content back by CID, pinning data so the garbage collector does not delete it, and inspecting the DAG statistics of an object. Note how every method guards against a missing client connection—IPFS is often not available in development or CI environments, and degrading gracefully is far more useful than crashing with a connection error.
Pinning deserves special attention because it is the mechanism that keeps content alive on the network. IPFS nodes automatically garbage-collect unpinned blocks, so data you add and never pin can silently disappear once the node restarts and the block cache is pruned. The get_stats method is equally important in production: it reports cumulative size and block count, which lets you understand how much actual storage a directory tree consumes and how efficiently it was chunked into blocks.
The second half of the code demonstrates content addressing in its purest form. calculate_cid_v0 reproduces the original IPFS scheme: a SHA-256 digest, prefixed with the multihash bytes that identify the hash function and digest length, then base58-encoded. calculate_cid_v1 builds on the newer multibase/multicodec framing that allows different codecs and hash algorithms while remaining self-describing. Knowing how CIDs are constructed matters because it explains why identical content always produces an identical address (deduplication) and why a one-bit change in the source bytes produces a completely different CID.
The create_merkle_dag function ties these ideas together by showing how nested directories become a Merkle DAG: each file is added as its own root, then wired into a directory tree via the MFS (Mutable File System) API, and finally collapsed into a single root CID. That root CID is the entire content-addressed address of the tree; fetching it retrieves the full structure, and verifying any block requires hashing only the blocks on its path back to the root.
One design decision in the code is worth highlighting: the manager returns plain dictionaries rather than custom objects. That keeps the class easy to test and easy to serialize, and it mirrors the JSON shape of the underlying HTTP API so there is no impedance mismatch between your application code and the protocol. In a larger codebase you would wrap these dictionaries in typed models with validation, but for a first integration the raw shape is a deliberate and defensible choice.
#!/usr/bin/env python3
"""IPFS interaction with Python."""
import ipfshttpclient
import hashlib
import json
from pathlib import Path
class IPFSManager:
"""Manage IPFS operations."""
def __init__(self, api_endpoint: str = "/ip4/127.0.0.1/tcp/5001"):
try:
self.client = ipfshttpclient.connect(api_endpoint)
except Exception as e:
print(f"IPFS not available: {e}")
self.client = None
def add_file(self, file_path: str) -> dict:
"""Add file to IPFS."""
if not self.client:
return {"error": "IPFS not connected"}
result = self.client.add(file_path)
return {
"cid": result["Hash"],
"name": result["Name"],
"size": result["Size"]
}
def add_directory(self, dir_path: str) -> dict:
"""Add directory recursively."""
if not self.client:
return {"error": "IPFS not connected"}
result = self.client.add(dir_path, recursive=True)
return {
"cid": result["Hash"],
"name": result["Name"],
"size": result["Size"]
}
def cat_file(self, cid: str) -> bytes:
"""Get file content from IPFS."""
if not self.client:
return b""
return self.client.cat(cid)
def pin_file(self, cid: str) -> bool:
"""Pin file to prevent garbage collection."""
if not self.client:
return False
self.client.pin.add(cid)
return True
def get_stats(self, cid: str) -> dict:
"""Get file statistics."""
if not self.client:
return {}
stat = self.client.files.stat(f"/ipfs/{cid}")
return {
"size": stat.get("Size"),
"cumulative_size": stat.get("CumulativeSize"),
"blocks": stat.get("Blocks"),
"type": stat.get("Type")
}
def list_pins(self) -> list:
"""List pinned files."""
if not self.client:
return []
pins = []
for pin in self.client.pin.ls():
pins.append({
"cid": pin["Cid"],
"type": pin["Type"]
})
return pins
def create_ipns_record(self, cid: str, key_name: str = "self") -> str:
"""Create IPNS (InterPlanetary Name System) record."""
if not self.client:
return ""
# Publish CID to IPNS
result = self.client.name.publish(cid, key=key_name)
return result["Name"]
class ContentAddressing:
"""Content addressing with CIDs."""
@staticmethod
def calculate_cid_v0(data: bytes) -> str:
"""Calculate CIDv0 (base58 multihash)."""
import base58
# SHA-256 hash
digest = hashlib.sha256(data).digest()
# Create multihash: <hash-func-id><digest-length><digest>
multihash = bytes([0x12, 0x20]) + digest # sha2-256 + 32 bytes
# Base58 encode
cid_v0 = base58.b58encode(multihash).decode()
return cid_v0
@staticmethod
def calculate_cid_v1(data: bytes) -> str:
"""Calculate CIDv1 (multibase + multicodec)."""
import multiformats
# Use multiformats library
cid = multiformats.cid.CIDv1(
version=1,
codec='raw',
mhcode='sha2-256',
mhlen=32,
digest=hashlib.sha256(data).digest()
)
return str(cid)
@staticmethod
def parse_cid(cid_string: str) -> dict:
"""Parse CID and extract components."""
# Use py-multiformats
try:
import multiformats
cid = multiformats.cid.CID.decode(cid_string)
return {
"version": cid.version,
"codec": cid.codec,
"mh_code": cid.mhcode,
"mh_length": cid.mhlen
}
except:
return {}
def create_merkle_dag():
"""Create Merkle DAG structure."""
import ipfshttpclient
client = ipfshttpclient.connect()
# Create nested structure
# File structure:
# root/
# ├── dir1/
# │ └── file1.txt
# └── dir2/
# └── file2.txt
# Add files first
file1 = client.add("file1.txt")
file2 = client.add("file2.txt")
# Create directory structure
root_cid = client.files.mkdir("/root/dir1")
client.files.cp(file1["Hash"], "/root/dir1/file1.txt")
client.files.mkdir("/root/dir2")
client.files.cp(file2["Hash"], "/root/dir2/file2.txt")
# Get root CID
stat = client.files.stat("/root")
print(f"Root CID: {stat['Cid']}")
A useful mental model from this section: the CID is to IPFS what a URL is to HTTP, except the CID also guarantees authenticity. The trade-off is that content addressing is immutable by default—updating a file changes its CID—which is exactly why IPNS exists. The create_ipns_record method publishes a mutable name that points at the current CID, so clients can resolve a stable name without knowing the latest content hash. This name-to-CID indirection is the foundation for any application that needs updatable content on an immutable storage layer.
There are two practical consequences you should carry forward. First, plan your pinning strategy before you add anything: decide which CIDs are pinned locally, which are delegated to a pinning service, and which are covered by Filecoin deals, because unpinned data is ephemeral. Second, treat CID changes as an application event rather than an error; when you republish a file you are creating a new object, and any system that referenced the old CID must be told about the new one explicitly.
Filecoin Storage
Filecoin extends IPFS from a best-effort caching network into an incentivized storage market with contractual guarantees. The integration below uses a Lotus client, which is the reference implementation that speaks Filecoin’s full node API. The critical concept is the storage deal: a client agrees to pay a miner a price denominated in attoFIL per epoch to store a specific piece of data for a fixed duration, and the network enforces the contract cryptographically rather than through any central authority. This changes the developer’s job from “put bytes somewhere” to “negotiate and monitor a contract,” which is a fundamentally different operational model.
The create_deal method shows the real workflow, and the first design decision is miner selection. In a marketplace of thousands of storage providers, blindly picking a miner is a recipe for lost data, so the implementation ranks candidates with a weighted reputation score. Capacity (Power) is weighted heaviest at 50%, followed by the number of successful deals at 30%, and finally node age capped at five years at 20%. This heuristic is deliberately simple; production systems layer on pledge, fault history, and geographic diversity on top of these three signals.
The retrieval path is the mirror image of the storage path. Whereas storing data requires agreeing on price and duration up front, retrieval creates a separate retrieval deal and streams the bytes back in chunks. The get_deal_status method surfaces the raw deal metadata—provider, piece CID, size, price, duration, and epoch—which is what operators need to audit whether their data is actually being stored as agreed. Because deals have fixed durations, any real integration must also run a renewal job that re-creates deals before they expire, or the data silently falls back to cold storage.
Finally, StoragePricing makes the economics concrete. Filecoin’s blockchain produces roughly 2,880 epochs per day (one every five minutes), and costs are quoted per GiB per epoch, which is an extremely fine granularity. The estimate converts that per-epoch price into a daily and monthly dollar figure using a fixed FIL-to-USD assumption. Notice the deliberate simplification: real pricing varies wildly by miner and market conditions, so treat the result as a planning estimate rather than a quote, and build in margin for the price discovery that a live marketplace will impose.
It is also worth noting what the code does not show, because the missing pieces are where real deployments succeed or fail. A production integration needs a deal monitoring loop that watches state transitions (proposal to active to expired), a fault-reporting path that files penalties against misbehaving miners, and a wallet that keeps a buffer of FIL to cover renewal and retrieval fees. None of this is protocol magic; it is ordinary distributed-systems engineering applied to a crypto-economic substrate, and the class structure above gives you the seams to hang those pieces on.
#!/usr/bin/env python3
"""Filecoin storage integration."""
from lotus import LotusClient
from pathlib import Path
from typing import Dict, List
import time
class FilecoinStorage:
"""Manage Filecoin storage deals."""
def __init__(self, api_token: str, api_endpoint: str):
self.client = LotusClient(api_endpoint, api_token)
def create_deal(self, cid: str,
duration: int = 180) -> Dict:
"""Create storage deal."""
# Find available miners
miners = self.client.state.list_miners()
# Filter by reputation/size
selected_miner = self._select_miner(miners)
# Create deal proposal
deal = self.client.client.deal(
cid=cid,
miner=selected_miner,
price="0.000000001", # AttoFIL per epoch
duration=duration, # Days
)
return {
"deal_id": deal["DealID"],
"miner": selected_miner,
"status": deal["State"]
}
def _select_miner(self, miners: List[str]) -> str:
"""Select best miner based on criteria."""
best_miner = None
best_score = 0
for miner in miners[:10]: # Check top 10
try:
info = self.client.state.miner_info(miner)
score = self._calculate_miner_score(info)
if score > best_score:
best_score = score
best_miner = miner
except:
continue
return best_miner or miners[0]
def _calculate_miner_score(self, info: Dict) -> float:
"""Calculate miner reputation score."""
score = 0
# Power (storage capacity)
score += info.get("Power", 0) * 0.5
# Reputation (number of successful deals)
score += info.get("SuccessfulDeals", 0) * 0.3
# Age (time in network)
score += min(info.get("Age", 0) / 365, 5) * 0.2
return score
def retrieve_data(self, cid: str, output_path: str):
"""Retrieve data from Filecoin."""
# Create retrieval deal
deal = self.client.client.retrieve(cid)
# Write to file
with open(output_path, 'wb') as f:
for chunk in deal:
f.write(chunk)
def get_deal_status(self, deal_id: int) -> Dict:
"""Get deal status."""
deal = self.client.client.get_deal(deal_id)
return {
"deal_id": deal_id,
"state": deal["State"],
"provider": deal["Provider"],
"piece_cid": deal["PieceCID"],
"size": deal["Size"],
"price": deal["Price"],
"duration": deal["Duration"],
"start_epoch": deal["StartEpoch"],
}
class StoragePricing:
"""Calculate Filecoin storage costs."""
@staticmethod
def estimate_cost(size_gb: float,
duration_days: int) -> Dict:
"""Estimate storage costs."""
# Average price (attoFIL per GiB per epoch)
avg_price_per_epoch = 0.0000001 # 0.0000001 FIL/GiB/epoch
epochs_per_day = 2880 # ~5 minutes per epoch
total_epochs = duration_days * epochs_per_epoch
# Calculate cost
cost = size_gb * avg_price_per_epoch * total_epochs
# Convert to FIL and USD (assuming $5/FIL)
cost_fil = cost
cost_usd = cost_fil * 5
return {
"size_gb": size_gb,
"duration_days": duration_days,
"cost_fil": cost_fil,
"cost_usd": cost_usd,
"per_month_usd": cost_usd / (duration_days / 30)
}
The key takeaway from the Filecoin section is that decentralized storage is a marketplace, not a filesystem. Cost, durability, and retrieval speed all depend on the deals you negotiate and the miners you select, which is why the reputation-scoring logic deserves as much engineering attention as the storage calls themselves. Treat the deal lifecycle—proposal, acceptance, sealing, and renewal—as a state machine you must monitor, not a fire-and-forget write. If you take nothing else from this section, remember that in the decentralized world you are always buying a service with a contract, and your code is the contract’s auditor.
Decentralized Database
A content-addressed filesystem gives you storage, but applications need queryable data structures, which is where OrbitDB enters the picture. OrbitDB is a peer-to-peer database built directly on top of IPFS: every database is itself a Merkle DAG, and updates are propagated as log entries that any peer can replay to reconstruct the same state. The connect method demonstrates the most important design decision, which is choosing the right data model. OrbitDB offers a key-value store for simple lookups, a document store for schema-like records that can be filtered, an append-only feed for event sourcing and activity streams, and a counter for the narrow but common case of distributed increments.
Document stores are the workhorse for most applications. The query_documents method shows how filters are expressed as predicates rather than SQL: every document is tested against a dictionary of equality constraints. This is both a strength and a limitation—it is trivially simple and fully local, but it does not scale to joins or aggregation the way a centralized SQL engine does, because there is no single node holding the whole dataset.
The second half of the code reveals the conflict-resolution machinery that makes a distributed database consistent at all. OrbitDB is built on CRDTs (Conflict-free Replicated Data Types), data structures engineered so that concurrent updates merge deterministically no matter the order they arrive. The G-Counter is a grow-only counter where each node keeps a per-node subtotal and the total is the sum; merging takes the maximum of each node’s value. The LWW-Register is a last-writer-wins field that compares timestamps. The OR-Set is the most interesting case: it tracks which nodes observed each element so that a remove operation only wins if it was observed after every concurrent add, preventing the classic problem of a deleted element reappearing.
Understanding these three CRDTs is worth the effort because they map directly onto real design choices. If you can express your conflict policy as “latest write wins,” the LWW register is your tool. If correctness requires that deletes never resurrect data, you need the OR-Set semantics. Choosing the wrong conflict policy is the decentralized equivalent of choosing the wrong sharding key—the system will work in tests and misbehave under real concurrency.
A common misconception is that CRDTs give you strong consistency; they do not. What they give you is eventual consistency with guaranteed convergence: every replica will eventually agree on a final state, but that state is defined by your merge policy, not by a global transaction order. That distinction matters because it changes what you can safely store. Counters, sets, and last-writer-wins fields are safe; anything requiring an atomic read-modify-write across the whole dataset, like a unique index or a balance check, needs an external coordination mechanism that OrbitDB alone does not provide.
#!/usr/bin/env python3
"""OrbitDB - Decentralized database."""
from orbitdb import OrbitDB
import asyncio
class DecentralizedDatabase:
"""Manage OrbitDB decentralized database."""
def __init__(self, ipfs_node):
self.db = None
async def connect(self, database_name: str):
"""Connect to OrbitDB."""
# Create OrbitDB instance
orbitdb = await OrbitDB.create_instance(ipfs_node)
# Create different types of databases
# Key-Value store
self.kv_db = await orbitdb.kvstore(database_name)
# Document store
self.doc_db = await orbitdb.docstore(f"{database_name}_docs")
# Feed (append-only log)
self.feed_db = await orbitdb.feed(f"{database_name}_feed")
# Counter
self.counter_db = await orbitdb.counter(f"{database_name}_counter")
async def put_key_value(self, key: str, value):
"""Store key-value."""
await self.kv_db.put(key, value)
async def get_key_value(self, key: str):
"""Retrieve key-value."""
return self.kv_db.get(key)
async def put_document(self, doc: dict):
"""Store document."""
await self.doc_db.put(doc)
async def query_documents(self, query: dict):
"""Query documents."""
return self.doc_db.query(lambda doc:
all(doc.get(k) == v for k, v in query.items())
)
async def append_feed(self, data):
"""Append to feed."""
await self.feed_db.add(data)
async def get_all_feed(self):
"""Get all feed entries."""
return list(self.feed_db.iterator())
async def increment_counter(self):
"""Increment counter."""
await self.counter_db.inc()
async def get_counter(self):
"""Get counter value."""
return self.counter_db.value
class CRDTDataStructure:
"""CRDT (Conflict-free Replicated Data Types)."""
# G-Counter (Grow-only counter)
class GCounter:
def __init__(self, node_id):
self.node_id = node_id
self.counts = {}
def increment(self):
self.counts[self.node_id] = \
self.counts.get(self.node_id, 0) + 1
def value(self):
return sum(self.counts.values())
def merge(self, other):
for node_id, count in other.counts.items():
self.counts[node_id] = max(
self.counts.get(node_id, 0),
count
)
# LWW-Register (Last-Writer-Wins)
class LWWRegister:
def __init__(self):
self.value = None
self.timestamp = 0
self.node_id = None
def set(self, value, node_id):
import time
ts = time.time()
if ts > self.timestamp:
self.value = value
self.timestamp = ts
self.node_id = node_id
def get(self):
return self.value
# OR-Set (Observed-Remove Set)
class ORSet:
def __init__(self):
self.elements = {} # element -> set of node_ids
def add(self, element, node_id):
if element not in self.elements:
self.elements[element] = set()
self.elements[element].add(node_id)
def remove(self, element, node_id):
if element in self.elements:
self.elements[element].discard(node_id)
if not self.elements[element]:
del self.elements[element]
def get(self):
return set(self.elements.keys())
The takeaway is that a decentralized database is not a drop-in replacement for Postgres; it is a different consistency model with different guarantees. You gain offline-first operation, replication without a master, and freedom from a central server, and you pay for it with weaker query semantics and conflict policies you must understand deeply. For most teams, the right move is hybrid: keep authoritative relational data centralized and use OrbitDB-style stores for collaborative, offline-first features where the consistency trade-off is acceptable.
Concretely, OrbitDB shines for applications like collaborative note-taking, peer-to-peer messaging, and device synchronization, where every client needs to work offline and merge changes later. It struggles when you need ad-hoc analytical queries, transactions spanning many documents, or a strict unique constraint enforced globally. Match the database to the workload, not the workload to the database, and you will get most of the benefit of decentralization without tripping over its limits.
P2P Protocols
Beneath IPFS, Filecoin, and OrbitDB sits the same foundation: the libp2p networking stack and its peer-to-peer protocols. This section implements two of the most important pieces. Bitswap is IPFS’s data-transfer protocol, and the BitswapMessage class shows its core abstraction: a want list, which is the set of blocks a node is looking for, and the blocks themselves that it is offering in exchange. The serialization logic matters because messages must be compact and self-describing; each want entry carries a priority so peers can decide what to send first, and every offered block includes a SHA-256 checksum so receivers can verify integrity on arrival.
The DHTClient class implements a simplified Kademlia distributed hash table, which is the lookup backbone of the entire ecosystem. Two constants define the algorithm’s shape. K (20) is the bucket size—the number of peers a node remembers for each region of the key space—and ALPHA (3) is the parallelism factor, the number of in-flight queries allowed at once. Routing in Kademlia is purely geometric: nodes are identified by their keys, and distance is computed as the XOR of the two identifiers. This unusual metric has a crucial property: because it is symmetric and the “closest” peers to a key are exactly the ones responsible for it, lookups converge logarithmically with the network size.
The find_peers method demonstrates the iterative lookup: start with the K closest nodes you know, query the ALPHA closest of them in parallel, fold the results back into the candidate list, and repeat until no closer peers remain. The put and get methods then show how these lookups are used in practice. Storing a value replicates it to the K closest peers, while retrieval checks the local store first—the Kademlia cache—and only falls back to the network when the value is absent locally. That local-store shortcut is a small detail with a large performance impact in production, since it turns repeated reads into zero-network operations.
Note what the implementation deliberately leaves as stubs. _query_peer, _store_on_peer, and _get_from_peer are network calls in a real deployment, implemented with libp2p streams or UDP messages. Structuring the code so these are isolated methods is itself a design decision: it keeps the protocol logic testable against in-memory peers and makes the network boundary explicit, which is exactly how the real implementations are organized.
The choice of XOR distance also deserves a moment of appreciation because it is what makes Kademlia self-balancing. Because keys are fixed-size, XOR distance is symmetric and satisfies the triangle inequality, so every node can compute its own position relative to every other node without global information. Combined with the K-bucket structure that keeps the closest observed peers for each bit prefix, the routing table stays fresh under churn without any central maintenance. When you see a production DHT serve millions of lookups a day, this geometric elegance is the reason it can do so.
#!/usrBitswap - IPFS's data transfer protocol
class BitswapMessage:
"""Bitswap message structure."""
def __init__(self):
self.want_list = []
self.blocks = []
self.pending = []
self.full = False
def add_want(self, cid: str, priority: int = 1):
"""Add block to want list."""
self.want_list.append({
"cid": cid,
"priority": priority,
"cancel": False,
"send_dont_have": True
})
def add_block(self, cid: str, data: bytes):
"""Add block to message."""
self.blocks.append({
"cid": cid,
"data": data,
"metadata": {
"block_size": len(data),
"checksum": self._calculate_checksum(data)
}
})
def _calculate_checksum(self, data: bytes) -> bytes:
"""Calculate block checksum."""
import hashlib
return hashlib.sha256(data).digest()
def serialize(self) -> bytes:
"""Serialize message for transmission."""
import protobuf
message = bitswap_pb2.Message()
for want in self.want_list:
entry = message.wantlist.add()
entry.block = want["cid"]
entry.priority = want["priority"]
entry.cancel = want["cancel"]
for block in self.blocks:
data = message.blocks.add()
data.prefix = block["cid"][:8] # CID prefix
data.data = block["data"]
return message.SerializeToString()
class DHTClient:
"""Distributed Hash Table client."""
# Kademlia DHT
K = 20 # Size of bucket
ALPHA = 3 # Parallel queries
def __init__(self, node_id: str):
self.node_id = node_id
self.routing_table = {}
self.local_store = {}
def find_peers(self, key: str) -> List[str]:
"""Find peers closest to key."""
# Get closest nodes from routing table
closest = self._get_closest(key, self.K)
# Query in parallel (alpha)
queried = set()
to_query = closest[:self.ALPHA]
while to_query:
node = to_query.pop(0)
if node in queried:
continue
queried.add(node)
# Query node for closer peers
nearer = self._query_peer(node, key)
# Add closer peers to query
for p in nearer:
if p not in queried:
to_query.append(p)
return list(queried)
def _get_closest(self, key: str, count: int) -> List[str]:
"""Get closest nodes to key."""
distances = []
for node_id in self.routing_table.keys():
dist = self._xor_distance(key, node_id)
distances.append((dist, node_id))
distances.sort()
return [n for _, n in distances[:count]]
def _xor_distance(self, key1: str, key2: str) -> int:
"""Calculate XOR distance between keys."""
import intset
k1 = intset.from_string(key1)
k2 = intset.from_string(key2)
return k1 ^ k2
def _query_peer(self, node: str, key: str) -> List[str]:
"""Query peer for closer nodes."""
# In practice: network call to peer
return []
def put(self, key: str, value: bytes):
"""Store value in DHT."""
# Find peers responsible for key
peers = self.find_peers(key)
# Store on k-closest peers
for peer in peers[:self.K]:
self._store_on_peer(peer, key, value)
def get(self, key: str) -> bytes:
"""Retrieve value from DHT."""
# Check local first
if key in self.local_store:
return self.local_store[key]
# Find peers
peers = self.find_peers(key)
# Query peers
for peer in peers[:self.ALPHA]:
value = self._get_from_peer(peer, key)
if value:
self.local_store[key] = value
return value
return None
def _store_on_peer(self, peer: str, key: str, value: bytes):
"""Store on peer (network call)."""
pass
def _get_from_peer(self, peer: str, key: str) -> bytes:
"""Get from peer (network call)."""
pass
Together, Bitswap and the Kademlia DHT explain how a network with no central index can still find data reliably at scale: the DHT answers “who has this block?” in logarithmic time, and Bitswap answers “give me this block” with a priority-aware exchange. Mastering these two protocols is the foundation for understanding every higher-level decentralized system, from file sharing to name resolution to distributed databases. When a decentralized application misbehaves, the root cause is almost always here—in routing tables that went stale or want lists that starved—so it pays to instrument these layers before anything else.
As a final mental exercise, trace what happens when a new node joins an IPFS network and requests a file it has never seen. It computes the CID locally from the content it wants, asks its bootstrap peers for the nodes closest to that CID, runs a Kademlia lookup to converge on providers, issues Bitswap wants for the individual blocks, verifies each block against the DAG, and finally pins what it cares to keep. Every step in that story has been a code example in this article, which is the clearest proof that decentralized systems are, at heart, a few elegant algorithms wired together with a lot of careful operational engineering.
External Resources
Related Articles
- Edge Computing: Cloudflare Workers, AWS Lambda@Edge
- WebAssembly (WASM): Production Deployment Patterns
- Quantum Computing: Algorithms and Simulators
Comments