MinIO Internals: Understanding the Distributed Object Store

Introduction

Understanding MinIO’s internal architecture helps you optimize deployments, troubleshoot issues, and design efficient data pipelines. MinIO is built from the ground up for distributed object storage, with every component designed for horizontal scalability and high performance. This article explores the key architectural components that make MinIO work.

Architecture Overview

MinIO follows a distributed architecture:

┌─────────────────────────────────────────────────────────┐
│                    MinIO Cluster                         │
├─────────────────────────────────────────────────────────┤
│  ┌─────────┐  ┌─────────┐  ┌─────────┐  ┌─────────┐ │
│  │ Server 1│  │ Server 2│  │ Server 3│  │ Server 4│ │
│  ├─────────┤  ├─────────┤  ├─────────┤  ├─────────┤ │
│  │ Drive 1 │  │ Drive 2 │  │ Drive 3 │  │ Drive 4 │ │
│  │ Drive 2 │  │ Drive 1 │  │ Drive 4 │  │ Drive 3 │ │
│  └─────────┘  └─────────┘  └─────────┘  └─────────┘ │
├─────────────────────────────────────────────────────────┤
│         Erasure Coding (EC:4) - 4 data, 4 parity       │
└─────────────────────────────────────────────────────────┘

Erasure Coding

How Erasure Coding Works

When you write an object, MinIO:

1. Splits the data into N fragments (data drives)
2. Generates M parity fragments (erasure code)
3. Distributes all fragments across drives/nodes

// Simplified erasure encoding
// N data drives, M parity drives
type EC struct {
    DataShards int
    ParityShards int
}

// Example: 8 drives (4 data + 4 parity)
// EC:4 can recover from losing any 4 drives

Recovery

When reading from a degraded cluster:

// Read from available fragments
// Use erasure decoding to reconstruct missing data
// Continue serving requests even during failures

Erasure Code Calculator

Distributed Hashing

MinIO uses consistent hashing for data placement:

Hash-Based Distribution

// Consistent hashing for object placement
type ConsistentHash struct {
    virtualNodes int
    hashRange int
}

// Object key determines placement
// hash(key) -> drive assignment
// Ensures even distribution across drives

Benefits

• Even distribution: Objects spread across all drives
• Minimal remapping: Adding drives only moves small amount of data
• Fault tolerance: Knows which drives hold each object’s fragments

Storage Engine

Object Storage Format

┌────────────────────────────────────┐
│         Object on Disk              │
├────────────────────────────────────┤
│  xl.meta (metadata)                │
│  - Checksums                       │
│  - Erasure coding info             │
│  - Version info                    │
│  - Custom metadata                 │
├────────────────────────────────────┤
│  part.1 (data fragment)            │
│  part.2 (data fragment)            │
│  ...                               │
└────────────────────────────────────┘

Metadata Structure

{
  "version": "v1",
  "format": "xl",
  "stat": {
    "size": 1024000,
    "mtime": "2026-01-01T00:00:00Z",
    "etag": "abc123"
  },
  "parts": [
    {"number": 1, "size": 512000},
    {"number": 2, "size": 512000}
  ],
  "erasure": {
    "data": 4,
    "parity": 4,
    "blockSize": 1048576
  }
}

Write Path

1. Client sends PUT request
   ↓
2. MinIO calculates hash
   ↓
3. Select drives using consistent hash
   ↓
4. Write data fragments to drives in parallel
   ↓
5. Write metadata
   ↓
6. Return success to client

Read Path

1. Client sends GET request
   ↓
2. Find drives holding object fragments
   ↓
3. Read fragments in parallel
   ↓
4. Decode and reconstruct data
   ↓
5. Return to client

Quorum and Consensus

Write Quorum

MinIO requires a minimum number of successful writes:

• Write quorum: N/2 + 1 drives must acknowledge write
• For EC:4 (8 drives): 5 writes required

Read Quorum

For reading from degraded cluster:

• Read quorum: N/2 + 1 fragments needed
• Can read even with 2 drive failures

Healing

// Automatic healing after drive failure
// When drive returns:
// 1. Check fragment checksums
// 2. Recalculate missing fragments
// 3. Write to missing locations
// 4. Update metadata

Performance Characteristics

Write Performance

# Sequential writes: ~5 GB/s per node
# Parallel writes: Linear scaling with nodes

# Example: 4 nodes, 8 drives each
# Write throughput: ~20 GB/s

Read Performance

# Sequential reads: ~10 GB/s per node
# Parallel reads: Scales with nodes

# Random reads: Depends on SSD
# Typical: ~100K IOPS per node

Latency

• Single object GET: <10ms (cached)
• Single object PUT: <20ms
• Metadata operations: <5ms

Caching

MinIO includes built-in caching:

// Cache configuration
type CacheConfig struct {
    Drives     []string  // Cache drives
    Quota     int       // Max % used
    Exclude   []string  // Exclude patterns
    After     int       // Access count before caching
}

// Cache works at:
// - Drive level
// - Cluster level

Cache Behavior

# Enable cache
mc admin config set myminio \
  cache drive=/mnt/cache \
  quota=80 \
  exclude="*.mp4,*.pdf"

# Cache hit serves from memory/disk
# Cache miss fetches from backend

Data Tiering

Overview

MinIO supports multiple storage tiers:

┌─────────────────────────────────────────┐
│            Hot (NVMe SSD)               │
│  Active data, frequent access           │
└─────────────────────────────────────────┘
         ↓ Transition (30 days)
┌─────────────────────────────────────────┐
│          Warm (HDD/ SATA SSD)           │
│  Less frequent access                   │
└─────────────────────────────────────────┘
         ↓ Transition (90 days)
┌─────────────────────────────────────────┐
│           Cold (Object Storage)          │
│  Glacier/cloud storage                   │
└─────────────────────────────────────────┘

Configuring Tiering

# Add remote tier
mc admin tier add minio s3 tier-1 \
  --endpoint https://s3.amazonaws.com \
  --access-key ACCESSKEY \
  --secret-key SECRETKEY \
  --bucket mybucket

# Set lifecycle
mc ilm add myminio/bucket \
  --transition-days 30 \
  --tier tier-1

Scalability

Horizontal Scaling

# Add new server to cluster
# Simply include new server in startup

./minio server \
  http://server1/minio{1...4} \
  http://server2/minio{1...4} \
  http://server3/minio{1...4} \
  http://server4/minio{1...4}  # New server

Capacity Planning

Conclusion

MinIO’s architecture is designed for simplicity and performance. Key takeaways: erasure coding provides data protection without replication overhead, consistent hashing ensures even data distribution, quorum-based operations guarantee consistency, and the distributed design scales horizontally. Understanding these internals helps you optimize deployments and troubleshoot issues.

In the next article, we’ll explore recent MinIO developments and trends for 2025-2026.

Resources

• MinIO Architecture
• Erasure Coding Deep Dive
• MinIO Internals