File system operations at scale run into three limits: throughput (I/O bandwidth), concurrency (file descriptor limits), and memory (loading too much at once). The patterns in this guide keep all three in check.
For basic file operations see Go file system operations. For streaming large files specifically see Go working with large datasets.
Parallel Directory Walk with Bounded Concurrency
filepath.WalkDir is sequential — one file at a time. For CPU-bound processing (hashing, parsing) across thousands of files, parallel processing is faster. The key is bounding concurrency to avoid exhausting file descriptors:
func processDirectoryParallel(root string, concurrency int, fn func(path string) error) error {
sem := make(chan struct{}, concurrency) // semaphore limits open files
var wg sync.WaitGroup
var mu sync.Mutex
var firstErr error
err := filepath.WalkDir(root, func(path string, d fs.DirEntry, err error) error {
if err != nil { return err }
if d.IsDir() { return nil }
// Check if a previous worker already failed
mu.Lock()
if firstErr != nil {
mu.Unlock()
return firstErr
}
mu.Unlock()
wg.Add(1)
sem <- struct{}{} // acquire slot (blocks if concurrency limit reached)
go func(p string) {
defer wg.Done()
defer func() { <-sem }() // release slot
if err := fn(p); err != nil {
mu.Lock()
if firstErr == nil { firstErr = err }
mu.Unlock()
}
}(path)
return nil
})
wg.Wait()
if firstErr != nil { return firstErr }
return err
}
// Usage: hash all files with 16 parallel workers
err := processDirectoryParallel("/data", 16, func(path string) error {
hash, err := hashFile(path)
if err != nil { return fmt.Errorf("hash %s: %w", path, err) }
fmt.Printf("%s %s\n", hash, path)
return nil
})
The semaphore channel bounds the number of simultaneously-open file descriptors. Set concurrency based on your workload: CPU-bound tasks (hashing, compression) → runtime.NumCPU(); I/O-bound → 2×–4× CPU count.
Streaming Large Files with Custom Buffers
The default bufio.Scanner buffer (64KB) silently truncates lines longer than that and returns an error. For log files with large JSON lines or structured data, increase the buffer:
func streamLargeFile(path string, maxLineBytes int, fn func(line string) error) error {
f, err := os.Open(path)
if err != nil { return err }
defer f.Close()
scanner := bufio.NewScanner(f)
// Pre-allocate a buffer that can hold maxLineBytes
// Initial capacity is smaller; it grows as needed up to maxLineBytes
scanner.Buffer(make([]byte, 0, min(maxLineBytes, 64*1024)), maxLineBytes)
for scanner.Scan() {
if err := fn(scanner.Text()); err != nil { return err }
}
return scanner.Err() // nil on clean EOF, non-nil on read errors or line-too-long
}
For binary file processing where you want fixed-size chunks rather than lines:
func streamChunks(path string, chunkSize int, fn func(chunk []byte, offset int64) error) error {
f, err := os.Open(path)
if err != nil { return err }
defer f.Close()
buf := make([]byte, chunkSize)
var offset int64
for {
n, err := io.ReadFull(f, buf)
if n > 0 {
if ferr := fn(buf[:n], offset); ferr != nil { return ferr }
offset += int64(n)
}
if err == io.EOF || err == io.ErrUnexpectedEOF { return nil }
if err != nil { return err }
}
}
io.ReadFull reads exactly len(buf) bytes, retrying partial reads. For the last chunk (smaller than chunkSize), it returns io.ErrUnexpectedEOF — treat that as EOF.
Atomic Directory Operations
When deploying files (configs, templates, static assets), write to a temp directory then rename to atomically replace the old directory:
func atomicReplaceDir(targetDir string, populate func(tempDir string) error) error {
// Create temp dir in the same filesystem as target (ensures rename works)
parent := filepath.Dir(targetDir)
tempDir, err := os.MkdirTemp(parent, ".tmp-")
if err != nil { return fmt.Errorf("create temp dir: %w", err) }
defer os.RemoveAll(tempDir) // cleanup if anything fails
// Populate the temp directory
if err := populate(tempDir); err != nil {
return fmt.Errorf("populate: %w", err)
}
// Rename old dir out of the way
oldDir := targetDir + ".old"
os.Rename(targetDir, oldDir) // move current aside (ignore error if doesn't exist)
defer os.RemoveAll(oldDir) // clean up old dir
// Atomically swap
if err := os.Rename(tempDir, targetDir); err != nil {
// Rollback: restore old dir
os.Rename(oldDir, targetDir)
return fmt.Errorf("rename: %w", err)
}
return nil
}
// Usage
err := atomicReplaceDir("/etc/myapp/config", func(dir string) error {
return downloadAndExtract(configURL, dir)
})
The two-step (old aside + rename new) pattern works around the fact that os.Rename fails if the target exists on some systems.
Batch Operations with Controlled Memory
When processing thousands of files into a database or API, batch them to avoid overwhelming downstream services:
func processBatch(ctx context.Context, dir string, batchSize int, fn func([]string) error) error {
var batch []string
err := filepath.WalkDir(dir, func(path string, d fs.DirEntry, walkErr error) error {
if walkErr != nil { return walkErr }
if d.IsDir() { return nil }
batch = append(batch, path)
if len(batch) >= batchSize {
if err := fn(batch); err != nil { return err }
batch = batch[:0] // reset slice, keep capacity
// Respect context cancellation between batches
select {
case <-ctx.Done(): return ctx.Err()
default:
}
}
return nil
})
if err != nil { return err }
// Process remaining items
if len(batch) > 0 {
return fn(batch)
}
return nil
}
Managing File Descriptor Limits
Each open file consumes a file descriptor. The OS limits how many a process can have open simultaneously (typically 1024 on Linux, ulimit -n). For utilities that process thousands of files:
import "golang.org/x/sys/unix"
func checkAndRaiseFDLimit(desired uint64) error {
var rlimit unix.Rlimit
if err := unix.Getrlimit(unix.RLIMIT_NOFILE, &rlimit); err != nil {
return err
}
if rlimit.Cur >= desired {
return nil // already sufficient
}
// Try to raise the soft limit
rlimit.Cur = min(desired, rlimit.Max)
if err := unix.Setrlimit(unix.RLIMIT_NOFILE, &rlimit); err != nil {
return fmt.Errorf("setrlimit to %d: %w", rlimit.Cur, err)
}
slog.Info("raised fd limit", slog.Uint64("new_soft_limit", rlimit.Cur))
return nil
}
// Call at startup for utilities that open many files
func main() {
if err := checkAndRaiseFDLimit(65536); err != nil {
slog.Warn("could not raise fd limit", slog.Any("error", err))
}
// rest of program...
}
Alternatively, configure the limit in the systemd unit file (LimitNOFILE=65536) or in Docker (--ulimit nofile=65536:65536).
Finding Files Efficiently
For pattern matching across large directories, filepath.WalkDir with early filepath.SkipDir is faster than loading all entries first:
func findFiles(root, ext string, maxResults int) ([]string, error) {
var results []string
err := filepath.WalkDir(root, func(path string, d fs.DirEntry, err error) error {
if err != nil { return err }
if d.IsDir() {
// Skip hidden directories and vendor
name := d.Name()
if len(name) > 1 && name[0] == '.' { return filepath.SkipDir }
if name == "vendor" || name == "node_modules" { return filepath.SkipDir }
return nil
}
if filepath.Ext(path) == ext {
results = append(results, path)
if len(results) >= maxResults {
return filepath.SkipAll // Go 1.20+: stop entire walk
}
}
return nil
})
return results, err
}
filepath.SkipAll (Go 1.20) stops the entire walk immediately — more efficient than returning an error and ignoring it.
Summary
- Bound parallel walks with a semaphore channel — file descriptor exhaustion crashes the process silently
scanner.Buffer(make([]byte, 0, initial), max)handles large lines; always checkscanner.Err()after the loop- Atomic directory replacement: populate temp dir → rename old aside → rename new → cleanup old
- Batch walk results and respect
ctx.Done()between batches for cancellable long-running operations - Check and raise
RLIMIT_NOFILEat startup for utilities that open many files concurrently filepath.SkipAll(Go 1.20) stops a walk immediately — more efficient than error-based early exit
Comments