The os, io, bufio, and path/filepath packages cover everything you need for file system work in Go. The standard library’s approach is deliberately low-level — you compose small, well-defined pieces rather than calling one magic function. Understanding each piece makes the compositions obvious.
The most common sources of bugs in file handling are: not closing files (resource leak), not checking scanner errors after a loop (silently missing the last lines), and building paths with string concatenation instead of filepath.Join (breaks on Windows).
For reading large files specifically see Go working with large datasets.
Reading Files
For small files where the whole content fits comfortably in memory, os.ReadFile is the simplest approach:
data, err := os.ReadFile("config.json")
if err != nil {
return fmt.Errorf("reading config: %w", err)
}
// data is []byte — convert to string if needed
fmt.Println(string(data))
For line-by-line processing of larger files, bufio.Scanner is the idiomatic choice. It buffers reads internally and handles line endings (\r\n and \n) across platforms:
func processLines(path string, fn func(line string) error) error {
f, err := os.Open(path)
if err != nil {
return fmt.Errorf("opening %s: %w", path, err)
}
defer f.Close()
scanner := bufio.NewScanner(f)
for scanner.Scan() {
if err := fn(scanner.Text()); err != nil {
return err
}
}
// scanner.Err() is nil on clean EOF; non-nil on read errors
// This is the easy-to-forget check that catches truncated files
return scanner.Err()
}
The scanner.Err() check after the loop is critical. bufio.Scanner.Scan returns false on both clean EOF and on errors — only scanner.Err() distinguishes them. Omitting it silently ignores read failures.
For very long lines (> 64KB default buffer), set a larger buffer:
scanner := bufio.NewScanner(f)
buf := make([]byte, 0, 1<<20) // 1 MB initial capacity
scanner.Buffer(buf, 10<<20) // up to 10 MB per line
Writing Files
For writing complete content at once:
data := []byte("content to write\n")
// os.WriteFile: create or truncate, then write, then close
if err := os.WriteFile("output.txt", data, 0644); err != nil {
return fmt.Errorf("writing file: %w", err)
}
For streaming writes (building content incrementally), bufio.Writer reduces the number of syscalls by buffering writes:
func writeLines(path string, lines []string) error {
f, err := os.Create(path)
if err != nil {
return fmt.Errorf("creating %s: %w", path, err)
}
defer f.Close()
w := bufio.NewWriter(f)
for _, line := range lines {
if _, err := fmt.Fprintln(w, line); err != nil {
return err
}
}
// Flush MUST be called — buffered data isn't written until then
return w.Flush()
}
defer f.Close() alone is not sufficient when using bufio.Writer — Close doesn’t flush the buffer. Always call w.Flush() explicitly before the function returns successfully. A deferred flush is risky because it swallows the error:
// ❌ Deferred flush — error from Flush is lost
defer w.Flush()
return nil
// ✅ Explicit flush with error check
if err := w.Flush(); err != nil {
return fmt.Errorf("flushing: %w", err)
}
return nil
Appending to Files
os.OpenFile with os.O_APPEND opens a file for appending without truncating it:
func appendLine(path, line string) error {
f, err := os.OpenFile(path, os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0644)
if err != nil {
return err
}
defer f.Close()
_, err = fmt.Fprintln(f, line)
return err
}
os.O_CREATE creates the file if it doesn’t exist. os.O_WRONLY opens for writing only. os.O_APPEND makes every write go to the end of the file, which is atomic on most Unix filesystems for writes under 4KB — safe for multiple processes writing to the same log file.
Directory Operations
// Create directory and all parents
if err := os.MkdirAll("data/2026/reports", 0755); err != nil {
return fmt.Errorf("creating dirs: %w", err)
}
// List directory entries (non-recursive)
entries, err := os.ReadDir("data")
if err != nil {
return err
}
for _, e := range entries {
fmt.Printf("%-20s dir=%-5v size=%d\n",
e.Name(), e.IsDir(), func() int64 {
info, _ := e.Info()
if info != nil { return info.Size() }
return 0
}())
}
// Remove a single file
os.Remove("temp.txt")
// Remove a directory tree
os.RemoveAll("data/old")
Walking Directory Trees
filepath.WalkDir (Go 1.16+) traverses a directory tree depth-first. It’s the replacement for the older filepath.Walk — more efficient because it passes fs.DirEntry (cheaper than os.FileInfo) to the callback:
err := filepath.WalkDir(".", func(path string, d fs.DirEntry, err error) error {
if err != nil {
// Permission error or broken symlink — log and continue
fmt.Fprintf(os.Stderr, "walk error at %s: %v\n", path, err)
return nil // return err to abort the walk
}
if d.IsDir() && d.Name() == "vendor" {
return filepath.SkipDir // skip entire subtree
}
if filepath.Ext(path) == ".go" {
info, _ := d.Info()
fmt.Printf("%s (%d bytes)\n", path, info.Size())
}
return nil
})
Return filepath.SkipDir from the callback to skip a directory (or the rest of a directory if returned for a file). Return any other non-nil error to abort the entire walk and return that error from WalkDir.
Atomic Writes
A plain write followed by a rename is the standard pattern for “write a file without risking partial content”:
// AtomicWrite writes data to path atomically.
// If the write fails mid-way, the original file is untouched.
func AtomicWrite(path string, data []byte, perm os.FileMode) error {
dir := filepath.Dir(path)
// Create temp file in the same directory — ensures same filesystem for rename
tmp, err := os.CreateTemp(dir, ".tmp-*")
if err != nil {
return fmt.Errorf("creating temp file: %w", err)
}
tmpName := tmp.Name()
defer os.Remove(tmpName) // cleanup if rename fails
if _, err := tmp.Write(data); err != nil {
tmp.Close()
return fmt.Errorf("writing temp file: %w", err)
}
if err := tmp.Sync(); err != nil { // flush to disk before rename
tmp.Close()
return err
}
if err := tmp.Close(); err != nil {
return err
}
if err := os.Chmod(tmpName, perm); err != nil {
return err
}
// Rename is atomic on the same filesystem
return os.Rename(tmpName, path)
}
The key insight: on the same filesystem, os.Rename is a single atomic syscall. Readers either see the old content or the new content — never partial data.
Temporary Files and Directories
Use os.CreateTemp and os.MkdirTemp for temporary files, always cleaning up with defer:
// Temporary file
tmp, err := os.CreateTemp("", "myapp-*.json")
if err != nil {
return err
}
defer os.Remove(tmp.Name()) // always clean up
// Write and use tmp...
tmp.Close()
// Temporary directory
dir, err := os.MkdirTemp("", "myapp-work-*")
if err != nil {
return err
}
defer os.RemoveAll(dir)
// Use dir...
In tests, prefer t.TempDir() over os.MkdirTemp — the test framework cleans it up automatically even if the test panics.
Path Safety
Never concatenate paths with + or fmt.Sprintf. Use filepath.Join — it handles OS separators and cleans . and .. components:
// ❌ Breaks on Windows (wrong separator), doesn't clean ".."
path := base + "/" + userInput
// ✅ Cross-platform, cleans path components
path := filepath.Join(base, userInput)
For user-provided paths in a CLI or web handler, validate that the resolved path is still inside the expected directory (path traversal prevention):
func safePath(base, userInput string) (string, error) {
full := filepath.Join(base, filepath.Clean(userInput))
// Ensure the result is still inside base
rel, err := filepath.Rel(base, full)
if err != nil || strings.HasPrefix(rel, "..") {
return "", fmt.Errorf("path traversal detected: %q", userInput)
}
return full, nil
}
Checking File Existence and Metadata
// Check existence (and get metadata in one call)
info, err := os.Stat(path)
if errors.Is(err, os.ErrNotExist) {
fmt.Println("file does not exist")
} else if err != nil {
fmt.Println("stat error:", err)
} else {
fmt.Printf("name=%s size=%d mode=%s dir=%v\n",
info.Name(), info.Size(), info.Mode(), info.IsDir())
}
// Follow symlinks: os.Stat follows, os.Lstat does not
linfo, _ := os.Lstat(path) // info about the symlink itself
A common mistake is checking os.Stat just to decide whether to create a file. This has a TOCTOU (time-of-check/time-of-use) race: another process may create the file between your check and your create. Use os.OpenFile with os.O_EXCL to atomically create-or-fail:
f, err := os.OpenFile(path, os.O_CREATE|os.O_EXCL|os.O_WRONLY, 0644)
if errors.Is(err, os.ErrExist) {
// File already exists — handle appropriately
}
Summary
os.ReadFilefor whole-file reads;bufio.Scannerfor line-by-line withscanner.Err()after the loop- Always
defer f.Close()and always callbufio.Writer.Flush()explicitly before returning success filepath.WalkDiris preferred overfilepath.Walk— usefilepath.SkipDirto prune subtrees- Atomic writes: write to a temp file in the same directory, sync, close, then
os.Rename filepath.Joinfor all path construction — never string concatenation- Validate user-provided paths with
filepath.Relto prevent directory traversal
Comments