Readers, Writers, and Buffers in Go

The io.Reader and io.Writer interfaces are the foundation of Go’s I/O system. Every function that reads from or writes to something — files, network connections, HTTP request bodies, compressed streams, cryptographic hashers — expresses its needs through these two interfaces. This composability is why you can gzip a file to an HTTP response, or hash a file while streaming it to disk, by wrapping one interface in another.

For file system operations specifically see Go file system operations. For encoding and serialization see Go encoding and decoding.

The Two Core Interfaces

type Reader interface {
    Read(p []byte) (n int, err error)
}

type Writer interface {
    Write(p []byte) (n int, err error)
}

Read fills p with bytes and returns how many it wrote and any error. It returns io.EOF when there’s no more data — not an error in the error sense, just a signal that the source is exhausted.

Write sends p and returns how many bytes were accepted. If n < len(p), the error is always non-nil — the Write contract requires that.

These two interfaces appear everywhere: os.File, net.Conn, bytes.Buffer, strings.Reader, http.Request.Body, gzip.Writer, crypto/sha256.New() — all satisfy one or both.

io.Copy: The Workhorse

io.Copy(dst, src) copies from any Reader to any Writer until src returns io.EOF. It uses an internal 32KB buffer and handles the read/write loop correctly:

// Copy a file to another file
src, _ := os.Open("input.txt")
defer src.Close()
dst, _ := os.Create("output.txt")
defer dst.Close()

n, err := io.Copy(dst, src)
fmt.Printf("Copied %d bytes\n", n)

io.Copy also copies an HTTP response body to stdout, streams a database query result to a network connection, or pipes any source to any sink — same function, any combination:

resp, _ := http.Get("https://example.com/data.json")
defer resp.Body.Close()
io.Copy(os.Stdout, resp.Body)  // stream response directly to terminal

bytes.Buffer and strings.Reader

bytes.Buffer is both a Reader and a Writer — an in-memory byte buffer useful for building content incrementally before sending:

var buf bytes.Buffer

fmt.Fprintf(&buf, "user: %s\n", name)
fmt.Fprintf(&buf, "email: %s\n", email)

// Now buf implements io.Reader — pass it to anything expecting a reader
req, _ := http.NewRequest("POST", url, &buf)
req.Header.Set("Content-Length", strconv.Itoa(buf.Len()))

strings.NewReader(s) wraps a string as a Reader without copying. Use it when you have a string but need a reader:

body := `{"action":"login","user":"alice"}`
req, _ := http.NewRequest("POST", url, strings.NewReader(body))
req.Header.Set("Content-Type", "application/json")

Buffered I/O with bufio

bufio.NewReader wraps any Reader with a buffer, enabling efficient line-by-line reading without many small Read syscalls:

f, _ := os.Open("data.csv")
defer f.Close()

scanner := bufio.NewScanner(f)
for scanner.Scan() {
    process(scanner.Text())
}
if err := scanner.Err(); err != nil {
    log.Fatal(err)
}

bufio.Scanner is the right tool for line-by-line reading — it handles \r\n and \n transparently. The default buffer is 64KB per line; for longer lines, call scanner.Buffer(make([]byte, 0, 1<<20), maxSize) before the loop.

bufio.NewWriter buffers writes — instead of one syscall per Write call, it accumulates in a buffer and flushes in larger chunks:

w := bufio.NewWriter(os.Stdout)
for _, line := range lines {
    fmt.Fprintln(w, line)  // buffered — no syscall yet
}
w.Flush()  // one syscall for all lines

Always call Flush() explicitly before the function returns — defer w.Flush() works but swallows the error. Use an explicit flush with error check for correctness.

Composing Readers and Writers

The power of the interfaces is composition. You can stack transformations:

// Hash a file while writing it to disk — zero extra memory
func copyAndHash(dst io.Writer, src io.Reader) (int64, []byte, error) {
    h := sha256.New()
    mw := io.MultiWriter(dst, h)  // writes to both dst and the hasher simultaneously
    n, err := io.Copy(mw, src)
    return n, h.Sum(nil), err
}

// Compress and write in one pass — no intermediate buffer
f, _ := os.Create("output.gz")
defer f.Close()

gzw := gzip.NewWriter(f)
defer gzw.Close()

io.Copy(gzw, input)  // input is compressed on the fly into f
gzw.Close()  // flush gzip footer before f.Close()

io.MultiWriter fans out to multiple writers simultaneously — each byte written goes to all of them. io.MultiReader chains readers in sequence:

// Prepend a header to a response body
header := strings.NewReader("HTTP/1.1 200 OK\r\n\r\n")
body   := bytes.NewReader(responseBody)
conn.Write(io.MultiReader(header, body))  // doesn't compile — needs io.Copy
combined := io.MultiReader(header, body)
io.Copy(conn, combined)

io.TeeReader: Read and Copy Simultaneously

io.TeeReader(r, w) returns a Reader that tees reads to w — every byte read from the returned reader is also written to w:

// Read an HTTP response body while logging it
var logBuf bytes.Buffer
tee := io.TeeReader(resp.Body, &logBuf)

var result MyStruct
json.NewDecoder(tee).Decode(&result)  // reads body AND copies to logBuf

if result.Status == "error" {
    log.Printf("server returned error; body: %s", logBuf.String())
}

This is invaluable for debugging — you can log the raw response body at the same time as you decode it, without buffering the whole body just to inspect it on error.

io.LimitedReader: Cap Maximum Read Size

io.LimitedReader reads at most N bytes and then returns io.EOF. Use it to enforce request size limits:

// Read at most 10MB from an upload
limited := &io.LimitedReader{R: r.Body, N: 10 << 20}
data, err := io.ReadAll(limited)
if limited.N == 0 && err == nil {
    http.Error(w, "request body too large", http.StatusRequestEntityTooLarge)
    return
}

http.MaxBytesReader is the HTTP-specific version — it also sets the response writer’s status on overflow.

io.Pipe: Goroutine Communication

io.Pipe() creates a synchronous in-memory pipe: writes on the PipeWriter block until someone reads from the PipeReader. This is useful for connecting a goroutine that produces data with one that consumes it:

pr, pw := io.Pipe()

// Producer goroutine — writes data
go func() {
    defer pw.Close()  // signals EOF to the reader when done
    json.NewEncoder(pw).Encode(data)
}()

// Consumer — sends as HTTP request body
req, _ := http.NewRequest("POST", url, pr)
resp, err := client.Do(req)

This avoids buffering the entire JSON in memory before sending — the HTTP client reads from the pipe as the encoder writes, streaming the data directly.

io.ReadAll vs Streaming

io.ReadAll(r) reads the entire reader into a byte slice. It’s simple and correct for small responses, but the wrong choice for large ones:

// ✅ Small API response — ReadAll is fine
body, err := io.ReadAll(resp.Body)
var result APIResponse
json.Unmarshal(body, &result)

// ✅ Large/streaming response — decode directly
json.NewDecoder(resp.Body).Decode(&result)

// ✅ Large file — stream with io.Copy
io.Copy(outputFile, responseBody)

The rule: if you need the raw bytes (for logging, hashing, base64 encoding), use ReadAll. If you only need the decoded value, decode directly from the reader.

Summary

• io.Reader and io.Writer are the foundation — any source or sink that implements them composes with everything else
• io.Copy is the correct way to stream from any reader to any writer — handles the read/write loop and buffer correctly
• bufio.Scanner for line-by-line reading; always check scanner.Err() after the loop
• io.MultiWriter fans out to multiple writers; io.MultiReader chains readers sequentially
• io.TeeReader reads and copies simultaneously — useful for logging raw data while decoding
• io.Pipe connects a producing goroutine to a consuming one synchronously — no intermediate buffer
• Prefer streaming (io.Copy, json.NewDecoder(r).Decode) over io.ReadAll for large payloads

Resources

• io package documentation
• bufio package documentation
• bytes package documentation
• Go Blog: Pipelines and cancellation