Encoding converts data from one representation to another — typically from an in-memory structure to a byte sequence suitable for storage or transmission, and back. Go’s standard library covers nearly every encoding format you’ll encounter: JSON, binary, base64, hex, CSV, gzip, and more. Choosing the right format is as important as using the API correctly.
For related topics see Go working with JSON, Go data serialization formats, and Go protocol buffers.
JSON: The Default for APIs
JSON is the lingua franca for HTTP APIs. Go’s encoding/json package handles the most common cases automatically via struct tags, and lets you override behavior with custom marshaler interfaces.
Basic Marshal and Unmarshal
Struct fields are encoded using their json tag. Fields without a tag use the field name as-is. The omitempty option omits zero-value fields from the output:
type User struct {
ID int `json:"id"`
Name string `json:"name"`
Email string `json:"email"`
CreatedAt time.Time `json:"created_at"`
Internal string `json:"-"` // always omitted
Bio string `json:"bio,omitempty"` // omitted if empty
}
user := User{ID: 1, Name: "Alice", Email: "[email protected]", CreatedAt: time.Now()}
data, err := json.Marshal(user)
if err != nil {
return fmt.Errorf("marshaling user: %w", err)
}
// {"id":1,"name":"Alice","email":"[email protected]","created_at":"2026-08-29T14:30:01Z"}
var decoded User
if err := json.Unmarshal(data, &decoded); err != nil {
return fmt.Errorf("unmarshaling user: %w", err)
}
For debugging or any output where humans read the JSON, json.MarshalIndent adds whitespace:
data, _ := json.MarshalIndent(user, "", " ")
fmt.Println(string(data))
Streaming JSON for Large Payloads
json.Marshal and json.Unmarshal work on complete byte slices — the entire JSON must fit in memory. For large responses or files, use json.Encoder and json.Decoder which operate on io.Writer and io.Reader directly:
// Encode directly to HTTP response writer — no intermediate buffer
func writeUsers(w http.ResponseWriter, users []User) error {
w.Header().Set("Content-Type", "application/json")
return json.NewEncoder(w).Encode(users)
}
// Decode from HTTP request body — streaming, memory-efficient
func readUser(r *http.Request) (*User, error) {
var user User
dec := json.NewDecoder(r.Body)
dec.DisallowUnknownFields() // reject unexpected fields — useful for strict APIs
if err := dec.Decode(&user); err != nil {
return nil, fmt.Errorf("decoding user: %w", err)
}
return &user, nil
}
DisallowUnknownFields() is worth enabling for request bodies — it turns typos in field names into errors rather than silently ignoring them.
Custom Marshaling for Non-Standard Types
When Go’s default JSON representation doesn’t match what your API expects — custom date formats, enums as strings, types that need special serialization — implement json.Marshaler and json.Unmarshaler:
// Custom date format: "2026-08-29" instead of Go's RFC3339 default
type Date struct {
time.Time
}
func (d Date) MarshalJSON() ([]byte, error) {
return json.Marshal(d.Format("2006-01-02"))
}
func (d *Date) UnmarshalJSON(data []byte) error {
var s string
if err := json.Unmarshal(data, &s); err != nil {
return err
}
t, err := time.Parse("2006-01-02", s)
if err != nil {
return fmt.Errorf("invalid date %q: %w", s, err)
}
d.Time = t
return nil
}
type Event struct {
Name string `json:"name"`
Date Date `json:"date"`
}
event := Event{Name: "Launch", Date: Date{time.Now()}}
data, _ := json.Marshal(event)
// {"name":"Launch","date":"2026-08-29"}
For enum-like types, the same pattern lets you serialize an integer constant as its string label — more readable in JSON than raw numbers, and more robust than parsing strings directly.
Base64: Encoding Binary for Text Contexts
Base64 encodes binary data into printable ASCII characters. Use it whenever binary needs to travel through a medium designed for text: JSON fields, HTTP headers, URLs, email. It’s not encryption — anyone can decode it.
Go provides three encodings:
data := []byte("binary\x00\xff\xfe data")
// Standard: uses + and /, adds = padding
encoded := base64.StdEncoding.EncodeToString(data)
// URL-safe: uses - and _ instead of + and / — safe in URL query strings
urlEncoded := base64.URLEncoding.EncodeToString(data)
// Raw URL-safe: no = padding — required by some protocols (JWT, for example)
rawEncoded := base64.RawURLEncoding.EncodeToString(data)
Decoding:
decoded, err := base64.StdEncoding.DecodeString(encoded)
if err != nil {
return fmt.Errorf("base64 decode: %w", err) // malformed input
}
The most common mistake is using StdEncoding in a URL and getting invalid characters. Use URLEncoding for anything that appears in a URL, and RawURLEncoding for compact tokens (like the header and payload parts of a JWT).
Hex Encoding: Human-Readable Binary
Hex encodes each byte as two hexadecimal characters. It’s twice the size of the input but completely printable and easy to inspect. Common uses: cryptographic hashes, checksums, binary protocol debugging:
import "encoding/hex"
hash := sha256.Sum256([]byte("hello"))
hexHash := hex.EncodeToString(hash[:])
fmt.Println(hexHash) // 2cf24dba5fb0a30e26e83b2ac5b9e29e1b161e5c1fa7425e73043362938b9824
decoded, err := hex.DecodeString(hexHash)
if err != nil {
// malformed hex string
}
hex.Dump produces formatted output useful for debugging binary protocols — it shows both the hex and the ASCII representation side by side.
Binary Encoding: Compact, Fast, Not Human-Readable
Binary encoding (using encoding/binary) writes integers and floats directly as bytes in a specified byte order. It’s compact (an int32 takes 4 bytes vs 8–12 bytes as a JSON number or hex string) and very fast to encode/decode.
Choose the byte order deliberately: binary.LittleEndian is standard for x86 (and most wire protocols); binary.BigEndian is the historical network byte order and used in many file formats.
import (
"bytes"
"encoding/binary"
)
type Header struct {
Magic uint32
Version uint16
Length uint32
}
func encodeHeader(h Header) ([]byte, error) {
var buf bytes.Buffer
if err := binary.Write(&buf, binary.LittleEndian, h); err != nil {
return nil, err
}
return buf.Bytes(), nil
}
func decodeHeader(data []byte) (Header, error) {
var h Header
r := bytes.NewReader(data)
if err := binary.Read(r, binary.LittleEndian, &h); err != nil {
return Header{}, fmt.Errorf("decoding header: %w", err)
}
return h, nil
}
binary.Write works with any fixed-size value (integers, floats, structs of fixed-size fields). For variable-length data like strings, write the length first, then the bytes — this is the same pattern used by length-prefixed protocols like gRPC’s framing.
For performance-critical binary serialization (game networking, financial data feeds), consider encoding/gob (Go-to-Go only) or Protocol Buffers (cross-language) over hand-rolling binary encoding with encoding/binary.
CSV: Tabular Data
encoding/csv handles the quoting, escaping, and line ending complexities of CSV that trip up naive string splitting:
import "encoding/csv"
type Record struct {
Name string
Score int
Notes string // may contain commas or quotes — csv handles it
}
func writeCSV(w io.Writer, records []Record) error {
cw := csv.NewWriter(w)
cw.Write([]string{"Name", "Score", "Notes"}) // header row
for _, r := range records {
err := cw.Write([]string{r.Name, strconv.Itoa(r.Score), r.Notes})
if err != nil {
return err
}
}
cw.Flush()
return cw.Error()
}
func readCSV(r io.Reader) ([]Record, error) {
cr := csv.NewReader(r)
cr.FieldsPerRecord = 3 // enforce column count — returns error on mismatch
rows, err := cr.ReadAll()
if err != nil {
return nil, err
}
var records []Record
for _, row := range rows[1:] { // skip header
score, _ := strconv.Atoi(row[1])
records = append(records, Record{Name: row[0], Score: score, Notes: row[2]})
}
return records, nil
}
For large CSVs (millions of rows), use cr.Read() in a loop instead of cr.ReadAll() — ReadAll loads everything into memory at once.
Compression: gzip
compress/gzip wraps any io.Writer or io.Reader to add transparent gzip compression. Compressing JSON before storage or transit typically achieves 70–90% size reduction:
func compress(data []byte) ([]byte, error) {
var buf bytes.Buffer
w := gzip.NewWriter(&buf)
if _, err := w.Write(data); err != nil {
return nil, err
}
if err := w.Close(); err != nil { // Close flushes and writes gzip footer
return nil, err
}
return buf.Bytes(), nil
}
func decompress(data []byte) ([]byte, error) {
r, err := gzip.NewReader(bytes.NewReader(data))
if err != nil {
return nil, err
}
defer r.Close()
return io.ReadAll(r)
}
For HTTP responses, use w.Header().Set("Content-Encoding", "gzip") and wrap the response writer — the browser decompresses automatically. Libraries like github.com/klauspost/compress/gzip offer better compression speed if gzip is on your hot path.
Choosing the Right Format
| Format | Use when | Size | Speed | Human-readable |
|---|---|---|---|---|
| JSON | HTTP APIs, configs, logs | Medium | Medium | ✅ |
| Protocol Buffers | High-throughput APIs, microservices | Small | Fast | ❌ |
Binary (encoding/binary) |
File formats, custom protocols | Smallest | Fastest | ❌ |
| Base64 | Binary in text contexts (JSON fields, headers) | +33% | Fast | Partial |
| Hex | Hashes, checksums, debugging | +100% | Fast | ✅ |
| CSV | Tabular data export/import | Medium | Fast | ✅ |
| gzip | Compressing any of the above | -70-90% | Medium | ❌ |
Summary
- Use
encoding/jsonwith typed struct tags for HTTP APIs; enableDisallowUnknownFields()on request bodies - Prefer
json.Encoder/json.DecoderoverMarshal/Unmarshalfor streaming or large payloads - Implement
json.Marshaler/json.Unmarshalerfor non-standard types — custom dates, enums, nested structures - Use
base64.URLEncodingfor URL-safe tokens andbase64.RawURLEncodingwhen padding must be omitted (JWT) - Use
encoding/binaryfor fixed-size binary protocols; always specify byte order explicitly compress/gzipwraps any writer/reader — JSON compressed to gzip is typically 70–90% smaller
Resources
- encoding/json documentation
- encoding/base64 documentation
- encoding/binary documentation
- Go Blog: JSON and Go
Comments