Data Serialization Formats in Go: JSON, Protobuf, MessagePack, and Avro

Every service serializes data — for HTTP responses, message queues, database storage, and inter-service communication. The choice of format is a tradeoff between human readability, compact size, serialization speed, schema evolution, and ecosystem support.

This guide covers the formats you’ll encounter in Go services, with enough detail to choose correctly and implement safely.

For JSON in depth see Go working with JSON. For Protocol Buffers in depth see Go protocol buffers serialization.

Format Decision Framework

The key question: who are the clients, and do you control all of them? If third parties or browsers will consume your API, JSON is the default. If you control all clients (internal services), binary formats give significant performance advantages.

JSON: The Universal Default

JSON’s strength is universality — every language, every tool, every developer understands it. Go’s encoding/json handles the common cases:

type Order struct {
    ID         string    `json:"id"`
    CustomerID string    `json:"customer_id"`
    Amount     int64     `json:"amount_cents"`  // avoid floats for money
    Status     string    `json:"status"`
    CreatedAt  time.Time `json:"created_at"`
    Items      []Item    `json:"items"`
}

// Marshal/unmarshal
data, err := json.Marshal(order)
var decoded Order
json.Unmarshal(data, &decoded)

// Streaming for HTTP (avoids buffering entire response)
json.NewEncoder(w).Encode(orders)
json.NewDecoder(r.Body).Decode(&req)

Performance characteristics: JSON is text, so encoding/decoding is CPU-bound (Unicode handling, number formatting). For a typical 200-byte API response, JSON serialization takes ~500ns. For 10MB batch payloads, the encoding time becomes meaningful.

Schema evolution: add new fields freely (old decoders ignore unknown fields with omitempty or DisallowUnknownFields: false). Removing or renaming fields is a breaking change requiring API versioning.

Protocol Buffers: Compact Binary

Protobuf is 3–10x smaller than JSON and 5–10x faster to encode. The tradeoff is the build step (schema compilation) and binary format (not human-readable):

// orders.proto
syntax = "proto3";
message Order {
  string id = 1;
  string customer_id = 2;
  int64 amount_cents = 3;
  OrderStatus status = 4;
  google.protobuf.Timestamp created_at = 5;
  repeated Item items = 6;
}

import (
    "google.golang.org/protobuf/proto"
    pb "myapp/gen/orders/v1"
)

order := &pb.Order{Id: "ord-1", AmountCents: 4999}

data, err := proto.Marshal(order)   // ~30 bytes
decoded := &pb.Order{}
proto.Unmarshal(data, decoded)

Protobuf is the right choice for internal microservice communication, gRPC, and anywhere all clients are under your control and message volume is high.

MessagePack: Binary JSON

MessagePack is a binary encoding of JSON-like data — the same types (strings, numbers, arrays, maps), but encoded compactly without field names repeated in every record. Decoding is faster than JSON because there’s no Unicode processing:

go get github.com/vmihailenco/msgpack/v5

import "github.com/vmihailenco/msgpack/v5"

type Event struct {
    UserID    string  `msgpack:"uid"`
    EventType string  `msgpack:"type"`
    Value     float64 `msgpack:"val"`
}

event := Event{UserID: "u123", EventType: "click", Value: 1.0}

// Marshal — struct tags control field names in encoded output
data, err := msgpack.Marshal(&event)
fmt.Printf("msgpack: %d bytes\n", len(data))  // ~25 bytes vs ~50 for JSON

// Unmarshal
var decoded Event
msgpack.Unmarshal(data, &decoded)

MessagePack is popular for Redis caching (smaller values = better memory efficiency), session storage, and internal binary protocols where you want JSON flexibility without JSON verbosity.

Schema evolution: same rules as JSON — add fields freely, removing requires coordination. Unlike protobuf, there are no field numbers; field names are the key.

Avro: Schema-First with Evolution Guarantees

Avro is the standard in Apache Kafka ecosystems, especially with Confluent Schema Registry. It supports forward and backward schema compatibility more formally than JSON or MessagePack:

• Forward compatible: new schema can read old data (new optional fields allowed)
• Backward compatible: old schema can read new data (added fields must have defaults)
• Full compatible: both directions — the strictest requirement

go get github.com/linkedin/goavro/v2

import "github.com/linkedin/goavro/v2"

schemaJSON := `{
  "type": "record",
  "name": "Order",
  "fields": [
    {"name": "id",           "type": "string"},
    {"name": "customer_id",  "type": "string"},
    {"name": "amount_cents", "type": "long"},
    {"name": "status",       "type": "string", "default": "pending"}
  ]
}`

codec, err := goavro.NewCodec(schemaJSON)
if err != nil { log.Fatal(err) }

// Encode
native := map[string]any{
    "id":           "ord-1",
    "customer_id":  "cust-1",
    "amount_cents": int64(4999),
    "status":       "paid",
}
binary, err := codec.BinaryFromNative(nil, native)

// Decode
decoded, _, err := codec.NativeFromBinary(binary)
order := decoded.(map[string]any)
fmt.Println(order["id"])

In a Kafka + Schema Registry setup, the schema ID is embedded in each message. Consumers retrieve the schema from the registry, ensuring they always decode with the correct schema version.

YAML and TOML: Human-Readable Config

For configuration files (not API payloads), YAML and TOML are more readable than JSON:

// YAML
import "gopkg.in/yaml.v3"

type Config struct {
    Server struct {
        Host string `yaml:"host"`
        Port int    `yaml:"port"`
    } `yaml:"server"`
}

data, _ := os.ReadFile("config.yaml")
var cfg Config
yaml.Unmarshal(data, &cfg)

// TOML
import "github.com/BurntSushi/toml"

var cfg Config
toml.DecodeFile("config.toml", &cfg)

Use YAML or TOML for config files, not for API serialization — they’re slow and have edge cases (YAML’s Norway problem: NO is parsed as false).

Performance Comparison

Rough benchmarks for a 200-byte payload on a modern machine:

These are order-of-magnitude figures — actual performance varies by payload structure. Profile your specific workload before choosing based on performance alone.

For JSON-heavy services where protobuf isn’t an option, github.com/bytedance/sonic is a drop-in replacement for encoding/json that’s 3–4x faster through SIMD acceleration.

Summary

• JSON: always correct for public APIs and browser clients; streaming with json.NewEncoder/Decoder for large payloads
• Protocol Buffers: best binary format for controlled internal services; 3–10x smaller, 5–10x faster than JSON
• MessagePack: binary JSON — use when you want JSON-style schema-less flexibility but smaller/faster encoding (Redis caching, session storage)
• Avro: use in Kafka ecosystems with schema registry; provides formal schema evolution compatibility guarantees
• YAML/TOML: for configuration files only, not API serialization

Resources

• encoding/json
• google.golang.org/protobuf
• vmihailenco/msgpack
• linkedin/goavro
• bytedance/sonic (faster JSON)