Protocol Buffers (protobuf) is Google’s binary serialization format: smaller than JSON, faster to encode/decode, and schema-enforced. Every field has a number and a wire type — that’s how the binary format identifies fields even as the schema evolves. The schema is the contract, and protoc generates type-safe code for any language.
In Go, protobuf is the natural serialization choice for gRPC services and for inter-service communication where size and speed matter. For public APIs where human readability and broad tooling matter, JSON is usually better.
For gRPC specifically see Go gRPC framework.
The Schema Workflow
You never write Go structs manually for protobuf — you define the schema in .proto files and generate Go code from them:
# Install protoc compiler + Go plugin
brew install protobuf
go install google.golang.org/protobuf/cmd/protoc-gen-go@latest
# Generate Go code from .proto file
protoc --go_out=. --go_opt=paths=source_relative path/to/messages.proto
The generated .pb.go file contains the Go structs, marshal/unmarshal methods, and all the plumbing — never edit it manually.
Defining Messages
// orders.proto
syntax = "proto3";
package orders.v1;
option go_package = "github.com/example/orders/pb/v1;orderspb";
import "google/protobuf/timestamp.proto";
message Order {
string id = 1;
string customer_id = 2;
repeated OrderItem items = 3;
double total_cents = 4; // use cents, not dollars — avoid float precision
OrderStatus status = 5;
google.protobuf.Timestamp created_at = 6;
map<string, string> metadata = 7;
}
message OrderItem {
string product_id = 1;
int32 quantity = 2;
int64 unit_price_cents = 3;
}
enum OrderStatus {
ORDER_STATUS_UNSPECIFIED = 0; // Always include a 0 value — it's the default
ORDER_STATUS_PENDING = 1;
ORDER_STATUS_PAID = 2;
ORDER_STATUS_SHIPPED = 3;
ORDER_STATUS_DELIVERED = 4;
ORDER_STATUS_CANCELLED = 5;
}
The field numbers (1, 2, 3…) are what the wire format uses to identify fields — never change or reuse a field number after deployment, even if you rename or remove a field. Renaming a field is safe (it’s a Go struct field name change in the generated code); renumbering is not.
Marshal and Unmarshal
import (
"google.golang.org/protobuf/proto"
pb "github.com/example/orders/pb/v1"
"google.golang.org/protobuf/types/known/timestamppb"
)
// Create a message
order := &pb.Order{
Id: "ord-abc123",
CustomerId: "cust-456",
TotalCents: 4999,
Status: pb.OrderStatus_ORDER_STATUS_PAID,
CreatedAt: timestamppb.Now(),
Items: []*pb.OrderItem{
{ProductId: "prod-1", Quantity: 2, UnitPriceCents: 1999},
{ProductId: "prod-2", Quantity: 1, UnitPriceCents: 1001},
},
}
// Marshal to bytes
data, err := proto.Marshal(order)
if err != nil {
return fmt.Errorf("marshaling order: %w", err)
}
fmt.Printf("serialized: %d bytes\n", len(data))
// Unmarshal from bytes
received := &pb.Order{}
if err := proto.Unmarshal(data, received); err != nil {
return fmt.Errorf("unmarshaling order: %w", err)
}
fmt.Printf("order ID: %s, status: %s\n", received.Id, received.Status)
Unlike JSON, protobuf has no field names in the wire format — it’s purely field-number-to-value pairs. A 50-field struct with mostly empty fields serializes much smaller than JSON because zero values aren’t written.
Timestamps and Well-Known Types
google.protobuf.Timestamp is protobuf’s standard timestamp type. The generated Go code uses *timestamppb.Timestamp:
// Convert time.Time to protobuf Timestamp
createdAt := timestamppb.New(time.Now())
order.CreatedAt = createdAt
// Convert back to time.Time
t := order.CreatedAt.AsTime() // time.Time
// Validate (timestamps can be invalid if constructed manually)
if err := order.CreatedAt.CheckValid(); err != nil {
return fmt.Errorf("invalid timestamp: %w", err)
}
Other Well Known Types: google.protobuf.Duration for durations, google.protobuf.StringValue for nullable strings (proto3 doesn’t have null — use wrapper types), google.protobuf.Any for arbitrary message types.
oneof for Sum Types
oneof means “exactly one of these fields is set.” Use it for events with different payloads, union types, and polymorphic messages:
message PaymentEvent {
string event_id = 1;
oneof payload {
ChargeInitiated charge_initiated = 2;
ChargeSucceeded charge_succeeded = 3;
ChargeFailed charge_failed = 4;
RefundInitiated refund_initiated = 5;
}
}
In Go, oneof becomes a type switch:
func handlePaymentEvent(event *pb.PaymentEvent) error {
switch p := event.Payload.(type) {
case *pb.PaymentEvent_ChargeInitiated:
return handleChargeInitiated(p.ChargeInitiated)
case *pb.PaymentEvent_ChargeSucceeded:
return handleChargeSucceeded(p.ChargeSucceeded)
case *pb.PaymentEvent_ChargeFailed:
return handleChargeFailed(p.ChargeFailed)
case *pb.PaymentEvent_RefundInitiated:
return handleRefundInitiated(p.RefundInitiated)
default:
return fmt.Errorf("unknown payment event type: %T", event.Payload)
}
}
Schema Evolution: Field Numbering Rules
These rules ensure old readers can understand new messages and new readers can understand old messages:
Safe changes:
- Add a new field with a new field number (old readers ignore unknown fields)
- Rename a field (the number is what matters, not the name)
- Add a new enum value
- Make a field
optionaltorepeatedor vice versa (with care)
Breaking changes — never do these:
- Remove a field number — use
reservedinstead - Change a field’s type
- Reuse a field number for a different field
message User {
string id = 1;
string name = 2;
// email was field 3 — now removed but number is reserved
reserved 3;
reserved "email"; // prevents accidental reuse by name
string phone = 4; // new field — safe to add
}
JSON Interop
Protobuf-generated Go structs support JSON encoding via protojson — the output uses field names from the proto schema (including camelCase conversion), not Go field names:
import "google.golang.org/protobuf/encoding/protojson"
// Marshal to JSON
marshaler := protojson.MarshalOptions{EmitUnpopulated: true}
jsonBytes, err := marshaler.Marshal(order)
// {"id":"ord-abc123","customerId":"cust-456","status":"ORDER_STATUS_PAID",...}
// Unmarshal from JSON (accepts both camelCase and snake_case)
unmarshaler := protojson.UnmarshalOptions{DiscardUnknown: true}
err = unmarshaler.Unmarshal(jsonBytes, received)
protojson is important for APIs that serve both gRPC clients (protobuf) and HTTP/REST clients (JSON) — the grpc-gateway library uses it to automatically translate between the two.
Comparing Size: Protobuf vs JSON
order := &pb.Order{
Id: "ord-abc123", CustomerId: "cust-456",
TotalCents: 4999, Status: pb.OrderStatus_ORDER_STATUS_PAID,
}
protoBytes, _ := proto.Marshal(order)
jsonBytes, _ := json.Marshal(order)
fmt.Printf("protobuf: %d bytes\n", len(protoBytes)) // ~30 bytes
fmt.Printf("json: %d bytes\n", len(jsonBytes)) // ~100 bytes
Protobuf is typically 3–10x smaller than equivalent JSON. The advantage grows with larger messages and more numeric fields (numbers are encoded compactly in varint format).
Summary
- Never write
.pb.gofiles by hand — define schema in.proto, runprotocto generate - Field numbers are the wire contract — never reuse or remove them; use
reservedfor removed fields oneofmodels sum types (mutually exclusive fields) — Go represents them as a type switch- Use
google.protobuf.Timestampfor times;timestamppb.New(t)converts fromtime.Time - Add new fields freely (old readers skip unknown fields); removing/renumbering breaks compatibility
protojson.Marshal/Unmarshalfor JSON interop — outputs camelCase field names from the schema
Comments