Distributed Tracing with OpenTelemetry and Jaeger in Go

When a user request touches five microservices before returning a response, and one of them is slow, traditional logs can’t tell you which one. A trace shows the complete call graph — every service, every database query, every outbound call — with precise timing at each step.

OpenTelemetry (OTel) is the current standard for instrumentation. It replaced OpenTracing and OpenCensus in 2021. The Jaeger backend collects, stores, and visualizes traces. This guide covers instrumenting Go services with OTel and shipping to Jaeger.

For logging context correlation see Go logging. For metrics see Go monitoring with Prometheus.

Setup: OpenTelemetry SDK

go get go.opentelemetry.io/otel
go get go.opentelemetry.io/otel/sdk/trace
go get go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp
go get go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp

Initialize the tracer at startup and shut it down cleanly on exit:

package tracing

import (
    "context"
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"
    "go.opentelemetry.io/otel/sdk/resource"
    sdktrace "go.opentelemetry.io/otel/sdk/trace"
    semconv "go.opentelemetry.io/otel/semconv/v1.21.0"
)

func Init(ctx context.Context, serviceName, jaegerEndpoint string) (func(context.Context) error, error) {
    // Exporter: ships spans to Jaeger via OTLP/HTTP
    exporter, err := otlptracehttp.New(ctx,
        otlptracehttp.WithEndpoint(jaegerEndpoint),  // e.g., "localhost:4318"
        otlptracehttp.WithInsecure(),
    )
    if err != nil {
        return nil, fmt.Errorf("creating exporter: %w", err)
    }

    // Resource: identifies this service in Jaeger's UI
    res := resource.NewWithAttributes(
        semconv.SchemaURL,
        semconv.ServiceName(serviceName),
        semconv.ServiceVersion("1.0.0"),
    )

    // TracerProvider: batches and ships spans
    tp := sdktrace.NewTracerProvider(
        sdktrace.WithBatcher(exporter),
        sdktrace.WithResource(res),
        sdktrace.WithSampler(sdktrace.ParentBased(sdktrace.TraceIDRatioBased(0.1))), // 10% sampling
    )

    // Set as global — all otel.Tracer("...") calls use this
    otel.SetTracerProvider(tp)

    // Return shutdown function — call in main() defer
    return tp.Shutdown, nil
}

In main.go:

func main() {
    ctx := context.Background()
    shutdown, err := tracing.Init(ctx, "order-service", "localhost:4318")
    if err != nil {
        log.Fatalf("tracing init: %v", err)
    }
    defer shutdown(ctx)

    // ... start server ...
}

Creating Spans

Every unit of work that you want to observe gets a span. A span has a name, start/end time, attributes (key-value tags), and optional events (log points within the span):

var tracer = otel.Tracer("order-service")

func processOrder(ctx context.Context, orderID string) error {
    // Start a span — ctx now contains the span, pass it to all downstream calls
    ctx, span := tracer.Start(ctx, "processOrder",
        trace.WithAttributes(attribute.String("order.id", orderID)),
    )
    defer span.End()  // End is called even if the function panics (via defer)

    // Record an event (log point within the span)
    span.AddEvent("validating order")

    if err := validateOrder(ctx, orderID); err != nil {
        // Record the error on the span — shows as red in Jaeger
        span.RecordError(err)
        span.SetStatus(codes.Error, err.Error())
        return err
    }

    span.AddEvent("charging payment")
    if err := chargePayment(ctx, orderID); err != nil {
        span.RecordError(err)
        span.SetStatus(codes.Error, "payment failed")
        return err
    }

    span.SetAttributes(attribute.Bool("order.completed", true))
    return nil
}

The key discipline: always pass ctx down the call chain. Child spans link to the parent through the context — if you lose the context, the trace becomes fragmented.

Instrumenting HTTP Servers

otelhttp.NewHandler wraps your handler to automatically create a span for each request:

import "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"

mux := http.NewServeMux()
mux.HandleFunc("/orders", handleOrders)
mux.HandleFunc("/users",  handleUsers)

// Wrap the entire mux — every route gets automatic tracing
handler := otelhttp.NewHandler(mux, "http-server",
    otelhttp.WithMessageEvents(otelhttp.ReadEvents, otelhttp.WriteEvents),
)

http.ListenAndServe(":8080", handler)

Each request creates a span named "http-server" with HTTP method, URL, and status code as attributes. Child spans created in handlers appear nested under this root span.

Instrumenting HTTP Clients

When your service calls another service, propagate the trace context via HTTP headers. otelhttp.NewTransport does this automatically:

import "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"

// Create a client that injects trace headers on every request
tracedClient := &http.Client{
    Transport: otelhttp.NewTransport(http.DefaultTransport),
    Timeout:   10 * time.Second,
}

func callUserService(ctx context.Context, userID string) (*User, error) {
    // ctx carries the active span — otelhttp.NewTransport extracts it
    req, err := http.NewRequestWithContext(ctx, "GET",
        "http://user-service/users/"+userID, nil)
    if err != nil {
        return nil, err
    }
    resp, err := tracedClient.Do(req)
    // The request will have W3C Trace Context headers:
    //   traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
}

The traceparent header carries the trace ID and parent span ID. The receiving service reads this header and creates a child span that links to the caller’s span — enabling the full trace to be assembled in Jaeger.

Instrumenting Database Calls

Manual span creation for database operations:

func getUserFromDB(ctx context.Context, db *sql.DB, id string) (*User, error) {
    ctx, span := tracer.Start(ctx, "db.query",
        trace.WithAttributes(
            attribute.String("db.system", "postgresql"),
            attribute.String("db.operation", "SELECT"),
            attribute.String("db.statement", "SELECT id, name FROM users WHERE id = ?"),
        ),
    )
    defer span.End()

    var u User
    err := db.QueryRowContext(ctx,
        "SELECT id, name, email FROM users WHERE id = $1", id,
    ).Scan(&u.ID, &u.Name, &u.Email)

    if err != nil {
        span.RecordError(err)
        span.SetStatus(codes.Error, "query failed")
        return nil, err
    }
    return &u, nil
}

For GORM, use the otelgorm plugin: go get gorm.io/plugin/opentelemetry/tracing.

Sampling Strategies

Every span has overhead — collecting 100% of spans in a high-traffic production service is expensive. Sampling controls what percentage of traces are captured:

// Development: sample everything
sdktrace.AlwaysSample()

// Production: sample 10% of traces
sdktrace.TraceIDRatioBased(0.1)

// Production (recommended): respect parent's sampling decision, else 10%
sdktrace.ParentBased(sdktrace.TraceIDRatioBased(0.1))

ParentBased is the right production default: if an upstream service is already tracing a request, continue tracing it all the way through the system. For new root spans, sample at the configured ratio. This ensures that traces are complete — not truncated at service boundaries — for the sampled portion.

For error-first sampling (always capture failing requests), use a custom sampler:

type errorSampler struct{ base sdktrace.Sampler }

func (s errorSampler) ShouldSample(p sdktrace.SamplingParameters) sdktrace.SamplingResult {
    // Always sample if request resulted in an error
    for _, attr := range p.Attributes {
        if attr.Key == semconv.HTTPStatusCodeKey && attr.Value.AsInt64() >= 500 {
            return sdktrace.SamplingResult{Decision: sdktrace.RecordAndSample}
        }
    }
    return s.base.ShouldSample(p)
}

Running Jaeger Locally

docker run -d --name jaeger \
  -p 4318:4318 \   # OTLP HTTP
  -p 4317:4317 \   # OTLP gRPC
  -p 16686:16686 \ # Jaeger UI
  jaegertracing/all-in-one:latest

Open http://localhost:16686 to see traces. Use the service dropdown to filter by service name, then click a trace to see the full span waterfall.

Common Mistakes

Not passing context. A span created without the parent context appears as a new root trace in Jaeger, disconnected from the upstream request. Always thread ctx through every function and create spans with that context.

Ending spans with span.End() before the work completes. defer span.End() is idiomatic — the span ends when the function returns, capturing the full duration.

Over-instrumentation. Not every function needs a span — focus on I/O operations (database, HTTP, queue), expensive computations, and business logic boundaries. Too many spans create noise and overhead.

Missing error recording. A slow span with no error indication looks like it succeeded. Always call span.RecordError(err) and span.SetStatus(codes.Error, ...) when an operation fails.

Summary

Initialize the OTel TracerProvider at startup with ParentBased(TraceIDRatioBased(0.1)) for production sampling
Create spans with tracer.Start(ctx, "operation") and always defer span.End()
Use otelhttp.NewHandler for automatic server instrumentation and otelhttp.NewTransport for automatic client context propagation
Record errors with span.RecordError(err) and span.SetStatus(codes.Error, ...) — don’t just let the span end without noting failure
Thread ctx through every function call — traces fragment if the context is dropped

Distributed Tracing with OpenTelemetry and Jaeger in Go

Setup: OpenTelemetry SDK

Creating Spans

Instrumenting HTTP Servers

Instrumenting HTTP Clients

Instrumenting Database Calls

Sampling Strategies

Running Jaeger Locally

Common Mistakes

Summary

Resources

Comments

Share this article

👍 Was this article helpful?