Centralized Logging in Go: ELK Stack, Structured Logs, and Production Patterns

Production logging in Go is more than fmt.Println. Effective logs are structured, carry context, ship to a centralized system, and make debugging fast when things go wrong at 3am. This guide covers the full stack: choosing a logger, structuring output, shipping to ELK, and patterns for distributed systems.

Go’s Standard Library: `log/slog` (Go 1.21+)

Since Go 1.21, the standard library includes log/slog — a structured logger that is fast, context-aware, and compatible with existing log handlers:

package main

import (
    "context"
    "log/slog"
    "os"
    "time"
)

func main() {
    // JSON handler for production
    logger := slog.New(slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{
        Level: slog.LevelInfo,
        AddSource: true, // include file:line
    }))
    slog.SetDefault(logger)

    // Basic structured logging
    slog.Info("server started", "port", 8080, "env", "production")

    // With context for request tracing
    ctx := context.WithValue(context.Background(), "request_id", "req-abc-123")
    slog.InfoContext(ctx, "request received",
        slog.String("method", "GET"),
        slog.String("path", "/api/users"),
        slog.Duration("latency", 42*time.Millisecond),
        slog.Int("status", 200),
    )

    // Error with stack context
    err := doSomething()
    if err != nil {
        slog.Error("operation failed",
            slog.String("operation", "doSomething"),
            slog.Any("error", err),
        )
    }
}

func doSomething() error { return nil }

Output:

{"time":"2026-08-28T10:00:00Z","level":"INFO","source":{"file":"main.go","line":14},"msg":"server started","port":8080,"env":"production"}

Custom Handler: Add Request ID from Context

type ContextHandler struct {
    slog.Handler
}

func (h ContextHandler) Handle(ctx context.Context, r slog.Record) error {
    if reqID, ok := ctx.Value("request_id").(string); ok {
        r.AddAttrs(slog.String("request_id", reqID))
    }
    if userID, ok := ctx.Value("user_id").(string); ok {
        r.AddAttrs(slog.String("user_id", userID))
    }
    return h.Handler.Handle(ctx, r)
}

func NewLogger() *slog.Logger {
    base := slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{Level: slog.LevelInfo})
    return slog.New(ContextHandler{base})
}

Zap: High-Performance Logging

For high-throughput services (>100k req/s), Uber’s zap is zero-allocation in the hot path:

import (
    "go.uber.org/zap"
    "go.uber.org/zap/zapcore"
    "os"
    "time"
)

func NewZapLogger(env string) (*zap.Logger, error) {
    var cfg zap.Config
    if env == "production" {
        cfg = zap.NewProductionConfig()
        cfg.EncoderConfig.TimeKey = "timestamp"
        cfg.EncoderConfig.EncodeTime = zapcore.ISO8601TimeEncoder
    } else {
        cfg = zap.NewDevelopmentConfig()
        cfg.EncoderConfig.EncodeLevel = zapcore.CapitalColorLevelEncoder
    }
    return cfg.Build()
}

func main() {
    logger, _ := NewZapLogger("production")
    defer logger.Sync()

    // Strongly-typed fields — zero allocation
    logger.Info("user login",
        zap.String("user_id", "u-123"),
        zap.String("ip", "10.0.0.1"),
        zap.Duration("latency", 12*time.Millisecond),
    )

    // Sugared logger for less critical paths
    sugar := logger.Sugar()
    sugar.Infow("request complete",
        "method", "POST",
        "path", "/api/orders",
        "status", 201,
    )

    // Logger with permanent fields (child logger)
    requestLogger := logger.With(
        zap.String("service", "order-service"),
        zap.String("version", "v2.1"),
    )
    requestLogger.Info("processing order", zap.String("order_id", "ord-456"))
}

HTTP Middleware: Log Every Request

package middleware

import (
    "log/slog"
    "net/http"
    "time"

    "github.com/google/uuid"
)

func RequestLogger(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        start := time.Now()

        // Assign request ID
        reqID := r.Header.Get("X-Request-ID")
        if reqID == "" {
            reqID = uuid.New().String()
        }
        w.Header().Set("X-Request-ID", reqID)

        // Wrap ResponseWriter to capture status code
        lrw := &loggingResponseWriter{ResponseWriter: w, status: 200}

        // Inject request ID into context
        ctx := context.WithValue(r.Context(), "request_id", reqID)

        next.ServeHTTP(lrw, r.WithContext(ctx))

        slog.InfoContext(ctx, "http request",
            slog.String("method", r.Method),
            slog.String("path", r.URL.Path),
            slog.String("remote_addr", r.RemoteAddr),
            slog.Int("status", lrw.status),
            slog.Duration("duration", time.Since(start)),
            slog.Int64("bytes", lrw.bytes),
        )
    })
}

type loggingResponseWriter struct {
    http.ResponseWriter
    status int
    bytes  int64
}

func (lrw *loggingResponseWriter) WriteHeader(code int) {
    lrw.status = code
    lrw.ResponseWriter.WriteHeader(code)
}

func (lrw *loggingResponseWriter) Write(b []byte) (int, error) {
    n, err := lrw.ResponseWriter.Write(b)
    lrw.bytes += int64(n)
    return n, err
}

Log Levels — When to Use Each

func demonstrateLogLevels(logger *slog.Logger) {
    // DEBUG: Verbose detail, off in production
    // Use for: variable values, loop iterations, SQL queries
    logger.Debug("cache miss", slog.String("key", "user:123"))

    // INFO: Normal operations, business events
    // Use for: request received, user logged in, job started
    logger.Info("order placed",
        slog.String("order_id", "ord-789"),
        slog.Float64("amount", 49.99),
    )

    // WARN: Unexpected but recoverable — investigate later
    // Use for: retry attempts, deprecated API usage, slow queries
    logger.Warn("slow database query",
        slog.String("query", "SELECT * FROM orders"),
        slog.Duration("duration", 2500*time.Millisecond),
        slog.Int("threshold_ms", 1000),
    )

    // ERROR: Operation failed, needs attention
    // Use for: failed requests, unexpected errors, data corruption
    logger.Error("payment failed",
        slog.String("order_id", "ord-789"),
        slog.String("error", "card declined"),
        slog.String("provider", "stripe"),
    )

    // Never log passwords, tokens, or PII in any level
}

Shipping Logs to ELK Stack

Docker Compose Setup

# docker-compose.yml
version: '3.8'

services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.11.0
    environment:
      - discovery.type=single-node
      - xpack.security.enabled=false
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ports:
      - "9200:9200"
    volumes:
      - es_data:/usr/share/elasticsearch/data

  logstash:
    image: docker.elastic.co/logstash/logstash:8.11.0
    volumes:
      - ./logstash/pipeline:/usr/share/logstash/pipeline
    ports:
      - "5044:5044"  # Beats input
      - "5000:5000/tcp"  # TCP JSON input
    depends_on:
      - elasticsearch

  kibana:
    image: docker.elastic.co/kibana/kibana:8.11.0
    ports:
      - "5601:5601"
    environment:
      - ELASTICSEARCH_HOSTS=http://elasticsearch:9200
    depends_on:
      - elasticsearch

  filebeat:
    image: docker.elastic.co/beats/filebeat:8.11.0
    user: root
    volumes:
      - ./filebeat.yml:/usr/share/filebeat/filebeat.yml:ro
      - /var/lib/docker/containers:/var/lib/docker/containers:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
    depends_on:
      - logstash

volumes:
  es_data:

Logstash Pipeline

# logstash/pipeline/go-app.conf
input {
  tcp {
    port => 5000
    codec => json_lines
  }
  beats {
    port => 5044
  }
}

filter {
  if [level] {
    mutate {
      lowercase => ["level"]
    }
  }

  # Parse Go duration strings
  if [duration] {
    ruby {
      code => 'event.set("duration_ms", event.get("duration").gsub(/[^0-9.]/, "").to_f)'
    }
  }

  # Add geo location from IP
  if [remote_addr] {
    geoip {
      source => "remote_addr"
      target => "geoip"
    }
  }

  # Tag slow requests
  if [duration_ms] and [duration_ms] > 1000 {
    mutate {
      add_tag => ["slow_request"]
    }
  }
}

output {
  elasticsearch {
    hosts => ["elasticsearch:9200"]
    index => "go-app-%{+YYYY.MM.dd}"
    document_type => "_doc"
  }
  # Also output to stdout for debugging
  stdout { codec => rubydebug }
}

Filebeat Config (Docker Log Shipping)

# filebeat.yml
filebeat.inputs:
- type: container
  paths:
    - '/var/lib/docker/containers/*/*.log'
  processors:
    - add_docker_metadata:
        host: "unix:///var/run/docker.sock"
    - decode_json_fields:
        fields: ["message"]
        target: ""
        overwrite_keys: true

output.logstash:
  hosts: ["logstash:5044"]

logging.level: warning

Sending Directly to Logstash from Go

import (
    "encoding/json"
    "net"
    "time"
)

type LogEntry struct {
    Timestamp string                 `json:"@timestamp"`
    Level     string                 `json:"level"`
    Message   string                 `json:"message"`
    Service   string                 `json:"service"`
    Fields    map[string]interface{} `json:"fields,omitempty"`
}

type LogstashWriter struct {
    conn    net.Conn
    service string
}

func NewLogstashWriter(addr, service string) (*LogstashWriter, error) {
    conn, err := net.Dial("tcp", addr)
    if err != nil {
        return nil, err
    }
    return &LogstashWriter{conn: conn, service: service}, nil
}

func (w *LogstashWriter) Write(p []byte) (n int, err error) {
    return w.conn.Write(append(p, '\n'))
}

Kibana Queries for Common Scenarios

Once logs are in Elasticsearch, use KQL (Kibana Query Language) to find problems:

# All errors in last hour
level: "error" AND @timestamp >= now-1h

# Slow requests (>1s)
duration_ms > 1000 AND level: "info"

# Specific user's activity
user_id: "u-123" AND @timestamp >= now-24h

# Payment failures
message: "payment failed" AND fields.provider: "stripe"

# 5xx errors
status >= 500 AND status < 600

# Service-specific errors
service: "order-service" AND level: "error"

Production Patterns

Never Log Sensitive Data

type User struct {
    ID       string
    Email    string
    Password string // NEVER log this
    Token    string // NEVER log this
}

// BAD: logs password
slog.Info("user created", slog.Any("user", user))

// GOOD: log only safe fields
slog.Info("user created",
    slog.String("user_id", user.ID),
    slog.String("email", maskEmail(user.Email)),
)

func maskEmail(email string) string {
    parts := strings.Split(email, "@")
    if len(parts) != 2 { return "***" }
    name := parts[0]
    if len(name) > 2 {
        name = name[:2] + strings.Repeat("*", len(name)-2)
    }
    return name + "@" + parts[1]
}

Correlation IDs Across Services

// Propagate request ID in HTTP calls between services
func callDownstream(ctx context.Context, url string) (*http.Response, error) {
    req, _ := http.NewRequestWithContext(ctx, "GET", url, nil)

    if reqID, ok := ctx.Value("request_id").(string); ok {
        req.Header.Set("X-Request-ID", reqID)
    }

    return http.DefaultClient.Do(req)
}

// Extract from incoming request
func extractRequestID(r *http.Request) string {
    if id := r.Header.Get("X-Request-ID"); id != "" {
        return id
    }
    return uuid.New().String()
}

Log Sampling for High-Volume Routes

import "math/rand"

// Log only 1% of successful health checks to avoid noise
func healthCheckLogger(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        next.ServeHTTP(w, r)
        if r.URL.Path == "/health" && rand.Float32() > 0.01 {
            return // skip 99% of health check logs
        }
        slog.Info("health check", slog.String("path", r.URL.Path))
    })
}

Choosing a Logger

	`log/slog`	`zap`	`logrus`
Go version	1.21+	Any	Any
Allocations	Low	Zero (core)	Medium
Throughput	High	Highest	Medium
API style	Simple	Typed / Sugared	Logrus fields
Ecosystem	Growing	Large	Large
Best for	New projects, stdlib preference	Ultra high-throughput	Existing codebases

For new Go 1.21+ projects, start with slog. For services doing millions of logs/second, use zap.

Summary

Use structured logging (JSON) so logs are queryable in ELK/Splunk/CloudWatch
Add request_id to every log line via context — makes tracing a request across services trivial
Log at appropriate levels: DEBUG off in production, INFO for business events, WARN/ERROR for problems
Never log passwords, tokens, or PII — mask or omit sensitive fields
Ship with Filebeat (Docker logs) or directly to Logstash for centralized aggregation
Use slog for new projects (stdlib, Go 1.21+), zap for ultra-high-throughput services

Centralized Logging in Go: ELK Stack, Structured Logs, and Production Patterns

Go’s Standard Library: `log/slog` (Go 1.21+)

Custom Handler: Add Request ID from Context

Zap: High-Performance Logging

HTTP Middleware: Log Every Request

Log Levels — When to Use Each

Shipping Logs to ELK Stack

Docker Compose Setup

Logstash Pipeline

Filebeat Config (Docker Log Shipping)

Sending Directly to Logstash from Go

Kibana Queries for Common Scenarios

Production Patterns

Never Log Sensitive Data

Correlation IDs Across Services

Log Sampling for High-Volume Routes

Choosing a Logger

Summary

Resources

Comments

Share this article

👍 Was this article helpful?

Go’s Standard Library: log/slog (Go 1.21+)

Custom Handler: Add Request ID from Context

Zap: High-Performance Logging

HTTP Middleware: Log Every Request

Log Levels — When to Use Each

Shipping Logs to ELK Stack

Docker Compose Setup

Logstash Pipeline

Filebeat Config (Docker Log Shipping)

Sending Directly to Logstash from Go

Kibana Queries for Common Scenarios

Production Patterns

Never Log Sensitive Data

Correlation IDs Across Services

Log Sampling for High-Volume Routes

Choosing a Logger

Summary

Resources

Comments

Share this article

👍 Was this article helpful?

Go’s Standard Library: `log/slog` (Go 1.21+)