Monitoring Go Apps with Prometheus and Grafana

Prometheus + Grafana is the standard observability stack for Go applications. Prometheus scrapes metrics from your app’s /metrics endpoint; Grafana visualizes them. The Go client library makes instrumentation straightforward — this guide covers everything from basic setup to production alerting.

Setup: `prometheus/client_golang`

go get github.com/prometheus/client_golang@latest

# go.mod
require (
    github.com/prometheus/client_golang v1.19.0
)

The Four Metric Types

Counter — Things That Only Increase

Use for: requests, errors, bytes sent, jobs processed

import "github.com/prometheus/client_golang/prometheus"

var (
    httpRequestsTotal = prometheus.NewCounterVec(
        prometheus.CounterOpts{
            Name: "http_requests_total",
            Help: "Total number of HTTP requests by method, path, and status.",
        },
        []string{"method", "path", "status"},
    )

    errorsTotal = prometheus.NewCounterVec(
        prometheus.CounterOpts{
            Namespace: "myapp",
            Name:      "errors_total",
            Help:      "Total errors by operation and type.",
        },
        []string{"operation", "error_type"},
    )
)

// Usage
httpRequestsTotal.WithLabelValues("GET", "/api/users", "200").Inc()
errorsTotal.WithLabelValues("db_query", "timeout").Inc()

Gauge — Current Value (Up and Down)

Use for: active connections, goroutine count, queue size, memory usage

var (
    activeConnections = prometheus.NewGauge(prometheus.GaugeOpts{
        Name: "active_connections",
        Help: "Number of currently active connections.",
    })

    jobQueueSize = prometheus.NewGaugeVec(
        prometheus.GaugeOpts{
            Name: "job_queue_size",
            Help: "Number of jobs waiting in queue.",
        },
        []string{"queue_name"},
    )
)

// Usage
activeConnections.Inc()
activeConnections.Dec()
activeConnections.Set(float64(currentCount))
jobQueueSize.WithLabelValues("email").Set(float64(len(emailQueue)))

Histogram — Distribution of Values

Use for: request duration, response size, DB query time

var requestDuration = prometheus.NewHistogramVec(
    prometheus.HistogramOpts{
        Name:    "http_request_duration_seconds",
        Help:    "HTTP request latency distribution.",
        // Buckets optimized for web API latency
        Buckets: []float64{.005, .01, .025, .05, .1, .25, .5, 1, 2.5, 5, 10},
    },
    []string{"method", "path"},
)

// Usage — observe a value
timer := prometheus.NewTimer(requestDuration.WithLabelValues("GET", "/api/users"))
defer timer.ObserveDuration()

Summary — Quantiles (Use Histogram Instead)

Summaries calculate quantiles client-side — they can’t be aggregated across instances. Prefer histograms with histogram_quantile() in PromQL for multi-instance deployments.

Full HTTP Middleware

A production-ready middleware that instruments every request:

package middleware

import (
    "net/http"
    "strconv"
    "time"

    "github.com/prometheus/client_golang/prometheus"
    "github.com/prometheus/client_golang/prometheus/promauto"
)

// promauto registers automatically — no manual MustRegister needed
var (
    httpRequests = promauto.NewCounterVec(prometheus.CounterOpts{
        Name: "http_requests_total",
        Help: "Total HTTP requests.",
    }, []string{"method", "path", "status"})

    httpDuration = promauto.NewHistogramVec(prometheus.HistogramOpts{
        Name:    "http_request_duration_seconds",
        Help:    "HTTP request duration.",
        Buckets: prometheus.DefBuckets,
    }, []string{"method", "path"})

    httpInFlight = promauto.NewGauge(prometheus.GaugeOpts{
        Name: "http_requests_in_flight",
        Help: "Current in-flight HTTP requests.",
    })

    httpResponseSize = promauto.NewHistogramVec(prometheus.HistogramOpts{
        Name:    "http_response_size_bytes",
        Help:    "HTTP response size in bytes.",
        Buckets: prometheus.ExponentialBuckets(100, 10, 6), // 100B to 100MB
    }, []string{"method", "path"})
)

func Prometheus(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        // Normalize path to prevent cardinality explosion
        // /api/users/123 → /api/users/:id
        path := normalizePath(r.URL.Path)

        httpInFlight.Inc()
        defer httpInFlight.Dec()

        start := time.Now()
        lw := &responseWriter{ResponseWriter: w, status: 200}

        next.ServeHTTP(lw, r)

        duration := time.Since(start).Seconds()
        status := strconv.Itoa(lw.status)

        httpRequests.WithLabelValues(r.Method, path, status).Inc()
        httpDuration.WithLabelValues(r.Method, path).Observe(duration)
        httpResponseSize.WithLabelValues(r.Method, path).Observe(float64(lw.bytes))
    })
}

type responseWriter struct {
    http.ResponseWriter
    status int
    bytes  int64
}

func (rw *responseWriter) WriteHeader(code int) {
    rw.status = code
    rw.ResponseWriter.WriteHeader(code)
}

func (rw *responseWriter) Write(b []byte) (int, error) {
    n, err := rw.ResponseWriter.Write(b)
    rw.bytes += int64(n)
    return n, err
}

func normalizePath(path string) string {
    // Replace numeric IDs with :id to prevent high cardinality
    // /api/users/123/orders/456 → /api/users/:id/orders/:id
    import "regexp"
    re := regexp.MustCompile(`/\d+`)
    return re.ReplaceAllString(path, "/:id")
}

Exposing the `/metrics` Endpoint

package main

import (
    "net/http"

    "github.com/prometheus/client_golang/prometheus/promhttp"
)

func main() {
    mux := http.NewServeMux()

    // Application routes
    mux.Handle("/api/users", middleware.Prometheus(usersHandler))
    mux.Handle("/api/orders", middleware.Prometheus(ordersHandler))

    // Prometheus metrics endpoint
    mux.Handle("/metrics", promhttp.Handler())

    // Health endpoints (not instrumented to avoid noise)
    mux.HandleFunc("/health/live",  livenessHandler)
    mux.HandleFunc("/health/ready", readinessHandler)

    http.ListenAndServe(":8080", mux)
}

Custom Business Metrics

Don’t just instrument infrastructure — instrument the business:

var (
    ordersProcessed = promauto.NewCounterVec(prometheus.CounterOpts{
        Namespace: "ecommerce",
        Name:      "orders_processed_total",
        Help:      "Orders processed by status.",
    }, []string{"status", "payment_method"})

    orderValue = promauto.NewHistogram(prometheus.HistogramOpts{
        Namespace: "ecommerce",
        Name:      "order_value_dollars",
        Help:      "Distribution of order values.",
        Buckets:   []float64{5, 10, 25, 50, 100, 250, 500, 1000},
    })

    activeSubscriptions = promauto.NewGaugeVec(prometheus.GaugeOpts{
        Namespace: "ecommerce",
        Name:      "active_subscriptions",
        Help:      "Current active subscriptions by plan.",
    }, []string{"plan"})
)

// Call these in your business logic, not just middleware
func ProcessOrder(order Order) error {
    err := doProcessOrder(order)

    status := "success"
    if err != nil {
        status = "failed"
    }

    ordersProcessed.WithLabelValues(status, order.PaymentMethod).Inc()
    if err == nil {
        orderValue.Observe(order.Total)
    }
    return err
}

Prometheus Configuration

# prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s
  external_labels:
    cluster: 'production'
    region: 'us-east-1'

alerting:
  alertmanagers:
    - static_configs:
        - targets: ['alertmanager:9093']

rule_files:
  - "alerts/*.yml"

scrape_configs:
  - job_name: 'go-app'
    static_configs:
      - targets: ['app:8080']
    metrics_path: '/metrics'
    scrape_interval: 10s

  - job_name: 'go-app-k8s'
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: 'true'

Alert Rules

# alerts/go-app.yml
groups:
  - name: go-app
    rules:
      - alert: HighErrorRate
        expr: |
          rate(http_requests_total{status=~"5.."}[5m]) /
          rate(http_requests_total[5m]) > 0.01
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "High error rate: {{ $value | humanizePercentage }}"
          description: "More than 1% of requests are returning 5xx errors."

      - alert: SlowP99Latency
        expr: |
          histogram_quantile(0.99,
            rate(http_request_duration_seconds_bucket[5m])
          ) > 1.0
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "P99 latency above 1s: {{ $value | humanizeDuration }}"

      - alert: HighGoroutineCount
        expr: go_goroutines > 10000
        for: 1m
        labels:
          severity: warning
        annotations:
          summary: "High goroutine count: {{ $value }}"

      - alert: MemoryUsageHigh
        expr: |
          go_memstats_alloc_bytes /
          go_memstats_sys_bytes > 0.9
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Memory usage above 90%"

Docker Compose: Full Stack

# docker-compose.yml
version: '3.8'

services:
  app:
    build: .
    ports:
      - "8080:8080"

  prometheus:
    image: prom/prometheus:v2.49.0
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - ./alerts:/etc/prometheus/alerts
      - prometheus_data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--storage.tsdb.retention.time=15d'
      - '--web.enable-lifecycle'

  grafana:
    image: grafana/grafana:10.3.0
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin123
      - GF_USERS_ALLOW_SIGN_UP=false
    volumes:
      - grafana_data:/var/lib/grafana
      - ./grafana/dashboards:/etc/grafana/provisioning/dashboards
      - ./grafana/datasources:/etc/grafana/provisioning/datasources

  alertmanager:
    image: prom/alertmanager:v0.26.0
    ports:
      - "9093:9093"
    volumes:
      - ./alertmanager.yml:/etc/alertmanager/alertmanager.yml

volumes:
  prometheus_data:
  grafana_data:

Key PromQL Queries

# Request rate (requests per second over last 5 minutes)
rate(http_requests_total[5m])

# Error rate as a percentage
rate(http_requests_total{status=~"5.."}[5m]) /
rate(http_requests_total[5m]) * 100

# P95 and P99 latency
histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))
histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m]))

# Average latency
rate(http_request_duration_seconds_sum[5m]) /
rate(http_request_duration_seconds_count[5m])

# Goroutine count over time
go_goroutines

# Heap memory in use (MB)
go_memstats_heap_inuse_bytes / 1024 / 1024

# GC pause time (P99)
histogram_quantile(0.99, rate(go_gc_duration_seconds_bucket[5m]))

# Active connections
active_connections

Grafana Dashboard Panels

For a standard Go service dashboard, create these panels:

Request Rate — rate(http_requests_total[5m]) (time series)
Error Rate % — error rate formula above (stat panel, alert threshold at 1%)
P50/P95/P99 Latency — histogram_quantile queries (time series, multiple)
In-Flight Requests — http_requests_in_flight (gauge)
Goroutines — go_goroutines (time series)
Heap Memory — go_memstats_heap_inuse_bytes (time series)
GC Pause — GC duration (time series)
Business Metrics — orders/second, active users, etc.

High-Cardinality Trap

The biggest Prometheus mistake: using high-cardinality labels like user ID, request ID, or full URL:

// BAD: Creates millions of time series — kills Prometheus
httpRequests.WithLabelValues(r.Method, r.URL.String(), userID).Inc()
//                                       ↑                 ↑
//                                 full URL           user ID
//                             (unlimited values)  (millions of users)

// GOOD: Low-cardinality labels only
httpRequests.WithLabelValues(r.Method, normalizePath(r.URL.Path), status).Inc()
// method: 10 values, path: ~50 normalized routes, status: ~10 codes
// Total: 10 × 50 × 10 = 5,000 time series — manageable

Rule: the product of all label cardinalities should stay under ~10,000 time series per metric.

Summary

Register metrics with promauto — no manual MustRegister boilerplate
Use counters for totals, gauges for current state, histograms for latency/size distributions
Normalize URL paths before using as labels — /api/users/123 → /api/users/:id
Always expose /metrics with promhttp.Handler()
Add business metrics alongside infrastructure metrics — they tell the story of whether your app is working, not just whether it’s running
Write alert rules for error rate, latency P99, and memory — these catch 90% of production incidents

Monitoring Go Apps with Prometheus and Grafana

Setup: `prometheus/client_golang`

The Four Metric Types

Counter — Things That Only Increase

Gauge — Current Value (Up and Down)

Histogram — Distribution of Values

Summary — Quantiles (Use Histogram Instead)

Full HTTP Middleware

Exposing the `/metrics` Endpoint

Custom Business Metrics

Prometheus Configuration

Alert Rules

Docker Compose: Full Stack

Key PromQL Queries

Grafana Dashboard Panels

High-Cardinality Trap

Summary

Resources

Comments

Share this article

👍 Was this article helpful?

Setup: prometheus/client_golang

The Four Metric Types

Counter — Things That Only Increase

Gauge — Current Value (Up and Down)

Histogram — Distribution of Values

Summary — Quantiles (Use Histogram Instead)

Full HTTP Middleware

Exposing the /metrics Endpoint

Custom Business Metrics

Prometheus Configuration

Alert Rules

Docker Compose: Full Stack

Key PromQL Queries

Grafana Dashboard Panels

High-Cardinality Trap

Summary

Resources

Comments

Share this article

👍 Was this article helpful?

Setup: `prometheus/client_golang`

Exposing the `/metrics` Endpoint