Introduction
Custom metrics provide deep insights into application behavior beyond standard infrastructure metrics. This article covers OpenTelemetry instrumentation patterns, metric types, and implementation best practices.
Key Statistics:
- Custom metrics: 3-5x more actionable than infrastructure metrics
- Proper instrumentation reduces MTTR by 60%
- OpenTelemetry: 500+ supported integrations
Metric Types
Before you write a single line of instrumentation, you must decide which metric type matches the signal you are trying to capture. Choosing the wrong type is a common source of misleading dashboards: for example, recording a request counter with a gauge will reset totals on every scrape, while using a counter for current memory usage will produce a permanently rising line that obscures the real value.
OpenTelemetry defines five core metric types, each suited to a specific category of measurement. Counters are monotonic and ideal for cumulative totals such as requests served or errors raised. Gauges represent point-in-time values that can move in either direction, like memory usage or active connections. Histograms sample a distribution and let you compute percentiles such as p95 or p99 without storing every individual observation. UpDownCounters behave like bidirectional counters and are the right choice for queue depth or concurrent requests. Observable metrics, finally, are populated by a callback that reads a value on demand, which suits system-level telemetry like CPU temperature or disk usage.
How to Choose the Right Metric Type
A practical way to think about the choice is to ask what question the metric will answer. If you need a rate per second, use a counter and let the query engine apply rate() or sum(rate()). If you need a percentile, use a histogram with explicit buckets. If you need a snapshot that can go up or down, use a gauge. The table-like diagram below summarizes these types and their canonical use cases; keep it handy when designing the metric surface for a new service.
┌─────────────────────────────────────────────────────────────────┐
│ OpenTelemetry Metric Types │
├─────────────────────────────────────────────────────────────────┤
│ │
│ Counter (monotonic) │
│ ├── Always increases (requests, errors) │
│ ├── Use for: counts, totals, cumulative values │
│ └── Example: total_requests, total_errors │
│ │
│ Gauge (point-in-time) │
│ ├── Can increase or decrease (memory, CPU) │
│ ├── Use for: current values, snapshots │
│ └── Example: memory_usage, active_connections │
│ │
│ Histogram (distribution) │
│ ├── Buckets for percentiles │
│ ├── Use for: latency, sizes, durations │
│ └── Example: request_duration, response_size │
│ │
│ UpDownCounter (bidirectional) │
│ ├── Can increase or decrease (counter) │
│ ├── Use for: queue depth, concurrent requests │
│ └── Example: queue_size, active_workers │
│ │
│ Observable (callback) │
│ ├── Values provided by callback function │
│ ├── Use for: system metrics, derived values │
│ └── Example: disk_usage, cpu_temp │
│ │
└─────────────────────────────────────────────────────────────────┘
These five types are the entire vocabulary you need to model almost any application signal. A common production pattern is to use counters for business events (orders, signups), histograms for latency and payload sizes, and up-down counters for in-flight work. Keep the metric surface small and consistent so that dashboards, alerts, and recording rules can all be written against a stable contract.
OpenTelemetry Python SDK
Now that the metric types are clear, we can see how they map onto the OpenTelemetry Python SDK. The SDK’s central abstraction is the meter, which is created from a global MeterProvider and is the factory for all individual metric instruments. You configure the provider once at process startup, wire it to a reader that periodically exports data, and then use metrics.get_meter() anywhere in your code to obtain instruments with the same name.
Basic Instrumentation
The example below sets up both tracing and metrics for a single service. Notice how the provider is bound to a resource that carries the service name — this single label is what allows the metrics backend to group telemetry by service and environment. The PeriodicExportingMetricReader pushes metric data to an OpenTelemetry Collector over OTLP on a fixed interval, which decouples your application from the storage backend and lets you change exporters without touching application code.
#!/usr/bin/env python3
"""OpenTelemetry Python instrumentation."""
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.sdk.resources import Resource
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry import metrics
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
from opentelemetry.exporter.otlp.proto.grpc.metric_exporter import OTLPMetricExporter
from opentelemetry.sdk.resources import Resource, SERVICE_NAME
# Setup tracing
trace.set_tracer_provider(
TracerProvider(
resource=Resource.create({SERVICE_NAME: "my-service"})
)
)
# Setup metrics
metric_reader = PeriodicExportingMetricReader(
OTLPMetricExporter(endpoint="localhost:4317", insecure=True)
)
metrics.set_meter_provider(
MeterProvider(
resource=Resource.create({SERVICE_NAME: "my-service"}),
metric_readers=[metric_reader]
)
)
# Get tracer and meter
tracer = trace.get_tracer(__name__)
meter = metrics.get_meter(__name__)
# ============== Custom Metrics ==============
# Counter: monotonically increasing
request_counter = meter.create_counter(
name="http.requests.total",
description="Total number of HTTP requests",
unit="1",
)
# Gauge: current value
active_connections = meter.create_gauge(
name="http.connections.active",
description="Number of active HTTP connections",
unit="1",
)
# Histogram: distribution
request_duration = meter.create_histogram(
name="http.request.duration",
description="HTTP request duration in seconds",
unit="s",
)
# UpDownCounter: bidirectional
queue_size = meter.create_up_down_counter(
name="queue.size",
description="Current queue size",
unit="1",
)
# Example instrumentation in HTTP handler
def handle_request(request):
# Add to counter
request_counter.add(1, {"method": request.method, "path": request.path})
# Record duration
with tracer.start_as_current_span("handle_request") as span:
span.set_attribute("http.method", request.method)
span.set_attribute("http.url", request.path)
start_time = time.time()
try:
result = process_request(request)
span.set_attribute("http.status_code", 200)
return result
except Exception as e:
span.set_attribute("http.status_code", 500)
span.record_exception(e)
raise
finally:
duration = time.time() - start_time
request_duration.record(duration, {"method": request.method})
The handler illustrates a key best practice: every metric is annotated with dimensions (labels) such as method and path, while high-cardinality values like user IDs are deliberately excluded. The histogram records duration in the finally block so it captures both successful and failed requests, and the span is used to attach the same attributes for correlation. When a request throws, the exception is recorded on the span — this lets you tie a latency outlier directly to the error that caused it in your tracing backend.
Advanced Custom Metrics
As your system grows, raw metric definitions scattered across handlers become hard to maintain. The next example organizes instruments into dedicated classes — one for business metrics, one for performance metrics, and a middleware that instruments every HTTP request automatically. This layered design keeps business telemetry (revenue, funnel steps, active users) separate from operational telemetry (latency, throughput, error rate), so each team can own the metrics relevant to it.
#!/usr/bin/env python3
"""Advanced OpenTelemetry metrics patterns."""
from opentelemetry import metrics
from opentelemetry.sdk.metrics.view import View, Aggregation
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import AggregationTemporality
from typing import Dict, List
import time
class BusinessMetrics:
"""Business-level custom metrics."""
def __init__(self, meter):
self.meter = meter
# Revenue tracking
self.revenue = meter.create_counter(
name="business.revenue.total",
description="Total revenue in USD",
unit="USD",
)
# User metrics
self.active_users = meter.create_up_down_counter(
name="business.users.active",
description="Number of active users",
unit="1",
)
# Order metrics
self.order_value = meter.create_histogram(
name="business.order.value",
description="Order value in USD",
unit="USD",
)
# Conversion funnel
self.funnel_steps = meter.create_counter(
name="business.funnel.step",
description="Funnel step completions",
unit="1",
)
def record_revenue(self, amount: float, currency: str,
product: str):
"""Record revenue event."""
self.revenue.add(
amount,
{
"currency": currency,
"product": product
}
)
def track_funnel(self, step: str, user_id: str):
"""Track funnel progression."""
self.funnel_steps.add(
1,
{
"step": step,
"user_id": user_id
}
)
class PerformanceMetrics:
"""Performance-focused custom metrics."""
def __init__(self, meter):
self.meter = meter
# Latency percentiles
self.latency = meter.create_histogram(
name="app.latency",
description="Operation latency in milliseconds",
unit="ms",
# Configure explicit bucket boundaries
# boundaries=[0, 10, 25, 50, 100, 250, 500, 1000, 2500, 5000, 10000]
)
# Throughput
self.throughput = meter.create_counter(
name="app.throughput",
description="Operations per second",
unit="ops",
)
# Error rate
self.errors = meter.create_counter(
name="app.errors.total",
description="Total number of errors",
unit="1",
)
# Size metrics
self.payload_size = meter.create_histogram(
name="app.payload.size",
description="Request/response payload size",
unit="bytes",
)
def record_latency(self, operation: str, duration_ms: float,
success: bool):
"""Record operation latency."""
self.latency.record(
duration_ms,
{
"operation": operation,
"success": str(success)
}
)
if not success:
self.errors.add(1, {"operation": operation})
class CustomMetricsMiddleware:
"""Middleware for automatic instrumentation."""
def __init__(self, app, meter):
self.app = app
self.request_counter = meter.create_counter(
name="http.server.requests.total",
description="Total HTTP requests",
unit="1",
)
self.request_duration = meter.create_histogram(
name="http.server.request.duration",
description="HTTP request duration",
unit="ms",
)
async def __call__(self, scope, receive, send):
"""Process HTTP request with instrumentation."""
start_time = time.perf_counter()
# Extract request info
method = scope.get('method', 'GET')
path = scope.get('path', '/')
# Add request
self.request_counter.add(
1,
{
"method": method,
"path": path,
"host": scope.get("headers", {}).get("host", "")
}
)
# Process request
await self.app(scope, receive, send)
# Record duration
duration_ms = (time.perf_counter() - start_time) * 1000
self.request_duration.record(
duration_ms,
{
"method": method,
"path": path
}
)
The middleware demonstrates the value of centralizing instrumentation. Rather than editing every route handler, a single ASGI middleware measures all HTTP traffic and attaches the same labels consistently. The commented bucket boundaries on the latency histogram hint at an important tuning lever: choose histogram buckets that match your realistic latency distribution, since default boundaries waste memory on empty buckets and lose precision for fast requests.
OpenTelemetry Collector
In production you rarely send telemetry straight from the application to the backend. The OpenTelemetry Collector acts as a vendor-neutral middle layer that receives OTLP data, transforms it, filters it, batches it, and forwards it to one or more exporters. This decoupling is what makes “swap the backend without touching the app” possible.
How the Pipeline Works
The Collector config below wires together receivers, processors, and exporters into named pipelines. The batch processor groups many small metric points into larger payloads, dramatically reducing egress cost and load on the backend. The metricstransform processor shows how to rename metrics and inject labels like environment during collection — transformations you may want to change frequently without rebuilding the application. Finally, the filter processor drops internal metrics early, so they never consume storage or query time.
# OpenTelemetry Collector configuration for custom metrics
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
prometheus:
config:
scrape_configs:
- job_name: 'node'
static_configs:
- targets: ['localhost:9090']
- job_name: 'custom-metrics'
static_configs:
- targets: ['localhost:8080']
processors:
batch:
timeout: 10s
send_batch_size: 1000
# Custom metrics transformations
metricstransform:
transforms:
- include: "http.requests.total"
action: update
operations:
- action: add_label
new_label: "environment"
value: "production"
- action: update_label
label: "method"
new_label: "http.method"
- include: "app.latency"
action: insert
new_name: "app.latency.histogram"
# Filter unwanted metrics
filter:
metric_views:
include:
match_type: "regexp"
metric_names:
- "http\..*"
- "app\..*"
- "business\..*"
exclude:
match_type: "regexp"
metric_names:
- "internal\..*"
exporters:
otlp:
endpoint: "https://tempo.example.com:4317"
tls:
insecure: false
prometheus:
endpoint: "0.0.0.0:8889"
namespace: "custom"
loki:
endpoint: "https://loki.example.com/loki/api/v1/push"
service:
pipelines:
metrics:
receivers: [otlp, prometheus]
processors: [batch, metricstransform, filter]
exporters: [otlp]
logs:
receivers: [otlp]
processors: [batch]
exporters: [loki]
Trade-offs of Collector-Based Collection
Routing everything through a Collector adds one moving part to your stack and requires you to operate and scale it. In exchange you gain centralized filtering, transformation, load smoothing, and the ability to run the same telemetry pipeline across every language your organization uses. For a small service, a direct exporter is simpler; for a platform with dozens of services, the Collector pays for itself quickly.
JavaScript/Node.js Instrumentation
The Python patterns translate directly to the OpenTelemetry JavaScript SDK, which is especially important because Node.js services are often the least instrumented part of a backend. The Node SDK can be configured almost entirely declaratively: a single NodeSDK object wires up resource attributes, trace and metric exporters, and even auto-instrumentation that captures HTTP, database, and framework traffic with zero manual edits.
// OpenTelemetry JavaScript instrumentation
const { NodeSDK } = require('@opentelemetry/sdk-node');
const { getNodeAutoInstrumentations } = require('@opentelemetry/auto-instrumentations-node');
const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-grpc');
const { OTLPMetricExporter } = require('@opentelemetry/exporter-metrics-otlp-grpc');
const { PeriodicExportingMetricReader } = require('@opentelemetry/sdk-metrics');
const { Resource } = require('@opentelemetry/resources');
const { ATTR_SERVICE_NAME } = require('@opentelemetry/semantic-conventions');
// Configure SDK
const sdk = new NodeSDK({
resource: new Resource({
[ATTR_SERVICE_NAME]: 'my-node-service',
'deployment.environment': 'production',
}),
traceExporter: new OTLPTraceExporter(),
metricReader: new PeriodicExportingMetricReader({
exporter: new OTLPMetricExporter(),
exportIntervalMillis: 10000,
}),
instrumentations: [
getNodeAutoInstrumentations(),
],
});
sdk.start();
// ============== Custom Metrics ==============
const { metrics } = require('@opentelemetry/api');
const meter = metrics.getMeter('my-service');
// Counter
const requestCounter = meter.createCounter('http.requests.total', {
description: 'Total HTTP requests',
});
// Gauge
const activeConnections = meter.createGauge('http.connections.active', {
description: 'Active connections',
});
// Histogram
const requestDuration = meter.createHistogram('http.request.duration', {
description: 'Request duration in ms',
unit: 'ms',
// Explicit bucket boundaries
boundaries: [10, 50, 100, 250, 500, 1000, 2500, 5000],
});
// UpDownCounter
const queueSize = meter.createUpDownCounter('queue.size', {
description: 'Queue size',
});
// Example middleware
function metricsMiddleware(req, res, next) {
const startTime = Date.now();
// Count request
requestCounter.add(1, {
method: req.method,
path: req.route?.path || req.path,
status_code: res.statusCode,
});
res.on('finish', () => {
// Record duration
const duration = Date.now() - startTime;
requestDuration.record(duration, {
method: req.method,
path: req.route?.path || req.path,
});
});
next();
}
// Custom business metrics
const businessMeter = meter.createMeter('business');
const revenueCounter = businessMeter.createCounter('revenue.total', {
description: 'Total revenue',
unit: 'USD',
});
function recordTransaction(amount, currency, product) {
revenueCounter.add(amount, {
currency,
product,
});
}
The JavaScript snippet mirrors the Python example: the same five instrument types are created from a meter, a middleware records request counts and durations, and business counters track revenue. This cross-language consistency is the core value proposition of OpenTelemetry — a metric called http.requests.total means the same thing whether it was emitted by a Python, Go, or Node.js service, so operators can reason about the whole system with one mental model. The explicit boundaries array on the histogram shows how to tune buckets for the latency profile of a web API.
Prometheus Integration
Once metrics leave your application, they land in a time-series database such as Prometheus. Prometheus is a pull-based system: it periodically scrapes an HTTP endpoint (/metrics) exposed by the application or, more commonly, by an OpenTelemetry Prometheus exporter. The scrape configuration below defines how often Prometheus collects data and how metric names are rewritten before they enter storage.
Relabeling and Recording Rules
The metric_relabel_configs section demonstrates a critical production technique: rewriting names as they are scraped. The first rule converts dotted names like http.requests into Prometheus-friendly app_http_requests and injects environment labels; the second rule drops internal metrics before they consume storage. Recording rules then precompute expensive queries such as error ratios and latency percentiles every 30 seconds, so dashboards query a cheap pre-aggregated time series instead of re-scanning raw data on every refresh.
# Prometheus scrape configuration for custom metrics
global:
scrape_interval: 15s
evaluation_interval: 15s
scrape_configs:
# Custom application metrics
- job_name: 'my-application'
metrics_path: '/metrics'
static_configs:
- targets: ['localhost:8080']
labels:
service: 'my-app'
environment: 'production'
# Metric relabeling
metric_relabel_configs:
# Add environment label
- source_labels: [__name__]
regex: 'http\.(.*)'
target_label: __name__
replacement: 'app_http_${1}'
# Drop internal metrics
- source_labels: [__name__]
regex: 'internal\..*'
action: drop
# Prometheus rules for custom metrics
- job_name: 'prometheus'
static_configs:
- targets: ['localhost:9090']
# Recording rules
rule_files:
- '/etc/prometheus/rules/*.yml'
The rules file below turns raw instrument data into the high-level metrics that on-call engineers actually alert on. http:requests:rate5m gives the request rate over a five-minute window; the error-ratio rule divides 5xx responses by total requests; and the histogram_quantile rules derive p95 and p99 latency from histogram buckets. Business rules aggregate revenue and order rates across all label dimensions, giving finance and product teams a single source of truth that matches the application’s own counters.
# Prometheus recording rules for custom metrics
groups:
- name: application.custom
interval: 30s
rules:
# HTTP request rate
- record: http:requests:rate5m
expr: rate(http_requests_total[5m])
# Error rate
- record: http:errors:rate5m
expr: |
sum(rate(http_requests_total{status=~"5.."}[5m]))
/
sum(rate(http_requests_total[5m]))
# Latency histogram quantiles
- record: http:latency:p95
expr: histogram_quantile(0.95, http_request_duration_seconds_bucket)
- record: http:latency:p99
expr: histogram_quantile(0.99, http_request_duration_seconds_bucket)
# Business metrics
- record: business:revenue:total
expr: sum(business_revenue_total)
- record: business:orders:rate1h
expr: sum(rate(business_orders_total[1h]))
With recording rules in place, building dashboards becomes declarative work. The Grafana dashboard below maps each pre-computed time series to a visual panel: a graph for request rate, a stat panel for the error percentage, a latency panel for p95, and gauges for active users and revenue. Because the hard work is done in Prometheus rules, the dashboard panels are simple one-line expressions that stay fast even across a year of retention.
{
"dashboard": {
"title": "Custom Metrics Overview",
"panels": [
{
"title": "Request Rate",
"type": "graph",
"targets": [
{
"expr": "rate(http_requests_total[5m])",
"legendFormat": "{{method}} {{path}}"
}
]
},
{
"title": "Error Rate",
"type": "stat",
"targets": [
{
"expr": "sum(rate(http_requests_total{status=~\"5..\"}[5m])) / sum(rate(http_requests_total[5m])) * 100",
"unit": "percent"
}
]
},
{
"title": "Latency p95",
"type": "graph",
"targets": [
{
"expr": "histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))",
"legendFormat": "p95"
}
]
},
{
"title": "Active Users",
"type": "gauge",
"targets": [
{
"expr": "business_users_active"
}
]
},
{
"title": "Revenue",
"type": "graph",
"targets": [
{
"expr": "increase(business_revenue_total[1h])",
"legendFormat": "{{product}}"
}
]
}
]
}
}
Best Practices
Instrumentation quality has more impact on observability than any single tool choice. Two recurring problems dominate: confusing metric names and runaway cardinality. The sections below address both, because once a naming or cardinality mistake is baked into production telemetry, it is painful to correct — historical data cannot be reliably renamed.
Metric Naming
Metric names are the public contract of your observability stack, so they deserve the same rigor as API design. The convention shown below is <domain>.<category>.<name> — for example http.request.duration — with dot separators, lowercase, and no units embedded in the name. Units belong in the unit field, not in the name: request_duration_ms mixes two concerns and breaks when you later switch to seconds. Names without a domain prefix, like requests, collide across services and make global aggregation meaningless.
┌─────────────────────────────────────────────────────────────────┐
│ Metric Naming Conventions │
├─────────────────────────────────────────────────────────────────┤
│ │
│ Structure: <domain>.<category>.<name> │
│ │
│ Examples: │
│ ├── http.request.duration (Good) │
│ ├── request_duration_ms (Bad - missing domain) │
│ ├── requests (Bad - too generic) │
│ │
│ Labels (dimensions): │
│ ├── method: GET, POST, PUT, DELETE │
│ ├── status_code: 200, 400, 500 │
│ ├── path: /api/users, /api/orders │
│ ├── environment: prod, staging, dev │
│ │
│ Units: │
│ ├── Duration: seconds (s), milliseconds (ms) │
│ ├── Bytes: bytes, kilobytes (kb), megabytes (mb) │
│ ├── Counts: 1 (no unit) │
│ └── Currency: USD, EUR │
│ │
└─────────────────────────────────────────────────────────────────┘
Cardinality Management
Cardinality refers to the number of unique label-value combinations a metric produces. Every combination consumes memory in Prometheus and creates a separate time series, so a label with millions of values — such as user_id — will exhaust memory and slow queries while providing almost no analytical value. The rule of thumb is to keep label cardinality bounded by design: a few thousand series per metric is acceptable, millions is not.
The code below contrasts the anti-patterns with their safe alternatives. Instead of a per-user label, aggregate by stable dimensions like country or plan. Instead of encoding a timestamp as a label, let the metric’s own time dimension represent time. And when you genuinely need to see per-user behavior, prefer a histogram or a log record with the user ID as an attribute, not a metric label.
#!/usr/bin/env python3
"""Avoid high cardinality in metrics."""
# BAD: High cardinality - every user becomes a label
user_counter.add(1, {"user_id": user.id}) # Millions of unique values!
# GOOD: Aggregate metrics instead
user_counter.add(1, {"country": user.country, "plan": user.plan})
# GOOD: Use histograms for distributions
request_duration.record(duration, {"endpoint": "/api/users"})
# BAD: Timestamp as label
counter.add(1, {"timestamp": "2026-02-18T10:00:00Z"})
# GOOD: Use time-based queries instead
rate(counter[5m])
By combining disciplined naming with bounded cardinality, you keep the metric surface small enough to query interactively and cheap enough to retain for the full year you will want. Enforce these conventions in code review: add a cardinality check to your CI pipeline that fails when a metric emits more than a few hundred distinct label combinations during tests.
External Resources
- OpenTelemetry Metrics Documentation
- Prometheus Metric Types
- OpenTelemetry Python SDK
- Grafana Dashboards
Related Articles
- Metrics Collection: Prometheus, InfluxDB, Telegraf
- SLO Implementation: Error Budgets
- Distributed Tracing: Jaeger, Zipkin
- Observability Cost Optimization
Comments