Observability Pipeline: OpenTelemetry vs Vector

Introduction

Modern observability requires collecting and correlating metrics, logs, and traces from diverse sources. OpenTelemetry and Vector are leading solutions for building observability pipelines, each with distinct approaches.

This guide compares these tools and helps you understand when to use each.

Understanding the Problem

Observability Data Types

Collection Challenges

• Multiple formats and protocols
• Different backends (Prometheus, Jaeger, Elasticsearch)
• High volume in production
• Complex routing requirements

OpenTelemetry

Overview

OpenTelemetry (OTel) is a vendor-neutral, open-source observability framework that provides APIs, libraries, and collection agents.

Architecture

┌─────────────────────────────────────────────────────────┐
│                  OpenTelemetry                          │
│                                                          │
│  ┌─────────────────────────────────────────────────┐   │
│  │                  Applications                     │   │
│  │  ┌─────────┐  ┌─────────┐  ┌─────────┐        │   │
│  │  │OTel SDK │  │OTel SDK │  │OTel SDK │        │   │
│  │  │ (Traces)│  │(Metrics)│  │ (Logs)  │        │   │
│  │  └────┬────┘  └────┬────┘  └────┬────┘        │   │
│  └───────┼──────────────┼──────────────┼──────────────┘   │
│          │              │              │                 │
│          └──────────────┼──────────────┘                 │
│                         ▼                               │
│  ┌─────────────────────────────────────────────────┐   │
│  │              OTel Collector                       │   │
│  │  ┌──────────┐  ┌──────────┐  ┌──────────────┐   │   │
│  │  │ Receivers│  │ Processors│  │  Exporters  │   │   │
│  │  └──────────┘  └──────────┘  └──────────────┘   │   │
│  └─────────────────────────────────────────────────┘   │
│                         │                               │
│         ┌──────────────┼──────────────┐               │
│         ▼              ▼              ▼               │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐         │
│  │Prometheus│  │ Jaeger   │  │Elastic   │         │
│  └──────────┘  └──────────┘  └──────────┘         │
└─────────────────────────────────────────────────────────┘

Installation

# Install OTel Collector
curl -sL https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v0.97.0/otelcol_0.97.0_linux_amd64.tar.gz | tar xz
sudo cp otelcol /usr/local/bin/

# Or via Docker
docker run otel/opentelemetry-collector-contrib:latest

Configuration

# otel-config.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318
  
  prometheus:
    config:
      scrape_configs:
        - job_name: 'prometheus'
          static_configs:
            - targets: ['localhost:9090']
  
  journald:
    directory: /var/log/journal

processors:
  batch:
    timeout: 10s
    send_batch_size: 1000
  
  memory_limiter:
    check_interval: 1s
    limit_mib: 4000
  
  resourcedetection:
    detectors: [env, ec2, k8s]

exporters:
  prometheus:
    endpoint: "0.0.0.0:8889"
  
  otlp:
    endpoint: "jaeger:4317"
    tls:
      insecure: true
  
  logging:
    loglevel: debug

service:
  pipelines:
    metrics:
      receivers: [prometheus]
      processors: [memory_limiter, batch]
      exporters: [prometheus]
    traces:
      receivers: [otlp]
      processors: [resourcedetection, batch]
      exporters: [otlp, logging]
    logs:
      receivers: [journald, otlp]
      processors: [resourcedetection, batch]
      exporters: [otlp, logging]

Application Integration

# Python instrumentation
pip install opentelemetry-api \
    opentelemetry-sdk \
    opentelemetry-exporter-otlp

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

# Setup tracing
provider = TracerProvider()
processor = BatchSpanProcessor(OTLPSpanExporter(endpoint="localhost:4317"))
provider.add_span_processor(processor)
trace.set_tracer_provider(provider)

# Create spans
tracer = trace.get_tracer(__name__)
with tracer.start_as_current_span("example-operation") as span:
    span.set_attribute("key", "value")

Vector

Overview

Vector is a high-performance, end-to-end observability data pipeline that collects, transforms, and routes observability data.

Architecture

┌─────────────────────────────────────────────────────────┐
│                        Vector                           │
│                                                          │
│  Sources ──► Transforms ──► Sinks                       │
│                                                          │
│  ┌──────────┐    ┌──────────┐    ┌──────────┐       │
│  │  Docker  │    │   Log    │    │ Prometheus│       │
│  │  Logs   │    │  Schema  │    │  Remote  │       │
│  └──────────┘    └──────────┘    └──────────┘       │
│                                                          │
│  ┌──────────┐    ┌──────────┐    ┌──────────┐       │
│  │   Syslog │    │   AWS    │    │  Click   │       │
│  │          │    │   Lambda │    │  House   │       │
│  └──────────┘    └──────────┘    └──────────┘       │
└─────────────────────────────────────────────────────────┘

Installation

# Install Vector
curl --proto '=https' --tlsv1.2 -sSf https://sh.vector.dev | sh

# Or via package manager
brew install vector

# Run via Docker
docker run -d \
  -v /vector.toml:/etc/vector/vector.toml:ro \
  vector:latest

Configuration

# vector.toml

[sources.docker]
type = "docker_logs"

[sources.syslog]
type = "syslog"
address = "0.0.0.0:514"

[transforms.parse_json]
type = "remap"
inputs = ["docker"]
source = '''
parsed, err = parse_json!(.message)
. = merge!(., parsed)
'''

[transforms.add_schema]
type = "lua"
inputs = ["parse_json"]
source = '''
.environment = "production"
.host = get_env_var!("HOST")
'''

[sinks.prometheus]
type = "prometheus"
inputs = ["add_schema"]
endpoint = "http://prometheus:9090"

[sinks.elasticsearch]
type = "elasticsearch"
inputs = ["add_schema"]
endpoint = "http://elasticsearch:9200"
aws = {region = "us-east-1"}

[sinks.datadog]
type = "datadog_logs"
inputs = ["add_schema"]
default_api_key = "${DATADOG_API_KEY}"

Transform Examples

# Filtering
[transforms.drop_debug]
type = "filter"
inputs = ["source"]
condition = '.level != "debug"'

# Routing
[transforms.route_logs]
type = "route"
inputs = ["source"]
route.app = 'starts_with!(.service, "app-")'
route.system = 'starts_with!(.service, "system-")'

# Enrichment
[transforms.enrich_metadata]
type = "add_fields"
inputs = ["source"]
fields.environment = "production"
fields.team = "platform"

Comparison Matrix

Use Cases

When to Use OpenTelemetry

• Need vendor-neutral instrumentation
• Collect traces and metrics
• Standardize across multiple languages
• Use with multiple backends

When to Use Vector

• High-volume log processing
• Complex routing and transformations
• Stream processing
• AWS ecosystem integration

Combined Architecture

# Hybrid setup
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  batch:
    timeout: 10s

exporters:
  vector:
    endpoint: "vector:9000"

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [vector]

Implementation Checklist

OpenTelemetry Setup

• Install OTel Collector
• Configure receivers
• Add instrumentation to apps
• Configure exporters

Vector Setup

• Install Vector
• Configure sources
• Add transforms
• Configure sinks

Integration

• Connect to backends
• Test end-to-end
• Monitor pipeline health
• Set up alerting

Summary

Both tools solve observability challenges:

1. OpenTelemetry standardizes telemetry collection across languages and provides traces and metrics natively.

2. Vector excels at high-performance log processing with flexible routing.

3. Combined approaches often use both—OTel for instrumentation, Vector for processing.

Start with OTel for standardization, add Vector for complex processing needs.

External Resources

• OpenTelemetry Documentation
• OpenTelemetry GitHub
• Vector Documentation
• Vector GitHub
• OpenTelemetry Collector