OpenTelemetry End to End: Traces, Metrics, and Logs in One Pipeline

OpenTelemetry has won. After years of vendor-specific agents and three competing standards for tracing alone, OTel is now the default way to instrument applications for traces, metrics, and logs. Major clouds, every serious APM vendor, and most language ecosystems ship OTel-native libraries.

What OpenTelemetry is not, despite the marketing, is a turnkey observability stack. It is a specification, a set of SDKs, a wire protocol (OTLP), and a collector. You still pick a backend. You still configure sampling. You still propagate context across queues and schedulers yourself. This guide walks the full pipeline end to end — what each piece does and where the failure modes live.

The Three Layers

A production OpenTelemetry pipeline has three distinct layers:

1. Instrumentation — the SDK in your application that produces spans, metrics, and log records.
2. Collection — one or more OpenTelemetry Collector processes that receive, transform, and export telemetry.
3. Backend — the storage and query system: Tempo, Jaeger, Honeycomb, Datadog, etc.

Teams routinely conflate these. The instrumentation choices and the backend choices are independent. You can send the same OTLP stream to Honeycomb today and Tempo tomorrow without touching application code.

Instrumentation

OTel instrumentation falls into two buckets: automatic and manual. Automatic instrumentation hooks into common libraries — HTTP clients, database drivers, queue consumers — and produces spans without code changes. Manual instrumentation lets you create spans around the business logic that automatic instrumentation cannot see.

For a Laravel application, the OTel PHP auto-instrumentation covers Guzzle, PDO, Redis, and the framework's HTTP kernel. What it does not cover is the line your product cares about: "how long did invoice generation take?" That needs a manual span.

use OpenTelemetry\API\Globals;

$tracer = Globals::tracerProvider()->getTracer('billing');
$span = $tracer->spanBuilder('invoice.generate')->startSpan();

try {
    $span->setAttribute('invoice.client_id', $clientId);
    $invoice = $this->generateInvoice($clientId);
    $span->setAttribute('invoice.line_count', $invoice->lines->count());
} finally {
    $span->end();
}

A few rules that pay off in production:

• Use semantic conventions for attribute names. http.request.method, db.system, messaging.destination.name — these are standardized so backends know how to render them. Inventing your own keys fragments your data.
• Avoid high-cardinality attributes on metrics. User IDs are fine on spans, but a counter with one label per user produces unbounded time series.
• Set the service name once, per process, via the OTEL_SERVICE_NAME env var or the Resource constructor. Do not pass it on every span.

The Collector

The OpenTelemetry Collector is the single most underused part of the project. It is a stateless process that receives telemetry, transforms it, and exports it to one or more backends. Most teams skip it and send OTLP directly from applications to a vendor — and pay for that decision later when they need to filter, redact, or switch backends.

A minimal collector config:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 5s
  memory_limiter:
    check_interval: 1s
    limit_mib: 512
  attributes/redact:
    actions:
      - key: user.email
        action: delete
      - key: http.request.header.authorization
        action: delete

exporters:
  otlp/tempo:
    endpoint: tempo:4317
    tls:
      insecure: true
  prometheusremotewrite:
    endpoint: http://prometheus:9090/api/v1/write

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, attributes/redact, batch]
      exporters: [otlp/tempo]
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [prometheusremotewrite]

This collector accepts OTLP over gRPC and HTTP, strips PII attributes, batches before export, and fans traces out to Tempo and metrics out to Prometheus. Same applications, same instrumentation — different routing logic moved into config.

Deployment Topologies

The collector can run in three places, often combined:

• Agent (sidecar or DaemonSet). One collector per node or per pod. Adds enrichment (host attributes, K8s pod labels) and lets applications send to localhost.
• Gateway (deployment). A cluster of collectors that aggregate, sample, and export to backends. Useful for tail-based sampling and cross-cluster aggregation.
• Edge (in-app). Some SDKs expose a built-in collector. Useful for serverless where you cannot run a separate process.

The canonical production setup is agents feeding a gateway. Agents handle resource enrichment and protocol translation; the gateway centralizes export configuration and sampling decisions.

Logs in the Pipeline

Logs were the last signal to land in OpenTelemetry, and the support is now stable. Two integration paths:

• Native OTLP logs. Emit logs directly via the OTel logging SDK. Each log record carries trace context automatically, linking it to the active span.
• Log appender / bridge. Keep your existing logger (Monolog, Logback, log/slog), and configure a bridge that pipes records into the OTLP pipeline.

The bridge path is the right call for any application that already has structured logging. Rewriting log calls for OTel does not pay off — what matters is that trace context (trace_id, span_id) ends up on the log records, so you can pivot between logs and traces in the backend.

Where Pipelines Break

Three failure modes account for most production OTel incidents:

Context loss across queues. When a worker picks up a queued job, the trace context from the producer does not propagate automatically. You have to serialize the context into the job payload and restore it in the consumer. Most language SDKs ship helpers for this — use them.

Cardinality explosion in metrics. A developer adds user.id as a metric attribute, the metric grows to millions of series, and the backend either rejects writes or sends a bill. The collector's transform processor can drop high-cardinality attributes as a safety valve, but the real fix is at the SDK.

Sampling cliffs. Head-based sampling at 10% means 90% of your error traces are also gone. Tail-based sampling at the gateway — keeping traces that contain errors or slow spans — solves this but requires the gateway topology.

Backend Choice Decoupled

Because everything speaks OTLP, the backend choice is no longer a 5-year commitment. Concretely:

• Self-hosted, low-cost: Tempo (traces) + Mimir or VictoriaMetrics (metrics) + Loki (logs)
• High-cardinality friendly: Honeycomb (traces and wide events)
• Single-vendor convenience: Datadog, New Relic, Grafana Cloud

Most teams should start with one of the all-in-one vendors, then revisit once spend gets uncomfortable. The instrumentation you write today moves with you.

Adoption Sequence

If you are starting from zero, this is the order that minimizes pain:

1. Add automatic instrumentation to one service. Pick the highest-traffic one.
2. Run a collector — even a single instance — between the service and the backend.
3. Add manual spans around two or three business operations that you actually want to measure.
4. Wire trace context propagation through queues and scheduled jobs.
5. Add metrics for the same business operations (counters for success/failure, histograms for duration).
6. Bridge your existing logger into the OTLP pipeline.

Stop after step three until the observability is actively driving decisions. Instrumentation without consumption is overhead.

Rolling out OpenTelemetry across a multi-service stack? We help engineering teams design observability pipelines that produce useful signal at sustainable cost. scopeforged.com