Trace Sampling: Head, Tail, and Adaptive Approaches

Tracing every request through a busy service is technically possible. It is also a fast way to triple your observability bill, fill your backend with noise, and make queries slow. Every production tracing setup samples — the question is how, and which traces survive.

This guide covers the three approaches in use today: head-based sampling, tail-based sampling, and adaptive sampling. They are not mutually exclusive — most mature pipelines combine two of them — and each makes different tradeoffs around cost, visibility, and operational complexity.

Why Sample at All

A trace is not a single record. A request that crosses ten services with two database calls each can produce 30+ spans, and each span carries attributes — request ID, user ID, HTTP path, response code, latency, sometimes payload size. At a thousand requests per second, that is over a hundred thousand spans per second. Backends charge by ingest volume or count of spans queried. Storage costs scale linearly. Query latency degrades.

The answer is to keep a representative sample, plus everything that looks interesting (errors, slow requests, specific users), and drop the bulk.

Head-Based Sampling

Head-based sampling decides at the start of the trace whether to keep it. The first service in the chain rolls a dice — "sample 5% of requests" — and propagates the decision through the trace context. Downstream services honor it.

This is the simplest sampling model and the only one that does not require buffering traces. It is also the default in most OTel SDKs.

processors:
  probabilistic_sampler:
    sampling_percentage: 5

The core limitation is that it samples blindly. A trace marked unsampled at the front door stays unsampled even if a database call deep in the stack times out. You lose the bug reports you most want to keep.

Head-based sampling works well when:

• Traffic is uniform and most requests look the same.
• You also have metrics and logs for error visibility — the missing traces hurt less.
• You need low operational overhead.

Tail-Based Sampling

Tail-based sampling waits until the trace finishes, then decides. The full trace is buffered in a sampler — usually the OpenTelemetry Collector running in a gateway topology — and rules evaluate it against criteria like "keep all traces with errors" or "keep 1% of traces over 500ms."

processors:
  tail_sampling:
    decision_wait: 10s
    num_traces: 50000
    policies:
      - name: errors
        type: status_code
        status_code:
          status_codes: [ERROR]
      - name: slow
        type: latency
        latency:
          threshold_ms: 500
      - name: baseline
        type: probabilistic
        probabilistic:
          sampling_percentage: 2

This catches the traces you care about — errors, latency outliers — while keeping the volume tractable.

The price is operational complexity:

• Memory. The sampler buffers active traces. A gateway sees thousands of in-flight traces; each consumes RAM until the decision is made.
• Decision wait time. The sampler waits a few seconds for late spans before deciding. Set the wait too short and you lose late-arriving spans; too long and memory blows up.
• Trace assembly. All spans of a trace must reach the same sampler instance. That means routing by trace_id — most gateway deployments use a load balancer with consistent hashing on the trace ID header.

Tail-based sampling pays off when you have meaningful traffic volume and care about catching specific request shapes. It does not pay off for small services where head-based at 100% is cheaper than running the gateway.

Adaptive Sampling

Adaptive sampling adjusts the rate dynamically based on traffic. Common variants:

• Per-endpoint adaptive. A low-traffic endpoint samples at 100%; a high-traffic one samples at 1%. Total volume stays roughly constant.
• Per-tenant adaptive. Same logic, partitioned by customer or account ID.
• Quota-based. Set a target rate — say, 1000 spans per second — and let the sampler pick a rate that lands near it.

Grafana's tracing stack supports this natively; OpenTelemetry's probabilistic_sampler does not, but the transform processor combined with routing can approximate it.

Adaptive sampling solves a real problem: head-based sampling at a fixed rate either over-samples low-traffic endpoints (you keep every request to a rare admin tool) or under-samples high-traffic ones (your busiest endpoint disappears from the data). Adapting per-endpoint smooths the distribution.

The complexity it introduces is reproducibility. When the rate is dynamic, calculating "how many real requests does this represented trace stand for" requires storing the sampling rate per trace. Most backends support this via a sampling.priority attribute or similar.

Combining Approaches

The most resilient setups combine sampling strategies at different stages.

Head-based to bound cost. Tail-based to keep what matters.

The application uses head-based sampling at 10–20%. The collector gateway then applies tail-based sampling on the survivors, keeping all errors and slow traces, plus a smaller baseline rate. The net effect: traces representing 1–2% of real requests reach the backend, but errors are over-represented.

The tradeoff is that head-based sampling at the front already drops 80–90% of error traces. To recover them, you can either:

• Bump head-based to 100% and rely entirely on tail-based — cost-effective only if your gateway has enough memory.
• Use a sticky head sampler that always keeps traces matching certain criteria — for example, a request header x-debug: 1 — alongside the random sample.

What to Watch For

A few traps that bite teams when they switch from head- to tail-based sampling:

• Span context loss across async boundaries. If your queue worker creates a fresh span without inheriting parent context, the tail sampler treats it as a separate trace. Errors in the worker may sample correctly; the producer request that triggered them does not.
• Incomplete traces. If one span of a trace arrives after the decision wait, it is dropped. Backends will silently show traces with missing pieces. Set the wait conservatively — 10–30 seconds is typical.
• Bias in latency analysis. If your tail sampler keeps slow traces preferentially, simple averages over sampled data overstate latency. Backends that account for sampling weight (Honeycomb's per-event sample rate, Datadog's stats) fix this; raw queries on stored spans do not.

Picking a Starting Point

If you are setting this up for the first time:

• Low traffic, single service. Head-based at 100%. Skip the gateway.
• Medium traffic, a few services. Head-based at 10–20%, send everything to the backend. Revisit if cost grows.
• High traffic, many services, errors matter. Head-based at 100%, tail-based at the gateway with error and latency policies. Plan for sampler memory.

The right answer changes with the bill. Sampling configuration is not a one-time decision — it is a parameter that gets tuned as traffic and budget evolve.

Running out of budget on traces while still missing the ones that matter? We help teams design sampling pipelines that keep useful signal at sustainable cost. scopeforged.com