Continuous Profiling in Production: Pyroscope and Parca

Application profiling used to be something you did once, in development, when a feature was slow. You attached a profiler, ran a benchmark, fixed the hot path, and moved on. The problem with that workflow is that production never looks like development. Real workloads hit pathological code paths that test data does not. Dependency versions drift. Garbage collection behaves differently under real load.

Continuous profiling closes that gap. Instead of profiling on demand, an always-on agent samples the entire fleet — at a low enough rate to be safe — and stores the profiles in a queryable backend. When something slows down, the profile is already there, ready to compare against last week.

This guide covers how continuous profiling works, the two open-source contenders (Pyroscope and Parca), and the operational considerations for running it on a real workload.

How Continuous Profiling Works

A traditional CPU profiler samples the program counter at high frequency — every 1ms or 10ms — to build a statistical picture of where time is spent. Continuous profiling does the same, but with three changes:

1. Low sampling rate. Typically 100Hz (every 10ms) — frequent enough to be useful, infrequent enough to add less than 1% overhead.
2. Long-running. The agent runs for the life of the process, not a fixed window.
3. Aggregated and stored. Profiles are flushed periodically — every 10–60 seconds — to a backend that can stitch them together over time.

The result is a four-dimensional dataset: service, instance, time, and stack trace. You can ask "what was the API gateway doing at 3:47 AM yesterday?" the same way you can ask "what was its CPU usage?" — except now the answer is at the function level.

eBPF vs Language Runtimes

There are two distinct approaches to collecting profiles.

Language runtime profilers hook into the language's own profiling APIs — runtime/pprof for Go, the JVM's AsyncProfiler, Python's py-spy. They produce accurate, native stack traces with full symbol resolution. The cost is per-language integration: you need an agent for each language in your stack.

eBPF-based profilers sample stacks from the kernel without modifying the application. They work across languages — Go, Java, Python, Rust, even C++ — from a single agent. The catch is symbolization: kernel-collected stacks contain raw memory addresses, and resolving those back to function names requires debug symbols or runtime hints.

For modern Linux-based fleets, eBPF profiling has become the default. The agent runs as a DaemonSet, profiles every container on the node, and resolves symbols using DWARF data or language-specific unwinders. The operational burden of "one profiler per language" disappears.

Pyroscope vs Parca

The two major open-source continuous profiling stacks have converged in capability but differ in lineage.

Pyroscope started as a standalone profiling backend with strong language-runtime integrations. Grafana Labs acquired it in 2023 and merged it into the broader Grafana stack. The eBPF support arrived later and is now first-class. Pyroscope speaks the pprof format natively and integrates with Grafana for visualization.

Parca is Polar Signals' eBPF-first profiler. It uses CO-RE BPF for kernel portability and ships its own UI. The data model is pprof, like Pyroscope, so profiles are portable between the two.

For a team already running Grafana, Pyroscope is the obvious pick — it integrates with traces (linking a slow span to its profile) and metrics in the same UI. For teams without Grafana, Parca is a clean, focused alternative with less to operate.

A Practical Setup

A Pyroscope DaemonSet collecting from every node, with the application doing nothing special:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: pyroscope-ebpf
spec:
  selector:
    matchLabels:
      app: pyroscope-ebpf
  template:
    spec:
      hostPID: true
      containers:
        - name: agent
          image: grafana/pyroscope-ebpf:latest
          securityContext:
            privileged: true
          env:
            - name: PYROSCOPE_SERVER_ADDRESS
              value: http://pyroscope:4040
            - name: PYROSCOPE_DETECT_SUBPROCESSES
              value: "true"

Three details that matter:

• hostPID: true and privileged: true give the agent access to all processes on the node. Less privileged setups work but require careful BPF capability configuration.
• DETECT_SUBPROCESSES lets the agent follow forked children, which is important for any worker pool or queue runner.
• The agent labels profiles with pod, namespace, and service automatically. The Pyroscope UI uses those for filtering.

From this point, the data flows in continuously. Nothing to configure per service.

Reading the Data

The primary view is a flame graph aggregated over a time range. Wide bars represent functions where the program spent the most time. You can drill in by clicking a function — Pyroscope re-renders the flame graph from that frame down — or compare two time ranges side by side.

Three query patterns earn their keep:

• Compare deployment versions. Pick a 1-hour window before and after a deploy. Diff the flame graphs. New hot frames jump out — a fresh framework version, a new dependency, an accidentally turned-on debug feature.
• Compare instances. When one pod is 10x slower than its siblings, comparing its profile against a healthy one shows whether it is GC, hot code, or something stuck in a poll loop.
• Slice by request type. If your profiling agent attaches labels — through a tracing-profile integration — you can filter to "profile data for slow /checkout requests only."

Cost and Overhead

In production deployments at scale:

• CPU overhead. 0.5–1% for eBPF profilers at 100Hz. Higher rates (1000Hz) push toward 5%.
• Storage. Raw profiles compress well. A 100-service fleet typically lands at tens of gigabytes per week of retention.
• Network. Flushes happen every 10–60 seconds; bandwidth is a non-issue for normal cluster sizes.

The primary cost is the operator's attention. Continuous profiling produces a lot of data; without queries that ask specific questions, it sits unread. Pair the rollout with a runbook entry — "when an API service's latency degrades, check the Pyroscope dashboard for that service" — so the tool gets used.

When the Setup Is Not Worth It

Continuous profiling is overkill for:

• Teams already comfortable with on-demand profiling and slow-moving codebases.
• Single-language stacks where a language-specific profiler is one command away (go tool pprof).
• Workloads dominated by I/O wait rather than CPU. Profilers see CPU time well; they see blocked time poorly. Distributed tracing is the better tool for I/O-bound systems.

Where it pays off is heterogeneous, high-traffic stacks where regressions appear weekly and "reproduce locally" is no longer a valid debugging strategy. At that scale, having the profile already collected when the regression hits saves an hour every time.

Figuring out where production CPU is actually going? We help teams stand up continuous profiling and connect it to the rest of their observability data. scopeforged.com