eBPF — extended Berkeley Packet Filter — has quietly become the most important technology in Linux observability over the last five years. It lets you run small, verified programs inside the kernel to observe and react to system events: syscalls, network packets, function entry and exit, scheduling decisions. The applications for observability are everywhere, and the operational model is fundamentally different from sidecars or in-process agents.
This guide is about what eBPF buys you for observability specifically — the questions it answers that other tools cannot, the production-ready projects worth using, and the gotchas that show up when you deploy it.
What Makes eBPF Different
The classic alternatives to eBPF are user-space agents (sidecars, in-process libraries) or kernel modules. Sidecars need to be deployed alongside every application. They require cooperation from the workload — instrumentation hooks, exported metrics, or proxied traffic. Kernel modules sidestep that but are dangerous: a bug crashes the kernel.
eBPF takes a third path. Programs are written in a restricted dialect of C (or Rust), compiled to bytecode, and loaded into the kernel. A verifier checks the program for safety — no unbounded loops, no out-of-bounds memory access, no privileged operations — before allowing it to run. The result is kernel-level visibility with user-space safety.
For observability, this means:
- You see what every process on the host is doing, without instrumentation.
- You can capture network traffic, syscall patterns, and function-level performance without modifying applications.
- Programs are dynamically loadable and unloadable. No reboot. No daemon restart.
What eBPF Sees
A partial list of what eBPF programs can observe:
- Syscall entry and exit. Every
open,read,write,socket,connect. Useful for security and forensic analysis. - Function entry and exit (kprobes/uprobes). Hook the start or end of any kernel or user-space function. Get the arguments. Measure the duration.
- Network events. Packets, sockets, TCP retransmits, DNS queries. Plug points exist throughout the network stack.
- Scheduler events. When a process runs, when it gets preempted, how long it waited.
- Block I/O. Disk reads and writes, latency, queue depth.
- Tracepoints and USDT probes. Statically defined points in the kernel and major applications (Postgres, MySQL, Node.js, JVM).
The range is wide enough that most production observability tools built on eBPF could be replaced by a few well-chosen programs. In practice, you do not write the programs yourself — you use a project that packages them.
Production-Ready Projects
The eBPF ecosystem now has a handful of mature tools targeting specific observability domains.
Cilium — A CNI plugin that uses eBPF for service routing, network policy, and load balancing inside Kubernetes. Its sub-project Hubble provides L3-L7 network observability: every service call, with timing and payload metadata, without sidecars or service mesh.
Parca and Pyroscope — Continuous profilers built on eBPF. They sample stacks across all processes on a node, producing flame graphs without per-language agent installation.
Falco — Security-focused. Watches syscall patterns and flags suspicious behavior (a database process spawning a shell, unexpected outbound connections). Replaces a lot of what host-based IDS used to do.
Pixie — Auto-instrumentation for Kubernetes applications. Tracing without code changes by hooking HTTP, gRPC, MySQL, and Postgres protocol parsing in the kernel.
bpftrace — A high-level scripting language for ad-hoc eBPF queries. The dtrace of Linux. Useful for one-off investigations: "show me every process that wrote to disk in the last 30 seconds" is a one-liner.
Each of these can be deployed as a DaemonSet, runs with elevated capabilities, and produces data that flows into the same backends (Prometheus, Grafana, Loki) you already use.
A Concrete Example: Network Latency
One of the recurring questions in distributed systems is "is the network slow, or is the service slow?" Service-level metrics often cannot answer this directly. eBPF can.
With Cilium Hubble running on the cluster:
hubble observe --protocol http --label app=checkout \
--since 5m -o json | jq '.l7.latency_ns'
This returns per-request L7 latency for every HTTP call into the checkout service, observed at the kernel level. Compared with the application's own latency metric, you can see if the gap is on the wire (TCP retransmits, slow DNS) or in user-space (slow GC, lock contention).
Nothing was added to the application. The data exists because Cilium's eBPF program saw every packet.
What eBPF Cannot Do
The restrictions on eBPF programs — the same ones that make them safe — limit what they can observe.
- No symbolization of user-space stacks without help. A profiler can capture stack traces, but resolving raw addresses back to function names requires either debug symbols on disk, frame pointers, or language-specific unwinders. Without those, your flame graph shows hex addresses.
- Limited iteration. Loops must be bounded. You cannot walk an arbitrarily long linked list inside an eBPF program. Workarounds exist (tail calls, BPF iterators), but they constrain how complex a program can be.
- No long-running state without maps. Programs are typically event-handlers. State persists in eBPF maps — key-value stores — that user-space agents read.
- Kernel version matters. eBPF features land continuously. Some advanced functionality (BPF iterators, ring buffers, CO-RE) requires a recent kernel. Practical floor is 5.10; for newer features, 5.15 or 6.x.
Operational Considerations
Deploying eBPF tooling at scale brings a few specific concerns.
Privilege. eBPF programs run with CAP_BPF or CAP_SYS_ADMIN. In Kubernetes, the agent pod runs privileged and accesses host resources. Treat this as part of your blast radius — a compromised eBPF agent has node-level access.
Verifier limits. The kernel verifier rejects programs it cannot prove are safe. As programs grow, you hit instruction count limits or path-explosion errors. Most projects manage this for you, but custom programs run into it quickly.
Symbol availability. Profiling and tracing rely on symbols. Make sure debug packages or DWARF data is available on production nodes — or use frame pointers (-fno-omit-frame-pointer) when building containers.
Performance impact. Most eBPF tools are tuned for low overhead — under 1% CPU at typical loads. Heavy syscall tracing (every read on a busy network process) can push higher. Measure on a canary before fleet-wide rollout.
Adoption Path
If you have no eBPF in your stack:
- Pick a single tool that solves a specific question you currently cannot answer. Continuous profiling and L7 network observability are the most common entry points.
- Run it on one cluster, with metrics flowing into your existing backend.
- Add a runbook entry that references it — when an incident hits a category the tool helps with, the on-call uses it.
- Expand to additional eBPF tools only when the next one solves a problem the team is actively feeling.
eBPF is powerful enough that it is tempting to deploy four eBPF tools at once. Resist that. Each adds privileged code to every node. Pace the rollout so the team builds operational competence with one tool before adding the next.
Deciding which eBPF observability tools fit your stack? We help teams pick what answers their next debugging question without adding sidecar sprawl. scopeforged.com
