How to Profile eBPF Code: Measuring CPU Cycles, Map Latency, and Kernel-Level Overhead

The Hidden Cost of Kernel Invisibility: Why eBPF Needs Profiling

Extended Berkeley Packet Filter (eBPF) has revolutionized network monitoring, security auditing, and system observability across modern Linux infrastructures. By allowing sandboxed bytecode to execute directly inside the kernel in response to system calls, network events, or tracepoints, eBPF eliminates the costly context switches traditionally required by user-space telemetry daemons.

However, this power comes with a crucial caveat: kernel execution is not free. As production nodes run increasingly dense suites of eBPF programs—combining CNI networking plugins, runtime security agents, and custom APM probes—the cumulative overhead on host CPU cycles and memory cache can become non-trivial.

A poorly optimized eBPF helper call or an inefficient BPF-to-BPF tail call executed millions of times per second can cause subtle CPU throttling, increased packet processing latency, and degraded application performance. Profiling eBPF bytecode itself—rather than using eBPF to profile user-space application code—is an essential discipline for systems software engineers operating high-throughput environments.

Kernel Measurement Mechanics: Turning On `bpf_stats_enabled`

By default, the Linux kernel disables detailed runtime execution statistics for eBPF programs to avoid introducing unnecessary instrumentation latency to every event trigger. To begin measuring BPF execution time, you must explicitly enable the kernel's internal BPF statistics collector.

This is controlled via the sysctl kernel parameter kernel.bpf_stats_enabled. You can enable it dynamically using sysctl:

sudo sysctl -w kernel.bpf_stats_enabled=1

Once activated, the kernel's JIT compiler inserts micro-instrumentation hooks that track two fundamental metrics for every loaded BPF program: total cumulative execution runtime in nanoseconds (run_time_ns) and total invocation count (run_cnt).

To inspect these raw statistics without writing custom toolchain extensions, you can query the BPF subsystem directly using bpftool:

sudo bpftool prog show

The command produces granular output for every loaded program ID:

124: kprobe  name trace_sys_enter  tag 8f4a1c02e12a  gpl
    loaded_at 2026-03-29T10:15:00+0000  uid 0
    xlated 184B  jited 120B  memlock 4096B  map_ids 42
    run_time_ns 45210982  run_cnt 120489

By dividing run_time_ns by run_cnt, you derive the average execution time per invocation. For high-frequency kprobes attached to hot network paths (such as tcp_v4_rcv), an average runtime exceeding 300 to 500 nanoseconds indicates an urgent need for bytecode optimization.

Sampling Hardware Performance Counters with `perf`

While bpf_stats_enabled provides macro-level execution timing, it lacks detailed microarchitectural visibility. It cannot tell you whether your eBPF program is bound by L1/L3 CPU cache misses, branch mispredictions, or instruction stall cycles inside the JIT-compiled native assembly.

To capture granular CPU metrics, you can combine the native Linux perf subsystem with BPF JIT symbol resolution. First, instruct the kernel to export JITed BPF program symbols to /proc/kallsyms:

sudo sysctl -w net.core.bpf_jit_kallsyms=1

With kernel symbols exposed, perf can resolve memory addresses inside the BPF JIT memory pool to human-readable program tags. You can record hardware CPU cycles globally across all CPU cores while targeting kernel execution space (:k):

sudo perf record -a -g -e cycles:k -- sleep 10

When analyzing the captured data via perf report, modern Linux kernel versions automatically demangle eBPF symbol entry points, displaying them as bpf_prog_<tag>_<name>:

# Samples: 42K of event 'cycles:k'
# Overhead  Command      Shared Object      Symbol
# ........  ...........  .................  ...................................
    12.40%  swapper      [kernel.kallsyms]  [k] bpf_prog_8f4a1c02e12a_trace_sys_enter
     4.15%  swapper      [kernel.kallsyms]  [k] bpf_map_lookup_elem
     1.82%  swapper      [kernel.kallsyms]  [k] ht_lookup_run

If helper functions such as bpf_map_lookup_elem or hash table lookups consume a disproportionate percentage of execution cycles compared to the parent BPF program logic, your workload is heavily bottlenecked by map access patterns rather than arithmetic instruction execution.

Generating Kernel-Level Flame Graphs for BPF Bytecode

Flame graphs render call stack distributions visually, letting developers identify execution bottlenecks at a glance. Generating a flame graph specifically scoped to eBPF runtime requires capturing kernel call stacks at high sampling frequencies.

Follow this pipeline to isolate BPF performance footprints:

Capture call stack traces at 99Hz during a representative stress workload:

sudo perf record -F 99 -a -g -- sleep 30

Extract the raw stack frames using standard script tooling:

sudo perf script > out.perf

Process the raw stack trace using Brendan Gregg's open-source FlameGraph toolsuite:

./stackcollapse-perf.pl out.perf > out.folded

Filter the folded stacks to isolate eBPF-specific frames and generate the SVG graphic:

grep "bpf_prog" out.folded | ./flamegraph.pl --title "eBPF Execution Profile" > ebpf_flamegraph.svg

The resulting SVG image highlights the precise ratio of time spent executing custom BPF byte logic versus native kernel helper routines. Widely expanded blocks under bpf_ringbuf_output or bpf_map_update_elem signal immediate opportunities for architectural refactoring.

Common eBPF Performance Anti-Patterns and Fixes

When profiling reveals excessive CPU consumption or high latency spikes inside an eBPF program, the root cause usually stems from three recurring design flaws:

1. Hash Map Contention Across CPU Cores

Using standard Hash Maps (BPF_MAP_TYPE_HASH) across multi-core systems forces the kernel to acquire spinlocks and handle CPU cache coherence invalidations across NUMA nodes during write operations.

Solution: Convert high-frequency state trackers to Per-CPU Hash Maps (BPF_MAP_TYPE_PERCPU_HASH) or Per-CPU Arrays (BPF_MAP_TYPE_PERCPU_ARRAY). Per-CPU maps allocate dedicated memory regions per CPU core, enabling lock-free local writes and moving aggregation duties to user-space read operations.

2. Excessive Ring Buffer Serialization

Emitting large data structures to user space via bpf_ringbuf_output on every intercepted kernel event quickly saturates memory buses and triggers ring buffer drop counts under heavy loads.

Solution: Filter events aggressively inside kernel space before pushing to the buffer. If collecting metric counts, accumulate totals inside a Per-CPU map within the kernel and push batch summaries to user space periodically via a timer array (BPF_MAP_TYPE_TIMER).

3. Deep Tail Call Chains

While bpf_tail_call enables modular program execution by jumping between distinct BPF contexts, chaining multiple tail calls invalidates CPU branch predictors and prevents the JIT compiler from optimizing stack allocations.

Solution: Refactor fragmented tail calls into modern BPF-to-BPF functions (__always_inline or static subprograms). Modern kernels (5.6+) support native function calls within BPF bytecode, allowing the JIT compiler to emit optimized, inlined assembly instructions.

Conclusion

eBPF has elevated Linux system observability to unprecedented levels, but observability tools themselves must remain performant. By systematically turning on bpf_stats_enabled, profiling hardware events with perf, and visualizing call stacks through targeted flame graphs, you can pinpoint microsecond-level latency regressions before they impact host workloads. Keeping your eBPF programs lean guarantees that your infrastructure retains its zero-overhead promise while scaling smoothly across high-throughput environments.