---
title: 'ProfInfer: eBPF LLM Inference Profiler'
url: https://www.emergentmind.com/topics/profinfer
type: topic
---

# ProfInfer: eBPF LLM Inference Profiler

ProfInfer is an eBPF-based fine-grained LLM inference profiler for modern inference engines such as llama.cpp, designed to provide operator-level, real-time visibility without modifying or recompiling the runtime source code [2601.20755]. It attaches probes dynamically across multiple layers of the inference stack, correlates operator semantics with timestamps, scheduler behavior, and hardware counters, and renders the resulting traces as operator DAGs, timelines, and statistical trend plots. The system is positioned as a response to a diagnostic gap in lightweight LLM runtimes, which typically expose token-level metrics such as throughput and TTFT but not per-operator, per-graph, or per-backend behavior [2601.20755].

## 1. Definition, scope, and diagnostic objective

ProfInfer targets modern LLM inference runtimes in which critical execution phenomena remain opaque at inference time: whether decoding is memory-bound or compute-bound, which operators dominate latency, how Mixture-of-Experts routing affects execution, and whether operator offloading is beneficial for particular tensor shapes or backend configurations [2601.20755]. The framework is explicitly described as **fine-grained** and **non-intrusive**. Its central design goal is to restore operator-level visibility to runtimes like llama.cpp while preserving low perturbation, with measured runtime overhead under 4% [2601.20755].

The profiling problem addressed by ProfInfer differs from the observability offered by several adjacent tool classes. General-purpose systems such as ONNX Runtime and TensorRT can expose some operator latencies and allocation information, but their visibility is described as coarse and intrusive, and they do not correlate low-level hardware metrics with LLM phases such as prefill and decode [2601.20755]. CPU and GPU profilers such as `perf`, `oprofile`, and Nsight provide low-level samples or vendor-specific traces, but not operator-correlated DAGs and timelines tailored to lightweight LLM runtimes [2601.20755]. PyTorch Profiler and serving traces are likewise oriented toward training or serving stacks rather than compact on-device libraries [2601.20755].

A plausible implication is that ProfInfer should be understood not merely as a timing tool, but as a systems-analysis layer specialized for the operational structure of LLM inference. The paper characterizes its outputs as useful for optimization, scheduling, and resource-aware deployment, indicating that it is meant to inform concrete runtime decisions rather than simply collect measurements [2601.20755].

## 2. Architecture and eBPF-based instrumentation

ProfInfer is organized as an end-to-end tracing pipeline spanning userspace control, kernelspace probe execution, userspace aggregation, and visualization [2601.20755]. A userspace controller compiles and loads eBPF programs through BCC on Ubuntu or libbpf on OpenHarmony, configures control flags such as tensor-dimension parsing and PMC collection, attaches uprobes and uretprobes to target runtime functions, enables kernel tracepoints, and opens `perf_event` file descriptors per thread for hardware-counter collection [2601.20755].

In kernelspace, the eBPF handlers record nanosecond timestamps, thread identifiers, CPU identifiers, and, when configured, parse arguments from userspace data structures such as `ggml_tensor` via `bpf_probe_read_user/kernel` to extract operator type, tensor pointers, names, and dimensions [2601.20755]. Counter values are read through BPF maps and `perf_read`, and events are emitted through perf buffers or ring buffers depending on the desired reliability-overhead trade-off [2601.20755]. Userspace then polls these buffers, filters events by probe type, merges events across threads, resolves symbols to operator instances, and computes per-operator durations and per-graph breakdowns [2601.20755].

The instrumentation strategy covers several runtime layers. At the token level, ProfInfer attaches probes to `llama_decode`, using call-return deltas to measure TTFT and TPOT [2601.20755]. At the graph level, it instruments `ggml_backend_graph_compute_async` to time each graph execution and extract backend identifiers so that CPU, OpenCL, and NPU graph segments can be distinguished [2601.20755]. At the operator level, it probes backend-dispatch functions such as `ggml_compute_forward`, backend-specific OpenCL forward functions, and an analogous Rockchip NPU integration point [2601.20755]. For MoE analysis it additionally probes `ggml_compute_forward_mul_mat_id` and extracts expert IDs through a two-level pointer dereference on the third source tensor [2601.20755].

Portability is a stated design feature. ProfInfer supports both BCC and libbpf, and where CO-RE and BTF are available it can be resilient across minor kernel-version changes without rebuilds [2601.20755]. Deployment nevertheless depends on standard eBPF and perf prerequisites: Linux with `CONFIG_BPF`, an enabled `perf_event` subsystem, and capabilities such as `CAP_BPF` or `CAP_SYS_ADMIN` [2601.20755]. In containers, privileged execution with BPF/perf capabilities is possible, but host-mode execution is recommended [2601.20755].

## 3. Data model, derived metrics, and visual analytics

ProfInfer collects several classes of raw signals: timestamps, process and thread identifiers, CPU identifiers, operator semantics, scheduler transitions, and per-thread hardware or software counters [2601.20755]. The operator semantics include operator type, operator name, source and target tensors, and tensor dimensions. The recorded counters include `cycles`, `stalled_backend_cycles`, `l3d_cache_refill`, `mem_access_wr`, and `major_faults` [2601.20755]. Scheduler state is reconstructed from `sched_switch` and `sched_wakeup` tracepoints, distinguishing preempted, runnable, and running states based on `prev_state` and switch-in events [2601.20755].

From these signals, ProfInfer derives a set of execution metrics. Operator, graph, and token latencies are computed as $T_i$, $T_{\text{graph}}$, and $T_{\text{token}}$, with total latency decomposed as $T_{\text{total}} = \sum_i T_i$ [2601.20755]. Memory traffic is estimated through cache-refill and write counters, with bytes fetched approximated as $64 \times \Delta l3d\_cache\_refill$ and write bytes as $16 \times \Delta mem\_access\_wr$, yielding an operator-level bandwidth estimate $B = \text{bytes}/\text{time}$ [2601.20755]. Stall ratio is defined as $r_{\text{stall}} = \Delta stalled\_backend\_cycles / \Delta cycles$, and thread utilization as active operator time divided by total traced time [2601.20755]. For matrix multiplication, arithmetic intensity is estimated in a Roofline-inspired manner as $I = \text{FLOPs}/\text{bytes}$, using $\text{FLOPs} \approx 2MNK$ and bytes from the observed counters [2601.20755].

These metrics feed three principal analysis views.

| View | Function | Representation |
|---|---|---|
| ProfDAG | Operator graph reconstruction | DAG from `ggml_tensor` sources and execution order |
| ProfTime | Temporal execution analysis | Chrome Trace Event Format, visualized via Perfetto |
| ProfStat | Statistical and trend analysis | Per-token, per-operator, per-expert plots and counter trends |

ProfDAG reconstructs the operator DAG from source tensor addresses, annotating nodes and edges with time and counter information [2601.20755]. ProfTime places operators on per-thread swimlanes, revealing intra-operator parallelism, backend partitioning, and scheduler interference [2601.20755]. ProfStat aggregates trends across tokens, operators, and experts, including time-vs-bytes signatures, stall-ratio trends, and page-fault correlations [2601.20755].

The workload-characterization methodology uses these views to distinguish compute-bound from memory-bound behavior. The paper describes rule-of-thumb classifications such as high arithmetic intensity with low stall ratio for compute-bound execution, versus high bandwidth and high stall ratio for memory-bound execution [2601.20755]. It also treats linear scaling of operator time with bytes, irrespective of FLOPs, as an indicator of memory-limited decoding [2601.20755]. This suggests that ProfInfer is not limited to descriptive tracing; it encodes an interpretive framework for identifying bottleneck classes directly from counter-correlated operator behavior.

## 4. Dense inference, Mixture-of-Experts, and offloading behavior

ProfInfer’s dense-inference analysis shows that attention and FFN paths are dominated by MatMul-family operators, with specific `MUL_MAT` instances appearing as core hotspots in different architectures [2601.20755]. In decode, the system reports that matrix-vector multiplications account for more than 97% of TTFT and TPOT, and that decoding becomes increasingly memory-bound as thread count rises, with stall ratios exceeding 80% at four threads on Cortex-A76 [2601.20755]. The profiler further shows that decode iteration time grows with context length because `KQ` and `KQV` operators increase stepwise with KV-cache growth, even though KV-cache removes some compute [2601.20755].

The operator graphs also expose architectural differences among models. According to the paper, Qwen2.5-1.5B adds `ADD` nodes relative to LLaMA3.2-1B, whereas Gemma2-2B adds multiple `SOFT_MAX`, `RMS_NORM`, `UNARY`, and `MUL` operators [2601.20755]. Execution order is not necessarily consecutive for operators sharing intermediates, which the paper associates with runtime optimizations or pipelining choices [2601.20755]. Timelines further show sequential inter-operator scheduling but visible intra-operator parallelism, particularly when MatMul work is partitioned across threads [2601.20755].

For Mixture-of-Experts workloads, ProfInfer uses `ggml_compute_forward_mul_mat_id` to expose top-$k$ expert IDs per FFN gating decision [2601.20755]. On Qwen1.5-MoE-A2.7B-Q4, which the paper describes as having 60 experts with $k=4$ active each iteration and total model size of approximately 8.9 GB with `mmap`, operator time correlates with the average reuse distance of selected experts [2601.20755]. Higher reuse distances imply eviction and storage reload, and the profiler observes that major page faults increase with expert eviction [2601.20755]. The paper concludes from this that the MoE bottleneck is dominated by disk I/O rather than DRAM bandwidth [2601.20755]. This is a specific example of how operator-level tracing combined with OS-level fault data reveals bottlenecks that would not be visible from token throughput alone.

Operator offloading is another principal use case. ProfInfer can display graph partitions executing on CPU plus NPU or CPU plus OpenCL backends, and can nest OpenCL kernel intervals beneath GGML-level timelines [2601.20755]. In the Rubik Pi experiments with Adreno GPU and 4-bit quantization, GPUs outperform CPUs for certain intermediate MatMul sizes but underperform on large LM Head dimensions because of OpenCL kernel-size constraints and bandwidth or latency issues [2601.20755]. The paper therefore argues for selective per-operator offloading rather than indiscriminate backend migration, since blanket offloading may worsen TTFT or TPOT [2601.20755].

## 5. Overhead, fidelity, and operational constraints

ProfInfer quantifies overhead as throughput degradation, using the formulation $\text{overhead} = (T_{\text{profile}} - T_{\text{base}}) / T_{\text{base}}$, equivalently the speed decrease relative to baseline tokens per second [2601.20755]. On Orange Pi platforms, the reported speed decrease ranges from 4.0% down to 2.8% for the BCC version and from 2.2% to 1.7% for the libbpf version, depending on enabled flags [2601.20755]. Average per-core probe-induced CPU load is reported as approximately 0.61–0.70% for BCC and approximately 0.41% for libbpf [2601.20755]. Token-plus-graph-only tracing reduces the speed decrease to about 0.1%, and the prefill stage is reported to have negligible overhead [2601.20755].

The system’s fidelity strategy is based on boundary measurements rather than statistical sampling. Timestamps are obtained with `bpf_ktime_get_ns`, while PMCs are read at operator entry and exit so that per-thread attribution is precise and sampling bias is minimized [2601.20755]. Buffer choice is also treated as a fidelity-overhead control: perf buffers provide reliable delivery with somewhat higher overhead, whereas ring buffers reduce overhead at the cost of possible event drops under pressure [2601.20755]. Userspace can dynamically throttle tracing by disabling heavy parsing such as tensor dimensions or PMC collection when tokens-per-second falls below a QoS threshold [2601.20755].

The evaluation includes qualitative comparisons with other profiling mechanisms. The architecture-only dump `ggml_graph_dump_dot` in llama.cpp incurs approximately 13% overhead for a single forward pass, and ONNX Runtime profiling is reported at approximately 8% overhead in preliminary tests [2601.20755]. By contrast, ProfInfer is intended to provide lower overhead together with operator semantics, hardware-counter correlation, and scheduler integration for llama.cpp-like runtimes [2601.20755].

Several practical limitations are stated explicitly. eBPF cannot read GPU PMCs, so GPU-side analysis relies on OpenCL kernel timestamps rather than vendor-specific hardware counters [2601.20755]. Current operator probing support is limited to CPU, OpenCL/CLBlast, and a Rockchip NPU backend, with CUDA, Metal, and other NPUs left as future work [2601.20755]. The system focuses on single-node on-device inference rather than distributed tracing, and stripped binaries complicate symbol-based attachment unless offsets or symbol maps are available [2601.20755]. Heavy per-operator parsing and PMC reading can also affect TPS, which is why dynamic throttling is part of the design [2601.20755].

## 6. Related naming, adjacent uses, and interpretive position

The name **ProfInfer** is not uniformly used across the broader literature represented here. The system actually introduced under that exact name is the eBPF-based LLM inference profiler described above [2601.20755]. By contrast, the shotgun-proteomics method in “A Combinatorial Perspective of the Protein Inference Problem” is named **ProteinInfer**, and the source explicitly states that “ProfInfer” appears to be a misnomer in that context [1211.6179]. Likewise, “Differentially Private Bayesian Programming” introduces **PrivInfer**, and its source similarly states that “ProfInfer” is very likely a misspelling there [1605.00283]. In privacy-preserving machine learning, **PINFER** denotes “Privacy-Preserving Inference for Machine Learning,” again a distinct system [1910.01865]. In cosmology, the relevant software is **PROSPECT**, a profile-likelihood code whose discussion frames “ProfInfer” only informally as profile-likelihood inference rather than as the code name itself [2312.02972].

This naming dispersion matters because it separates several unrelated research lines: protein inference in proteomics, privacy-preserving inference, differentially private Bayesian programming, profile-likelihood inference in cosmology, and fine-grained LLM systems profiling. ProfInfer in the strict bibliographic sense therefore refers to the 2026 eBPF profiler, not to these neighboring frameworks [2601.20755].

Within systems research, ProfInfer’s distinctive contribution is the alignment of low-level tracing with high-level LLM operator semantics. The paper positions it as adding non-intrusive operator-semantic visibility for lightweight runtimes, per-operator PMC integration, MoE-specific expert-ID tracing, and multi-backend observability across CPU, GPU, and NPU execution paths [2601.20755]. This suggests a broader methodological role: profiling is treated not as a post hoc performance audit, but as an operational basis for backend partitioning, thread tuning, interference diagnosis, caching strategy, and admission or QoS control [2601.20755].

In that sense, ProfInfer occupies a specific niche in the emerging tooling ecosystem around production LLM inference. It is not an inference engine, model optimizer, or scheduler. It is a diagnostic substrate that makes dense inference, MoE routing, and offloading behavior visible at the operator level with bounded perturbation, and it does so using runtime attachment rather than recompilation or intrusive instrumentation [2601.20755].

Source: https://www.emergentmind.com/topics/profinfer