ProfInfer: eBPF LLM Inference Profiler

• ProfInfer is an eBPF-based fine-grained LLM inference profiler that attaches dynamic probes to capture operator-level behavior in modern inference engines.
• It correlates operator semantics with hardware counters and scheduler events, enabling detailed performance and bottleneck analysis for various backend configurations.
• Designed as a non-intrusive diagnostic tool, ProfInfer informs optimization decisions and resource allocation with precise, real-time operator tracing while maintaining low overhead.

ProfInfer is an eBPF-based fine-grained LLM inference profiler for modern inference engines such as llama.cpp, designed to provide operator-level, real-time visibility without modifying or recompiling the runtime source code (Zou et al., 28 Jan 2026). It attaches probes dynamically across multiple layers of the inference stack, correlates operator semantics with timestamps, scheduler behavior, and hardware counters, and renders the resulting traces as operator DAGs, timelines, and statistical trend plots. The system is positioned as a response to a diagnostic gap in lightweight LLM runtimes, which typically expose token-level metrics such as throughput and TTFT but not per-operator, per-graph, or per-backend behavior (Zou et al., 28 Jan 2026).

1. Definition, scope, and diagnostic objective

ProfInfer targets modern LLM inference runtimes in which critical execution phenomena remain opaque at inference time: whether decoding is memory-bound or compute-bound, which operators dominate latency, how Mixture-of-Experts routing affects execution, and whether operator offloading is beneficial for particular tensor shapes or backend configurations (Zou et al., 28 Jan 2026). The framework is explicitly described as fine-grained and non-intrusive. Its central design goal is to restore operator-level visibility to runtimes like llama.cpp while preserving low perturbation, with measured runtime overhead under 4% (Zou et al., 28 Jan 2026).

The profiling problem addressed by ProfInfer differs from the observability offered by several adjacent tool classes. General-purpose systems such as ONNX Runtime and TensorRT can expose some operator latencies and allocation information, but their visibility is described as coarse and intrusive, and they do not correlate low-level hardware metrics with LLM phases such as prefill and decode (Zou et al., 28 Jan 2026). CPU and GPU profilers such as perf, oprofile, and Nsight provide low-level samples or vendor-specific traces, but not operator-correlated DAGs and timelines tailored to lightweight LLM runtimes (Zou et al., 28 Jan 2026). PyTorch Profiler and serving traces are likewise oriented toward training or serving stacks rather than compact on-device libraries (Zou et al., 28 Jan 2026).

A plausible implication is that ProfInfer should be understood not merely as a timing tool, but as a systems-analysis layer specialized for the operational structure of LLM inference. The paper characterizes its outputs as useful for optimization, scheduling, and resource-aware deployment, indicating that it is meant to inform concrete runtime decisions rather than simply collect measurements (Zou et al., 28 Jan 2026).

2. Architecture and eBPF-based instrumentation

ProfInfer is organized as an end-to-end tracing pipeline spanning userspace control, kernelspace probe execution, userspace aggregation, and visualization (Zou et al., 28 Jan 2026). A userspace controller compiles and loads eBPF programs through BCC on Ubuntu or libbpf on OpenHarmony, configures control flags such as tensor-dimension parsing and PMC collection, attaches uprobes and uretprobes to target runtime functions, enables kernel tracepoints, and opens perf_event file descriptors per thread for hardware-counter collection (Zou et al., 28 Jan 2026).

In kernelspace, the eBPF handlers record nanosecond timestamps, thread identifiers, CPU identifiers, and, when configured, parse arguments from userspace data structures such as ggml_tensor via bpf_probe_read_user/kernel to extract operator type, tensor pointers, names, and dimensions (Zou et al., 28 Jan 2026). Counter values are read through BPF maps and perf_read, and events are emitted through perf buffers or ring buffers depending on the desired reliability-overhead trade-off (Zou et al., 28 Jan 2026). Userspace then polls these buffers, filters events by probe type, merges events across threads, resolves symbols to operator instances, and computes per-operator durations and per-graph breakdowns (Zou et al., 28 Jan 2026).

The instrumentation strategy covers several runtime layers. At the token level, ProfInfer attaches probes to llama_decode, using call-return deltas to measure TTFT and TPOT (Zou et al., 28 Jan 2026). At the graph level, it instruments ggml_backend_graph_compute_async to time each graph execution and extract backend identifiers so that CPU, OpenCL, and NPU graph segments can be distinguished (Zou et al., 28 Jan 2026). At the operator level, it probes backend-dispatch functions such as ggml_compute_forward, backend-specific OpenCL forward functions, and an analogous Rockchip NPU integration point (Zou et al., 28 Jan 2026). For MoE analysis it additionally probes ggml_compute_forward_mul_mat_id and extracts expert IDs through a two-level pointer dereference on the third source tensor (Zou et al., 28 Jan 2026).

Portability is a stated design feature. ProfInfer supports both BCC and libbpf, and where CO-RE and BTF are available it can be resilient across minor kernel-version changes without rebuilds (Zou et al., 28 Jan 2026). Deployment nevertheless depends on standard eBPF and perf prerequisites: Linux with CONFIG_BPF, an enabled perf_event subsystem, and capabilities such as CAP_BPF or CAP_SYS_ADMIN (Zou et al., 28 Jan 2026). In containers, privileged execution with BPF/perf capabilities is possible, but host-mode execution is recommended (Zou et al., 28 Jan 2026).

3. Data model, derived metrics, and visual analytics

ProfInfer collects several classes of raw signals: timestamps, process and thread identifiers, CPU identifiers, operator semantics, scheduler transitions, and per-thread hardware or software counters (Zou et al., 28 Jan 2026). The operator semantics include operator type, operator name, source and target tensors, and tensor dimensions. The recorded counters include cycles, stalled_backend_cycles, l3d_cache_refill, mem_access_wr, and major_faults (Zou et al., 28 Jan 2026). Scheduler state is reconstructed from sched_switch and sched_wakeup tracepoints, distinguishing preempted, runnable, and running states based on prev_state and switch-in events (Zou et al., 28 Jan 2026).

From these signals, ProfInfer derives a set of execution metrics. Operator, graph, and token latencies are computed as $T_i$, $T_{\text{graph}}$, and $T_{\text{token}}$, with total latency decomposed as $T_{\text{total}} = \sum_i T_i$ (Zou et al., 28 Jan 2026). Memory traffic is estimated through cache-refill and write counters, with bytes fetched approximated as $64 \times \Delta l3d\_cache\_refill$ and write bytes as $16 \times \Delta mem\_access\_wr$, yielding an operator-level bandwidth estimate $B = \text{bytes}/\text{time}$ (Zou et al., 28 Jan 2026). Stall ratio is defined as $$r_{\text{stall}} = \Delta stalled\_backend\_cycles / \Delta cycles$$, and thread utilization as active operator time divided by total traced time (Zou et al., 28 Jan 2026). For matrix multiplication, arithmetic intensity is estimated in a Roofline-inspired manner as $I = \text{FLOPs}/\text{bytes}$, using $\text{FLOPs} \approx 2MNK$ and bytes from the observed counters (Zou et al., 28 Jan 2026).

These metrics feed three principal analysis views.

View	Function	Representation
ProfDAG	Operator graph reconstruction	DAG from ggml_tensor sources and execution order
ProfTime	Temporal execution analysis	Chrome Trace Event Format, visualized via Perfetto
ProfStat	Statistical and trend analysis	Per-token, per-operator, per-expert plots and counter trends

ProfDAG reconstructs the operator DAG from source tensor addresses, annotating nodes and edges with time and counter information (Zou et al., 28 Jan 2026). ProfTime places operators on per-thread swimlanes, revealing intra-operator parallelism, backend partitioning, and scheduler interference (Zou et al., 28 Jan 2026). ProfStat aggregates trends across tokens, operators, and experts, including time-vs-bytes signatures, stall-ratio trends, and page-fault correlations (Zou et al., 28 Jan 2026).

The workload-characterization methodology uses these views to distinguish compute-bound from memory-bound behavior. The paper describes rule-of-thumb classifications such as high arithmetic intensity with low stall ratio for compute-bound execution, versus high bandwidth and high stall ratio for memory-bound execution (Zou et al., 28 Jan 2026). It also treats linear scaling of operator time with bytes, irrespective of FLOPs, as an indicator of memory-limited decoding (Zou et al., 28 Jan 2026). This suggests that ProfInfer is not limited to descriptive tracing; it encodes an interpretive framework for identifying bottleneck classes directly from counter-correlated operator behavior.

4. Dense inference, Mixture-of-Experts, and offloading behavior

ProfInfer’s dense-inference analysis shows that attention and FFN paths are dominated by MatMul-family operators, with specific MUL_MAT instances appearing as core hotspots in different architectures (Zou et al., 28 Jan 2026). In decode, the system reports that matrix-vector multiplications account for more than 97% of TTFT and TPOT, and that decoding becomes increasingly memory-bound as thread count rises, with stall ratios exceeding 80% at four threads on Cortex-A76 (Zou et al., 28 Jan 2026). The profiler further shows that decode iteration time grows with context length because KQ and KQV operators increase stepwise with KV-cache growth, even though KV-cache removes some compute (Zou et al., 28 Jan 2026).

The operator graphs also expose architectural differences among models. According to the paper, Qwen2.5-1.5B adds ADD nodes relative to LLaMA3.2-1B, whereas Gemma2-2B adds multiple SOFT_MAX, RMS_NORM, UNARY, and MUL operators (Zou et al., 28 Jan 2026). Execution order is not necessarily consecutive for operators sharing intermediates, which the paper associates with runtime optimizations or pipelining choices (Zou et al., 28 Jan 2026). Timelines further show sequential inter-operator scheduling but visible intra-operator parallelism, particularly when MatMul work is partitioned across threads (Zou et al., 28 Jan 2026).

For Mixture-of-Experts workloads, ProfInfer uses ggml_compute_forward_mul_mat_id to expose top-$T_{\text{graph}}$0 expert IDs per FFN gating decision (Zou et al., 28 Jan 2026). On Qwen1.5-MoE-A2.7B-Q4, which the paper describes as having 60 experts with $T_{\text{graph}}$1 active each iteration and total model size of approximately 8.9 GB with mmap, operator time correlates with the average reuse distance of selected experts (Zou et al., 28 Jan 2026). Higher reuse distances imply eviction and storage reload, and the profiler observes that major page faults increase with expert eviction (Zou et al., 28 Jan 2026). The paper concludes from this that the MoE bottleneck is dominated by disk I/O rather than DRAM bandwidth (Zou et al., 28 Jan 2026). This is a specific example of how operator-level tracing combined with OS-level fault data reveals bottlenecks that would not be visible from token throughput alone.

Operator offloading is another principal use case. ProfInfer can display graph partitions executing on CPU plus NPU or CPU plus OpenCL backends, and can nest OpenCL kernel intervals beneath GGML-level timelines (Zou et al., 28 Jan 2026). In the Rubik Pi experiments with Adreno GPU and 4-bit quantization, GPUs outperform CPUs for certain intermediate MatMul sizes but underperform on large LM Head dimensions because of OpenCL kernel-size constraints and bandwidth or latency issues (Zou et al., 28 Jan 2026). The paper therefore argues for selective per-operator offloading rather than indiscriminate backend migration, since blanket offloading may worsen TTFT or TPOT (Zou et al., 28 Jan 2026).

5. Overhead, fidelity, and operational constraints

ProfInfer quantifies overhead as throughput degradation, using the formulation $T_{\text{graph}}$2, equivalently the speed decrease relative to baseline tokens per second (Zou et al., 28 Jan 2026). On Orange Pi platforms, the reported speed decrease ranges from 4.0% down to 2.8% for the BCC version and from 2.2% to 1.7% for the libbpf version, depending on enabled flags (Zou et al., 28 Jan 2026). Average per-core probe-induced CPU load is reported as approximately 0.61–0.70% for BCC and approximately 0.41% for libbpf (Zou et al., 28 Jan 2026). Token-plus-graph-only tracing reduces the speed decrease to about 0.1%, and the prefill stage is reported to have negligible overhead (Zou et al., 28 Jan 2026).

The system’s fidelity strategy is based on boundary measurements rather than statistical sampling. Timestamps are obtained with bpf_ktime_get_ns, while PMCs are read at operator entry and exit so that per-thread attribution is precise and sampling bias is minimized (Zou et al., 28 Jan 2026). Buffer choice is also treated as a fidelity-overhead control: perf buffers provide reliable delivery with somewhat higher overhead, whereas ring buffers reduce overhead at the cost of possible event drops under pressure (Zou et al., 28 Jan 2026). Userspace can dynamically throttle tracing by disabling heavy parsing such as tensor dimensions or PMC collection when tokens-per-second falls below a QoS threshold (Zou et al., 28 Jan 2026).

The evaluation includes qualitative comparisons with other profiling mechanisms. The architecture-only dump ggml_graph_dump_dot in llama.cpp incurs approximately 13% overhead for a single forward pass, and ONNX Runtime profiling is reported at approximately 8% overhead in preliminary tests (Zou et al., 28 Jan 2026). By contrast, ProfInfer is intended to provide lower overhead together with operator semantics, hardware-counter correlation, and scheduler integration for llama.cpp-like runtimes (Zou et al., 28 Jan 2026).

Several practical limitations are stated explicitly. eBPF cannot read GPU PMCs, so GPU-side analysis relies on OpenCL kernel timestamps rather than vendor-specific hardware counters (Zou et al., 28 Jan 2026). Current operator probing support is limited to CPU, OpenCL/CLBlast, and a Rockchip NPU backend, with CUDA, Metal, and other NPUs left as future work (Zou et al., 28 Jan 2026). The system focuses on single-node on-device inference rather than distributed tracing, and stripped binaries complicate symbol-based attachment unless offsets or symbol maps are available (Zou et al., 28 Jan 2026). Heavy per-operator parsing and PMC reading can also affect TPS, which is why dynamic throttling is part of the design (Zou et al., 28 Jan 2026).

6. Related naming, adjacent uses, and interpretive position

The name ProfInfer is not uniformly used across the broader literature represented here. The system actually introduced under that exact name is the eBPF-based LLM inference profiler described above (Zou et al., 28 Jan 2026). By contrast, the shotgun-proteomics method in “A Combinatorial Perspective of the Protein Inference Problem” is named ProteinInfer, and the source explicitly states that “ProfInfer” appears to be a misnomer in that context (Yang et al., 2012). Likewise, “Differentially Private Bayesian Programming” introduces PrivInfer, and its source similarly states that “ProfInfer” is very likely a misspelling there (Barthe et al., 2016). In privacy-preserving machine learning, PINFER denotes “Privacy-Preserving Inference for Machine Learning,” again a distinct system (Joye et al., 2019). In cosmology, the relevant software is PROSPECT, a profile-likelihood code whose discussion frames “ProfInfer” only informally as profile-likelihood inference rather than as the code name itself (Holm et al., 2023).

This naming dispersion matters because it separates several unrelated research lines: protein inference in proteomics, privacy-preserving inference, differentially private Bayesian programming, profile-likelihood inference in cosmology, and fine-grained LLM systems profiling. ProfInfer in the strict bibliographic sense therefore refers to the 2026 eBPF profiler, not to these neighboring frameworks (Zou et al., 28 Jan 2026).

Within systems research, ProfInfer’s distinctive contribution is the alignment of low-level tracing with high-level LLM operator semantics. The paper positions it as adding non-intrusive operator-semantic visibility for lightweight runtimes, per-operator PMC integration, MoE-specific expert-ID tracing, and multi-backend observability across CPU, GPU, and NPU execution paths (Zou et al., 28 Jan 2026). This suggests a broader methodological role: profiling is treated not as a post hoc performance audit, but as an operational basis for backend partitioning, thread tuning, interference diagnosis, caching strategy, and admission or QoS control (Zou et al., 28 Jan 2026).

In that sense, ProfInfer occupies a specific niche in the emerging tooling ecosystem around production LLM inference. It is not an inference engine, model optimizer, or scheduler. It is a diagnostic substrate that makes dense inference, MoE routing, and offloading behavior visible at the operator level with bounded perturbation, and it does so using runtime attachment rather than recompilation or intrusive instrumentation (Zou et al., 28 Jan 2026).