bpfix: Diagnostic Tool for eBPF Verification

• bpfix is a diagnostic tool for eBPF that reconstructs the verifier’s proof lifecycle, identifying where essential proof information is established and later lost.
• It analyzes detailed kernel logs to bridge the diagnostic gap between terminal errors and the underlying proof failures in eBPF programs.
• Empirical evaluations demonstrate that bpfix improves error localization and enhances automated repair strategies by clarifying root causes in verification rejections.

bpfix is a diagnostic tool for eBPF verifier failures that reads the verifier’s existing log, reconstructs the proof the verifier was trying to maintain, and pinpoints where that proof was first established and later lost, rather than only reporting where verification stopped. In the formulation of "Characterizing and Bridging the Diagnostic Gap in eBPF Verifier Rejections" (Zheng et al., 2 Jul 2026), bpfix is designed to bridge a diagnostic gap in eBPF development: verifier rejections are expressed as low-level terminal errors, but repair depends on recovering the verifier-visible proof lifecycle that made a later operation unverifiable.

1. eBPF verification and the diagnostic gap

eBPF allows developers to load small programs into the Linux kernel for networking, security, observability, and related purposes. The workflow described for the paper is: write code in C or Rust, compile to eBPF bytecode, load it into the kernel, and have the kernel’s eBPF verifier check it before execution. The verifier is a static analyzer that symbolically executes the program along all paths and tracks an abstract state for each instruction and each register or stack slot. Its reasoning includes pointer types and provenance, bounds, scalar ranges, resource lifetimes, and stack initialization and bounds (Zheng et al., 2 Jul 2026).

The paper frames verification as maintenance of a proof over execution. Typical proof obligations include statements such as whether a pointer is within packet bounds when dereferenced, whether a scalar length lies within an allowed range, or whether a resource lifetime is still valid. A rejection occurs when, along some path, the verifier can no longer prove a property that a later operation requires.

The reported difficulty is not merely that programs are rejected, but that verifier diagnostics are misaligned with the structure of the underlying proof failure. When logging is enabled with log_level=2, the verifier provides a per-instruction trace and a terminal error line such as R5 invalid mem access 'scalar', only read from bpf_array is supported, invalid access to packet, or R2 invalid mem access 'scalar'. These diagnostics identify the immediate failing condition and the instruction where checking stopped, but they do not explicitly expose which proof was needed, where that proof had previously been established, or where it was later invalidated.

The paper defines this mismatch as a diagnostic gap: the terminal error reports where verification stopped, whereas the root cause is often where the proof was lost earlier, potentially many instructions away or in a different part of the control-flow graph. bpfix is built around that distinction. Its core objective is to reconstruct, from the existing verifier log, the proof required at the rejecting operation, the source or instruction span where that proof was established, and the point where it was lost.

2. Empirical characterization of verifier rejections

To characterize verifier failures, the paper constructs bpfix-empirical, a dataset of 235 reproduced eBPF verifier rejections. The source pool comprised 936 candidate reports collected from Stack Overflow questions about eBPF verifier failures, GitHub issues, GitHub fix commits, and Linux kernel selftests. Each candidate was rebuilt and loaded under a fixed toolchain consisting of Linux kernel 6.15.11, clang 18, and verifier logging at log_level=2; only cases that reproduced a verifier rejection under that setup were retained (Zheng et al., 2 Jul 2026).

Each retained case includes the faulty source and the developer’s fix. That before/after pairing is used as ground truth for identifying the root cause and locating where the eventual fix was applied. The study first separates failures by repair location. Among the 235 reproduced rejections, 191/235 (81%) are program bugs where the fix changes the eBPF program’s source, whereas 44/235 (19%) reject correct source and are fixed elsewhere: 18 are compiler issues, 14 are environment misconfiguration, and 12 are verifier limitations that require changes in the verifier or kernel.

For the 191 program bugs, the paper defines the root cause as the specific source-level mistake and groups them into 12 categories. The listed categories and counts are:

• Unclamped scalar used as offset or length — 24
• Corrupted or stale dynptr object — 23
• Packet access without a bound on every path — 22
• Missing null check — 19
• Pointer type or provenance mismatch — 16
• Unverified address dereferenced — 16
• Index exceeds object capacity — 15
• Context or contract misuse — 15
• Unpaired resource reference — 15
• Interrupt flag not restored in order — 11
• Probe signature mismatched with the ABI — 9
• Oversized or uninitialized stack buffer — 6

A central result is that 10 of these 12 root causes are eBPF-specific, so interpreting and repairing many rejections requires domain knowledge beyond ordinary C or Rust experience.

The study also measures the coarseness of terminal error reporting. Across all reproduced rejections, 47% of rejections return only EINVAL. To analyze message ambiguity, the paper normalizes error strings by masking registers and offsets, reducing 167 raw strings to 82 templates. Among those templates, 15 of the 82 templates each map to more than one root cause. The most frequent template, R# invalid mem access 'scalar', appears in 28 cases and spans 9 distinct root-cause categories; invalid access to packet appears in 26 cases across 5 categories; invalid access to map value appears in 18 cases across 4 categories; and R# !read_ok appears in 13 cases across 4 categories (Zheng et al., 2 Jul 2026).

These results establish the paper’s central empirical claim: terminal verifier errors are often too coarse to identify the root cause of a rejection, because they reveal where verification stopped rather than where the proof was lost.

3. Proof reconstruction and localization strategy

bpfix is described as a standalone Rust tool of approximately 23k LOC that does not change the kernel or the verifier. Its input is the verifier log from a failed program, with optional source code and debug information for improved source mapping. Its output is a Rust-style diagnostic containing a stable error ID of the form error[BPFOCUS-Exxx], a class such as source_bug or lowering_artifact, the required proof at the failing operation, the spans where the proof was established and where it was lost, and help suggestions oriented toward restoring that proof (Zheng et al., 2 Jul 2026).

The architecture has three stages. The first is Log Analysis, which parses the verifier log, extracts the terminal error line and the per-instruction abstract states, and normalizes them into an evidence stream. The second is Proof Reconstruction, which uses the terminal error and nearby abstract state to determine which proof family is required at the rejected operation. The proof families enumerated in the paper include pointer provenance, packet bounds, scalar range constraints, map value pointer derivation, and dynptr lifetime. Once the required proof is inferred, bpfix tracks evidence for that proof through the abstract state sequence, determining when the proof was established, when it became incompatible or disappeared, and where the verifier later required it. The third stage is Diagnostic Generation, which selects the rejection location, the loss point if visible, source spans for context, and next-action guidance.

The central abstraction is a proof lifecycle consisting of establishment, persistence, loss, and rejection. Establishment corresponds to instructions that create proof evidence, such as a successful bounds check, a helper call returning a map value pointer, or a dynptr initialization. Persistence corresponds to later instructions that preserve or refine that evidence. Loss corresponds to operations that make the proof incompatible or unusable, such as casting a pointer to an integer and back, arithmetic that breaks pointer provenance, overwriting a dynptr, or using a reference after its lifetime ends. Rejection occurs when a later instruction requires that proof and the verifier can no longer establish it.

The localization rule described in the paper is to choose the earliest instruction that kills the required proof as the reported loss point. If there is no earlier evidence that the proof was ever established, bpfix interprets the failure as one in which the required proof was never created, as in the absence of a packet bounds check or a null check. The tool therefore shifts diagnosis from an error-at-use model to a lost-proof model.

Conceptually, bpfix’s analysis uses the abstract state recorded in the log: register types such as pkt(), ctx, scalar, and ptr_to_map_value; scalar ranges; and reference or dynptr identifiers tied to lifetimes. The paper does not provide explicit pseudocode, but it describes the tool as tracking transitions such as pointer-to-scalar conversion, range widening to unknown, and dynptr reference corruption or staleness, then connecting those transitions to a particular proof family and to a source-level repair target.

4. Diagnostics, classes, and illustrative cases

The diagnostic format is intentionally Rust-like. It includes annotated source snippets, underlines for relevant spans, an error ID, a class label, the verifier’s terminal line, an explicit statement of the required proof, and one or more help: lines. The paper’s running example uses a packet-parsing program in which udph is assigned from either an IPv4 or IPv6 header, checked against data_end, and then dereferenced as ((struct udphdr *)udph)->dest. The verifier rejects the later load with R5 invalid mem access 'scalar', even though the source performed a packet bounds check (Zheng et al., 2 Jul 2026).

bpfix reconstructs the needed proof as a pointer-provenance obligation: the register used at the dereference must still be a verifier-recognized packet pointer with valid bounds. In the example, bpfix reports:

• error[BPFOCUS-E006]: verifier-visible compiler lowering hides the required proof
• class: lowering_artifact
• required proof: preserve a verifier-recognized pointer type at the operation that requires a pointer

It also marks the rejecting source line, indicates that verifier state changes from pkt to scalar before the access, and suggests reacquiring a verifier-tracked pointer before the dereference. The paper characterizes this case as one in which the source had the right semantics and bounds check, but compiler lowering obscured the proof as seen by the verifier.

The paper contrasts this with a source bug in which a pointer is constructed purely from an integer offset:

if (eth->h_proto == bpf_htons(ETH_P_IP)) { struct iphdr *iph2 = (void *)(sizeof(*eth) + nh_off); // no packet base return iph2->protocol; // rejected }

For that case, bpfix reports:

• error[BPFOCUS-E011]: scalar or pkt_end value is used where the verifier requires a real pointer
• class: source_bug
• verifier[31]: R1 invalid mem access 'scalar'

The distinction between source_bug and lowering_artifact is one of the tool’s explicit classification goals. A rejection may share the same terminal verifier string as another failure yet arise from a different layer. In the authors’ categorization, source_bug denotes a problem in user source that should be repaired by changing source logic, whereas lowering_artifact denotes a problem that arises from compiler lowering while the source is logically correct.

The paper also states that bpfix uses its reconstruction of proof families and state transitions to map failures into categories such as stack bounds, pointer aliasing or provenance, uninitialized stack, and dynptr lifetime. The diagnostic’s required-proof statement and next-action guidance are phrased in those terms, for example by advising a developer to derive a map-value pointer with the proper map helper before accessing map contents.

5. Benchmarking LLM-assisted repair

Beyond human-facing diagnostics, the paper evaluates whether localized proof-oriented diagnostics improve automatic repair by LLMs. For that purpose it introduces bpfix-bench, a benchmark of 75 repair tasks: 40 focused on specific verifier proofs and 35 minimized from real open-source projects, including Cilium, xdp-tools, and bpftime. Each task includes the buggy program and an executable test suite that compiles the candidate fix, attempts to load the eBPF program through the real kernel verifier, runs functional tests, and in some tasks checks source semantics. A fix is counted as successful only if it passes all such checks (Zheng et al., 2 Jul 2026).

The evaluation uses three models at temperature 0: Qwen3.6 27B, GLM 5.2, and Qwen2.5 3B. Each model is tested under two input conditions—raw verifier log and bpfix diagnostic—and in two interaction modes: one-shot and retry, where retry permits one failure-informed second attempt.

The headline result is that current models achieve 0–37% one-shot success with raw verifier logs, whereas replacing the log with bpfix localization improves repair by 11–21 percentage points. The per-model results reported in the paper are:

• Qwen3.6 27B: raw log one-shot 22/75; bpfix diagnostic one-shot 38/75; raw log +1 retry 30/75; bpfix diagnostic +1 retry 44/75
• GLM 5.2: raw log one-shot 28/75; bpfix diagnostic one-shot 38/75; raw log +1 retry 47/75; bpfix diagnostic +1 retry 52/75
• Qwen2.5 3B: raw log one-shot 0/75; bpfix diagnostic one-shot 8/75; raw log +1 retry 0/75; bpfix diagnostic +1 retry 10/75

The paper further categorizes unsuccessful attempts by failure stage. In one-shot runs, Qwen3.6 27B under raw logs has 3 compile failures, 19 load failures, 9 functional failures, and 22 proof failures; with bpfix diagnostics, the corresponding counts are 1 compile, 10 load, 10 functional, and 16 proof. For GLM 5.2, raw logs yield 1 compile, 10 load, 11 functional, and 25 proof failures, while bpfix diagnostics yield 1 compile, 5 load, 9 functional, and 22 proof. For Qwen2.5 3B, raw logs yield 7 compile, 62 load, 0 functional, 3 proof, and 3 call failures; with bpfix diagnostics the counts are 14 compile, 39 load, 6 functional, 8 proof, and 0 call failures (Zheng et al., 2 Jul 2026).

The interpretation given is that localized proof-aware diagnostics provide the right signal to guide repair: what proof is missing, where it was lost, and what kind of change is needed. The observed improvements are concentrated in fewer verifier-load failures and fewer proof or source-semantics failures. For the smallest model, the paper additionally notes that raw logs are often too large and unstructured, causing context-related call failures that disappear when bpfix diagnostics are substituted.

6. Implementation, use, limitations, and implications

The implementation is in Rust and consumes verifier logs captured from the Linux kernel’s eBPF verifier at log_level=2, optionally together with source code and object or debug information. The empirical study is conducted with Linux 6.15.11 and clang 18. The paper does not claim the tool is fully version-agnostic; rather, it states that bpfix must understand the format of verifier logs and abstract states for the relevant kernel series, while remaining log-driven rather than dependent on internal verifier APIs (Zheng et al., 2 Jul 2026).

Because bpfix does not modify the kernel, it can be used with existing kernels so long as their log format is supported. The practical workflow described in the paper is to compile an eBPF program, attempt to load it with detailed verifier logging, capture the log, run bpfix on that log, inspect the resulting Rust-like diagnostic, modify the source accordingly, and then recompile and reload. The tool is available at https://github.com/eunomia-bpf/bpfix.

The paper identifies several limitations. bpfix relies on the printed log, so its reconstruction may be incomplete or inaccurate if the kernel does not print sufficient state, if the log format changes in unexpected ways, or if relevant verifier behavior is not reflected in the log. Some failure categories, including extremely complex control flow or obscure verifier heuristics, may not yield a clear loss point. The diagnostic includes a confidence rating, exemplified in the paper by a medium confidence output. The authors also state that bpfix is not a formal proof checker and does not guarantee that the reported loss point is the only or earliest cause when multiple transitions weaken the required proof.

The empirical study and the LLM evaluation also carry explicit threats to validity. The 235-case corpus is toolchain-specific, relying on kernel 6.15.11 and clang 18; the dataset may reflect selection bias because it is drawn from Stack Overflow, GitHub, and selftests; and the root-cause labels as well as layer attribution between source_bug and lowering_artifact involve manual judgment grounded in developers’ fixes. On the LLM side, only three models are tested, prompt presentation may affect performance, the benchmark size is specialized though nontrivial, and the evaluation focuses on one-shot repair plus a single retry rather than richer interactive workflows.

Within those constraints, the paper’s broader implication is that richer verifier diagnostics can be reconstructed from existing logs without modifying the kernel. This suggests a more general principle for eBPF tooling: diagnostics are more actionable when they expose proof lifecycles rather than only failure points. In the paper’s framing, bpfix demonstrates that localizing where the proof was lost is useful both for human developers and for automated repair systems, and that this localization can materially reduce the trial-and-error involved in repairing verifier rejections (Zheng et al., 2 Jul 2026).