LLVM-AutoFix: Agentic Harness for LLVM Bug Repair

• LLVM-AutoFix is a specialized agentic harness that automates debugging and repair of LLVM middle-end bugs through controlled interactions with LLVM-specific artifacts.
• Its architecture combines structured tool invocations, a curated benchmark (llvm-bench), and a minimal repair agent (llvm-autofix-mini) to iteratively generate and validate patches.
• Empirical evaluations show a 22% improvement in pre-review success rates, though expert review remains essential to ensure semantic correctness of the patches.

LLVM-AutoFix, written in the primary paper as llvm-autofix, is an agentic harness for LLVM that is designed to help LLM agents understand, reproduce, debug, and repair real compiler bugs rather than toy defects or generic repository issues. Its core elements are agent-friendly LLVM tools, a curated benchmark of reproducible LLVM middle-end bugs named llvm-bench, and a specialized minimal agent named llvm-autofix-mini. In the evaluation reported for the system, frontier models suffer a roughly 60% performance decline when moving from common software-bug benchmarks to LLVM compiler bugs, while the specialized harness and agent recover a substantial portion of that gap, underscoring the need for LLVM-specific tooling, validation, and debugging workflows (Zheng et al., 20 Mar 2026).

1. Definition and problem setting

LLVM-AutoFix addresses the automated repair of LLVM compiler bugs under conditions that differ sharply from ordinary software maintenance. The target defects are middle-end LLVM bugs, especially crash bugs and miscompilation bugs, and the input is not a rich natural-language issue description but typically a minimal LLVM IR reproducer, an opt invocation, an assertion or stack trace, or an Alive2 counterexample showing that an optimization is unsound. In this setting, a successful repair must modify LLVM C++ sources so that the reproducer no longer crashes or miscompiles while also passing relevant regression tests (Zheng et al., 20 Mar 2026).

The motivating difficulty is structural. LLVM bugs are associated with sparse, non-descriptive reports, deep cross-domain expertise requirements, a large and fragile C++ build environment, and validation regimes in which passing a narrow regression suite does not imply semantic correctness. The reported benchmark results make this difference concrete: with mini-SWE-agent, GPT-5 drops from 65.0% on SWE-bench Verified to 21.0% on llvm-bench live, Gemini 2.5 Pro from 53.6% to 9.2%, Qwen3 Max from 69.6% to 24.4%, DeepSeek V3.2 from 60.0% to 38.9%, and GPT-4o from 21.6% to 8.3%, yielding an average performance decline of about 60% when moving from common software bugs to LLVM compiler bugs (Zheng et al., 20 Mar 2026).

A defining characteristic of LLVM-AutoFix is that it treats LLVM not as a generic repository to be edited with shell access, but as a specialized debugging environment. This shifts the repair problem from open-ended code generation toward controlled interaction with LLVM-specific artifacts such as IR semantics, pass internals, GDB state, Alive2 validation, and llvm-lit regression infrastructure. The result is a harness-centered conception of automated repair rather than a prompt-only conception (Zheng et al., 20 Mar 2026).

2. System architecture and harness design

LLVM-AutoFix is organized as an agentic harness: a layer that standardizes reproduction, debugging, patch generation, and validation for LLVM bugs. The harness exposes constrained tools rather than unconstrained shell access, enabling the agent to search code, inspect functions, query the LangRef, debug a live opt process, edit files, preview patches, reset the tree, and run an integrated online test procedure (Zheng et al., 20 Mar 2026).

Component	Function	Concrete contents
Agent-friendly tools	Provide structured interaction with LLVM	find, grep, read, edit, preview, code, docs, langref, debug, eval, reset, test
Benchmark layer	Supply reproducible repair tasks	llvm-bench and llvm-bench live
Specialized agent	Execute LLVM-centered repair workflow	llvm-autofix-mini

The tool interface is deliberately LLVM-specific. Generic utilities include recursive search and line-based file reading, but the distinctive tools are code(func: str), which shows the source of an LLVM C++ function; langref(inst: str), which retrieves the relevant LLVM IR language-reference entry; debug(cmd: str) and eval(expr: str), which operate against a paused opt instance under GDB; and test(), which rebuilds LLVM and runs the reproducer and selected regression suites. The harness also orchestrates debugger setup by placing breakpoints at the crashing function for crash bugs or at the first transformation point for a miscompilation bug (Zheng et al., 20 Mar 2026).

This architecture is motivated by failure modes observed in generic agents. Mini-SWE-agent, even when augmented with LLVM-AutoFix tools, retains unrestricted shell behavior and can corrupt the repository state, trigger context blowups, or exit without substantive repair attempts. By contrast, llvm-autofix-mini is limited to the harness toolset, which reduces destructive behaviors and makes the agent’s interaction space closer to the workflow of a human LLVM engineer diagnosing a pass-level defect (Zheng et al., 20 Mar 2026).

3. Benchmark construction and evaluation protocol

The benchmark associated with LLVM-AutoFix is llvm-bench, a curated dataset of 334 reproducible LLVM middle-end bugs collected from fixed LLVM issues. It contains 222 crash bugs and 112 miscompilation bugs, with a live subset of 229 issues intended to reduce data-leakage concerns by focusing on the last year’s bugs. Each benchmark instance includes a base commit, one or more minimal IR reproducers, a validated golden patch derived from the LLVM developers’ fix, and extensive regression tests (Zheng et al., 20 Mar 2026).

The construction pipeline has three stages. First, issue collection filters LLVM GitHub issues closed as fixed since January 1, 2024, keeping only middle-end bugs labeled miscompilation or crash-on-valid and excluding wontfix, duplicate, and invalid. Second, reproducer validation checks that the base commit builds and that the extracted IR reproducer actually triggers the failure. Third, golden patch validation applies the developer patch and confirms that the repaired build passes the harness reproduction and test pipeline. Issues failing any stage are dropped (Zheng et al., 20 Mar 2026).

The benchmark is unusually detailed. Across the full set, bugs directly affect 64 middle-end components, with the most frequent being LoopVectorize at 79 cases, SLPVectorizer at 78, and InstCombine at 53. Each bug has on average 1.4 reproducers, about 722 component-specific regression tests, and more than 10,000 shared tests. Patch complexity is stratified into easy, medium, and hard splits: easy bugs comprise 255 instances and typically require a single-function, single-file fix; medium bugs comprise 44 instances with multi-function fixes in one file; hard bugs comprise 35 instances and involve cross-file repair (Zheng et al., 20 Mar 2026).

Evaluation uses pass@1-style resolution accounting. An accepted patch is one generated within resource limits that passes the harness’s online and offline tests. The paper then adds a stricter expert-validation layer: % Correct measures the proportion of accepted patches judged semantically correct by an LLVM maintainer, and a post-review resolved rate measures the proportion of all attempted bugs that are both accepted by the harness and confirmed correct by expert review. This distinction is central because compiler-repair evaluation is highly vulnerable to false positives that survive ordinary regression testing (Zheng et al., 20 Mar 2026).

4. The llvm-autofix-mini agent and empirical performance

llvm-autofix-mini follows a four-stage workflow: Setup, Reason, Generate, and Validate. In Setup, the harness validates the reproducer, builds LLVM, launches opt under GDB, and infers the erroneous component. In Reason, the agent uses ReAct-style tool invocation to inspect stack frames, local variables, C++ implementations, and LangRef entries until it converges on a candidate root cause and edit point. In Generate, it issues edit operations and calls test() iteratively. In Validate, the harness performs more exhaustive offline regression checking (Zheng et al., 20 Mar 2026).

This specialization materially improves pre-review success for most models. On llvm-bench live, GPT-5 rises from 21.0% resolved with mini-SWE-agent to 51.5% with llvm-autofix-mini; Qwen3 Max rises from 24.4% to 35.8%; Gemini 2.5 Pro from 9.2% to 14.4%; and GPT-4o from 8.3% to 12.2%. DeepSeek V3.2 is the exception, falling from 38.9% to 10.5% because it systematically struggled with the tool-calling format under the specialized agent. The paper summarizes the overall gain as approximately 22% over the state of the art, and a McNemar test reported in the appendix finds the improvement statistically significant for GPT-5, Gemini 2.5 Pro, and Qwen3 Max at $\alpha = 0.05$ (Zheng et al., 20 Mar 2026).

The harder result is that pre-review resolution substantially overstates real repair capability. Expert review shows that many accepted patches are semantically wrong. For mini-SWE-agent, GPT-5 falls from 21.0% pre-review resolved to 6.6% genuinely resolved, and DeepSeek V3.2 falls from 38.9% to 14.4%. For llvm-autofix-mini, GPT-5 falls from 51.5% to 20.1%, while Qwen3 Max falls from 35.8% to 13.1%. The reported % Correct values for accepted patches remain low: 31.3% for GPT-5 with mini-SWE-agent, 39.0% for GPT-5 with llvm-autofix-mini, and 36.6% for Qwen3 Max with llvm-autofix-mini (Zheng et al., 20 Mar 2026).

The qualitative error taxonomy explains this gap. Some patches overfit the reproducer by disabling an optimization for a narrow observed pattern rather than restoring the underlying invariant. Others bypass assertions or insert early exits that merely avoid the failing path. Still others localize the bug to the wrong file or function and produce coherent but irrelevant edits. The paper also reports a positive case: for an InstCombine miscompilation, llvm-autofix-mini with GPT-5 produced a patch judged by the LLVM expert to be correct and even superior to the golden patch for that reproducer, demonstrating that the harness can sometimes support genuinely nontrivial compiler reasoning (Zheng et al., 20 Mar 2026).

5. Validation, trustworthiness, and limits of test-based repair

LLVM-AutoFix is built around the observation that compiler patches are unusually easy to validate incorrectly. A patch can pass the reproducer and even a large regression pool while still being semantically unsound, for example by weakening an assertion, suppressing a transformation in an overbroad but test-compatible way, or exploiting incomplete coverage in existing tests. The harness therefore distinguishes online validation, offline validation, and expert review rather than treating test-suite success as a sufficient endpoint (Zheng et al., 20 Mar 2026).

This concern aligns with earlier formal work on LLVM reasoning. The Codewalker-based ACL2 study models a subset of LLVM instructions in an interpreter ll2, derives semantic functions with correctness theorems of the form $\text{LL2}(s,\text{CLK}(s)) = \text{SEM}(s)$ under explicit invariants, and proves correctness of an LLVM-level occurrence-counting loop against a higher-level specification. That work is limited in instruction coverage and requires manual LLVM-to-LL2 translation, but it provides a contrasting validation regime in which patch acceptance is tied to theorem-prover semantics rather than regression adequacy (Hardin, 2015).

The limitations stated for LLVM-AutoFix are correspondingly broad. The current harness is centered on LLVM middle-end crash and miscompilation bugs, not yet on front-end defects, backend code-generation defects, missed optimizations, slow compiles, or hangs as a primary evaluation target. Resource costs remain substantial, at roughly 15–25 minutes and 1.3–4.7 million tokens per bug. Medium and hard bugs frequently hit token or tool-call limits, and hard bugs are almost never repaired correctly; among the reported results, only GPT-5 with llvm-autofix-mini fixes one hard issue correctly after expert review (Zheng et al., 20 Mar 2026).

These results imply that LLVM-AutoFix is best understood as an infrastructure layer for disciplined experimentation in compiler repair rather than as a fully autonomous maintainer. Its central contribution is not merely higher pass rates, but a methodology for exposing where current agents fail: in localization, invariant recovery, semantic validation, and cost-effective context management (Zheng et al., 20 Mar 2026).

6. Broader LLVM-AutoFix ecosystem and adjacent research directions

Although llvm-autofix is the first paper in the provided corpus to name a concrete LLVM harness directly, adjacent work outlines a broader LLVM-AutoFix ecosystem in which different subsystems address test validity, localization, optimization repair, warning repair, and CI reconstruction. ReFuzzer, for example, sits between LLM-based program generation and LLVM/Clang, compiles and runs generated C/C++ tests under sanitizers, and uses a local LLM to repair static and dynamic invalidities. On dual-GPU hardware it raises test validity from 47.0–49.4% to 96.6–97.3%, with vectorization coverage gains of 9.2%, 2.3%, and 7.1% for black-, grey-, and white-box fuzzing respectively. The paper explicitly presents this as a blueprint for a broader LLVM-AutoFix framework in which repaired tests become the foundation for downstream bug detection and possibly compiler-patch synthesis (Shree et al., 5 Aug 2025).

A complementary localization front-end is provided by AutoCBI. That system uses LLM-generated summaries of compiler source files, coverage-derived suspicious-file rankings, failing test programs, and compilation outputs to rerank LLVM or GCC source files for bug isolation. On 60 LLVM bugs it reports Top-1 localization for 22 bugs, Top-5 for 30, Top-10 for 39, and Top-20 for 50, with MFR 8.70 and MAR 10.25. The synthesis explicitly frames this as a file-level narrowing stage for LLVM-AutoFix, reducing the search space before patch generation (Qi et al., 21 Jun 2025).

LLVM-AutoFix can also refer to automated repair of optimization behavior rather than source-level bug correction. The New Pass Manager auto-tuning framework defines a context-free grammar for valid nested pass pipelines, represents pipelines as forests, mines a Synergy Knowledge Graph of pass interactions, and uses a structure-aware genetic algorithm to search only valid candidates. On LLVM 18.1.6 it reports an average 13.62% additional instruction-count reduction over opt -Oz. The paper’s synthesis states that an LLVM-AutoFix-style system can use this machinery to “repair” suboptimal optimization behavior by generating valid nested pipelines tailored to a workload (Pan et al., 15 Oct 2025).

Other adjacent lines target narrower but industrially relevant repair scopes. An LLM-based repair pipeline for C++ implicit data loss warnings uses clangd via LSP, Tree-sitter context extraction, self-consistent decisions about whether range checks are needed, and iterative validation through clangd diagnostics; on 110 code snippets from SAP HANA it achieves a 92.73% human-acceptance rate and reduces the number of fixes that add instructions by 39.09% relative to a baseline strategy (You et al., 21 Jan 2026). PhantomRun reconstructs CI jobs for embedded projects in Docker, parses build logs, uses LLMs plus historical human fix examples to synthesize patches, and repairs up to 45% of CI compilation failures across the targeted projects. Its synthesis presents containerized CI reproduction, structured log parsing, and bounded iterative validation as design elements readily transferable to LLVM/Clang-centric repair systems (Fu et al., 23 Feb 2026).

Taken together, these systems suggest that “LLVM-AutoFix” denotes not only a single harness but a research program. One branch focuses on real LLVM bug repair with agentic tooling; others provide high-validity test generation, file-level localization, optimization-pipeline synthesis, warning remediation, CI reconstruction, and formal semantic checking. A plausible implication is that future LLVM repair systems will be compositional rather than monolithic, combining these capabilities into a multi-stage pipeline (Zheng et al., 20 Mar 2026).

7. Prospects and open directions

The main open problem for LLVM-AutoFix is trustworthy validation. The harness results show that compiler-specific tools substantially improve repair attempts, yet expert review still rejects a large fraction of accepted patches. This suggests that future progress depends less on unconstrained code generation than on stronger oracles: detection of assertion bypasses, methods to surface missed optimizations or silent semantic weakening, and broader automated challenge suites that complement existing LLVM regression tests (Zheng et al., 20 Mar 2026).

Several concrete extensions are already suggested by adjacent work. ReFuzzer points to a test-side foundation in which sanitizer-clean, compiler-interesting inputs are generated and repaired automatically before they are used for crash or miscompilation discovery (Shree et al., 5 Aug 2025). AutoCBI suggests an interactive or multi-source localization stage that ranks suspicious LLVM files before patch synthesis (Qi et al., 21 Jun 2025). The New Pass Manager auto-tuning framework suggests that performance regressions may sometimes be “fixed” not by editing LLVM source but by synthesizing a better valid nested pipeline (Pan et al., 15 Oct 2025). Codewalker indicates a longer-term route toward semantics-aware validation in which at least some LLVM transformations can be reasoned about with machine-checked invariants rather than regression outcomes alone (Hardin, 2015).

The conceptual trajectory is therefore twofold. In the narrow sense, LLVM-AutoFix is a concrete agentic harness, benchmark, and minimal agent for real LLVM middle-end bugs. In the broader sense, it is an organizing label for automated LLVM repair research that spans valid test construction, bug isolation, pass-pipeline repair, warning remediation, CI reconstruction, and semantic validation. The present evidence indicates that specialized harnesses are necessary, generic software-repair benchmarks are insufficient, and expert review remains indispensable; it also indicates that the required substrate for richer LLVM-centered auto-repair now exists in interoperable form across several research strands (Zheng et al., 20 Mar 2026).