---
title: 'LLVM-AutoFix: Agentic Harness for LLVM Bug Repair'
url: https://www.emergentmind.com/topics/llvm-autofix
type: topic
---

# LLVM-AutoFix: Agentic Harness for LLVM Bug Repair

LLVM-AutoFix, written in the primary paper as `llvm-autofix`, is an agentic harness for LLVM that is designed to help large language model agents understand, reproduce, debug, and repair real compiler bugs rather than toy defects or generic repository issues. Its core elements are agent-friendly LLVM tools, a curated benchmark of reproducible LLVM middle-end bugs named `llvm-bench`, and a specialized minimal agent named `llvm-autofix-mini`. In the evaluation reported for the system, frontier models suffer a roughly 60% performance decline when moving from common software-bug benchmarks to LLVM compiler bugs, while the specialized harness and agent recover a substantial portion of that gap, underscoring the need for LLVM-specific tooling, validation, and debugging workflows [2603.20075].

## 1. Definition and problem setting

LLVM-AutoFix addresses the automated repair of LLVM compiler bugs under conditions that differ sharply from ordinary software maintenance. The target defects are middle-end LLVM bugs, especially crash bugs and miscompilation bugs, and the input is not a rich natural-language issue description but typically a minimal LLVM IR reproducer, an `opt` invocation, an assertion or stack trace, or an Alive2 counterexample showing that an optimization is unsound. In this setting, a successful repair must modify LLVM C++ sources so that the reproducer no longer crashes or miscompiles while also passing relevant regression tests [2603.20075].

The motivating difficulty is structural. LLVM bugs are associated with sparse, non-descriptive reports, deep cross-domain expertise requirements, a large and fragile C++ build environment, and validation regimes in which passing a narrow regression suite does not imply semantic correctness. The reported benchmark results make this difference concrete: with mini-SWE-agent, GPT-5 drops from 65.0% on SWE-bench Verified to 21.0% on `llvm-bench live`, Gemini 2.5 Pro from 53.6% to 9.2%, Qwen3 Max from 69.6% to 24.4%, DeepSeek V3.2 from 60.0% to 38.9%, and GPT-4o from 21.6% to 8.3%, yielding an average performance decline of about 60% when moving from common software bugs to LLVM compiler bugs [2603.20075].

A defining characteristic of LLVM-AutoFix is that it treats LLVM not as a generic repository to be edited with shell access, but as a specialized debugging environment. This shifts the repair problem from open-ended code generation toward controlled interaction with LLVM-specific artifacts such as IR semantics, pass internals, GDB state, Alive2 validation, and `llvm-lit` regression infrastructure. The result is a harness-centered conception of automated repair rather than a prompt-only conception [2603.20075].

## 2. System architecture and harness design

LLVM-AutoFix is organized as an agentic harness: a layer that standardizes reproduction, debugging, patch generation, and validation for LLVM bugs. The harness exposes constrained tools rather than unconstrained shell access, enabling the agent to search code, inspect functions, query the LangRef, debug a live `opt` process, edit files, preview patches, reset the tree, and run an integrated online test procedure [2603.20075].

| Component | Function | Concrete contents |
|---|---|---|
| Agent-friendly tools | Provide structured interaction with LLVM | `find`, `grep`, `read`, `edit`, `preview`, `code`, `docs`, `langref`, `debug`, `eval`, `reset`, `test` |
| Benchmark layer | Supply reproducible repair tasks | `llvm-bench` and `llvm-bench live` |
| Specialized agent | Execute LLVM-centered repair workflow | `llvm-autofix-mini` |

The tool interface is deliberately LLVM-specific. Generic utilities include recursive search and line-based file reading, but the distinctive tools are `code(func: str)`, which shows the source of an LLVM C++ function; `langref(inst: str)`, which retrieves the relevant LLVM IR language-reference entry; `debug(cmd: str)` and `eval(expr: str)`, which operate against a paused `opt` instance under GDB; and `test()`, which rebuilds LLVM and runs the reproducer and selected regression suites. The harness also orchestrates debugger setup by placing breakpoints at the crashing function for crash bugs or at the first transformation point for a miscompilation bug [2603.20075].

This architecture is motivated by failure modes observed in generic agents. Mini-SWE-agent, even when augmented with LLVM-AutoFix tools, retains unrestricted shell behavior and can corrupt the repository state, trigger context blowups, or exit without substantive repair attempts. By contrast, `llvm-autofix-mini` is limited to the harness toolset, which reduces destructive behaviors and makes the agent’s interaction space closer to the workflow of a human LLVM engineer diagnosing a pass-level defect [2603.20075].

## 3. Benchmark construction and evaluation protocol

The benchmark associated with LLVM-AutoFix is `llvm-bench`, a curated dataset of 334 reproducible LLVM middle-end bugs collected from fixed LLVM issues. It contains 222 crash bugs and 112 miscompilation bugs, with a live subset of 229 issues intended to reduce data-leakage concerns by focusing on the last year’s bugs. Each benchmark instance includes a base commit, one or more minimal IR reproducers, a validated golden patch derived from the LLVM developers’ fix, and extensive regression tests [2603.20075].

The construction pipeline has three stages. First, issue collection filters LLVM GitHub issues closed as fixed since January 1, 2024, keeping only middle-end bugs labeled `miscompilation` or `crash-on-valid` and excluding `wontfix`, `duplicate`, and `invalid`. Second, reproducer validation checks that the base commit builds and that the extracted IR reproducer actually triggers the failure. Third, golden patch validation applies the developer patch and confirms that the repaired build passes the harness reproduction and test pipeline. Issues failing any stage are dropped [2603.20075].

The benchmark is unusually detailed. Across the full set, bugs directly affect 64 middle-end components, with the most frequent being LoopVectorize at 79 cases, SLPVectorizer at 78, and InstCombine at 53. Each bug has on average 1.4 reproducers, about 722 component-specific regression tests, and more than 10,000 shared tests. Patch complexity is stratified into easy, medium, and hard splits: easy bugs comprise 255 instances and typically require a single-function, single-file fix; medium bugs comprise 44 instances with multi-function fixes in one file; hard bugs comprise 35 instances and involve cross-file repair [2603.20075].

Evaluation uses pass@1-style resolution accounting. An accepted patch is one generated within resource limits that passes the harness’s online and offline tests. The paper then adds a stricter expert-validation layer: `% Correct` measures the proportion of accepted patches judged semantically correct by an LLVM maintainer, and a post-review resolved rate measures the proportion of all attempted bugs that are both accepted by the harness and confirmed correct by expert review. This distinction is central because compiler-repair evaluation is highly vulnerable to false positives that survive ordinary regression testing [2603.20075].

## 4. The `llvm-autofix-mini` agent and empirical performance

`llvm-autofix-mini` follows a four-stage workflow: Setup, Reason, Generate, and Validate. In Setup, the harness validates the reproducer, builds LLVM, launches `opt` under GDB, and infers the erroneous component. In Reason, the agent uses ReAct-style tool invocation to inspect stack frames, local variables, C++ implementations, and LangRef entries until it converges on a candidate root cause and edit point. In Generate, it issues `edit` operations and calls `test()` iteratively. In Validate, the harness performs more exhaustive offline regression checking [2603.20075].

This specialization materially improves pre-review success for most models. On `llvm-bench live`, GPT-5 rises from 21.0% resolved with mini-SWE-agent to 51.5% with `llvm-autofix-mini`; Qwen3 Max rises from 24.4% to 35.8%; Gemini 2.5 Pro from 9.2% to 14.4%; and GPT-4o from 8.3% to 12.2%. DeepSeek V3.2 is the exception, falling from 38.9% to 10.5% because it systematically struggled with the tool-calling format under the specialized agent. The paper summarizes the overall gain as approximately 22% over the state of the art, and a McNemar test reported in the appendix finds the improvement statistically significant for GPT-5, Gemini 2.5 Pro, and Qwen3 Max at $\alpha = 0.05$ [2603.20075].

The harder result is that pre-review resolution substantially overstates real repair capability. Expert review shows that many accepted patches are semantically wrong. For mini-SWE-agent, GPT-5 falls from 21.0% pre-review resolved to 6.6% genuinely resolved, and DeepSeek V3.2 falls from 38.9% to 14.4%. For `llvm-autofix-mini`, GPT-5 falls from 51.5% to 20.1%, while Qwen3 Max falls from 35.8% to 13.1%. The reported `% Correct` values for accepted patches remain low: 31.3% for GPT-5 with mini-SWE-agent, 39.0% for GPT-5 with `llvm-autofix-mini`, and 36.6% for Qwen3 Max with `llvm-autofix-mini` [2603.20075].

The qualitative error taxonomy explains this gap. Some patches overfit the reproducer by disabling an optimization for a narrow observed pattern rather than restoring the underlying invariant. Others bypass assertions or insert early exits that merely avoid the failing path. Still others localize the bug to the wrong file or function and produce coherent but irrelevant edits. The paper also reports a positive case: for an InstCombine miscompilation, `llvm-autofix-mini` with GPT-5 produced a patch judged by the LLVM expert to be correct and even superior to the golden patch for that reproducer, demonstrating that the harness can sometimes support genuinely nontrivial compiler reasoning [2603.20075].

## 5. Validation, trustworthiness, and limits of test-based repair

LLVM-AutoFix is built around the observation that compiler patches are unusually easy to validate incorrectly. A patch can pass the reproducer and even a large regression pool while still being semantically unsound, for example by weakening an assertion, suppressing a transformation in an overbroad but test-compatible way, or exploiting incomplete coverage in existing tests. The harness therefore distinguishes online validation, offline validation, and expert review rather than treating test-suite success as a sufficient endpoint [2603.20075].

This concern aligns with earlier formal work on LLVM reasoning. The Codewalker-based ACL2 study models a subset of LLVM instructions in an interpreter `ll2`, derives semantic functions with correctness theorems of the form $\text{LL2}(s,\text{CLK}(s)) = \text{SEM}(s)$ under explicit invariants, and proves correctness of an LLVM-level occurrence-counting loop against a higher-level specification. That work is limited in instruction coverage and requires manual LLVM-to-LL2 translation, but it provides a contrasting validation regime in which patch acceptance is tied to theorem-prover semantics rather than regression adequacy [1509.06083].

The limitations stated for LLVM-AutoFix are correspondingly broad. The current harness is centered on LLVM middle-end crash and miscompilation bugs, not yet on front-end defects, backend code-generation defects, missed optimizations, slow compiles, or hangs as a primary evaluation target. Resource costs remain substantial, at roughly 15–25 minutes and 1.3–4.7 million tokens per bug. Medium and hard bugs frequently hit token or tool-call limits, and hard bugs are almost never repaired correctly; among the reported results, only GPT-5 with `llvm-autofix-mini` fixes one hard issue correctly after expert review [2603.20075].

These results imply that LLVM-AutoFix is best understood as an infrastructure layer for disciplined experimentation in compiler repair rather than as a fully autonomous maintainer. Its central contribution is not merely higher pass rates, but a methodology for exposing where current agents fail: in localization, invariant recovery, semantic validation, and cost-effective context management [2603.20075].

## 6. Broader LLVM-AutoFix ecosystem and adjacent research directions

Although `llvm-autofix` is the first paper in the provided corpus to name a concrete LLVM harness directly, adjacent work outlines a broader LLVM-AutoFix ecosystem in which different subsystems address test validity, localization, optimization repair, warning repair, and CI reconstruction. ReFuzzer, for example, sits between LLM-based program generation and LLVM/Clang, compiles and runs generated C/C++ tests under sanitizers, and uses a local LLM to repair static and dynamic invalidities. On dual-GPU hardware it raises test validity from 47.0–49.4% to 96.6–97.3%, with vectorization coverage gains of 9.2%, 2.3%, and 7.1% for black-, grey-, and white-box fuzzing respectively. The paper explicitly presents this as a blueprint for a broader LLVM-AutoFix framework in which repaired tests become the foundation for downstream bug detection and possibly compiler-patch synthesis [2508.03603].

A complementary localization front-end is provided by AutoCBI. That system uses LLM-generated summaries of compiler source files, coverage-derived suspicious-file rankings, failing test programs, and compilation outputs to rerank LLVM or GCC source files for bug isolation. On 60 LLVM bugs it reports Top-1 localization for 22 bugs, Top-5 for 30, Top-10 for 39, and Top-20 for 50, with MFR 8.70 and MAR 10.25. The synthesis explicitly frames this as a file-level narrowing stage for LLVM-AutoFix, reducing the search space before patch generation [2506.17647].

LLVM-AutoFix can also refer to automated repair of optimization behavior rather than source-level bug correction. The New Pass Manager auto-tuning framework defines a context-free grammar for valid nested pass pipelines, represents pipelines as forests, mines a Synergy Knowledge Graph of pass interactions, and uses a structure-aware genetic algorithm to search only valid candidates. On LLVM 18.1.6 it reports an average 13.62% additional instruction-count reduction over `opt -Oz`. The paper’s synthesis states that an LLVM-AutoFix-style system can use this machinery to “repair” suboptimal optimization behavior by generating valid nested pipelines tailored to a workload [2510.13184].

Other adjacent lines target narrower but industrially relevant repair scopes. An LLM-based repair pipeline for C++ implicit data loss warnings uses clangd via LSP, Tree-sitter context extraction, self-consistent decisions about whether range checks are needed, and iterative validation through clangd diagnostics; on 110 code snippets from SAP HANA it achieves a 92.73% human-acceptance rate and reduces the number of fixes that add instructions by 39.09% relative to a baseline strategy [2601.14936]. PhantomRun reconstructs CI jobs for embedded projects in Docker, parses build logs, uses LLMs plus historical human fix examples to synthesize patches, and repairs up to 45% of CI compilation failures across the targeted projects. Its synthesis presents containerized CI reproduction, structured log parsing, and bounded iterative validation as design elements readily transferable to LLVM/Clang-centric repair systems [2602.20284].

Taken together, these systems suggest that “LLVM-AutoFix” denotes not only a single harness but a research program. One branch focuses on real LLVM bug repair with agentic tooling; others provide high-validity test generation, file-level localization, optimization-pipeline synthesis, warning remediation, CI reconstruction, and formal semantic checking. A plausible implication is that future LLVM repair systems will be compositional rather than monolithic, combining these capabilities into a multi-stage pipeline [2603.20075].

## 7. Prospects and open directions

The main open problem for LLVM-AutoFix is trustworthy validation. The harness results show that compiler-specific tools substantially improve repair attempts, yet expert review still rejects a large fraction of accepted patches. This suggests that future progress depends less on unconstrained code generation than on stronger oracles: detection of assertion bypasses, methods to surface missed optimizations or silent semantic weakening, and broader automated challenge suites that complement existing LLVM regression tests [2603.20075].

Several concrete extensions are already suggested by adjacent work. ReFuzzer points to a test-side foundation in which sanitizer-clean, compiler-interesting inputs are generated and repaired automatically before they are used for crash or miscompilation discovery [2508.03603]. AutoCBI suggests an interactive or multi-source localization stage that ranks suspicious LLVM files before patch synthesis [2506.17647]. The New Pass Manager auto-tuning framework suggests that performance regressions may sometimes be “fixed” not by editing LLVM source but by synthesizing a better valid nested pipeline [2510.13184]. Codewalker indicates a longer-term route toward semantics-aware validation in which at least some LLVM transformations can be reasoned about with machine-checked invariants rather than regression outcomes alone [1509.06083].

The conceptual trajectory is therefore twofold. In the narrow sense, LLVM-AutoFix is a concrete agentic harness, benchmark, and minimal agent for real LLVM middle-end bugs. In the broader sense, it is an organizing label for automated LLVM repair research that spans valid test construction, bug isolation, pass-pipeline repair, warning remediation, CI reconstruction, and semantic validation. The present evidence indicates that specialized harnesses are necessary, generic software-repair benchmarks are insufficient, and expert review remains indispensable; it also indicates that the required substrate for richer LLVM-centered auto-repair now exists in interoperable form across several research strands [2603.20075].

Source: https://www.emergentmind.com/topics/llvm-autofix