LLVM-AutoFix-Mini: Minimal LLVM Bug Repair Agent
- LLVM-AutoFix-Mini is a minimal, LLVM-specific LLM agent designed to repair middle-end crashes and miscompilations using a four-stage pipeline.
- It employs a structured workflow—Setup, Reason, Generate, and Validate—to reproduce bugs, perform root-cause analysis, synthesize patches, and execute rigorous regression tests.
- Evaluation shows that LLVM-AutoFix-Mini outperforms baseline agents by up to 1.22×, although challenges remain in ensuring semantically correct patch validation.
LLVM-AutoFix-Mini is a minimal agent tailored for fixing LLVM middle-end bugs inside the broader llvm-autofix harness. It is designed as a simple but effective agent that drives compiler-specific tools to reproduce, understand, patch, and validate real-world LLVM bugs, especially crashes and miscompilations arising in LLVM’s middle-end passes (Zheng et al., 20 Mar 2026).
1. Definition and problem setting
LLVM-AutoFix-Mini is defined in the literature as the paper’s “tailored, minimal LLM agent for fixing LLVM bugs” within an agentic harness for real-world compilers. Its target domain is not generic software maintenance but LLVM middle-end bug repair, where issue reports are typically sparse, non-descriptive, and centered on reproducers, stack traces, or translation-validation counterexamples rather than explanatory prose. The benchmark used alongside it focuses on crash bugs, such as compiler assertions or crashes during optimization, and miscompilation bugs, where an optimization transforms one LLVM IR program into a non-equivalent one (Zheng et al., 20 Mar 2026).
The motivating claim is that compiler bugs are qualitatively harder than ordinary software bugs. The reported evaluation shows a performance decline of about 60% in frontier models when tackling compiler bugs compared with common software bugs, and the same study attributes this to sparse reports, specialized compiler knowledge requirements, and the need for compiler-specific runtime context. LLVM is the chosen substrate because it is “one of the most widely used compiler infrastructures,” and the middle-end is both target-independent and heavily populated with subtle optimization logic, including components such as LoopVectorize, SLPVectorizer, and InstCombine (Zheng et al., 20 Mar 2026).
This framing places LLVM-AutoFix-Mini in a distinct category from general-purpose repair agents. It is not merely an LLM wrapped around a repository; it is an LLVM-aware agent whose design presupposes familiarity with opt, LLVM IR semantics, translation validation, component-specific regression tests, and pass-local debugging.
2. Four-stage architecture and control model
LLVM-AutoFix-Mini is a single-agent system organized around an explicit four-stage pipeline: Setup, Reason, Generate, and Validate. The agent is deliberately minimal rather than multi-agent, and it uses ReAct-style loops in the Reason and Generate stages. Its design restricts interaction to a limited, LLVM-focused tool interface rather than unrestricted shell access (Zheng et al., 20 Mar 2026).
| Stage | Primary role | Salient operations |
|---|---|---|
| Setup | Reproduce and instrument bug context | validate reproducer, launch opt under gdb, set pass-specific breakpoint |
| Reason | Root-cause analysis | inspect IR, source, docs, stack frames, internal state |
| Generate | Patch synthesis and iteration | edit source, preview diff, run online tests, reset if needed |
| Validate | Final acceptance | offline regression testing and semantic checking |
In Setup, the harness validates that the reproducer still triggers the bug at the buggy base commit. For crash bugs, it sets a breakpoint just before the crashing function; for miscompilation bugs, it pauses before the first transformation in the implicated pass. The harness also infers the erroneous component from the stack trace or pass pipeline and passes that information to the model (Zheng et al., 20 Mar 2026).
In Reason, the agent enters a ReAct loop over compiler-specific evidence. The system prompt includes the reproducer, the failure signal, and the paused debug context. For crashes, the signal is typically a stack trace; for miscompilations, it can be an Alive2 mismatch together with a concrete counterexample. The purpose of this stage is root-cause analysis rather than patch production. The prompt explicitly instructs the model not to treat assertion suppression as a legitimate fix, because the assertion failure is regarded as a symptom of an earlier semantic defect (Zheng et al., 20 Mar 2026).
In Generate, the agent starts a new ReAct loop focused on patch synthesis. The prompt includes the Stage II root-cause analysis and suspected code locations. The agent is instructed to propose minimal changes, apply them through controlled edits, and validate them incrementally. The reported policy is to test early and often, while avoiding broad refactoring or assertion weakening (Zheng et al., 20 Mar 2026).
In Validate, the harness performs offline checking after online tests succeed. This stage uses broader regression tests and, for miscompilations, Alive2-based semantic validation. A patch is accepted only if it passes all online and offline checks (Zheng et al., 20 Mar 2026).
3. Tooling, prompts, and execution discipline
LLVM-AutoFix-Mini operates through a structured tool interface. The generic tools are find, grep, list, read, edit, and preview. The LLVM-specific tools are code, docs, langref, debug, eval, reset, and test. This interface is narrower than the one given to mini-SWE-agent, and the restriction is intentional: the agent is not allowed to use arbitrary shell commands directly, which reduces failure modes such as corrupting the repository state (Zheng et al., 20 Mar 2026).
The debug and eval tools are central to the design. debug runs gdb commands in the current debug session, while eval evaluates expressions in the current debug context. The prompts explicitly encourage using eval first when inspecting optimizer state. This makes the agent’s reasoning grounded in pass-local invariants, IR objects, and analysis results rather than in source reading alone (Zheng et al., 20 Mar 2026).
The test tool encapsulates online validation. It rebuilds LLVM as needed and runs the current patch against the reproducer and relevant component tests. This allows the agent to search within a constrained patch space while receiving compiler-specific feedback after each edit. In the reported configuration, the agent operates with temperature 0, a 64k context window, up to 500 chat rounds, a 5M-token budget, and limits of at most 25 edit calls and 25 test calls (Zheng et al., 20 Mar 2026).
Prompt design is also specialized. The Reason-stage prompt instructs the model to understand the issue, analyze the opt log, connect the symptom to a root cause, propose detailed edit points, and summarize the intended fix effect. The Generate-stage prompt repeats the reproducer and symptom, injects the prior root-cause analysis, highlights suspected code locations, and directs the model to iterate through edit–test cycles. This prompt discipline encodes a specific repair philosophy: localization and semantic diagnosis should precede patch synthesis, and patch synthesis should be continuously constrained by regression and semantic tests (Zheng et al., 20 Mar 2026).
4. Benchmark, evaluation protocol, and empirical performance
LLVM-AutoFix-Mini is evaluated on llvm-bench, a benchmark of reproducible LLVM middle-end bugs. The full set contains 334 issues, including 222 crashes and 112 miscompilations, spanning 64 directly affected middle-end components. The largest affected components are LoopVectorize with 79 bugs, SLPVectorizer with 78, and InstCombine with 53. The benchmark also records average golden patch complexity as 17.1 edited lines, 1.6 functions, and 1.2 files (Zheng et al., 20 Mar 2026).
The benchmark defines difficulty splits. In the full set, the easy split contains 255 issues with an average patch size of 9.1 lines and modifies a single function in a single file; the medium split contains 44 issues with an average patch size of 38.8 lines across 2.6 functions in one file; the hard split contains 35 issues with an average patch size of 47.6 lines across 4.9 functions in 2.7 files. The live subset, llvm-bench-live, contains 229 issues, including 160 crashes and 69 miscompilations (Zheng et al., 20 Mar 2026).
The principal success metric is percentage resolved, defined as the fraction of issues for which the agent produces an accepted patch. The reported comparison against mini-SWE-agent on llvm-bench-live is summarized below.
| Model | mini-SWE-agent | LLVM-AutoFix-Mini |
|---|---|---|
| GPT-4o | 8.3% | 12.2% |
| GPT-5 | 21.0% | 51.5% |
| Gemini 2.5 Pro | 9.2% | 14.4% |
| Qwen 3 Max | 24.4% | 35.8% |
| DeepSeek V3.2 | 38.9% | 10.5% |
The abstract reports that LLVM-AutoFix-Mini outperforms the state of the art by approximately 22% per model, while the main text states that its performance is generally 1.22× that of mini-SWE-agent, with the greatest improvement occurring with GPT-5, where resolution rises from 21.0% to 51.5% (Zheng et al., 20 Mar 2026). The same study reports that generic agents experience an average 62% drop in resolution rate when տեղափոխed from SWE-bench Verified to llvm-bench-live, reinforcing the claim that compiler-specific harnesses are necessary (Zheng et al., 20 Mar 2026).
The evaluation also includes expert review of accepted patches. For GPT-5 with LLVM-AutoFix-Mini, 51.5% of issues receive accepted patches, 39.0% of accepted patches are judged truly correct, and the resulting genuine resolution rate is 20.1% of issues. The study further notes that miscompilations are harder than crashes by roughly 13–14 percentage points, and that hard multi-file bugs remain particularly challenging (Zheng et al., 20 Mar 2026).
5. Relationship to adjacent LLVM auto-fixing and transformation research
LLVM-AutoFix-Mini, as explicitly defined, is the minimal agent inside llvm-autofix. At the same time, adjacent LLVM research describes several compact subsystems that can be read as complementary design patterns for an extended LLVM-AutoFix-Mini ecosystem.
A file-localization front end is suggested by AutoCBI, which performs compiler bug isolation at file level by combining four information sources: the failing test program, source-file function summaries, suspicious file lists from coverage-based localization, and compilation configurations with related outputs. On 60 LLVM bugs, AutoCBI localizes 22 bugs at Top-1, 30 at Top-5, 39 at Top-10, and 50 at Top-20, and improves Top-1 isolation over RecBi by 69.23% for LLVM (Qi et al., 21 Jun 2025). This suggests that a localization stage of that kind could precede LLVM-AutoFix-Mini’s patch synthesis stage when the faulty component is not already known.
A validity-refinement front end is suggested by ReFuzzer, which inserts a feedback loop between LLM-generated tests and LLVM/Clang. ReFuzzer uses clang diagnostics and sanitizer outputs to refine erroneous C/C++ tests with a local LLM, improving validity from 47.0–49.4% to 96.6–97.3%, with an average processing time of 2.9–3.5 s per test program on a dual-GPU machine (Shree et al., 5 Aug 2025). A plausible implication is that similar static and dynamic validity filters could be used to sanitize generated reproducers or regression tests before they are fed into LLVM-AutoFix-Mini.
A binary-lifting and normalization variant is suggested by SATURN, an LLVM-based deobfuscation framework that lifts binaries to LLVM-IR with Remill, reconstructs control flow, uses LLVM optimizations together with Souper and Z3, performs stack aliasing and argument recovery, and recompiles deobfuscated code (Garba et al., 2019). This suggests a path for extending LLVM-AutoFix-Mini beyond source-visible LLVM bugs toward “binary-to-IR-to-binary” repair workflows.
A pass-pipeline repair or tuning variant is suggested by the work on synergy-guided compiler auto-tuning of nested LLVM pass pipelines. That framework introduces a formal grammar for valid nested New Pass Manager pipelines, represents pipelines as forests, and uses a structure-aware genetic algorithm that manipulates those forests directly, guaranteeing syntactic validity by construction. On seven benchmark datasets with LLVM 18.1.6, it reports an average 13.62% additional instruction count reduction compared to opt -Oz (Pan et al., 15 Oct 2025). This suggests that validity-by-construction search over New Pass Manager pipelines could complement bug repair when the defect is rooted in pass sequencing rather than in pass implementation.
A higher-level analysis substrate is suggested by NOELLE, which provides LLVM extensions such as a Program Dependence Graph, complete call graph, loop abstractions, a Data Flow Engine, schedulers, and a Loop Builder. The paper reports that ten custom tools built on NOELLE achieve a 33.2% to 99.2% reduction in code size compared to counterparts without it (Matni et al., 2021). A plausible implication is that NOELLE-style abstractions could reduce engineering cost for analysis-heavy variants of LLVM-AutoFix-Mini.
6. Limitations, failure modes, and future directions
The principal limitation reported for LLVM-AutoFix-Mini is patch validation insufficiency. Existing LLVM regression tests and Alive2 checks do not catch all incorrect patches, and expert review shows that many accepted patches are semantically wrong. The study explicitly identifies failure modes such as ChangeAssert, WrongLocalization, and WrongFix, indicating that passing the available validation harness is not equivalent to producing a correct compiler repair (Zheng et al., 20 Mar 2026).
The current scope is also narrow. The main evaluation targets middle-end crashes and miscompilations; front-end bugs, back-end bugs, and broader performance issues are not comprehensively covered. The paper additionally notes context-rot effects in long Reason/Generate loops and shows that even frontier models still lack robust understanding of LLVM invariants and semantics. Increasing token budgets does not eliminate these difficulties (Zheng et al., 20 Mar 2026).
The reported future directions include more sophisticated agents, richer tool integration, automatic detection of “cheating” patches that weaken assertions or bypass components, better context management, and extension to more bug classes and compiler subsystems (Zheng et al., 20 Mar 2026). Related localization work points toward more precise function-level or statement-level isolation rather than file-level ranking alone (Qi et al., 21 Jun 2025). Related transformation work suggests that pairwise synergy models, structural refinement policies, and richer IR abstractions remain incomplete approximations rather than final solutions (Pan et al., 15 Oct 2025, Matni et al., 2021).
Taken together, these results position LLVM-AutoFix-Mini as both a practical baseline and a research scaffold. In its strict sense, it is a minimal, compiler-specific LLM agent for LLVM middle-end repair. In a broader methodological sense, it sits at the intersection of LLVM-specific localization, validity filtering, IR normalization, analysis-driven transformation, and pass-pipeline search, with the common theme that compiler repair requires domain-specific structure rather than generic code-editing capability alone.