---
title: 'LLVM-AutoFix-Mini: Minimal LLVM Bug Repair Agent'
url: https://www.emergentmind.com/topics/llvm-autofix-mini
type: topic
---

# LLVM-AutoFix-Mini: Minimal LLVM Bug Repair Agent

LLVM-AutoFix-Mini is a minimal agent tailored for fixing LLVM middle-end bugs inside the broader llvm-autofix harness. It is designed as a simple but effective agent that drives compiler-specific tools to reproduce, understand, patch, and validate real-world LLVM bugs, especially crashes and miscompilations arising in LLVM’s middle-end passes [2603.20075].

## 1. Definition and problem setting

LLVM-AutoFix-Mini is defined in the literature as the paper’s “tailored, minimal LLM agent for fixing LLVM bugs” within an agentic harness for real-world compilers. Its target domain is not generic software maintenance but LLVM middle-end bug repair, where issue reports are typically sparse, non-descriptive, and centered on reproducers, stack traces, or translation-validation counterexamples rather than explanatory prose. The benchmark used alongside it focuses on crash bugs, such as compiler assertions or crashes during optimization, and miscompilation bugs, where an optimization transforms one LLVM IR program into a non-equivalent one [2603.20075].

The motivating claim is that compiler bugs are qualitatively harder than ordinary software bugs. The reported evaluation shows a performance decline of about 60% in frontier models when tackling compiler bugs compared with common software bugs, and the same study attributes this to sparse reports, specialized compiler knowledge requirements, and the need for compiler-specific runtime context. LLVM is the chosen substrate because it is “one of the most widely used compiler infrastructures,” and the middle-end is both target-independent and heavily populated with subtle optimization logic, including components such as LoopVectorize, SLPVectorizer, and InstCombine [2603.20075].

This framing places LLVM-AutoFix-Mini in a distinct category from general-purpose repair agents. It is not merely an LLM wrapped around a repository; it is an LLVM-aware agent whose design presupposes familiarity with `opt`, LLVM IR semantics, translation validation, component-specific regression tests, and pass-local debugging.

## 2. Four-stage architecture and control model

LLVM-AutoFix-Mini is a single-agent system organized around an explicit four-stage pipeline: **Setup**, **Reason**, **Generate**, and **Validate**. The agent is deliberately minimal rather than multi-agent, and it uses ReAct-style loops in the Reason and Generate stages. Its design restricts interaction to a limited, LLVM-focused tool interface rather than unrestricted shell access [2603.20075].

| Stage | Primary role | Salient operations |
|---|---|---|
| Setup | Reproduce and instrument bug context | validate reproducer, launch `opt` under `gdb`, set pass-specific breakpoint |
| Reason | Root-cause analysis | inspect IR, source, docs, stack frames, internal state |
| Generate | Patch synthesis and iteration | edit source, preview diff, run online tests, reset if needed |
| Validate | Final acceptance | offline regression testing and semantic checking |

In **Setup**, the harness validates that the reproducer still triggers the bug at the buggy base commit. For crash bugs, it sets a breakpoint just before the crashing function; for miscompilation bugs, it pauses before the first transformation in the implicated pass. The harness also infers the erroneous component from the stack trace or pass pipeline and passes that information to the model [2603.20075].

In **Reason**, the agent enters a ReAct loop over compiler-specific evidence. The system prompt includes the reproducer, the failure signal, and the paused debug context. For crashes, the signal is typically a stack trace; for miscompilations, it can be an Alive2 mismatch together with a concrete counterexample. The purpose of this stage is root-cause analysis rather than patch production. The prompt explicitly instructs the model not to treat assertion suppression as a legitimate fix, because the assertion failure is regarded as a symptom of an earlier semantic defect [2603.20075].

In **Generate**, the agent starts a new ReAct loop focused on patch synthesis. The prompt includes the Stage II root-cause analysis and suspected code locations. The agent is instructed to propose minimal changes, apply them through controlled edits, and validate them incrementally. The reported policy is to test early and often, while avoiding broad refactoring or assertion weakening [2603.20075].

In **Validate**, the harness performs offline checking after online tests succeed. This stage uses broader regression tests and, for miscompilations, Alive2-based semantic validation. A patch is accepted only if it passes all online and offline checks [2603.20075].

## 3. Tooling, prompts, and execution discipline

LLVM-AutoFix-Mini operates through a structured tool interface. The generic tools are `find`, `grep`, `list`, `read`, `edit`, and `preview`. The LLVM-specific tools are `code`, `docs`, `langref`, `debug`, `eval`, `reset`, and `test`. This interface is narrower than the one given to mini-SWE-agent, and the restriction is intentional: the agent is not allowed to use arbitrary shell commands directly, which reduces failure modes such as corrupting the repository state [2603.20075].

The `debug` and `eval` tools are central to the design. `debug` runs `gdb` commands in the current debug session, while `eval` evaluates expressions in the current debug context. The prompts explicitly encourage using `eval` first when inspecting optimizer state. This makes the agent’s reasoning grounded in pass-local invariants, IR objects, and analysis results rather than in source reading alone [2603.20075].

The `test` tool encapsulates online validation. It rebuilds LLVM as needed and runs the current patch against the reproducer and relevant component tests. This allows the agent to search within a constrained patch space while receiving compiler-specific feedback after each edit. In the reported configuration, the agent operates with temperature 0, a 64k context window, up to 500 chat rounds, a 5M-token budget, and limits of at most 25 `edit` calls and 25 `test` calls [2603.20075].

Prompt design is also specialized. The Reason-stage prompt instructs the model to understand the issue, analyze the `opt` log, connect the symptom to a root cause, propose detailed edit points, and summarize the intended fix effect. The Generate-stage prompt repeats the reproducer and symptom, injects the prior root-cause analysis, highlights suspected code locations, and directs the model to iterate through edit–test cycles. This prompt discipline encodes a specific repair philosophy: localization and semantic diagnosis should precede patch synthesis, and patch synthesis should be continuously constrained by regression and semantic tests [2603.20075].

## 4. Benchmark, evaluation protocol, and empirical performance

LLVM-AutoFix-Mini is evaluated on llvm-bench, a benchmark of reproducible LLVM middle-end bugs. The full set contains 334 issues, including 222 crashes and 112 miscompilations, spanning 64 directly affected middle-end components. The largest affected components are LoopVectorize with 79 bugs, SLPVectorizer with 78, and InstCombine with 53. The benchmark also records average golden patch complexity as 17.1 edited lines, 1.6 functions, and 1.2 files [2603.20075].

The benchmark defines difficulty splits. In the full set, the easy split contains 255 issues with an average patch size of 9.1 lines and modifies a single function in a single file; the medium split contains 44 issues with an average patch size of 38.8 lines across 2.6 functions in one file; the hard split contains 35 issues with an average patch size of 47.6 lines across 4.9 functions in 2.7 files. The live subset, llvm-bench-live, contains 229 issues, including 160 crashes and 69 miscompilations [2603.20075].

The principal success metric is percentage resolved, defined as the fraction of issues for which the agent produces an accepted patch. The reported comparison against mini-SWE-agent on llvm-bench-live is summarized below.

| Model | mini-SWE-agent | LLVM-AutoFix-Mini |
|---|---:|---:|
| GPT-4o | 8.3% | 12.2% |
| GPT-5 | 21.0% | 51.5% |
| Gemini 2.5 Pro | 9.2% | 14.4% |
| Qwen 3 Max | 24.4% | 35.8% |
| DeepSeek V3.2 | 38.9% | 10.5% |

The abstract reports that LLVM-AutoFix-Mini outperforms the state of the art by approximately 22% per model, while the main text states that its performance is generally 1.22× that of mini-SWE-agent, with the greatest improvement occurring with GPT-5, where resolution rises from 21.0% to 51.5% [2603.20075]. The same study reports that generic agents experience an average 62% drop in resolution rate when տեղափոխed from SWE-bench Verified to llvm-bench-live, reinforcing the claim that compiler-specific harnesses are necessary [2603.20075].

The evaluation also includes expert review of accepted patches. For GPT-5 with LLVM-AutoFix-Mini, 51.5% of issues receive accepted patches, 39.0% of accepted patches are judged truly correct, and the resulting genuine resolution rate is 20.1% of issues. The study further notes that miscompilations are harder than crashes by roughly 13–14 percentage points, and that hard multi-file bugs remain particularly challenging [2603.20075].

## 5. Relationship to adjacent LLVM auto-fixing and transformation research

LLVM-AutoFix-Mini, as explicitly defined, is the minimal agent inside llvm-autofix. At the same time, adjacent LLVM research describes several compact subsystems that can be read as complementary design patterns for an extended LLVM-AutoFix-Mini ecosystem.

A file-localization front end is suggested by AutoCBI, which performs compiler bug isolation at file level by combining four information sources: the failing test program, source-file function summaries, suspicious file lists from coverage-based localization, and compilation configurations with related outputs. On 60 LLVM bugs, AutoCBI localizes 22 bugs at Top-1, 30 at Top-5, 39 at Top-10, and 50 at Top-20, and improves Top-1 isolation over RecBi by 69.23% for LLVM [2506.17647]. This suggests that a localization stage of that kind could precede LLVM-AutoFix-Mini’s patch synthesis stage when the faulty component is not already known.

A validity-refinement front end is suggested by ReFuzzer, which inserts a feedback loop between LLM-generated tests and LLVM/Clang. ReFuzzer uses clang diagnostics and sanitizer outputs to refine erroneous C/C++ tests with a local LLM, improving validity from 47.0–49.4% to 96.6–97.3%, with an average processing time of 2.9–3.5 s per test program on a dual-GPU machine [2508.03603]. A plausible implication is that similar static and dynamic validity filters could be used to sanitize generated reproducers or regression tests before they are fed into LLVM-AutoFix-Mini.

A binary-lifting and normalization variant is suggested by SATURN, an LLVM-based deobfuscation framework that lifts binaries to LLVM-IR with Remill, reconstructs control flow, uses LLVM optimizations together with Souper and Z3, performs stack aliasing and argument recovery, and recompiles deobfuscated code [1909.01752]. This suggests a path for extending LLVM-AutoFix-Mini beyond source-visible LLVM bugs toward “binary-to-IR-to-binary” repair workflows.

A pass-pipeline repair or tuning variant is suggested by the work on synergy-guided compiler auto-tuning of nested LLVM pass pipelines. That framework introduces a formal grammar for valid nested New Pass Manager pipelines, represents pipelines as forests, and uses a structure-aware genetic algorithm that manipulates those forests directly, guaranteeing syntactic validity by construction. On seven benchmark datasets with LLVM 18.1.6, it reports an average 13.62% additional instruction count reduction compared to `opt -Oz` [2510.13184]. This suggests that validity-by-construction search over New Pass Manager pipelines could complement bug repair when the defect is rooted in pass sequencing rather than in pass implementation.

A higher-level analysis substrate is suggested by NOELLE, which provides LLVM extensions such as a Program Dependence Graph, complete call graph, loop abstractions, a Data Flow Engine, schedulers, and a Loop Builder. The paper reports that ten custom tools built on NOELLE achieve a 33.2% to 99.2% reduction in code size compared to counterparts without it [2102.05081]. A plausible implication is that NOELLE-style abstractions could reduce engineering cost for analysis-heavy variants of LLVM-AutoFix-Mini.

## 6. Limitations, failure modes, and future directions

The principal limitation reported for LLVM-AutoFix-Mini is patch validation insufficiency. Existing LLVM regression tests and Alive2 checks do not catch all incorrect patches, and expert review shows that many accepted patches are semantically wrong. The study explicitly identifies failure modes such as `ChangeAssert`, `WrongLocalization`, and `WrongFix`, indicating that passing the available validation harness is not equivalent to producing a correct compiler repair [2603.20075].

The current scope is also narrow. The main evaluation targets middle-end crashes and miscompilations; front-end bugs, back-end bugs, and broader performance issues are not comprehensively covered. The paper additionally notes context-rot effects in long Reason/Generate loops and shows that even frontier models still lack robust understanding of LLVM invariants and semantics. Increasing token budgets does not eliminate these difficulties [2603.20075].

The reported future directions include more sophisticated agents, richer tool integration, automatic detection of “cheating” patches that weaken assertions or bypass components, better context management, and extension to more bug classes and compiler subsystems [2603.20075]. Related localization work points toward more precise function-level or statement-level isolation rather than file-level ranking alone [2506.17647]. Related transformation work suggests that pairwise synergy models, structural refinement policies, and richer IR abstractions remain incomplete approximations rather than final solutions [2510.13184][2102.05081].

Taken together, these results position LLVM-AutoFix-Mini as both a practical baseline and a research scaffold. In its strict sense, it is a minimal, compiler-specific LLM agent for LLVM middle-end repair. In a broader methodological sense, it sits at the intersection of LLVM-specific localization, validity filtering, IR normalization, analysis-driven transformation, and pass-pipeline search, with the common theme that compiler repair requires domain-specific structure rather than generic code-editing capability alone.

Source: https://www.emergentmind.com/topics/llvm-autofix-mini