---
title: 'LLVM-Bench: LLVM Compiler Issue Resolution Benchmark'
url: https://www.emergentmind.com/topics/llvm-bench
type: topic
---

# LLVM-Bench: LLVM Compiler Issue Resolution Benchmark

Searching arXiv for the specified LLVM-Bench paper and closely related work to ground the article.
Tool call: arxiv_search(query="2607.00700 OR \"LLVM-Bench: Benchmarking and Advancing Large Language Models for LLVM Compiler Issue Resolution\" OR LLVM-Bench LLVM issue resolution", max_results=10, sort_by="relevance")
LLVM-Bench is a benchmark and execution framework for LLVM compiler issue resolution that was introduced as a large-scale, system-level testbed for evaluating large language models and agents on real LLVM development tasks [2607.00700]. In its current, explicit sense, it comprises 423 real-world, validated tasks collected from the LLVM project, together with LLVM-Gym, an automated platform for issue reproduction, patch application, compiler building, and test execution. The benchmark targets repository-scale program repair and implementation under the constraints of the LLVM ecosystem: patches must apply to the target tree, build successfully, and pass the full LLVM test suite. Earlier LLVM-focused autotuning work used “LLVM-Bench” in an informal pipeline sense to denote Clang/Polly-based benchmarking and autotuning over PolyBench kernels, but the benchmark proper is the 2026 issue-resolution suite centered on LLVM development tasks [2010.08040].

## 1. Definition and scope

LLVM-Bench was introduced to evaluate whether methods that perform strongly on application-level software engineering benchmarks transfer to a compiler infrastructure whose scale and constraints are materially different from those of typical repositories used in prior LLM studies [2607.00700]. The motivation is tied to the characteristics of LLVM itself: the project spans approximately 159K files and approximately 53M LOC at the upper end of the benchmark snapshots, and its issue tracker has accumulated nearly 100,000 issues, with approximately 30,000 unresolved. Issue resolution in this setting requires repository-scale comprehension across front-end, mid-end, and back-end components, as well as conformance to build scripts, generated code, LLVM IR tests, and target-specific constraints.

The benchmark is explicitly positioned against application-level benchmarks such as SWE-bench and SWE-bench-Verified. LLVM-Bench differs in repository size, language mix, and validation regime. SWE-bench repositories are described as far smaller, with the largest SWE-bench repository at approximately 6,000 files and 0.9M LOC, whereas LLVM-Bench tasks are grounded in a system-level compiler codebase with deterministic end-to-end validation via build and test execution [2607.00700]. This suggests that LLVM-Bench is not merely a domain transfer benchmark; it is also a stress test for applicability, compilability, and behavioral preservation under compiler-scale dependencies.

The benchmark’s task scope is broader than bug fixing alone. It includes bug fix, optimization, and new feature tasks. A common misconception is to treat LLVM-Bench as a pure bug-repair dataset analogous to classical defect corpora. The benchmark definition contradicts that view: optimization and feature tasks are part of the task distribution, and evaluation remains patch-centric rather than explanation-centric [2607.00700].

## 2. Dataset construction and task taxonomy

LLVM-Bench was constructed from the `llvm/llvm-project` repository over versions 18, 19, 20, and 21 across 2023–2025 [2607.00700]. The initial crawl covered approximately 561,000 commits, which were down-selected to approximately 70,000 commits tied to the relevant period and components. From these, issue-resolution instances were mined from pairs of consecutive commits, `Commit_bug` and `Commit_fix`, under three conditions: the pair had to be linked to a merged pull request, the pull request had to reference a reported GitHub issue, and the pull request had to introduce verifiable tests.

Each mined instance was decomposed into two diffs. The **Test Patch** is the diff in test files between `Commit_bug` and `Commit_fix`, and it encodes reproduction and verification conditions such as targets, flags, and test programs. The **Golden Patch** is the remaining diff implementing the fix or feature, excluding tests. File filtering retained core source extensions `.c`, `.h`, `.cpp`, `.hpp`, and `.ll`, and removed “meta-changes” such as renames or moves only. This process yielded 1,222 candidate instances, which were then filtered by automated execution-based validation and subsequently reviewed manually.

The automated validation had two stages. First, issue reproducibility was checked by applying the Test Patch to `Commit_bug` and retaining the instance only if the tests failed as expected. Second, patch correctness was checked by applying both the Golden Patch and the Test Patch to `Commit_bug`, building LLVM, and running the full test suite; the instance was retained only if all tests passed. After this stage, 492 candidates remained. Manual validation then removed issues lacking self-contained, unambiguous descriptions and discarded issues with explicit solution snippets or direct pointers to the fix in order to reduce leakage. Multiple authors independently verified the retained instances, and Gemini-3 was used as a secondary checker. The final benchmark contains 423 validated tasks [2607.00700].

The task distribution is skewed toward bug fixing and mid-end work. Of the 423 tasks, 319 are bug fixes (75.41%), 82 are optimizations (19.39%), and 22 are new features (5.20%). By LLVM component, 144 tasks are front-end (34.04%), 246 are mid-end (58.16%), 28 are back-end (6.62%), and 5 are categorized as others (1.18%). By version, the benchmark includes 111 tasks from LLVM 18.x (26.24%), 121 from 19.x (28.61%), 99 from 20.x (23.40%), and 92 from 21.x (21.75%) [2607.00700].

The benchmark also quantifies scale and human effort. Issue text length has mean 1,323.67 tokens, median 457, and maximum 21,580. The codebase snapshot per task has mean 145.49K files and mean 47.95M LOC, with maxima of 159.92K files and 53.23M LOC. Golden Patches change a mean of 3.36 files and 129.97 lines, with medians 2 and 52 respectively, and a maximum of 53 files and 3,775 lines. Human resolution effort, measured from issue open to merge, has mean 72.46 days, median 5.99 days, and maximum 4,016.95 days; discussion rounds have mean 3.70, median 3, and maximum 45. The duration distribution is also reported: 25.53% resolved within 1 day, 27.19% within 1 week, 21.04% within 1 month, 18.68% within 6 months, and 7.57% over 6 months [2607.00700].

## 3. Task format, evaluation protocol, and metrics

Models evaluated on LLVM-Bench receive the issue title and description, the full repository snapshot at `Commit_bug`, and retrieval corpora derived from repository files indexed for search [2607.00700]. They do not receive the Golden Patch or Test Patch; those artifacts are reserved for internal execution and verification in LLVM-Gym to prevent leakage. The expected output is a unified diff that applies cleanly to `Commit_bug`, compiles LLVM successfully, and passes the full test suite including the newly added tests encoded by the Test Patch.

Evaluation is hierarchical. A patch is first judged for **validity**, meaning that it applies cleanly to the target tree. It is then judged for **compilability**, meaning that the patched LLVM builds successfully with Ninja. Finally, it is judged for **functional correctness**, meaning that the entire LLVM test suite passes, including the new tests from the Test Patch. LLVM-Bench formalizes three task-level rates over the full benchmark:
$$
A = \frac{N_{\text{applied}}}{N_{\text{total}}} \times 100\%
$$
$$
B = \frac{N_{\text{built}}}{N_{\text{total}}} \times 100\%
$$
$$
R = \frac{N_{\text{resolved}}}{N_{\text{total}}} \times 100\%
$$
where $A$ is patch application rate, $B$ is build success rate, and $R$ is resolution rate [2607.00700].

The benchmark also reports efficiency metrics: input tokens per task, output tokens per task, and average monetary inference cost per task as reported by each API and model. This separation between effectiveness and efficiency matters because larger contexts and more agentic workflows can increase resolution rates while also increasing token usage and cost. A common misreading is to interpret resolution alone as the full evaluation target. The benchmark explicitly treats applicability, build success, and cost as distinct dimensions [2607.00700].

LLVM-Bench further formalizes **complementarity** among techniques. For a set of techniques $T=\{t_1,\dots,t_k\}$ with resolved sets $S(t_i)$, union coverage is
$$
U = \frac{\left|\bigcup_i S(t_i)\right|}{N_{\text{total}}} \times 100\%
$$
and complementarity gain over the best single technique is
$$
G = U - \max_i \left(\frac{|S(t_i)|}{N_{\text{total}}} \times 100\%\right).
$$
These definitions are operational rather than purely descriptive: they motivate the benchmark’s ensemble method and provide a way to quantify non-overlapping success across models and agents [2607.00700].

## 4. LLVM-Gym and executable validation

LLVM-Gym is the benchmark’s execution platform and automates four stages: issue reproduction, patch application, compiler building, and test execution [2607.00700]. In the reproduction stage, LLVM-Gym checks out `Commit_bug`, applies the Test Patch, and runs the tests; the task is retained only if the test patch fails, thereby confirming that the issue is reproducible. In the patch application stage, a candidate patch produced by an LLM or agent is applied to `Commit_bug`. In the build stage, LLVM is built with Ninja, and build failures are recorded distinctly from patch-apply failures. In the test stage, LLVM-Gym runs the full LLVM test suite, including both existing regression tests and the added tests from the Test Patch, and captures pass/fail results together with logs and artifacts.

The platform’s reproducibility controls include fixed commits, verified test patches, deterministic automated steps, and multi-author validation [2607.00700]. Error capture is explicitly stratified: failed hunks correspond to invalidity, non-zero compiler or linker exits correspond to build failures, and non-zero test-runner exits or failing assertions correspond to functional failures. The paper does not specify containerization, timeouts, or hardware, and it notes that practitioners typically use clean environments such as Docker and per-step timeouts to manage evaluation throughput. Because these operational details are not part of the benchmark’s formal specification, they should be treated as practical guidance rather than benchmark-defining properties.

LLVM-Gym is central to the benchmark’s epistemic stance. LLVM-Bench does not assess whether a model can describe a plausible fix; it assesses whether the model can produce an executable patch that survives a system-level workflow. This suggests that LLVM-Bench is aligned more closely with repository-constrained program synthesis and repair than with text-only issue reasoning [2607.00700].

## 5. Empirical results, dominant failure modes, and complementarity

LLVM-Bench evaluates four representative LLMs—DeepSeek v3.2, Qwen 3 Coder Plus, Gemini 3 Flash, and Grok Code Fast 1—under six retrieval configurations and three agents, all at temperature 0 and maximum generation length 8,192 [2607.00700]. The six retrieval configurations are Sparse Retrieval using BM25 with context budgets of 13K, 27K, and 50K tokens, and Oracle Retrieval with the same three budgets, where the context is taken from files modified by the Golden Patch. The latter is explicitly an upper bound and is not available in practice.

For retrieval-augmented LLMs, the best Sparse Retrieval results were 41.61% applied (Gemini, 27K), 18.44% built (Gemini, 50K), and 1.89% resolved (Grok, 50K). Under Oracle Retrieval, the best results were 53.66% applied (Gemini, 50K), 35.22% built (Gemini, 50K), and 4.02% resolved (Grok, 50K). Averaged across LLMs, Oracle Retrieval outperformed Sparse Retrieval on all three metrics: 2.78% versus 1.04% resolved, 19.72% versus 9.32% built, and 38.00% versus 28.27% applied [2607.00700]. Context length also mattered: average resolution increased from 1.69% at 13K to 1.74% at 27K and 2.31% at 50K, though the paper notes increased token and cost overhead.

The three agents are SWE-agent, Trae-agent, and Live-SWE-agent, each capped at 50 turns [2607.00700]. Agents improved substantially over LLM-only techniques. Average agent performance was 6.21% resolved, 22.36% built, and 36.96% applied, compared with 1.91%, 14.52%, and 33.14% for LLM-only methods. Relative gains were reported as +206.21% in resolution, +35.57% in build success, and +9.69% in application. SWE-agent had the strongest average balance among the agents, with peak results of 10.87% resolved using Grok, 40.19% built using DeepSeek, and 55.32% applied using DeepSeek [2607.00700].

A central finding is that dominant errors occur early in the pipeline. Among incorrect patches, only 35.77% even apply to the repository. Of those that apply, only 17.88% compile. Within the compilable subset, 9.96% still target wrong files and 7.39% wrong lines, while only 0.53% reach correct locations but fail functional tests [2607.00700]. This hierarchy shows that patch invalidity and build failure dominate functional misbehavior. An objective reading is that repository structure and dependency constraints, rather than last-mile semantic correctness, are the principal bottlenecks at LLVM scale.

The results also show strong complementarity across techniques. With Sparse Retrieval, the best single LLM resolved between 0.24% and 1.89% of tasks, but the union across LLMs reached 5.91%. With Oracle Retrieval, the best single LLM reached 4.02%, while the union reached 17.02%. Across agents, individual resolution ranged from 2.60% to 10.87%, but the union reached 36.17%. Across all LLMs and agents combined, union coverage was 45.39% [2607.00700]. A common misconception is that the best single technique approximates the frontier of solvable tasks. The union analysis shows the opposite: different methods solve substantially different subsets.

## 6. LLVM-Ens and ensemble-based issue resolution

LLVM-Ens is the ensemble method introduced alongside LLVM-Bench to exploit the complementarity revealed by the union analyses [2607.00700]. It is described as a lightweight ensemble that aggregates candidate patches from diverse LLMs and agents, filters incorrect and redundant candidates, and then uses an LLM-based ensemble agent to choose the most promising patch for test execution.

Its pipeline has three stages. First, **incorrect patch filtering** removes patches that fail to apply to `Commit_bug` or fail to build LLVM. Second, **redundant patch filtering** normalizes unified diffs with `unidiff`, removes formatting/comments, canonicalizes hunks, groups candidates into equivalence classes, and retains one representative per class. The equivalence relation is defined by
$$
p_1 \equiv p_2 \iff \text{Normalize}(p_1)=\text{Normalize}(p_2).
$$
Third, **ensemble reasoning** uses an agent to analyze the issue text and retained candidates, rank them, and test them until one passes the full test suite, at which point it is accepted as the resolution [2607.00700].

The paper also provides an illustrative formalization. Let $P$ be the full candidate set, $F_{\text{app}}$ the subset that apply, $F_{\text{build}}$ the subset that both apply and build, and $E$ the equivalence classes after normalization. The final selection chooses a representative patch from $E$, with terminal success determined by full test-suite passage. An illustrative score is
$$
S(p)=\alpha\cdot s_{\text{build}}(p)+\beta\cdot s_{\text{test}}(p),
$$
with $\beta>\alpha\ge 0$, although the operational criterion is ultimately whether the candidate passes the full test suite [2607.00700]. The paper explicitly notes that the similarity term sometimes used in general ensemble formulations is effectively a no-op here because equivalence-based filtering already removes redundant candidates before ranking.

LLVM-Ens was evaluated over four candidate pools: LLMs plus Sparse Retrieval, LLMs plus Oracle Retrieval, agents, and all techniques combined [2607.00700]. The best per-pool results were 4.26% resolved, 54.14% built, and 61.94% applied for LLMs plus Sparse; 12.29% resolved, 79.43% built, and 82.74% applied for LLMs plus Oracle; 21.99% resolved, 87.47% built, and 90.07% applied for agents; and 20.80% resolved, 95.51% built, and 95.74% applied for all techniques combined. On average across pools, LLVM-Ens improved resolution by 339.38%, build success by 361.91%, and application by 140.11% relative to the best individual within each pool. The paper also reports that a random-ensemble baseline was consistently much weaker, indicating that the gains do not arise merely from expanding the candidate set [2607.00700].

An important interpretive point is that the highest reported resolution, 21.99%, was achieved on the 12-patch agent pool, not on the 36-patch pool containing all techniques. The paper attributes this to a selection trade-off: larger candidate pools increase diversity but also introduce more incorrect patches, which can make final selection harder. This is a methodological nuance rather than a contradiction.

## 7. Relation to earlier LLVM-focused benchmarking and autotuning

Before the 2026 benchmark, the phrase “LLVM-Bench” appeared in an earlier, informal sense in work on Clang/Polly autotuning over PolyBench kernels [2010.08040]. That line of work developed an LLVM/Clang/Polly autotuning framework in which user-directed loop-transformation pragmas from the SOLLVE development branch were combined with Bayesian optimization in `ytopt` to minimize execution time over constrained pragma parameter spaces. The framework used Clang 10.0, Polly from the `pragma-clang-loop` branch, ConfigSpace for constrained search spaces, and surrogate models including Random Forests, Extra Trees, Gradient Boosted Regression Trees, and Gaussian Process regression [2104.13242].

The earlier workflow centered on six PolyBench kernels—`syr2k`, `3mm`, `lu`, `heat-3d`, `covariance`, and Floyd–Warshall—and searched spaces as large as 170,368 configurations under a budget of 200 code evaluations [2104.13242]. It reported that autotuning produced the smallest execution times for `syr2k`, `3mm`, `heat-3d`, `lu`, and `covariance`, and it diagnosed a serious regression in Floyd–Warshall caused by Polly’s default heuristic scheduling. The mitigation required compiler flags such as `-mllvm -polly-reschedule=0`, `-mllvm -polly-postopts=0`, and `-mllvm -polly-pragma-ignore-depcheck`, after which autotuning became effective [2010.08040].

This earlier usage is related but conceptually distinct. It treats LLVM as an optimization stack for loop transformations and studies performance tuning of generated code rather than repository-level issue resolution. A plausible implication is that the term “LLVM-Bench” evolved from an informal label for an LLVM-centric benchmarking/autotuning pipeline into a specific benchmark suite for LLM-driven compiler issue resolution. The 2026 benchmark’s formal scope, dataset design, evaluation metrics, and platform support make it the canonical meaning of LLVM-Bench in the literature represented here [2607.00700].

Source: https://www.emergentmind.com/topics/llvm-bench