LLVM-Bench: LLVM Compiler Issue Resolution Benchmark

• LLVM-Bench is a comprehensive benchmark and execution framework for LLVM compiler issue resolution, featuring 423 real-world validated tasks.
• It evaluates patch application, build success, and functional correctness across bug fixes, optimizations, and new feature tasks in an expansive LLVM codebase.
• The benchmark leverages LLVM-Gym and ensemble methods to systematically address challenges in patch validity, build failures, and repository-scale evaluation.

Searching arXiv for the specified LLVM-Bench paper and closely related work to ground the article. Tool call: arxiv_search(query="(Tian et al., 1 Jul 2026) OR \"LLVM-Bench: Benchmarking and Advancing LLMs for LLVM Compiler Issue Resolution\" OR LLVM-Bench LLVM issue resolution", max_results=10, sort_by="relevance") LLVM-Bench is a benchmark and execution framework for LLVM compiler issue resolution that was introduced as a large-scale, system-level testbed for evaluating LLMs and agents on real LLVM development tasks (Tian et al., 1 Jul 2026). In its current, explicit sense, it comprises 423 real-world, validated tasks collected from the LLVM project, together with LLVM-Gym, an automated platform for issue reproduction, patch application, compiler building, and test execution. The benchmark targets repository-scale program repair and implementation under the constraints of the LLVM ecosystem: patches must apply to the target tree, build successfully, and pass the full LLVM test suite. Earlier LLVM-focused autotuning work used “LLVM-Bench” in an informal pipeline sense to denote Clang/Polly-based benchmarking and autotuning over PolyBench kernels, but the benchmark proper is the 2026 issue-resolution suite centered on LLVM development tasks (Wu et al., 2020).

1. Definition and scope

LLVM-Bench was introduced to evaluate whether methods that perform strongly on application-level software engineering benchmarks transfer to a compiler infrastructure whose scale and constraints are materially different from those of typical repositories used in prior LLM studies (Tian et al., 1 Jul 2026). The motivation is tied to the characteristics of LLVM itself: the project spans approximately 159K files and approximately 53M LOC at the upper end of the benchmark snapshots, and its issue tracker has accumulated nearly 100,000 issues, with approximately 30,000 unresolved. Issue resolution in this setting requires repository-scale comprehension across front-end, mid-end, and back-end components, as well as conformance to build scripts, generated code, LLVM IR tests, and target-specific constraints.

The benchmark is explicitly positioned against application-level benchmarks such as SWE-bench and SWE-bench-Verified. LLVM-Bench differs in repository size, language mix, and validation regime. SWE-bench repositories are described as far smaller, with the largest SWE-bench repository at approximately 6,000 files and 0.9M LOC, whereas LLVM-Bench tasks are grounded in a system-level compiler codebase with deterministic end-to-end validation via build and test execution (Tian et al., 1 Jul 2026). This suggests that LLVM-Bench is not merely a domain transfer benchmark; it is also a stress test for applicability, compilability, and behavioral preservation under compiler-scale dependencies.

The benchmark’s task scope is broader than bug fixing alone. It includes bug fix, optimization, and new feature tasks. A common misconception is to treat LLVM-Bench as a pure bug-repair dataset analogous to classical defect corpora. The benchmark definition contradicts that view: optimization and feature tasks are part of the task distribution, and evaluation remains patch-centric rather than explanation-centric (Tian et al., 1 Jul 2026).

2. Dataset construction and task taxonomy

LLVM-Bench was constructed from the llvm/llvm-project repository over versions 18, 19, 20, and 21 across 2023–2025 (Tian et al., 1 Jul 2026). The initial crawl covered approximately 561,000 commits, which were down-selected to approximately 70,000 commits tied to the relevant period and components. From these, issue-resolution instances were mined from pairs of consecutive commits, Commit_bug and Commit_fix, under three conditions: the pair had to be linked to a merged pull request, the pull request had to reference a reported GitHub issue, and the pull request had to introduce verifiable tests.

Each mined instance was decomposed into two diffs. The Test Patch is the diff in test files between Commit_bug and Commit_fix, and it encodes reproduction and verification conditions such as targets, flags, and test programs. The Golden Patch is the remaining diff implementing the fix or feature, excluding tests. File filtering retained core source extensions .c, .h, .cpp, .hpp, and .ll, and removed “meta-changes” such as renames or moves only. This process yielded 1,222 candidate instances, which were then filtered by automated execution-based validation and subsequently reviewed manually.

The automated validation had two stages. First, issue reproducibility was checked by applying the Test Patch to Commit_bug and retaining the instance only if the tests failed as expected. Second, patch correctness was checked by applying both the Golden Patch and the Test Patch to Commit_bug, building LLVM, and running the full test suite; the instance was retained only if all tests passed. After this stage, 492 candidates remained. Manual validation then removed issues lacking self-contained, unambiguous descriptions and discarded issues with explicit solution snippets or direct pointers to the fix in order to reduce leakage. Multiple authors independently verified the retained instances, and Gemini-3 was used as a secondary checker. The final benchmark contains 423 validated tasks (Tian et al., 1 Jul 2026).

The task distribution is skewed toward bug fixing and mid-end work. Of the 423 tasks, 319 are bug fixes (75.41%), 82 are optimizations (19.39%), and 22 are new features (5.20%). By LLVM component, 144 tasks are front-end (34.04%), 246 are mid-end (58.16%), 28 are back-end (6.62%), and 5 are categorized as others (1.18%). By version, the benchmark includes 111 tasks from LLVM 18.x (26.24%), 121 from 19.x (28.61%), 99 from 20.x (23.40%), and 92 from 21.x (21.75%) (Tian et al., 1 Jul 2026).

The benchmark also quantifies scale and human effort. Issue text length has mean 1,323.67 tokens, median 457, and maximum 21,580. The codebase snapshot per task has mean 145.49K files and mean 47.95M LOC, with maxima of 159.92K files and 53.23M LOC. Golden Patches change a mean of 3.36 files and 129.97 lines, with medians 2 and 52 respectively, and a maximum of 53 files and 3,775 lines. Human resolution effort, measured from issue open to merge, has mean 72.46 days, median 5.99 days, and maximum 4,016.95 days; discussion rounds have mean 3.70, median 3, and maximum 45. The duration distribution is also reported: 25.53% resolved within 1 day, 27.19% within 1 week, 21.04% within 1 month, 18.68% within 6 months, and 7.57% over 6 months (Tian et al., 1 Jul 2026).

3. Task format, evaluation protocol, and metrics

Models evaluated on LLVM-Bench receive the issue title and description, the full repository snapshot at Commit_bug, and retrieval corpora derived from repository files indexed for search (Tian et al., 1 Jul 2026). They do not receive the Golden Patch or Test Patch; those artifacts are reserved for internal execution and verification in LLVM-Gym to prevent leakage. The expected output is a unified diff that applies cleanly to Commit_bug, compiles LLVM successfully, and passes the full test suite including the newly added tests encoded by the Test Patch.

Evaluation is hierarchical. A patch is first judged for validity, meaning that it applies cleanly to the target tree. It is then judged for compilability, meaning that the patched LLVM builds successfully with Ninja. Finally, it is judged for functional correctness, meaning that the entire LLVM test suite passes, including the new tests from the Test Patch. LLVM-Bench formalizes three task-level rates over the full benchmark:

$$A = \frac{N_{\text{applied}}}{N_{\text{total}}} \times 100\%$$

$$B = \frac{N_{\text{built}}}{N_{\text{total}}} \times 100\%$$

$$R = \frac{N_{\text{resolved}}}{N_{\text{total}}} \times 100\%$$

where $A$ is patch application rate, $B$ is build success rate, and $R$ is resolution rate (Tian et al., 1 Jul 2026).

The benchmark also reports efficiency metrics: input tokens per task, output tokens per task, and average monetary inference cost per task as reported by each API and model. This separation between effectiveness and efficiency matters because larger contexts and more agentic workflows can increase resolution rates while also increasing token usage and cost. A common misreading is to interpret resolution alone as the full evaluation target. The benchmark explicitly treats applicability, build success, and cost as distinct dimensions (Tian et al., 1 Jul 2026).

LLVM-Bench further formalizes complementarity among techniques. For a set of techniques $T=\{t_1,\dots,t_k\}$ with resolved sets $S(t_i)$, union coverage is

$$U = \frac{\left|\bigcup_i S(t_i)\right|}{N_{\text{total}}} \times 100\%$$

and complementarity gain over the best single technique is

$$G = U - \max_i \left(\frac{|S(t_i)|}{N_{\text{total}}} \times 100\%\right).$$

These definitions are operational rather than purely descriptive: they motivate the benchmark’s ensemble method and provide a way to quantify non-overlapping success across models and agents (Tian et al., 1 Jul 2026).

4. LLVM-Gym and executable validation

LLVM-Gym is the benchmark’s execution platform and automates four stages: issue reproduction, patch application, compiler building, and test execution (Tian et al., 1 Jul 2026). In the reproduction stage, LLVM-Gym checks out Commit_bug, applies the Test Patch, and runs the tests; the task is retained only if the test patch fails, thereby confirming that the issue is reproducible. In the patch application stage, a candidate patch produced by an LLM or agent is applied to Commit_bug. In the build stage, LLVM is built with Ninja, and build failures are recorded distinctly from patch-apply failures. In the test stage, LLVM-Gym runs the full LLVM test suite, including both existing regression tests and the added tests from the Test Patch, and captures pass/fail results together with logs and artifacts.

The platform’s reproducibility controls include fixed commits, verified test patches, deterministic automated steps, and multi-author validation (Tian et al., 1 Jul 2026). Error capture is explicitly stratified: failed hunks correspond to invalidity, non-zero compiler or linker exits correspond to build failures, and non-zero test-runner exits or failing assertions correspond to functional failures. The paper does not specify containerization, timeouts, or hardware, and it notes that practitioners typically use clean environments such as Docker and per-step timeouts to manage evaluation throughput. Because these operational details are not part of the benchmark’s formal specification, they should be treated as practical guidance rather than benchmark-defining properties.

LLVM-Gym is central to the benchmark’s epistemic stance. LLVM-Bench does not assess whether a model can describe a plausible fix; it assesses whether the model can produce an executable patch that survives a system-level workflow. This suggests that LLVM-Bench is aligned more closely with repository-constrained program synthesis and repair than with text-only issue reasoning (Tian et al., 1 Jul 2026).

5. Empirical results, dominant failure modes, and complementarity

LLVM-Bench evaluates four representative LLMs—DeepSeek v3.2, Qwen 3 Coder Plus, Gemini 3 Flash, and Grok Code Fast 1—under six retrieval configurations and three agents, all at temperature 0 and maximum generation length 8,192 (Tian et al., 1 Jul 2026). The six retrieval configurations are Sparse Retrieval using BM25 with context budgets of 13K, 27K, and 50K tokens, and Oracle Retrieval with the same three budgets, where the context is taken from files modified by the Golden Patch. The latter is explicitly an upper bound and is not available in practice.

For retrieval-augmented LLMs, the best Sparse Retrieval results were 41.61% applied (Gemini, 27K), 18.44% built (Gemini, 50K), and 1.89% resolved (Grok, 50K). Under Oracle Retrieval, the best results were 53.66% applied (Gemini, 50K), 35.22% built (Gemini, 50K), and 4.02% resolved (Grok, 50K). Averaged across LLMs, Oracle Retrieval outperformed Sparse Retrieval on all three metrics: 2.78% versus 1.04% resolved, 19.72% versus 9.32% built, and 38.00% versus 28.27% applied (Tian et al., 1 Jul 2026). Context length also mattered: average resolution increased from 1.69% at 13K to 1.74% at 27K and 2.31% at 50K, though the paper notes increased token and cost overhead.

The three agents are SWE-agent, Trae-agent, and Live-SWE-agent, each capped at 50 turns (Tian et al., 1 Jul 2026). Agents improved substantially over LLM-only techniques. Average agent performance was 6.21% resolved, 22.36% built, and 36.96% applied, compared with 1.91%, 14.52%, and 33.14% for LLM-only methods. Relative gains were reported as +206.21% in resolution, +35.57% in build success, and +9.69% in application. SWE-agent had the strongest average balance among the agents, with peak results of 10.87% resolved using Grok, 40.19% built using DeepSeek, and 55.32% applied using DeepSeek (Tian et al., 1 Jul 2026).

A central finding is that dominant errors occur early in the pipeline. Among incorrect patches, only 35.77% even apply to the repository. Of those that apply, only 17.88% compile. Within the compilable subset, 9.96% still target wrong files and 7.39% wrong lines, while only 0.53% reach correct locations but fail functional tests (Tian et al., 1 Jul 2026). This hierarchy shows that patch invalidity and build failure dominate functional misbehavior. An objective reading is that repository structure and dependency constraints, rather than last-mile semantic correctness, are the principal bottlenecks at LLVM scale.

The results also show strong complementarity across techniques. With Sparse Retrieval, the best single LLM resolved between 0.24% and 1.89% of tasks, but the union across LLMs reached 5.91%. With Oracle Retrieval, the best single LLM reached 4.02%, while the union reached 17.02%. Across agents, individual resolution ranged from 2.60% to 10.87%, but the union reached 36.17%. Across all LLMs and agents combined, union coverage was 45.39% (Tian et al., 1 Jul 2026). A common misconception is that the best single technique approximates the frontier of solvable tasks. The union analysis shows the opposite: different methods solve substantially different subsets.

6. LLVM-Ens and ensemble-based issue resolution

LLVM-Ens is the ensemble method introduced alongside LLVM-Bench to exploit the complementarity revealed by the union analyses (Tian et al., 1 Jul 2026). It is described as a lightweight ensemble that aggregates candidate patches from diverse LLMs and agents, filters incorrect and redundant candidates, and then uses an LLM-based ensemble agent to choose the most promising patch for test execution.

Its pipeline has three stages. First, incorrect patch filtering removes patches that fail to apply to Commit_bug or fail to build LLVM. Second, redundant patch filtering normalizes unified diffs with unidiff, removes formatting/comments, canonicalizes hunks, groups candidates into equivalence classes, and retains one representative per class. The equivalence relation is defined by

$$B = \frac{N_{\text{built}}}{N_{\text{total}}} \times 100\%$$0

Third, ensemble reasoning uses an agent to analyze the issue text and retained candidates, rank them, and test them until one passes the full test suite, at which point it is accepted as the resolution (Tian et al., 1 Jul 2026).

The paper also provides an illustrative formalization. Let $$B = \frac{N_{\text{built}}}{N_{\text{total}}} \times 100\%$$1 be the full candidate set, $$B = \frac{N_{\text{built}}}{N_{\text{total}}} \times 100\%$$2 the subset that apply, $$B = \frac{N_{\text{built}}}{N_{\text{total}}} \times 100\%$$3 the subset that both apply and build, and $$B = \frac{N_{\text{built}}}{N_{\text{total}}} \times 100\%$$4 the equivalence classes after normalization. The final selection chooses a representative patch from $$B = \frac{N_{\text{built}}}{N_{\text{total}}} \times 100\%$$5, with terminal success determined by full test-suite passage. An illustrative score is

$$B = \frac{N_{\text{built}}}{N_{\text{total}}} \times 100\%$$6

with $$B = \frac{N_{\text{built}}}{N_{\text{total}}} \times 100\%$$7, although the operational criterion is ultimately whether the candidate passes the full test suite (Tian et al., 1 Jul 2026). The paper explicitly notes that the similarity term sometimes used in general ensemble formulations is effectively a no-op here because equivalence-based filtering already removes redundant candidates before ranking.

LLVM-Ens was evaluated over four candidate pools: LLMs plus Sparse Retrieval, LLMs plus Oracle Retrieval, agents, and all techniques combined (Tian et al., 1 Jul 2026). The best per-pool results were 4.26% resolved, 54.14% built, and 61.94% applied for LLMs plus Sparse; 12.29% resolved, 79.43% built, and 82.74% applied for LLMs plus Oracle; 21.99% resolved, 87.47% built, and 90.07% applied for agents; and 20.80% resolved, 95.51% built, and 95.74% applied for all techniques combined. On average across pools, LLVM-Ens improved resolution by 339.38%, build success by 361.91%, and application by 140.11% relative to the best individual within each pool. The paper also reports that a random-ensemble baseline was consistently much weaker, indicating that the gains do not arise merely from expanding the candidate set (Tian et al., 1 Jul 2026).

An important interpretive point is that the highest reported resolution, 21.99%, was achieved on the 12-patch agent pool, not on the 36-patch pool containing all techniques. The paper attributes this to a selection trade-off: larger candidate pools increase diversity but also introduce more incorrect patches, which can make final selection harder. This is a methodological nuance rather than a contradiction.

7. Relation to earlier LLVM-focused benchmarking and autotuning

Before the 2026 benchmark, the phrase “LLVM-Bench” appeared in an earlier, informal sense in work on Clang/Polly autotuning over PolyBench kernels (Wu et al., 2020). That line of work developed an LLVM/Clang/Polly autotuning framework in which user-directed loop-transformation pragmas from the SOLLVE development branch were combined with Bayesian optimization in ytopt to minimize execution time over constrained pragma parameter spaces. The framework used Clang 10.0, Polly from the pragma-clang-loop branch, ConfigSpace for constrained search spaces, and surrogate models including Random Forests, Extra Trees, Gradient Boosted Regression Trees, and Gaussian Process regression (Wu et al., 2021).

The earlier workflow centered on six PolyBench kernels—syr2k, 3mm, lu, heat-3d, covariance, and Floyd–Warshall—and searched spaces as large as 170,368 configurations under a budget of 200 code evaluations (Wu et al., 2021). It reported that autotuning produced the smallest execution times for syr2k, 3mm, heat-3d, lu, and covariance, and it diagnosed a serious regression in Floyd–Warshall caused by Polly’s default heuristic scheduling. The mitigation required compiler flags such as -mllvm -polly-reschedule=0, -mllvm -polly-postopts=0, and -mllvm -polly-pragma-ignore-depcheck, after which autotuning became effective (Wu et al., 2020).

This earlier usage is related but conceptually distinct. It treats LLVM as an optimization stack for loop transformations and studies performance tuning of generated code rather than repository-level issue resolution. A plausible implication is that the term “LLVM-Bench” evolved from an informal label for an LLVM-centric benchmarking/autotuning pipeline into a specific benchmark suite for LLM-driven compiler issue resolution. The 2026 benchmark’s formal scope, dataset design, evaluation metrics, and platform support make it the canonical meaning of LLVM-Bench in the literature represented here (Tian et al., 1 Jul 2026).