BUILD-BENCH: C/C++ Compilation Benchmark

• BUILD-BENCH is a benchmark that evaluates whether LLM agents can compile heterogeneous, real-world C/C++ projects with incomplete documentation and varied build systems.
• It employs a long-tail GitHub sample and expert-validated binary naming to distinguish between strict and flexible build success metrics.
• It leverages iterative, multi-agent workflows with documentation retrieval and repair strategies to overcome dependency issues and environment-specific errors.

BUILD-BENCH is a benchmark introduced by Zhang et al. for evaluating whether LLM-based agents can compile real-world open-source C/C++ projects under realistic conditions rather than on toy programs or selectively chosen repositories. It treats compilation as a complex software engineering task in which build instructions may be incomplete or missing, dependencies may be undocumented or unavailable, build systems vary widely, and successful builds may require environment-specific fixes or even source or build-script patches. Its defining design choice is to sample the long tail of GitHub rather than disproportionately sampling popular, well-maintained repositories, and to evaluate success with expert-generated expected binary file names rather than with the mere appearance of some binary artifact (Zhang et al., 27 Sep 2025).

1. Problem setting and rationale

BUILD-BENCH was proposed to measure compilation ability in the setting that practitioners actually encounter: heterogeneous open-source software with incomplete documentation, fragile environments, and nontrivial repair requirements. The paper argues that prior compilation benchmarks and systems substantially understate this difficulty because they disproportionately sample popular repositories that are better documented, more maintained, and more likely to already be “LLM familiar.” BUILD-BENCH instead targets repositories in the 50–500 star range, which the paper describes as closer to GitHub’s long-tail distribution (Zhang et al., 27 Sep 2025).

The benchmark is restricted to real-world C/C++ repositories, but the difficulty is not purely language-local. Repositories may use many different build systems, may require subdirectory reasoning, may hide instructions outside the README, and may fail for reasons that only become clear after iterative troubleshooting. The benchmark therefore evaluates more than command synthesis. It evaluates documentation search, environment setup, dependency repair, source or build-script modification, and root-cause diagnosis.

A common misconception is that BUILD-BENCH primarily measures whether a model can emit a plausible shell script. The benchmark is stricter than that. The paper explicitly criticizes earlier settings in which “success” can mean only that at least one binary appears after a build, even if the intended target does not compile. BUILD-BENCH instead ties success to the expected project outputs.

2. Corpus construction and annotation

The dataset construction begins with a large GitHub corpus collected through the REST API, covering repositories created between April 1, 2008 and January 1, 2024. After removing homework, assignment, tutorial, and demo-style repositories, repositories with fewer than 50 stars, and forks, the paper reports a raw pool of 6.57 million repositories. From this population, the authors randomly sampled 385 projects using Cochran’s sample-size formula for a proportion estimate at 95% confidence with a 5% margin of error:

$n_0 = \frac{Z^2 p (1 - p)}{E^2},$

with $Z=1.96$, $p=0.5$, and $E=0.05$, yielding $n_0=384.16$, then rounded up after finite population correction to obtain 385 (Zhang et al., 27 Sep 2025).

Human experts then manually attempted to build every sampled repository in a best-effort setting. Repositories were excluded if they targeted an incompatible OS and could not be cross-compiled, were trivial or unbuildable, lacked critical source files or had irreparable dependencies, or had compilation or linking errors that humans could not resolve. This process yielded 148 compilable repositories as the final BUILD-BENCH test set. The annotation effort also produced ground-truth binary file names and build-instruction URLs when available. The paper states that 12 graduate students with more than three years of systems research experience performed the labeling, and that the effort took around 150 hours (Zhang et al., 27 Sep 2025).

The resulting test set is heterogeneous in build infrastructure. The paper reports 62 Make-based projects, 60 CMake, 29 Autotools, 14 Visual Studio/MSBuild, and smaller numbers using custom scripts, QMake, Meson, or no explicit build system at all. This diversity is central to the benchmark’s purpose: a system that succeeds only when a standard toolchain is obvious is not sufficient for BUILD-BENCH.

3. Evaluation protocol and success metrics

BUILD-BENCH uses a separate validation set of 70 popular repositories for developing the baseline agent, while the final test set remains the random long-tail sample. This separation is intended to reduce overfitting of agent design choices to the test distribution (Zhang et al., 27 Sep 2025).

The evaluation protocol is stricter than earlier compilation settings because it distinguishes between unvalidated artifact creation and validated project success. After execution, all produced binary files are compared against an expert-generated list of expected binary file names. The benchmark reports three metrics.

Metric	Criterion	Interpretation
Strict Success	All expected binary names are present	Strongest validated success
Flexible Success	At least one expected binary exists	Looser validated success
Unvalidated completion rate	Traditional looser completion metric	Not trusted as final success

This distinction matters because vendored subprojects or intermediate build products can otherwise create false positives. The paper explicitly frames Strict Success and Flexible Success as validated metrics, in contrast to unvalidated completion.

Another important feature is that BUILD-BENCH is expert-grounded not only at the binary-output level but also at the instruction level. For many repositories, the authors annotated build-instruction URLs, enabling direct study of retrieval accuracy. This makes the benchmark suitable both for end-to-end evaluation and for component-level analysis of documentation retrieval.

4. OSS-BUILD-AGENT and the retrieval-centered compilation workflow

The paper proposes OSS-BUILD-AGENT as a baseline system tailored to BUILD-BENCH. It has an optional LLM-Assisted Retrieval front-end and a two-agent compilation loop. The retrieval module begins with the README, has the LLM iteratively extract candidate compilation instructions, judge whether they are sufficient, and, if not, identify promising links. It can recursively fetch and summarize up to three newly identified links, repeating this for up to three iterations or until the system is confident that the build knowledge is complete (Zhang et al., 27 Sep 2025).

This retrieval design is explicitly documentation-first. The paper argues that real build instructions may reside in the README, other repository files, subdirectories, or external webpages, and that immediately diving into build scripts can distract the model with noisy but incomplete information. On 130 repositories from the test set for which the authors had ground-truth build-instruction URLs, this module achieved 73.8% retrieval accuracy, compared with 46.2% for CompileAgent (Zhang et al., 27 Sep 2025).

After retrieval, the compilation stage proceeds through a multi-agent loop involving a Bash Command Generator and an Execution Agent. The generator receives the repository context and any retrieved instructions, synthesizes a candidate sequence of shell commands, and the Execution Agent runs them in a fresh containerized Ubuntu 22.04 environment with minimal preinstalled packages. The process is iterative: an initial command set $S_0$ is generated from input context $C_0$, and at each turn the generator uses the current context $C_k$ and feedback $f_k$ from execution to produce revised commands $S_k$. The loop stops when compilation succeeds or the maximum number of turns is reached.

The paper presents this as a ReAct-style observation–repair–rebuild loop specialized for compilation. Its goal is not only to emit commands, but to support the kinds of local repair that BUILD-BENCH makes necessary: installing dependencies, changing directories correctly, patching source or build files, and responding to downstream errors without losing sight of the original root cause.

5. Empirical results, baseline comparisons, and failure modes

BUILD-BENCH evaluates rule-based systems, single-turn LLM baselines, prior multi-agent systems, and OSS-BUILD-AGENT. The reported results show both the difficulty of the benchmark and the large gap between this long-tail distribution and prior easier settings (Zhang et al., 27 Sep 2025).

Method	Model	Reported performance
GHCC	Rule-based	30.2% unvalidated completion, 13.4% flexible validated success
Assemblage	Rule-based	10.7% completion, 9.4% flexible validated success
Single-turn baseline	o3-mini	7.4% strict, 8.1% flexible
Single-turn baseline	Claude 3.7-Sonnet	21.5% strict, 22.1% flexible
CompileAgent	GPT-4o	49.7% strict, 55.7% flexible
OSS-BUILD-AGENT (no retrieval)	GPT-4o	38.5% strict, 41.9% flexible
OSS-BUILD-AGENT (no retrieval)	o3-mini	48.0% strict, 50.7% flexible
OSS-BUILD-AGENT + retrieval	GPT-4o	53.0% strict, 57.6% flexible
OSS-BUILD-AGENT + retrieval	Claude 3.7-Sonnet	66.4% strict, 71.8% flexible

One of the paper’s strongest empirical observations is distribution shift. CompileAgent reaches 49.7% strict validated success on BUILD-BENCH, which the paper contrasts with the 89% strict validated success reported on CompileAgentBench. The authors interpret this drop as evidence that BUILD-BENCH is materially harder and more realistic.

The paper also studies stochasticity. With GPT-4o and retrieval, three repeated runs produced 45.6%/50.3%, 54.7%/59.5%, and 58.8%/62.9% strict/flexible validated success, averaged as 53.0% ± 6.8 strict and 57.6% ± 6.5 flexible. Reported pass@k values rise to 59.5% strict and 64.2% flexible at pass@2, and to 65.5% strict and 70.3% flexible at pass@3. This suggests that multi-attempt evaluation captures a practical aspect of agentic compilation that single-run metrics miss.

The failure analysis emphasizes that compilation is a long-horizon debugging problem. The most common failure mode occurs when the agent recognizes an error but still cannot devise a correct fix after many turns and terminates. Dependency errors are another frequent class. The case studies are concrete. In s9xie/hed, the agent used execution feedback to patch source code from deprecated OpenCV v3 symbols such as CV_LOAD_IMAGE_COLOR and CV_LOAD_IMAGE_GRAYSCALE to IMREAD_COLOR and IMREAD_GRAYSCALE, after which the repository compiled. Other examples show failure from invoking cmake in the wrong directory because CMakeLists.txt was actually in a subdirectory, and failure from chasing a downstream “No rule to make target 'install'” message instead of diagnosing an earlier submodule initialization problem caused by dubious ownership. These examples support the paper’s claim that root-cause diagnosis, not just command generation, is central to the task.

6. Position within the benchmark landscape

BUILD-BENCH belongs to a broader family of execution-grounded software benchmarks, but its unit of evaluation is specific: compiling existing real-world open-source C/C++ repositories. It is therefore distinct from benchmarks that evaluate from-scratch repository generation or architecture-aware repair.

A common source of confusion is the similarity of names. Build-bench, for example, is a different benchmark that studies repair of package build failures during migration across instruction set architectures such as x86_64 and aarch64, using 268 real-world failed packages and an iterative tool-augmented repair loop (Zhao et al., 2 Nov 2025). BUILD-BENCH, by contrast, evaluates compilation of existing repositories drawn from a random GitHub sample and focuses on validated binary production rather than cross-ISA migration.

It also differs from repository-generation benchmarks such as NL2Repo-Bench, in which an agent receives a single natural-language requirements document and an empty workspace and must autonomously produce a fully installable Python library, with success measured by execution of hidden upstream pytest suites (Ding et al., 14 Dec 2025). BUILD-BENCH assumes an existing repository and asks whether an agent can make that repository build.

This suggests that BUILD-BENCH occupies a specific niche within agentic software engineering evaluation. It is neither a generic benchmarking framework nor a general “build anything” benchmark. It is a compilation benchmark whose main contribution is to operationalize the long-tail reality of open-source build engineering: scattered documentation, heterogeneous build systems, environment repair, and validated end artifacts. Within that scope, its strongest result—66.4% strict validated success and 71.8% flexible validated success for OSS-BUILD-AGENT with Claude 3.7-Sonnet—shows substantial headroom remains (Zhang et al., 27 Sep 2025).