---
title: 'BUILD-BENCH: C/C++ Compilation Benchmark'
url: https://www.emergentmind.com/topics/build-bench
type: topic
---

# BUILD-BENCH: C/C++ Compilation Benchmark

BUILD-BENCH is a benchmark introduced by Zhang et al. for evaluating whether LLM-based agents can compile real-world open-source C/C++ projects under realistic conditions rather than on toy programs or selectively chosen repositories. It treats compilation as a complex software engineering task in which build instructions may be incomplete or missing, dependencies may be undocumented or unavailable, build systems vary widely, and successful builds may require environment-specific fixes or even source or build-script patches. Its defining design choice is to sample the long tail of GitHub rather than disproportionately sampling popular, well-maintained repositories, and to evaluate success with expert-generated expected binary file names rather than with the mere appearance of some binary artifact [2509.25248].

## 1. Problem setting and rationale

BUILD-BENCH was proposed to measure compilation ability in the setting that practitioners actually encounter: heterogeneous open-source software with incomplete documentation, fragile environments, and nontrivial repair requirements. The paper argues that prior compilation benchmarks and systems substantially understate this difficulty because they disproportionately sample popular repositories that are better documented, more maintained, and more likely to already be “LLM familiar.” BUILD-BENCH instead targets repositories in the 50–500 star range, which the paper describes as closer to GitHub’s long-tail distribution [2509.25248].

The benchmark is restricted to real-world **C/C++** repositories, but the difficulty is not purely language-local. Repositories may use many different build systems, may require subdirectory reasoning, may hide instructions outside the README, and may fail for reasons that only become clear after iterative troubleshooting. The benchmark therefore evaluates more than command synthesis. It evaluates documentation search, environment setup, dependency repair, source or build-script modification, and root-cause diagnosis.

A common misconception is that BUILD-BENCH primarily measures whether a model can emit a plausible shell script. The benchmark is stricter than that. The paper explicitly criticizes earlier settings in which “success” can mean only that at least one binary appears after a build, even if the intended target does not compile. BUILD-BENCH instead ties success to the expected project outputs.

## 2. Corpus construction and annotation

The dataset construction begins with a large GitHub corpus collected through the REST API, covering repositories created between **April 1, 2008** and **January 1, 2024**. After removing homework, assignment, tutorial, and demo-style repositories, repositories with fewer than **50 stars**, and forks, the paper reports a raw pool of **6.57 million repositories**. From this population, the authors randomly sampled **385 projects** using Cochran’s sample-size formula for a proportion estimate at **95% confidence** with a **5% margin of error**:

$$
n_0 = \frac{Z^2 p (1 - p)}{E^2},
$$

with $Z=1.96$, $p=0.5$, and $E=0.05$, yielding $n_0=384.16$, then rounded up after finite population correction to obtain **385** [2509.25248].

Human experts then manually attempted to build every sampled repository in a best-effort setting. Repositories were excluded if they targeted an incompatible OS and could not be cross-compiled, were trivial or unbuildable, lacked critical source files or had irreparable dependencies, or had compilation or linking errors that humans could not resolve. This process yielded **148 compilable repositories** as the final BUILD-BENCH test set. The annotation effort also produced ground-truth binary file names and build-instruction URLs when available. The paper states that **12 graduate students** with more than **three years of systems research experience** performed the labeling, and that the effort took around **150 hours** [2509.25248].

The resulting test set is heterogeneous in build infrastructure. The paper reports **62 Make-based projects**, **60 CMake**, **29 Autotools**, **14 Visual Studio/MSBuild**, and smaller numbers using custom scripts, QMake, Meson, or no explicit build system at all. This diversity is central to the benchmark’s purpose: a system that succeeds only when a standard toolchain is obvious is not sufficient for BUILD-BENCH.

## 3. Evaluation protocol and success metrics

BUILD-BENCH uses a separate **validation set of 70 popular repositories** for developing the baseline agent, while the final test set remains the random long-tail sample. This separation is intended to reduce overfitting of agent design choices to the test distribution [2509.25248].

The evaluation protocol is stricter than earlier compilation settings because it distinguishes between unvalidated artifact creation and validated project success. After execution, all produced binary files are compared against an expert-generated list of expected binary file names. The benchmark reports three metrics.

| Metric | Criterion | Interpretation |
|---|---|---|
| **Strict Success** | All expected binary names are present | Strongest validated success |
| **Flexible Success** | At least one expected binary exists | Looser validated success |
| **Unvalidated completion rate** | Traditional looser completion metric | Not trusted as final success |

This distinction matters because vendored subprojects or intermediate build products can otherwise create false positives. The paper explicitly frames Strict Success and Flexible Success as validated metrics, in contrast to unvalidated completion.

Another important feature is that BUILD-BENCH is expert-grounded not only at the binary-output level but also at the instruction level. For many repositories, the authors annotated build-instruction URLs, enabling direct study of retrieval accuracy. This makes the benchmark suitable both for end-to-end evaluation and for component-level analysis of documentation retrieval.

## 4. OSS-BUILD-AGENT and the retrieval-centered compilation workflow

The paper proposes **OSS-BUILD-AGENT** as a baseline system tailored to BUILD-BENCH. It has an optional **LLM-Assisted Retrieval** front-end and a two-agent compilation loop. The retrieval module begins with the README, has the LLM iteratively extract candidate compilation instructions, judge whether they are sufficient, and, if not, identify promising links. It can recursively fetch and summarize up to **three** newly identified links, repeating this for up to **three iterations** or until the system is confident that the build knowledge is complete [2509.25248].

This retrieval design is explicitly documentation-first. The paper argues that real build instructions may reside in the README, other repository files, subdirectories, or external webpages, and that immediately diving into build scripts can distract the model with noisy but incomplete information. On **130 repositories** from the test set for which the authors had ground-truth build-instruction URLs, this module achieved **73.8% retrieval accuracy**, compared with **46.2%** for CompileAgent [2509.25248].

After retrieval, the compilation stage proceeds through a multi-agent loop involving a **Bash Command Generator** and an **Execution Agent**. The generator receives the repository context and any retrieved instructions, synthesizes a candidate sequence of shell commands, and the Execution Agent runs them in a fresh containerized **Ubuntu 22.04** environment with minimal preinstalled packages. The process is iterative: an initial command set $S_0$ is generated from input context $C_0$, and at each turn the generator uses the current context $C_k$ and feedback $f_k$ from execution to produce revised commands $S_k$. The loop stops when compilation succeeds or the maximum number of turns is reached.

The paper presents this as a ReAct-style observation–repair–rebuild loop specialized for compilation. Its goal is not only to emit commands, but to support the kinds of local repair that BUILD-BENCH makes necessary: installing dependencies, changing directories correctly, patching source or build files, and responding to downstream errors without losing sight of the original root cause.

## 5. Empirical results, baseline comparisons, and failure modes

BUILD-BENCH evaluates rule-based systems, single-turn LLM baselines, prior multi-agent systems, and OSS-BUILD-AGENT. The reported results show both the difficulty of the benchmark and the large gap between this long-tail distribution and prior easier settings [2509.25248].

| Method | Model | Reported performance |
|---|---|---|
| **GHCC** | Rule-based | **30.2%** unvalidated completion, **13.4%** flexible validated success |
| **Assemblage** | Rule-based | **10.7%** completion, **9.4%** flexible validated success |
| **Single-turn baseline** | o3-mini | **7.4%** strict, **8.1%** flexible |
| **Single-turn baseline** | Claude 3.7-Sonnet | **21.5%** strict, **22.1%** flexible |
| **CompileAgent** | GPT-4o | **49.7%** strict, **55.7%** flexible |
| **OSS-BUILD-AGENT (no retrieval)** | GPT-4o | **38.5%** strict, **41.9%** flexible |
| **OSS-BUILD-AGENT (no retrieval)** | o3-mini | **48.0%** strict, **50.7%** flexible |
| **OSS-BUILD-AGENT + retrieval** | GPT-4o | **53.0%** strict, **57.6%** flexible |
| **OSS-BUILD-AGENT + retrieval** | Claude 3.7-Sonnet | **66.4%** strict, **71.8%** flexible |

One of the paper’s strongest empirical observations is distribution shift. CompileAgent reaches **49.7%** strict validated success on BUILD-BENCH, which the paper contrasts with the **89%** strict validated success reported on CompileAgentBench. The authors interpret this drop as evidence that BUILD-BENCH is materially harder and more realistic.

The paper also studies stochasticity. With **GPT-4o** and retrieval, three repeated runs produced **45.6%/50.3%**, **54.7%/59.5%**, and **58.8%/62.9%** strict/flexible validated success, averaged as **53.0% ± 6.8** strict and **57.6% ± 6.5** flexible. Reported pass@k values rise to **59.5%** strict and **64.2%** flexible at **pass@2**, and to **65.5%** strict and **70.3%** flexible at **pass@3**. This suggests that multi-attempt evaluation captures a practical aspect of agentic compilation that single-run metrics miss.

The failure analysis emphasizes that compilation is a long-horizon debugging problem. The most common failure mode occurs when the agent recognizes an error but still cannot devise a correct fix after many turns and terminates. Dependency errors are another frequent class. The case studies are concrete. In **s9xie/hed**, the agent used execution feedback to patch source code from deprecated OpenCV v3 symbols such as `CV_LOAD_IMAGE_COLOR` and `CV_LOAD_IMAGE_GRAYSCALE` to `IMREAD_COLOR` and `IMREAD_GRAYSCALE`, after which the repository compiled. Other examples show failure from invoking `cmake` in the wrong directory because `CMakeLists.txt` was actually in a subdirectory, and failure from chasing a downstream “No rule to make target 'install'” message instead of diagnosing an earlier submodule initialization problem caused by dubious ownership. These examples support the paper’s claim that root-cause diagnosis, not just command generation, is central to the task.

## 6. Position within the benchmark landscape

BUILD-BENCH belongs to a broader family of execution-grounded software benchmarks, but its unit of evaluation is specific: **compiling existing real-world open-source C/C++ repositories**. It is therefore distinct from benchmarks that evaluate from-scratch repository generation or architecture-aware repair.

A common source of confusion is the similarity of names. **Build-bench**, for example, is a different benchmark that studies repair of package build failures during migration across instruction set architectures such as **x86_64** and **aarch64**, using **268 real-world failed packages** and an iterative tool-augmented repair loop [2511.00780]. BUILD-BENCH, by contrast, evaluates compilation of existing repositories drawn from a random GitHub sample and focuses on validated binary production rather than cross-ISA migration.

It also differs from repository-generation benchmarks such as **NL2Repo-Bench**, in which an agent receives a single natural-language requirements document and an empty workspace and must autonomously produce a fully installable Python library, with success measured by execution of hidden upstream pytest suites [2512.12730]. BUILD-BENCH assumes an existing repository and asks whether an agent can make that repository build.

This suggests that BUILD-BENCH occupies a specific niche within agentic software engineering evaluation. It is neither a generic benchmarking framework nor a general “build anything” benchmark. It is a compilation benchmark whose main contribution is to operationalize the long-tail reality of open-source build engineering: scattered documentation, heterogeneous build systems, environment repair, and validated end artifacts. Within that scope, its strongest result—**66.4%** strict validated success and **71.8%** flexible validated success for OSS-BUILD-AGENT with **Claude 3.7-Sonnet**—shows substantial headroom remains [2509.25248].

Source: https://www.emergentmind.com/topics/build-bench