OSS-BUILD-AGENT: Repo-Level Build Automation

• OSS-BUILD-AGENT is a two-stage, multi-agent system that automates the compilation of open-source C/C++ projects by integrating LLM-assisted build instruction retrieval and iterative error repair.
• It utilizes a Bash Command Generator and an Execution Agent that work in tandem within a clean Ubuntu Docker environment to generate, execute, and refine complete command sequences.
• The system effectively handles scattered documentation and complex dependency issues, leading to significant improvements in compilation success on diverse, uncurated repositories.

Searching arXiv for the cited OSS build-agent papers to ground the article. arXiv search: BuildBench / OSS-BUILD-AGENT. Searching for BuildBench and related compilation-agent papers. OSS-BUILD-AGENT is a two-stage, multi-agent, LLM-based system for compiling real-world open-source C/C++ projects from cloned repositories in a clean Ubuntu Docker environment. Introduced as the strong baseline for BUILD-BENCH, it treats compilation as a repo-level software engineering task in which build instructions may be absent, scattered, outdated, or entangled with undocumented dependencies and environment assumptions. Its core design combines LLM-assisted build-instruction retrieval with an iterative compilation loop in which one agent proposes a full bash command sequence and another executes it, returning logs for subsequent repair. The system is intended to operate on the long tail of open-source software rather than only on highly curated repositories, and it is evaluated on a benchmark of 148 manually-verified compilable C/C++-heavy repositories sampled randomly from 6.57M GitHub repositories (Zhang et al., 27 Sep 2025).

1. Problem setting and benchmark definition

OSS-BUILD-AGENT addresses the task: given a cloned OSS repository in a clean Ubuntu Docker container, automatically produce its intended binary artifacts, as defined by human-labeled ground truth, starting from whatever documentation or build scripts exist in the repository or on the web. The formulation is explicitly repo-level rather than file-level: the system must reason over repository structure, build files, auxiliary documentation, CI artifacts, external web pages, and execution feedback, rather than merely generating a short build script from a single prompt (Zhang et al., 27 Sep 2025).

BUILD-BENCH was constructed to stress this setting. Its test set contains 148 manually-verified compilable C/C++-heavy repositories, and its validation set contains 70 repositories used for agent design and tuning. The benchmark is deliberately long-tail in character: most projects have 50–500 stars rather than the highly curated top-star repositories used in earlier evaluations. Build-system diversity is central to the benchmark: 62 repositories use Make, 60 use CMake, 29 use Autotools, 14 use MSBuild/Visual Studio, and smaller subsets use QMake, Meson, custom scripts, while 10 repositories have “no explicit build system.” Each repository is annotated with expected binary names and with labeled URLs that actually contain build instructions, enabling both build validation and retrieval evaluation (Zhang et al., 27 Sep 2025).

The benchmark uses two success notions. Let $B_\text{produced}$ denote the set of binary filenames produced by the agent and $B_\text{gt}$ the expert-labeled ground-truth binary set. Then strict success is defined as

$$\text{StrictSuccess} = \begin{cases} 1, & \text{if } B_\text{gt} \subseteq B_\text{produced} \ 0, & \text{otherwise} \end{cases}$$

and flexible success as

$$\text{FlexibleSuccess} = \begin{cases} 1, & \text{if } B_\text{gt} \cap B_\text{produced} \neq \varnothing \ 0, & \text{otherwise}. \end{cases}$$

This distinction is important because naive “completion” signals can be misleading: a repository may emit some binary artifact while still failing to produce the intended deliverables (Zhang et al., 27 Sep 2025).

2. System architecture and iterative compilation loop

The architecture has two main stages: LLM-Assisted Build Instruction Retrieval and a Multi-Agent Compilation System. The first stage consolidates build instructions from README and linked documentation; the second stage iteratively generates and executes full build-command sequences until success or budget exhaustion. The compilation system itself contains two cooperating agents: a Bash Command Generator and an Execution Agent. The Bash Command Generator receives retrieved instructions, repository context, Docker-environment information, and previous execution feedback, and emits a full sequence of bash commands intended to install dependencies, configure the project, build it, and install resulting binaries. The Execution Agent runs that command sequence in the container and returns logs and exit codes (Zhang et al., 27 Sep 2025).

The paper formalizes the refinement loop as follows. For refinement steps $k = 0, 1, \dots, K$, let $C_k$ be the context at step $k$, with $C_0$ including the initial prompt, repository description, directory structure, retrieved instructions, and environment description. The Bash Command Generator produces

$S_k = \text{LLM}_\text{build}(C_k, f_{k-1}),$

and the Execution Agent returns

$f_k = \text{Execute}(S_k).$

Execution stops when $B_\text{gt}$0 or when the attempt budget is exhausted (Zhang et al., 27 Sep 2025).

A notable design choice is that the Bash Command Generator typically emits a full batch of commands rather than a single shell action. A single iteration may therefore include dependency installation, directory creation, configuration, build, and installation. This macro-style interaction reduces trivial low-information steps and yields error logs that correspond to a coherent build attempt rather than to isolated shell actions. The system is also model-agnostic: experiments instantiate it with GPT‑4o, GPT o3-mini, Claude 3.7 Sonnet, Gemini‑2.5-flash, and Qwen3 variants (Zhang et al., 27 Sep 2025).

The architecture implies a particular conception of compilation: it is not a single-pass synthesis problem but an iterative control problem over a mutable execution environment. A plausible implication is that repo-level build automation benefits less from monolithic planning than from repeated command generation conditioned on grounded execution traces.

3. LLM-assisted build-instruction retrieval

The retrieval stage is one of the defining features of OSS-BUILD-AGENT. It begins with the repository README and iteratively distills explicit compilation steps, assesses whether the instructions are sufficient for a generic Ubuntu environment, and, if necessary, follows promising documentation links. At each iteration, the LLM performs three tasks: extract compilation-related steps such as dependency installations and build commands, judge whether the gathered instructions are sufficiently complete, and identify up to three promising links that may contain additional build or installation information. The system then fetches the content of those links, integrates the new information into an aggregated context, and repeats this process for at most three iterations (Zhang et al., 27 Sep 2025).

This retrieval module is explicitly documentation-first. Rather than prioritizing raw build artifacts such as Makefile or CMakeLists.txt, it starts from human-oriented documentation and traverses linked docs, wiki pages, and external web pages. The paper argues that this reduces derailment by “noisy” build scripts and better handles cases where instructions are split across multiple files or pages. On the 130 repositories in BUILD-BENCH where annotators labeled the true URL hosting the build instructions, OSS-BUILD-AGENT’s retrieval module achieves 73.8% retrieval accuracy, compared with 46.2% for CompileAgent’s retrieval module, using GPT‑4o in both cases (Zhang et al., 27 Sep 2025).

The retrieved output is a single consolidated “final compilation instructions” artifact that becomes part of the prompt context for the Bash Command Generator. This design matters quantitatively. With the same model, adding retrieval substantially improves downstream compilation: for GPT‑4o, strict success rises from 38.5% without retrieval to 53.0% with retrieval; for o3-mini, strict success rises from 48.0% to 63.1% (Zhang et al., 27 Sep 2025).

A broader inference suggested by these results is that documentation traversal is not a peripheral convenience layer but a first-order determinant of agentic build performance. In repo-level compilation, the retrieval problem is not separable from the build problem.

4. Error handling, repair behavior, and representative cases

OSS-BUILD-AGENT’s refinement loop is driven by execution feedback. After each attempt, the Bash Command Generator receives compiler errors, missing-dependency messages, configuration failures, and path errors, and must decide how to modify the next command sequence. Typical repairs include installing missing apt packages, changing working directories, adding or modifying compiler flags such as -g -O0, or patching source and build scripts when the repository is incompatible with modern toolchains (Zhang et al., 27 Sep 2025).

The paper reports that OSS-BUILD-AGENT attempts 6.6 error-resolution steps on average, excluding retrieval, whereas CompileAgent averages 7.5. The comparison is not merely numeric: CompileAgent often operates at single-command granularity, whereas OSS-BUILD-AGENT’s full-command history gives the model a more coherent root-cause signal. The authors argue that quality of reasoning and contextual view matter more than step count alone (Zhang et al., 27 Sep 2025).

Three case studies illustrate the system’s behavior. In the s9xie/hed repository, an approximately 10-year-old HED edge-detection system based on Caffe originally targeted Ubuntu 14 and OpenCV v3. Compilation fails because constants such as CV_LOAD_IMAGE_COLOR and CV_LOAD_IMAGE_GRAYSCALE are undefined in modern OpenCV v4. OSS-BUILD-AGENT interprets the error logs, applies sed substitutions to replace them with IMREAD_COLOR and IMREAD_GRAYSCALE, reruns the build, and succeeds. In bernhard-schmitzer/optimal-transport, a CMake invocation fails because the source directory does not contain CMakeLists.txt; the correct file resides in src/. The paper uses this as an example of a conceptually simple root cause that the agent may nevertheless fail to resolve. In blitz3d-ng/package, a failed git submodule update --init --recursive due to “detected dubious ownership” causes later make install failure; the agent focuses on the downstream make error rather than the true submodule root cause, illustrating a recurrent weakness in root-cause analysis (Zhang et al., 27 Sep 2025).

The system is also unstable across runs. Three independent GPT‑4o runs yield mean strict success of 53.0% ± 6.8 and mean flexible success of 57.6% ± 6.5. However, retrying helps materially: pass@1, pass@2, and pass@3 for strict success are 54.7%, 59.5%, and 65.5%, respectively, while flexible pass@1, pass@2, and pass@3 are 59.5%, 64.2%, and 70.3%. This suggests that stochasticity in retrieval paths and repair trajectories is a structural characteristic of agentic compilation rather than incidental noise (Zhang et al., 27 Sep 2025).

5. Empirical performance and comparisons

On BUILD-BENCH, OSS-BUILD-AGENT substantially outperforms rule-based baselines, single-turn LLM baselines, and the more complex CompileAgent system. The most important results are summarized below.

System	Strict success	Flexible success
GHCC	10.1%	13.4%
Assemblage	6.0%	9.4%
CompileAgent (GPT‑4o)	49.7%	55.7%
OSS-BUILD-AGENT + retrieval (Claude 3.7 Sonnet)	66.4%	71.8%

The rule-based systems are far weaker: GHCC reaches 10.1% strict and 13.4% flexible success, while Assemblage reaches 6.0% strict and 9.4% flexible success. Single-turn LLM baselines also remain limited: o3-mini achieves 7.4% strict and 8.1% flexible success, and Claude 3.7 Sonnet reaches 21.5% strict and 22.1% flexible success. CompileAgent, using GPT‑4o and its own retrieval, reaches 49.7% strict and 55.7% flexible success on BUILD-BENCH, markedly below the approximately 89% strict success it reports on its own CompileAgentBench, which the paper interprets as evidence that BUILD-BENCH is substantially harder (Zhang et al., 27 Sep 2025).

Within OSS-BUILD-AGENT itself, retrieval is decisive, and model capability matters. Without retrieval, GPT‑4o reaches 38.5% strict and 41.9% flexible success, while o3-mini reaches 48.0% strict and 50.7% flexible success. With retrieval, GPT‑4o reaches 53.0% ± 6.8 strict and 57.6% ± 6.5 flexible; o3-mini reaches 63.1% strict and 68.5% flexible; Gemini‑2.5-flash reaches 57.0% strict and 61.1% flexible; Qwen3 235B reaches 59.7% strict and 66.4% flexible; and Claude 3.7 Sonnet yields the best overall results with 85.2% completions, 66.4% strict success, and 71.8% flexible success. The code-specialized Qwen3 Coder 485B underperforms general models, which the authors attribute to the retrieval module’s reliance on documentation comprehension rather than pure code modeling (Zhang et al., 27 Sep 2025).

The paper therefore characterizes OSS-BUILD-AGENT as state of the art on BUILD-BENCH. The result is notable not only because of the absolute success rate, but because the system achieves it with a simpler two-agent architecture than CompileAgent’s seven-agent, five-tool design. A plausible implication is that retrieval quality and coherent repair context can dominate architectural complexity.

6. Broader context, limitations, and significance

OSS-BUILD-AGENT sits within a broader shift from rule-based compilation heuristics toward execution-centered, agentic software operationalization. CompileAgent, for example, is also a repo-level LLM-based agent framework dedicated to compilation and integrates five tools under a flow-based strategy, but BUILD-BENCH shows a lower strict validated success rate for CompileAgent on a more realistic benchmark (Hu et al., 7 May 2025). ExecutionAgent extends similar ideas beyond C/C++ to 50 open-source projects across 14 programming languages, successfully executing the test suites of 33/50 projects and matching ground-truth test results with a deviation of only 7.5%, reinforcing the view that build-and-test automation is a cross-ecosystem problem rather than a C/C++ anomaly (Bouzenia et al., 2024).

Adjacent work on deployment rather than compilation reaches much larger scales. Deploy-Master performs repository discovery, build-spec inference, execution-based validation, and publication for scientific software, reporting 52,550 build attempts and 50,112 successfully validated tools in roughly a single day. That work suggests that execution-centered build-agent principles can be extended from benchmarked compilation to web-scale software operationalization (Wang et al., 7 Jan 2026).

The need for such systems is also supported by empirical studies of human builders. A study of non-contributors documented 303 build issues across 12 OSS projects and found that only 39.9% were settled, with environment mismatch, missing tools, dependency problems, and test failures dominating the failure landscape (Huang et al., 2024). A separate study of 330 student build tasks reported an overall verified build success rate of 45.5% before intervention and showed that targeted guidance about versions, dependencies, environment variables, and resource configurations could dramatically raise success rates, especially for trap issues involving compatibility and environment assumptions (Huang et al., 21 Feb 2025). These findings align closely with OSS-BUILD-AGENT’s emphasis on build-instruction retrieval and iterative error resolution.

The limitations of OSS-BUILD-AGENT are explicit. BUILD-BENCH includes only 148 compilable repositories in the test set, a trade-off accepted for intensive manual verification. The retrieval module reaches 73.8% URL accuracy rather than full coverage. Roughly one third of repositories remain unbuilt even in the best configuration. Error handling is reactive and local, with no long-range memory across repositories. Common failure modes include recognizing an error but failing to eliminate it after several turns, overlooking easy dependency fixes, and chasing secondary errors while missing the root cause. The authors therefore suggest improved agent architectures, better retrieval beyond three iterations and three links per iteration, richer benchmarks, and stability-oriented evaluation such as pass@k as natural next steps (Zhang et al., 27 Sep 2025).

Taken together, OSS-BUILD-AGENT formalizes an important transition in software automation. Compilation is treated not as a static rule-application problem but as an interactive, evidence-grounded engineering task in which documentation retrieval, command synthesis, execution, and repair are inseparable. In that sense, the system is both a benchmark baseline and a concrete model of how repo-level software agents can operate on messy, heterogeneous open-source codebases.