NL2Repo Bench: Repository Code Synthesis

• The paper introduces NL2Repo Bench, which rigorously evaluates LLMs' ability to execute repository-scale tasks based solely on natural language instructions.
• NL2Repo Bench is a benchmark paradigm that measures end-to-end code synthesis by converting detailed documentation into fully deployable multi-file repositories.
• It employs multi-agent pipelines and precise metrics, such as command match and execution success rates, to capture robustness and iterative refinement in code automation.

Natural Language to Code Repository Benchmarks (NL2Repo Bench) comprise a rigorous class of evaluations measuring LLMs' (LLMs) ability to autonomously perform repository-scale software engineering tasks specified in natural language. Differentiated from prior function- and snippet-level code benchmarks, NL2Repo Bench paradigms require models to synthesize, edit, or deploy entire repositories—often involving multi-file dependency resolution, cross-component reasoning, and realistic environment setup—using only natural-language specifications, documentation, or instructions. These benchmarks provide a critical testbed for the next generation of coding agents, emphasizing autonomy, robustness, and agentic planning rather than short-horizon code completion.

1. Conceptual Foundations and Benchmark Design

NL2Repo Bench builds upon gaps identified in existing code evaluation suites, such as HumanEval, MBPP, SWE-bench, and RepoBench, which predominantly measure function-level synthesis or isolated bug fixing (Xiao et al., 10 Feb 2025). It leverages principles from CSR-Bench, which originally targeted computer science research repository deployment via markdown-driven automation, but generalizes broadly to any project with sufficiently detailed markdown or documentation instructions.

A prototypical NL2Repo Bench instance requires a model to parse repository-level natural-language directives—README, tutorials, requirements/spec documents—then translate these into a sequenced, executable workflow. This workflow may encompass environment provisioning, dependency installation, artifact generation, dataset/model retrieval, configuration editing, running complex training pipelines, inference execution, and validation/evaluation procedures. The process stresses the translation fidelity of natural language to heterogeneous code artifacts and automation scripts distributed over multiple files and project components.

2. Dataset Construction, Taxonomy, and Scope

Benchmarks are assembled by mining open-source repositories from platforms such as GitHub, applying strict filtering for:

• Self-containment (documentation must describe full setup, execution, and evaluation without manual gaps),
• Instruction granularity (tutorial-style markdown, not minimal example or underspecified specs),
• Diverse language, framework, and build-system coverage (e.g., Python, Rust, Go; PyTorch, TensorFlow; pip, conda, Docker, make).

A representative NL2Repo Bench instantiation may select 100 repositories covering computer vision, web development, and ML, extracting and segmenting all markdown files. Each repository is decomposed into distinctive task stages (S: Setup, D: Download, T: Training, I: Inference, E: Evaluation), where each stage is mapped to one or many shell commands sourced from the documentation (Xiao et al., 10 Feb 2025).

3. Multi-Agent Automation and Iterative Refinement

The deployment pipeline leverages agent-based architectures inspired by CSR-Agents:

• Command Drafter: parses natural-language sections and directory structure to emit initial command sequences partitioned by stage.
• Script Executor: runs these commands in isolated container environments, capturing stdout and stderr.
• Log Analyzer: inspects failures, prerequisites, and resolves path and environment conflicts.
• Issue Retriever: retrieves historical solutions using BM25-based similarity search over repository issue logs.
• Web Searcher: supplements internal retrieval with external web queries when repository-local data is insufficient.

This multi-agent, iterative refinement loop allows up to $R_{max}$ rounds per task stage, recording command traces, error patterns, and eventual execution success. Representative pseudocode encapsulates the full process:

for stage in {S,D,T,I,E}: cmds ← Drafter(stage) r ← 0 while r < Rmax: success, logs ← Executor(cmds) if success: break hints ← Analyzer(logs) ∪ Retriever(logs) ∪ WebSearch(logs) cmds ← Refine(cmds, hints) r ← r + 1 record(stage, cmds, success, r)

4. Evaluation Metrics: Formalization and Implementation

NL2Repo Bench employs precise execution-based and syntactic metrics for evaluation:

• Command Match Rate: $\mathrm{Acc} = (1/N) \sum_{i} 𝟙\{\hat{c}_i = c_i\}$, quantifying exact correspondence between model-generated and reference commands.
• Stage Success Rate: $\mathrm{Success}_k = (\#\, \text{commands in stage$k$exit-code 0}) / (\#\, \text{commands in$k$})$.
• Average Rounds: $\mathrm{Rounds} = (1/N) \sum_{i} r_i$, the mean refinement steps.
• Syntactic Correctness Rate: $\mathrm{Syn} = (1/N) \sum_i 𝟙\{\text{parseable}(\hat{c}_i)\}$.
• Execution Success Rate: $\mathrm{Exec} = (1/N) \sum_i 𝟙\{\text{exit-code}_{\hat{c}_i} = 0\}$.

These metrics capture not only raw functional correctness but also the quality and efficiency of code deployment scripts, parsing robustness, and iterative repair performance.

Aggregate cross-model results on 100 repositories (e.g., Claude 3-Sonnet, GPT-4-Turbo, Llama 3 70B, Mistral Large-2) reveal command-level accuracies between 0.27–0.37, execution success rates 0.33–0.44, and high syntactic correctness rates (0.83–0.91). Setup and download stages typically achieve success rates above 0.40, while training, inference, and evaluation stages remain challenging (≈0.14–0.18) (Xiao et al., 10 Feb 2025).

5. Failure Modalities and Bottleneck Analysis

Detailed manual analysis of 500 failures identifies:

Failure Category	Prevalence (%)	Typical Manifestations
Missing dependencies	32	Absent libraries (“cmake”, “libomp”)
Path/environment errors	25	Incorrect working directory, env activation
Version conflicts	18	Python version mismatch, dependency clash
Network/data errors	15	Unreachable URLs, failed downloads
Parsing/system errors	10	YAML syntax, misconfigured files

This suggests persistent fragility in upstream environment configuration, cross-component linking, and external resource handling. Challenges are exacerbated for non-trivial build systems, cross-platform edge cases (e.g., Windows, GPU drivers), and multi-language, multi-toolchain scenarios.

6. Limitations and Extension Pathways

NL2Repo Bench, while providing a broad and realistic testbed, exhibits limitations:

• Underrepresentation of sophisticated build pipelines (e.g., CMake, Bazel) and polyglot codebases.
• Bash-centric execution limits benchmarking to Unix-like environments, excluding PowerShell or fish shell.
• Context limitations in LLMs hinder ultra-large repository generalization and Jupyter/multi-step tutorial workflows.
• No support for advanced containerization pipelines (e.g., Helm, Kubernetes), or markup-driven documentation formats (Sphinx, Javadoc).

Future directions include broadening language and platform coverage, enriching testbed diversity with notebook and cloud-native workflows, and integrating reasoning over structured documentation and build artifacts.

7. Significance and Outlook for NL2Repo Evaluation

NL2Repo Bench and related repository-level benchmarks delineate a new evaluation paradigm for LLM-based coding agents, requiring sustained comprehension, reasoning, and execution over complex, multi-stage development workflows. Unlike prior code generation evaluations, success in NL2Repo tasks depends on holistic repository understanding, cross-file logic integration, and robust automation/tool calling.

The standardization of agent-based multi-round deployment evaluation, combined with rigorous test harnesses and curated failure taxonomies, is positioned to drive future research toward autonomous, reliable software synthesis and intelligent repository navigation. Immediate practical implications include improved developer productivity, more reproducible software science, and diagnostic benchmarks for LLM capabilities in real-world environments. NL2Repo Bench serves as both a measure and catalyst for progress in repository-scale code automation (Xiao et al., 10 Feb 2025).