DeNovoSWE: Scaling Long-Horizon Environments for Generating Entire Repositories from Scratch

Abstract: As the capabilities of LLM-based code agents continue to advance, their expected role is expanding beyond localized bug fixing in existing codebases toward architecting and implementing complete software repositories from high-level specifications. However, training agents for such long-horizon software engineering tasks remains difficult due to the scarcity of large-scale, verifiable whole-repository generation data. In this paper, we introduce \textbf{DeNovoSWE}, a large-scale dataset for whole-repository generation. DeNovoSWE comprises 4,818 high-quality instances, where each instance requires generating a complete repository from documentation. Our dataset is automatically constructed through a carefully designed sandboxed agentic workflow, enabling scalable curation without human annotation. DeNovoSWE is constructed with "divide and conquer" and critic-repair philosophy. To balance data quality and diversity, we further introduce a difficulty-aware trajectory filtering strategy. Fine-tuning Qwen3-30B-A3B on DeNovoSWE substantially improves long-horizon SWE performance, raising its score on the challenging BeyondSWE-Doc2Repo benchmark from 5.8% to 47.2%.

• The paper introduces DeNovoSWE, a scalable platform that automatically creates complete, verifiable repositories from documentation using a multi-agent divide-and-conquer and critic-repair framework.
• The methodology involves partitioning repository tasks and employing dynamic, difficulty-aware filtering to retain challenging instances while ensuring high behavioral coverage and precise testing.
• Experimental results show significant performance gains, with fine-tuned models achieving dramatic improvements on benchmarks like Doc2Repo and NL2RepoBench compared to vanilla baselines.

DeNovoSWE: A Large-Scale Platform for Long-Horizon Document-to-Repository Software Engineering

Motivation and Positioning

The increasing sophistication of LLM-based code agents drives demand for benchmarks and training sets that extend beyond localized, issue-level code repair towards architecting and implementing entire repositories from high-level documentation. Existing benchmarks such as SWE-bench-Verified have become saturated among top-performing systems and insufficiently challenge agentic systems on long-horizon reasoning, global dependency management, and environment-level composition. The lack of large-scale, well-structured, verifiable document-to-repository datasets poses a key bottleneck for progress.

DeNovoSWE is introduced to address these issues by framing whole-repository creation from documentation as a large-scale, long-horizon, verifiable software engineering environment. The dataset extends prior efforts such as NL2RepoBench and BeyondSWE-Doc2Repo by 1–2 orders of magnitude in terms of covered tasks, as well as systematically increasing instance difficulty and behavioral coverage.

Figure 1: DeNovoSWE scales software engineering environments along the axes of task scope and difficulty, moving from localized bug fixing to full repository synthesis. The benchmark consists of 4,818 repository-generation tasks, vastly outscaling previous datasets.

Automated, Structured Data Synthesis Pipeline

DeNovoSWE is constructed via a fully automated agentic workflow that enforces rigorous separation between data construction, documentation synthesis, and evaluation, thereby ensuring scalability and minimizing contamination or leakage.

Divide-and-Conquer Framework

The core data construction methodology is grounded in a multi-agent divide-and-conquer framework, augmented by a critic-repair mechanism to ensure precise, targeted, and test-aligned documentation:

1. Divide Phase: The repository is partitioned into high-level component capabilities via automatic code and test analysis. Each capability is mapped to relevant modules, APIs, and interfaces using runtime code profiling and LLM-based component-classification. This modularizes documentation requirements and enables targeted coverage of critical repository functionality.
2. Conquer Phase: Dedicated agents iteratively draft, critique, and repair ability-level documentation sections. The Draft agent synthesizes candidate documentation for each decomposed capability. The Critic agent evaluates documentation coverage, flags underspecification or omission risks (especially for test-exercised code), while the Repair agent revises accordingly. This loop converges when behavioral and structural constraints are satisfied per observed test and coverage signals.

   Figure 2: The DeNovoSWE construction pipeline leverages a partition-then-iterate workflow, combining code profiling, LLM-based mapping, and iterative agentic documentation refinement. Agents generate and merge ability-level documentation, with final evaluation occurring in an isolated golden environment.

Evaluation and Robustness

To guarantee a true ‘closed-book’ scenario and prevent agentic code recovery via both direct and indirect channels, DeNovoSWE applies rigorous environment cleansing, including:

• Complete stripping of source and test code from evaluation containers
• Multi-channel erasure of caches, artifacts, and site-package residues
• Shell/network-based source-blocking and package name obfuscation
• LLM-empowered audits of trace logs to catch sophisticated circumvention attempts

Repository selection is further controlled via coverage and pass-rate thresholds, ensuring that each instance is both behaviorally rich and reliably verifiable.

Scaling and Analysis

DeNovoSWE provides a corpus of 4,818 whole-repository generation tasks, with strong variability in repository size, number of exercised unit tests, and test-file coverage. Median repository instances exhibit high behavioral complexity and multi-file/unit-test engagement (median 79 unit tests, 12 test files, 89.6% coverage). The dataset is license-filtered for open-source compliance.

Difficulty-Aware Trajectory Filtering

For this inherently challenging environment, a key innovation is the adoption of a difficulty-aware filtering strategy for curating high-quality generation trajectories. Instead of globally discarding all non-fully-passing generations—an approach that disproportionately penalizes harder instances—the paper introduces a unified difficulty metric (integrating code size, agent pass rates, and independent LLM judgments), which is then used to set interval-specific score thresholds.

Figure 3: Trajectory pass rates drop as task difficulty increases; DeNovoSWE difficulty-aware filtering retains high-quality, challenging samples across the spectrum, avoiding selection bias inherent in static thresholds.

This dynamic filtering procedure enables the training set to retain valuable partial or near-successes on hard tasks, while enforcing higher precision for easy cases. Empirical ablations confirm that this filtering strategy improves both diversity and downstream agent performance.

Experimental Validation

DeNovoSWE's utility is validated through comprehensive experiments on two state-of-the-art repository-synthesis benchmarks: BeyondSWE-Doc2Repo and NL2RepoBench. Models fine-tuned purely on DeNovoSWE exhibit dramatic performance gains over both vanilla and issue-level-SWE-pretrained backbones. Specifically:

• Qwen3-30B-A3B-Instruct, when fine-tuned with DeNovoSWE, achieves a Doc2Repo score of 47.2% versus 5.8% for its vanilla counterpart (+41.4 points), and 23.0% versus 4.3% on NL2RepoBench.
• On the stronger Qwen3.5-35B-A3B backbone, fine-tuning on DeNovoSWE elevates performance to 50.0% (Doc2Repo) and 27.1% (NL2RepoBench), closely approaching proprietary foundation models (Gemini3-Pro at 52.0%, Qwen3.5-Plus at 52.4%) despite the smaller scale.
• Difficulty-aware filtering further boosts absolute accuracy over static-threshold baselines, confirming its benefit for high-diversity, long-horizon environments.

Practical and Theoretical Implications

DeNovoSWE is positioned as a critical platform for advancing general-purpose, long-horizon code agents capable of synthesizing coherent, functional repositories from high-level natural language or structured documentation. The divide-and-conquer, critic-repair approach enables scalable, verifiable task construction while difficulty-aware data curation preserves both behavioral diversity and high execution fidelity.

The dataset is likely to catalyze research on:

• Hierarchical or modular code agent architectures
• Long-agentic planning and tool-use in multi-step synthesis
• Realistic engineering workflows combining specification, documentation, and open-ended implementation
• Robust, scalable evaluation methodologies for future LLM-based engineering systems

Conclusion

DeNovoSWE establishes a new standard for real-world, long-horizon document-to-repository generation tasks. By combining an automated, principled data construction workflow with nuanced instance-level difficulty measures and scalable trajectory filtering, DeNovoSWE enables the development and assessment of code agents with advanced reasoning, planning, and software synthesis abilities. The empirical findings underscore its efficacy in improving agentic performance across challenging repository-synthesis benchmarks and highlight its promise as a foundational asset for future research in autonomous software engineering (2606.10728).