BeyondSWE: Benchmark for Repository-Scale Code Agents
- BeyondSWE is a benchmark that tests code agents on large-scale, cross-repository tasks rather than localized bug fixes.
- It categorizes evaluations along resolution scope and knowledge scope, quantifying modifications with metrics like files changed and lines of code.
- The benchmark includes varied settings (CrossRepo, DomainFix, DepMigrate, Doc2Repo) that simulate real-world issues from dependency migration to repository generation.
Searching arXiv for the cited BeyondSWE paper and closely related repository-generation work to ground the article in current literature. BeyondSWE is a software-engineering benchmark designed to evaluate whether contemporary code agents can operate beyond localized bug fixing inside a single repository. It was introduced to measure capabilities that prior SWE-style benchmarks largely underemphasize: cross-repository reasoning, domain-specialized issue resolution, dependency-driven migration, and full repository generation from specifications. The benchmark contains 500 real-world instances from 246 GitHub repositories and organizes them along two explicit dimensions—resolution scope and knowledge scope—thereby extending evaluation from “Local Func / Within Repo” tasks toward repository-wide transformation and specification-driven construction (Chen et al., 3 Mar 2026).
1. Conceptual scope and benchmark taxonomy
The central premise of BeyondSWE is that existing code-agent evaluation is dominated by what the paper characterizes as a localized regime: function-level or issue-level repair within one codebase, with most necessary knowledge assumed to be already present inside that repository. BeyondSWE broadens this regime along two axes.
The first axis is resolution scope, which measures how much of the codebase must be modified or produced. The benchmark distinguishes Local Func tasks from Global Repo tasks. CrossRepo and DomainFix remain close to issue-resolution settings, whereas DepMigrate requires coordinated repository-wide edits and Doc2Repo requires full repository construction.
The second axis is knowledge scope, which measures where the required information comes from. BeyondSWE explicitly enumerates Within Repo, Cross Repo, Domain, Official Docs, and Human Spec. This makes the benchmark’s organizing principle more explicit than prior SWE-style benchmarks, which the paper places almost entirely at “Local Func / Within Repo” (Chen et al., 3 Mar 2026).
| Setting | Resolution / Knowledge | Instances |
|---|---|---|
| CrossRepo | Local Func / Cross Repo | 200 |
| DomainFix | Local Func / Domain | 72 |
| DepMigrate | Global Repo / Official Docs | 178 |
| Doc2Repo | Global Repo / Human Spec | 50 |
The comparison table reported for prior benchmarks sharpens the contrast. SWE-bench-Verified, SWE-bench-Live, and SWE-bench Pro are all categorized as Local Func in resolution scope and Within Repo in knowledge scope. BeyondSWE overall is described as Mix / Mix, because it deliberately spans both local and global engineering tasks and both internal and external knowledge sources.
The benchmark statistics also indicate a materially broader task profile. Averaged over non-Doc2Repo issue tasks, BeyondSWE instances involve 5.6 files and 209.9 lines. By setting, the paper reports 4.1 files / 190.7 lines for CrossRepo, 4.2 files / 157.6 lines for DomainFix, and 8.4 files / 281.6 lines for DepMigrate. This suggests that BeyondSWE is not merely broad in description, but also broader in the scale of modifications required.
2. Task structure and the four settings
Each BeyondSWE instance has three components: a problem statement, a Docker environment, and a test suite. For CrossRepo, DomainFix, and DepMigrate, the problem statement is issue-style; for Doc2Repo, it is a specification document. For the issue-resolution settings, the test suite is divided into P2P tests, which should remain passing, and F2P tests, which should become passing. For Doc2Repo, the full test suite determines success (Chen et al., 3 Mar 2026).
CrossRepo evaluates cross-repository issue resolution. Agents must fix an issue in a target repository by leveraging information from external repositories, upstream implementations, or related projects. The expected output is a patch that causes all P2P and F2P tests to pass. The curation pipeline began from 3,000 candidates, retained about 800 after environment construction and stability inspection, and finalized 200 issues across 67 repositories after manual verification. The paper reports an average of 1.3 links per issue.
DomainFix targets issue resolution in scientific or technically specialized repositories where correctness depends on real domain knowledge rather than generic programming priors. Named fields include quantum computing, molecular dynamics, materials science, bioinformatics, convex optimization, astronomy, seismology, plasma physics, geospatial analysis, photonic IC design, and computational chemistry. The appendix lists repositories such as astroplan, biotite, Biopython, cclib, PlasmaPy, qutip, obspy, cvxpy, geopandas, pymatgen, mdanalysis, and gdsfactory. Construction involved collaboration with experts across 11 scientific disciplines, identification of 21 high-quality repositories, collection of about 800 candidate PRs, reduction to about 200 after environment construction and stability checks, and final unanimous review by three domain experts per instance, yielding 72 issues across 12 repositories.
DepMigrate addresses dependency-driven migration. Here the repository is checked out to the pre-migration commit, but the Docker environment is configured with the upgraded dependency. Agents must read official documentation and migration guides, locate affected call sites, and perform coordinated multi-file edits so that all P2P and F2P tests pass. The paper gives migration examples such as Pydantic v1 to v2, Django 4.x to 5.0, and NumPy 1.x to 2.x. The sourcing pipeline identified 23 widely used packages with significant version upgrades, collected about 7,000 candidate PRs, retained about 1,000 after environment construction and stability inspection, and finalized 178 issues across 120 repositories through manual verification by four software engineering experts.
Doc2Repo is the benchmark’s document-to-repository generation setting. The input consists only of a specification document, an empty workspace, and a test suite. The specification includes purpose, usage examples, and API details such as function or class signatures, parameters, return types, and behavior descriptions, but explicitly excludes implementation details and directory structure. Repository names are masked as target_repo, forcing agents to infer project structure from usage context. Repositories were collected from projects created between January and November 2025, required to remain active after August 2025, and to have at least three contributors and more than 20 stars. After environment construction, the authors obtained 60 instances and manually selected 50 high-quality instances. The appendix reports repository scales ranging from about 1,000 to over 16,000 lines of code, with 40 out of 50 exceeding 1,500 lines.
3. Construction pipeline, environment reconstruction, and benchmark hygiene
A major contribution of BeyondSWE is its benchmark-construction pipeline, especially for historical environment reconstruction. The paper describes an agent-based environment construction process that starts from a base Ubuntu Docker container, instantiates an LLM agent (Gemini 3 Pro), clones the repository, checks out the pre-PR commit, runs tests in a loop, iteratively fixes environment and dependency issues—including system-level commands such as apt-get—and finally distills the working command history into a minimal reproducible Dockerfile (Chen et al., 3 Mar 2026).
Generated Dockerfiles are then subjected to stability inspection. Each Dockerfile is built and tested five times. Instances are retained only if the behavior is deterministic: before patching, P2P tests pass and F2P tests fail; after applying the gold patch, both P2P and F2P tests pass. Any instance with non-determinism or failed criteria is removed.
Because pull requests often contain explicit solution information, the benchmark sanitizes task descriptions by rewriting PR descriptions into issue-style problem statements using Gemini 3 Pro. The stated intent is to preserve symptoms, reproduction steps, logs, and environment information while removing explicit fix logic, commits, and author references. This is especially important for CrossRepo, where original PR histories may include external links that directly expose solution patterns.
Human review is extensive. The paper reports 3 domain experts in scientific computing for DomainFix repository selection, 5 senior software engineers for environment construction and data cleaning, and 5 senior PhDs specializing in software engineering and LLMs for final auditing. For Doc2Repo, tests are adapted from original repositories with LLM assistance and human review, and stricter-than-source cases are documented in repository README files.
The evaluation protocol also introduces explicit isolation safeguards. Agents operate in one execution environment, but scoring is performed by extracting the generated patch via git diff, applying it to a fresh Docker container, and running tests there. Two anti-cheating mechanisms are highlighted. First, git-history sanitization removes all commits, logs, and metadata after the target commit while retaining earlier history for realistic inspection. Second, test restoration resets all test files to their original state after patch application, so any test modifications made by the agent are discarded. These measures are intended to prevent answer leakage and harness gaming.
The public release includes the BeyondSWE benchmark, the SearchSWE implementation, and the AweAgent scaffold. Each instance exposes structured JSON fields including instance_id, dataset_id, task, user, repo, language, workdir, image_url, patch, pr_commit, parent_commit, problem_statement, github_url, pre_commands, FAIL_TO_PASS, and PASS_TO_PASS. For reproducibility, this is a comparatively rich schema (Chen et al., 3 Mar 2026).
4. Evaluation protocol, metrics, and benchmark scale
BeyondSWE is evaluated with two agent frameworks: OpenHands and SearchSWE. The evaluated models are Gemini 3 Pro, GPT-5.2, DeepSeek-V3.2, Kimi-K2, GLM-4.7, MiniMax-M2.1, Seed-Coder, Qwen3-Coder-Plus, and Qwen3-235B-A22B-Instruct-2507. The appendix specifies a maximum of 200 interaction turns and a maximum context length equal to each model’s native limit (Chen et al., 3 Mar 2026).
Under OpenHands, the toolset consists of ExecuteBashTool, StrReplaceEditorTool, and FinishTool. SearchSWE adds SearchTool and BrowserTool. Search is implemented through Google Search via SerpAPI, webpage content extraction through Jina Reader, and webpage summarization by DeepSeek-V3.2.
For CrossRepo, DomainFix, and DepMigrate, the principal metric is Resolved Rate, defined as the percentage of instances where all P2P and F2P tests pass. For Doc2Repo, the paper reports Pass Rate, defined as the average percentage of tests passed, and (Almost) Correct Count, defined by the number of instances with all tests passed and the number with at least \geq 90\% of tests passed. The paper does not provide full formal equations for resolved rate, pass rate, pass@k, normalized cost, or aggregate score.
The benchmark’s aggregate scale is intended to cover heterogeneous software-engineering regimes. Final composition is 200 CrossRepo instances, 72 DomainFix instances, 178 DepMigrate instances, and 50 Doc2Repo instances, for a total of 500 instances over 246 repositories. This distribution makes BeyondSWE simultaneously an issue-resolution benchmark, a migration benchmark, and a repository-generation benchmark.
5. Empirical findings and capability gaps
The headline empirical result is that BeyondSWE exposes a substantial capability gap for current code agents. Even frontier systems “plateau below 45% success.” Under OpenHands, the best average score is Gemini 3 Pro: 41.82 AVG. Other strong average results are GLM-4.7: 41.20, DeepSeek-V3.2: 40.01, and Kimi-K2: 39.81. The paper contrasts this with reported “80%+” performance on SWE-bench Verified, arguing that broader software-engineering tasks remain far from solved (Chen et al., 3 Mar 2026).
Performance is heterogeneous across settings. Under OpenHands, Seed-Coder is best on CrossRepo with 44.72, while Gemini 3 Pro is second with 41.50. On DomainFix, GLM-4.7 is best with 36.11, and the table marks DeepSeek-V3.2: 30.56 as second-best. On DepMigrate, Gemini 3 Pro is best with 41.81, followed by GLM-4.7: 39.89. On Doc2Repo Pass Rate, DeepSeek-V3.2 is best with 54.99, followed closely by Kimi-K2: 54.91.
Doc2Repo illustrates a distinct failure mode. Moderate pass rates do not imply complete repository correctness. The table reports Gemini 3 Pro: (8) / 2, GPT-5.2: (6) / 2, Kimi-K2: (6) / 2, and DeepSeek-V3.2: (3) / 0 for (#Almost Correct) / #Correct. This indicates that many generated repositories partially satisfy the test suite while very few are fully correct.
No single model dominates all settings. The benchmark instead reveals capability specialization: Gemini 3 Pro is strongest overall and on DepMigrate, Seed-Coder is strongest on CrossRepo, GLM-4.7 is strongest on DomainFix under OpenHands, and DeepSeek-V3.2 is strongest on Doc2Repo pass rate. This suggests that present-day code agents do not exhibit a unified, robust software-engineering competence across heterogeneous task classes.
The most difficult setting in resolved-rate terms is DomainFix, where performance is described as “rarely exceeding 36%.” The paper attributes this to persistent bottlenecks in domain and scientific reasoning. Doc2Repo represents a different difficulty regime: pass rates are often around 45–55%, yet fully correct repositories are rare, indicating problems with architectural coherence and end-to-end system construction. CrossRepo and DepMigrate are somewhat easier, often in the 35–45% range, but still far below performance levels associated with single-repository bug fixing.
A common misconception is that repository-level generation and issue-level repair are simply different scales of the same capability. The reported results argue against that simplification. BeyondSWE’s own Doc2Repo findings show that passing many individual tests is not equivalent to constructing a coherent repository, and the cross-setting heterogeneity indicates that local debugging skill does not automatically transfer to cross-repo reasoning, domain reasoning, or migration workflows.
6. SearchSWE, search–code integration, and subsequent training on Doc2Repo-style tasks
SearchSWE was introduced as a search-augmented extension of a coding agent because many BeyondSWE tasks require information beyond the local repository. Its architecture combines a local context—a Docker container for inspecting files, editing code, executing shell commands, and running tests—with a global context exposed through SearchTool and BrowserTool. The intended workflow is to interleave reasoning about the task, local inspection, selective search, browsing, synthesis into code changes, local testing, and iteration, thereby approximating developer workflows that alternate between code and external information sources (Chen et al., 3 Mar 2026).
Empirically, search augmentation produces inconsistent gains. Under SearchSWE, the best average score is Gemini 3 Pro: 43.84 AVG, a gain of +2.0 over OpenHands, while GLM-4.7 reaches 41.75, a gain of +0.6. Search helps broadly on CrossRepo for several models—such as GPT-5.2: +3.2, GLM-4.7: +5.2, MiniMax-M2.1: +3.5, and Kimi-K2: +2.9—but hurts Seed-Coder: -5.8. DomainFix shows especially high variance: Gemini 3 Pro: +7.5, MiniMax-M2.1: +3.2, Qwen3-Coder-Plus: +11.6, but GLM-4.7: -3.7 and Seed-Coder: -2.8. DepMigrate is mixed, and Doc2Repo is generally harmed or only weakly helped by search, with Gemini 3 Pro: -1.3, Kimi-K2: -5.6, DeepSeek-V3.2: -1.4, and GPT-5.2: +2.0.
The paper’s interpretation is that there remains a “critical disconnect” between search and code. Search and coding abilities have matured separately in LLM systems, but not yet fused into a reliable developer-like workflow. Tool-usage analysis supports this view. Gemini 3 Pro achieves the most consistent gains with only 0.8–1.1 search calls per instance, whereas DeepSeek-V3.2 uses 4.2–5.4 search calls per instance and shows an unstable -0.2 average effect. This suggests that selective, well-targeted search is more valuable than frequent search.
The appendix identifies three qualitative search failure modes. The Information Landscape Gap arises when code-relevant knowledge is contained in source files or diffs, but search engines prioritize user-facing documentation. Temporal Misalignment and Version Bias occurs when search surfaces recent documentation that does not match the version in the local environment. Semantic Drift and Context Contamination appears when polysemous or niche technical queries return irrelevant material from unrelated domains. These analyses make the benchmark useful not only as an evaluation suite but also as a diagnostic instrument for retrieval-augmented code agents.
Subsequent work positions BeyondSWE-Doc2Repo as an important downstream benchmark for whole-repository generation. DeNovoSWE introduces a dataset of 4,818 high-quality instances for complete repository generation from documentation and uses BeyondSWE-Doc2Repo as a primary evaluation target. On that benchmark, the paper reports that fine-tuning Qwen3-30B-A3B-Instruct on DeNovoSWE raises its score from 0.058 = 5.8% to 0.472 = 47.2%. It also reports that issue-level training on Scale-SWE improves the same backbone only to 0.292 = 29.2%, indicating that task-aligned whole-repository generation data transfer more effectively to BeyondSWE-Doc2Repo than issue-level supervision (Zhao et al., 9 Jun 2026).
This relationship clarifies BeyondSWE’s broader significance. BeyondSWE is not merely a harder benchmark; it helps separate qualitatively different software-engineering capabilities. The DeNovoSWE results suggest that Doc2Repo-style performance depends strongly on aligned long-horizon training data rather than on patch-level SWE training alone. A plausible implication is that BeyondSWE, especially its Doc2Repo component, functions both as an evaluation benchmark and as a target for shaping future datasets and training regimes for repository-scale code agents.