---
title: 'BeyondSWE: Benchmark for Repository-Scale Code Agents'
url: https://www.emergentmind.com/topics/beyondswe
type: topic
---

# BeyondSWE: Benchmark for Repository-Scale Code Agents

Searching arXiv for the cited BeyondSWE paper and closely related repository-generation work to ground the article in current literature.
BeyondSWE is a software-engineering benchmark designed to evaluate whether contemporary code agents can operate beyond localized bug fixing inside a single repository. It was introduced to measure capabilities that prior SWE-style benchmarks largely underemphasize: cross-repository reasoning, domain-specialized issue resolution, dependency-driven migration, and full repository generation from specifications. The benchmark contains 500 real-world instances from 246 GitHub repositories and organizes them along two explicit dimensions—resolution scope and knowledge scope—thereby extending evaluation from “Local Func / Within Repo” tasks toward repository-wide transformation and specification-driven construction [2603.03194].

## 1. Conceptual scope and benchmark taxonomy

The central premise of BeyondSWE is that existing code-agent evaluation is dominated by what the paper characterizes as a localized regime: function-level or issue-level repair within one codebase, with most necessary knowledge assumed to be already present inside that repository. BeyondSWE broadens this regime along two axes.

The first axis is **resolution scope**, which measures how much of the codebase must be modified or produced. The benchmark distinguishes **Local Func** tasks from **Global Repo** tasks. CrossRepo and DomainFix remain close to issue-resolution settings, whereas DepMigrate requires coordinated repository-wide edits and Doc2Repo requires full repository construction.

The second axis is **knowledge scope**, which measures where the required information comes from. BeyondSWE explicitly enumerates **Within Repo**, **Cross Repo**, **Domain**, **Official Docs**, and **Human Spec**. This makes the benchmark’s organizing principle more explicit than prior SWE-style benchmarks, which the paper places almost entirely at “Local Func / Within Repo” [2603.03194].

| Setting | Resolution / Knowledge | Instances |
|---|---|---:|
| CrossRepo | Local Func / Cross Repo | 200 |
| DomainFix | Local Func / Domain | 72 |
| DepMigrate | Global Repo / Official Docs | 178 |
| Doc2Repo | Global Repo / Human Spec | 50 |

The comparison table reported for prior benchmarks sharpens the contrast. SWE-bench-Verified, SWE-bench-Live, and SWE-bench Pro are all categorized as **Local Func** in resolution scope and **Within Repo** in knowledge scope. BeyondSWE overall is described as **Mix / Mix**, because it deliberately spans both local and global engineering tasks and both internal and external knowledge sources.

The benchmark statistics also indicate a materially broader task profile. Averaged over non-Doc2Repo issue tasks, BeyondSWE instances involve **5.6 files** and **209.9 lines**. By setting, the paper reports **4.1 files / 190.7 lines** for CrossRepo, **4.2 files / 157.6 lines** for DomainFix, and **8.4 files / 281.6 lines** for DepMigrate. This suggests that BeyondSWE is not merely broad in description, but also broader in the scale of modifications required.

## 2. Task structure and the four settings

Each BeyondSWE instance has three components: a **problem statement**, a **Docker environment**, and a **test suite**. For CrossRepo, DomainFix, and DepMigrate, the problem statement is issue-style; for Doc2Repo, it is a specification document. For the issue-resolution settings, the test suite is divided into **P2P** tests, which should remain passing, and **F2P** tests, which should become passing. For Doc2Repo, the full test suite determines success [2603.03194].

**CrossRepo** evaluates cross-repository issue resolution. Agents must fix an issue in a target repository by leveraging information from external repositories, upstream implementations, or related projects. The expected output is a patch that causes all P2P and F2P tests to pass. The curation pipeline began from **3,000** candidates, retained about **800** after environment construction and stability inspection, and finalized **200 issues across 67 repositories** after manual verification. The paper reports an average of **1.3 links per issue**.

**DomainFix** targets issue resolution in scientific or technically specialized repositories where correctness depends on real domain knowledge rather than generic programming priors. Named fields include quantum computing, molecular dynamics, materials science, bioinformatics, convex optimization, astronomy, seismology, plasma physics, geospatial analysis, photonic IC design, and computational chemistry. The appendix lists repositories such as `astroplan`, `biotite`, `Biopython`, `cclib`, `PlasmaPy`, `qutip`, `obspy`, `cvxpy`, `geopandas`, `pymatgen`, `mdanalysis`, and `gdsfactory`. Construction involved collaboration with experts across **11 scientific disciplines**, identification of **21 high-quality repositories**, collection of about **800 candidate PRs**, reduction to about **200** after environment construction and stability checks, and final unanimous review by **three domain experts** per instance, yielding **72 issues across 12 repositories**.

**DepMigrate** addresses dependency-driven migration. Here the repository is checked out to the **pre-migration commit**, but the Docker environment is configured with the **upgraded dependency**. Agents must read official documentation and migration guides, locate affected call sites, and perform coordinated multi-file edits so that all P2P and F2P tests pass. The paper gives migration examples such as **Pydantic v1 to v2**, **Django 4.x to 5.0**, and **NumPy 1.x to 2.x**. The sourcing pipeline identified **23 widely used packages** with significant version upgrades, collected about **7,000** candidate PRs, retained about **1,000** after environment construction and stability inspection, and finalized **178 issues across 120 repositories** through manual verification by **four software engineering experts**.

**Doc2Repo** is the benchmark’s document-to-repository generation setting. The input consists only of a specification document, an **empty workspace**, and a test suite. The specification includes purpose, usage examples, and API details such as function or class signatures, parameters, return types, and behavior descriptions, but explicitly excludes implementation details and directory structure. Repository names are masked as `target_repo`, forcing agents to infer project structure from usage context. Repositories were collected from projects created between **January and November 2025**, required to remain active after **August 2025**, and to have at least **three contributors** and more than **20 stars**. After environment construction, the authors obtained **60 instances** and manually selected **50 high-quality instances**. The appendix reports repository scales ranging from about **1,000** to over **16,000 lines of code**, with **40 out of 50** exceeding **1,500 lines**.

## 3. Construction pipeline, environment reconstruction, and benchmark hygiene

A major contribution of BeyondSWE is its benchmark-construction pipeline, especially for historical environment reconstruction. The paper describes an **agent-based environment construction** process that starts from a base Ubuntu Docker container, instantiates an LLM agent (**Gemini 3 Pro**), clones the repository, checks out the pre-PR commit, runs tests in a loop, iteratively fixes environment and dependency issues—including system-level commands such as `apt-get`—and finally distills the working command history into a minimal reproducible Dockerfile [2603.03194].

Generated Dockerfiles are then subjected to **stability inspection**. Each Dockerfile is built and tested **five times**. Instances are retained only if the behavior is deterministic: before patching, P2P tests pass and F2P tests fail; after applying the gold patch, both P2P and F2P tests pass. Any instance with non-determinism or failed criteria is removed.

Because pull requests often contain explicit solution information, the benchmark sanitizes task descriptions by rewriting PR descriptions into issue-style problem statements using Gemini 3 Pro. The stated intent is to preserve symptoms, reproduction steps, logs, and environment information while removing explicit fix logic, commits, and author references. This is especially important for CrossRepo, where original PR histories may include external links that directly expose solution patterns.

Human review is extensive. The paper reports **3 domain experts in scientific computing** for DomainFix repository selection, **5 senior software engineers** for environment construction and data cleaning, and **5 senior PhDs specializing in software engineering and LLMs** for final auditing. For Doc2Repo, tests are adapted from original repositories with LLM assistance and human review, and stricter-than-source cases are documented in repository README files.

The evaluation protocol also introduces explicit isolation safeguards. Agents operate in one execution environment, but scoring is performed by extracting the generated patch via `git diff`, applying it to a **fresh Docker container**, and running tests there. Two anti-cheating mechanisms are highlighted. First, **git-history sanitization** removes all commits, logs, and metadata after the target commit while retaining earlier history for realistic inspection. Second, **test restoration** resets all test files to their original state after patch application, so any test modifications made by the agent are discarded. These measures are intended to prevent answer leakage and harness gaming.

The public release includes the BeyondSWE benchmark, the SearchSWE implementation, and the AweAgent scaffold. Each instance exposes structured JSON fields including `instance_id`, `dataset_id`, `task`, `user`, `repo`, `language`, `workdir`, `image_url`, `patch`, `pr_commit`, `parent_commit`, `problem_statement`, `github_url`, `pre_commands`, `FAIL_TO_PASS`, and `PASS_TO_PASS`. For reproducibility, this is a comparatively rich schema [2603.03194].

## 4. Evaluation protocol, metrics, and benchmark scale

BeyondSWE is evaluated with two agent frameworks: **OpenHands** and **SearchSWE**. The evaluated models are **Gemini 3 Pro**, **GPT-5.2**, **DeepSeek-V3.2**, **Kimi-K2**, **GLM-4.7**, **MiniMax-M2.1**, **Seed-Coder**, **Qwen3-Coder-Plus**, and **Qwen3-235B-A22B-Instruct-2507**. The appendix specifies a maximum of **200** interaction turns and a maximum context length equal to each model’s native limit [2603.03194].

Under OpenHands, the toolset consists of `ExecuteBashTool`, `StrReplaceEditorTool`, and `FinishTool`. SearchSWE adds `SearchTool` and `BrowserTool`. Search is implemented through **Google Search via SerpAPI**, webpage content extraction through **Jina Reader**, and webpage summarization by **DeepSeek-V3.2**.

For CrossRepo, DomainFix, and DepMigrate, the principal metric is **Resolved Rate**, defined as the percentage of instances where **all P2P and F2P tests pass**. For Doc2Repo, the paper reports **Pass Rate**, defined as the average percentage of tests passed, and **(Almost) Correct Count**, defined by the number of instances with all tests passed and the number with at least `\geq 90\%` of tests passed. The paper does not provide full formal equations for resolved rate, pass rate, pass@k, normalized cost, or aggregate score.

The benchmark’s aggregate scale is intended to cover heterogeneous software-engineering regimes. Final composition is **200** CrossRepo instances, **72** DomainFix instances, **178** DepMigrate instances, and **50** Doc2Repo instances, for a total of **500** instances over **246** repositories. This distribution makes BeyondSWE simultaneously an issue-resolution benchmark, a migration benchmark, and a repository-generation benchmark.

## 5. Empirical findings and capability gaps

The headline empirical result is that BeyondSWE exposes a substantial capability gap for current code agents. Even frontier systems “plateau below 45% success.” Under OpenHands, the best average score is **Gemini 3 Pro: 41.82 AVG**. Other strong average results are **GLM-4.7: 41.20**, **DeepSeek-V3.2: 40.01**, and **Kimi-K2: 39.81**. The paper contrasts this with reported “80%+” performance on SWE-bench Verified, arguing that broader software-engineering tasks remain far from solved [2603.03194].

Performance is heterogeneous across settings. Under OpenHands, **Seed-Coder** is best on CrossRepo with **44.72**, while **Gemini 3 Pro** is second with **41.50**. On DomainFix, **GLM-4.7** is best with **36.11**, and the table marks **DeepSeek-V3.2: 30.56** as second-best. On DepMigrate, **Gemini 3 Pro** is best with **41.81**, followed by **GLM-4.7: 39.89**. On Doc2Repo Pass Rate, **DeepSeek-V3.2** is best with **54.99**, followed closely by **Kimi-K2: 54.91**.

Doc2Repo illustrates a distinct failure mode. Moderate pass rates do not imply complete repository correctness. The table reports **Gemini 3 Pro: `(8) / 2`**, **GPT-5.2: `(6) / 2`**, **Kimi-K2: `(6) / 2`**, and **DeepSeek-V3.2: `(3) / 0`** for `(#Almost Correct) / #Correct`. This indicates that many generated repositories partially satisfy the test suite while very few are fully correct.

No single model dominates all settings. The benchmark instead reveals capability specialization: Gemini 3 Pro is strongest overall and on DepMigrate, Seed-Coder is strongest on CrossRepo, GLM-4.7 is strongest on DomainFix under OpenHands, and DeepSeek-V3.2 is strongest on Doc2Repo pass rate. This suggests that present-day code agents do not exhibit a unified, robust software-engineering competence across heterogeneous task classes.

The most difficult setting in resolved-rate terms is DomainFix, where performance is described as “rarely exceeding 36%.” The paper attributes this to persistent bottlenecks in domain and scientific reasoning. Doc2Repo represents a different difficulty regime: pass rates are often around **45–55%**, yet fully correct repositories are rare, indicating problems with architectural coherence and end-to-end system construction. CrossRepo and DepMigrate are somewhat easier, often in the **35–45%** range, but still far below performance levels associated with single-repository bug fixing.

A common misconception is that repository-level generation and issue-level repair are simply different scales of the same capability. The reported results argue against that simplification. BeyondSWE’s own Doc2Repo findings show that passing many individual tests is not equivalent to constructing a coherent repository, and the cross-setting heterogeneity indicates that local debugging skill does not automatically transfer to cross-repo reasoning, domain reasoning, or migration workflows.

## 6. SearchSWE, search–code integration, and subsequent training on Doc2Repo-style tasks

SearchSWE was introduced as a search-augmented extension of a coding agent because many BeyondSWE tasks require information beyond the local repository. Its architecture combines a **local context**—a Docker container for inspecting files, editing code, executing shell commands, and running tests—with a **global context** exposed through `SearchTool` and `BrowserTool`. The intended workflow is to interleave reasoning about the task, local inspection, selective search, browsing, synthesis into code changes, local testing, and iteration, thereby approximating developer workflows that alternate between code and external information sources [2603.03194].

Empirically, search augmentation produces **inconsistent** gains. Under SearchSWE, the best average score is **Gemini 3 Pro: 43.84 AVG**, a gain of **+2.0** over OpenHands, while **GLM-4.7** reaches **41.75**, a gain of **+0.6**. Search helps broadly on CrossRepo for several models—such as **GPT-5.2: +3.2**, **GLM-4.7: +5.2**, **MiniMax-M2.1: +3.5**, and **Kimi-K2: +2.9**—but hurts **Seed-Coder: -5.8**. DomainFix shows especially high variance: **Gemini 3 Pro: +7.5**, **MiniMax-M2.1: +3.2**, **Qwen3-Coder-Plus: +11.6**, but **GLM-4.7: -3.7** and **Seed-Coder: -2.8**. DepMigrate is mixed, and Doc2Repo is generally harmed or only weakly helped by search, with **Gemini 3 Pro: -1.3**, **Kimi-K2: -5.6**, **DeepSeek-V3.2: -1.4**, and **GPT-5.2: +2.0**.

The paper’s interpretation is that there remains a “critical disconnect” between search and code. Search and coding abilities have matured separately in LLM systems, but not yet fused into a reliable developer-like workflow. Tool-usage analysis supports this view. **Gemini 3 Pro** achieves the most consistent gains with only **0.8–1.1 search calls per instance**, whereas **DeepSeek-V3.2** uses **4.2–5.4 search calls per instance** and shows an unstable **-0.2 average** effect. This suggests that selective, well-targeted search is more valuable than frequent search.

The appendix identifies three qualitative search failure modes. **The Information Landscape Gap** arises when code-relevant knowledge is contained in source files or diffs, but search engines prioritize user-facing documentation. **Temporal Misalignment and Version Bias** occurs when search surfaces recent documentation that does not match the version in the local environment. **Semantic Drift and Context Contamination** appears when polysemous or niche technical queries return irrelevant material from unrelated domains. These analyses make the benchmark useful not only as an evaluation suite but also as a diagnostic instrument for retrieval-augmented code agents.

Subsequent work positions BeyondSWE-Doc2Repo as an important downstream benchmark for whole-repository generation. **DeNovoSWE** introduces a dataset of **4,818 high-quality instances** for complete repository generation from documentation and uses BeyondSWE-Doc2Repo as a primary evaluation target. On that benchmark, the paper reports that fine-tuning **Qwen3-30B-A3B-Instruct** on DeNovoSWE raises its score from **0.058 = 5.8%** to **0.472 = 47.2%**. It also reports that issue-level training on Scale-SWE improves the same backbone only to **0.292 = 29.2%**, indicating that task-aligned whole-repository generation data transfer more effectively to BeyondSWE-Doc2Repo than issue-level supervision [2606.10728].

This relationship clarifies BeyondSWE’s broader significance. BeyondSWE is not merely a harder benchmark; it helps separate qualitatively different software-engineering capabilities. The DeNovoSWE results suggest that Doc2Repo-style performance depends strongly on aligned long-horizon training data rather than on patch-level SWE training alone. A plausible implication is that BeyondSWE, especially its Doc2Repo component, functions both as an evaluation benchmark and as a target for shaping future datasets and training regimes for repository-scale code agents.

Source: https://www.emergentmind.com/topics/beyondswe