- The paper introduces a sandboxed multi-agent pipeline that builds reproducible environments, generates validated tests, and writes solution-free issue descriptions from GitHub pull requests, creating 100,000 verified tasks across 5,200 Python repositories.
- The resulting dataset contains realistic, diverse tasks with a median of three modified files, 5.7 fail-to-pass tests, and 209 pass-to-pass tests per instance, while a manual audit found 94% of 100 sampled instances valid.
- Training Qwen3-30B-A3B on 71,498 successful agent trajectories improves SWE-bench Verified performance from 22.0% to 64.0%, outperforming controlled baselines trained on SWE-Gym and SWE-smith data.
Motivation and problem statement
The paper addresses the data bottleneck in training LLM-based software engineering (SWE) agents. Unlike conventional code generation, SWE tasks require executable environments, dependency management, and validated test suites, which makes systematic dataset curation difficult. Existing approaches either rely on labor-intensive manual curation, as in SWE-Gym (Pan et al., 2024), or on synthetic and rule-based generation, as in SWE-smith (Yang et al., 30 Apr 2025) and SWE-Mirror (Wang et al., 10 Sep 2025). The resulting datasets are limited in scale, repository diversity, or task realism. Scale-SWE (2602.09892) proposes an automated, sandboxed multi-agent pipeline that converts raw GitHub pull requests into verified, executable SWE task instances at scale.
The sandboxed multi-agent construction system
The framework coordinates three specialized agents built on the SWE-agent scaffold [(Wei et al., 25 Feb 2025) is not this; the scaffold citation is yang2024sweagent], powered by DeepSeek v3.1/v3.2 (DeepSeek-AI et al., 2 Dec 2025) and Gemini 3 Pro:
- Environment Builder Agent (EBA): initialized inside a base Docker container with the cloned repository, it autonomously inspects configuration files (
setup.py, pyproject.toml, README), resolves dependencies interactively from terminal feedback, and distills its command trajectory into a reproducible Dockerfile. To control cost, full environment construction is performed for at most ten PRs per repository; remaining PRs are evaluated in the "nearest" environment by PR-ID proximity, and discarded if tests fail there. This yields an average of 19 verified instances per repository.
- Unit-test Creator Agent (UCA): generates Fail-to-Pass (F2P) and Pass-to-Pass (P2P) suites via an interactive execute–analyze–refine loop inside the EBA-built container, verifying that F2P tests fail on the parent commit and pass on the merge commit. An anti-overfitting rule requires tests to invoke public APIs rather than newly introduced functions directly.
- Problem Statement Writer Agent (PSWA): uses Gemini 3 Pro to synthesize self-contained issue descriptions grounded in the generated test suite, deliberately avoiding solution leakage. Grounding in F2P tests is necessary because those tests may reference interfaces absent from the pre-fix codebase; descriptions derived only from PR metadata would otherwise be untractable or misaligned.
Each instance follows the SWE-bench schema, extended with a f2p_script field for agent-generated reproduction scripts when developer-written test patches are unavailable.
Data curation and quality assurance
Repositories are sourced from the top 15,000 PyPI packages plus SEART queries (Python, ≥5 contributors, ≥500 stars), yielding ~23,000 candidates after filtering. All PRs from SWE-bench Verified repositories are excluded to prevent contamination, only permissive licenses are retained, and LLM-as-a-judge filters remove GPU-dependent, tutorial-style repositories and low-value PRs (documentation-only, typo fixes, lock-file changes). From roughly 6 million merged PRs, the pipeline produces 100,000 verified instances across 5,200 real repositories—a 50% increase in repository count over SWE-rebench's 3.5k (Badertdinov et al., 26 May 2025) and far beyond SWE-Gym's 11 repositories.
Quality assurance combines a rule-based check (all P2P pass and F2P fail on the buggy code; all pass on the golden patch) with a manual audit of 100 sampled instances by four Ph.D. students under cross-validation, of which 94 were judged valid. A fixed P2P-then-F2P execution order prevents test pollution. An aggressive git sanitization script (hard reset, deletion of remote refs, tags, logs, packed-refs, followed by git gc --prune=now --aggressive) removes any trace of future commits, blocking solution leakage via commands such as git log --all.
The dataset exhibits substantial task complexity: median instances modify 3 files and add 43 lines, with a mean of 5.7 F2P and 209 P2P tests per instance. Bug-type annotation using the BugPilot taxonomy (Sonwane et al., 22 Oct 2025) shows a balanced distribution across ten categories, whereas SWE-smith is heavily skewed toward logic errors and SWE-Gym lacks variety due to its small repository pool.
Training results
The authors distilled trajectories from 25,000 instances using DeepSeek-V3.2 (five samples per instance, temperature 0.95, up to 100 turns), retaining only submissions passing all unit tests. This produced 71,498 trajectories totaling ~3.5 billion tokens. Fine-tuning Qwen3-30B-A3B-Instruct on this corpus (SFT with loss masking on assistant action turns, learning rate 1e-5, batch size 128, context length 131,072) yields Scale-SWE-Agent, evaluated on SWE-bench Verified with OpenHands as scaffolding and inference context extended to 262,144 tokens.
| Model |
Base |
SWE-bench Verified |
| Qwen3-30B-A3B-Instruct |
— |
22.0 |
| Qwen3-Coder-30B-A3B-Instruct |
— |
51.6 |
| GLM-4.7-Flash-30A3B |
— |
59.2 |
| KAT-Dev-32B |
— |
62.4 |
| Scale-SWE-Agent |
Qwen3-30B-A3B-Instruct |
64.0 |
The headline result is a 42-point absolute improvement over the base model (22% → 64%), exceeding same-size competitors and larger models such as SWE-RL (Llama3-70B, 41.0) and SWE-Fixer-72B (32.8). Under a controlled comparison using the identical distillation-and-SFT pipeline, Scale-SWE reaches 64.0 versus 54.8 for SWE-Gym and 54.6 for SWE-smith. Notably, SWE-smith—with ten times more instances than SWE-Gym—performs slightly worse than SWE-Gym, which the authors interpret as evidence of diminishing returns from purely synthetic data and a clear advantage for high-fidelity real-world corpora. Trajectory statistics further show that Scale-SWE tasks require more interaction turns and tokens than alternatives, consistent with their higher difficulty.
Limitations and open questions
Several caveats bear on these results. First, the expert check covers only 100 of 100,000 instances (94% validity), so the dataset-wide error rate is estimated rather than measured exhaustively. Second, the "nearest-environment" reuse strategy discards PRs whose tests fail outside their native timeline position, introducing a selection bias toward environments robust to nearby state drift; the magnitude of this bias is not quantified. Third, the corpus is Python-only, and the pipeline's reliance on conda/pip-based environment construction does not obviously transfer to compiled languages—the authors explicitly list Java, C, C++, and Rust extension as future work. Fourth, the comparison against SWE-Gym and SWE-smith holds the distillation pipeline fixed but not the underlying teacher model or instance-selection procedure, so the contribution of data realism versus other factors cannot be fully isolated. Finally, whether the gains persist under reinforcement learning rather than SFT distillation, and how performance scales beyond 71k trajectories, remain open questions the paper does not answer.
Conclusion
Scale-SWE demonstrates that a sandboxed multi-agent workflow can automate the full lifecycle of SWE dataset construction—environment building, test synthesis, and problem statement writing—at a scale (100k verified instances, 5.2k repositories) previously unattained with real-world data. The resulting fine-tuned agent achieves a 64% resolve rate on SWE-bench Verified, nearly tripling its base model and surpassing prior open-source systems of comparable or larger size. The controlled ablation indicating that real-world data outperforms larger synthetic corpora is arguably the paper's most consequential empirical finding, though its generalization beyond Python and beyond supervised fine-tuning remains to be established.
