SWE-Universe Framework

• SWE-Universe is a scalable, automated framework that constructs high-fidelity, reproducible software engineering environments derived from GitHub pull requests.
• It employs iterative self-verification loops and in-loop hacking detection to ensure script validity and optimized build yields.
• The framework leverages an 80B parameter Mixture-of-Experts model to refine build trajectories, enhance reproducibility and support multilingual environments.

SWE-Universe is a scalable, automated framework designed to construct real-world verifiable software engineering (SWE) environments at million-scale, leveraging large-scale data ingestion from GitHub pull requests (PRs) and a custom-trained model-based agentic workflow. This system addresses prevailing challenges in automated SWE environment construction, specifically low production yield, weak verification, and operational cost, by combining iterative self-verification loops and in-loop hacking detection. The output is over 800,000 containerized, high-fidelity environments covering a broad spectrum of programming languages, underpinning advances in agentic mid-training and reinforcement learning for coding agents (Chen et al., 2 Feb 2026).

1. Conceptual Architecture and Workflow

SWE-Universe operates as a sequential modular pipeline converting public PRs into isolated, reproducible “gyms” for software engineering tasks. Its architecture comprises the following components:

Module	Function	Technology/Method
Data Ingestion & Filtering	Crawls ~33.3M PRs (2021–2025), filters corpus	GitHub API, LLM-based filter
Patch Separation	Distinguishes test patches from fix patches	LLM-based semantic analysis
Autonomous Building Agent	Applies patches, generates verifiers	Qwen-Next-80A3 (80B MoE model)
Iterative Self-Verification Loop	Verifies patch validity	Bash scripts, script toggling procedures
In-Loop Hacking Detector	Detects superficial verifiers (“grep”, etc.)	Pattern matching, anomaly scoring
Quality Judge & Storage	Final scoring, Dockerization, storage	LLM agent, Alibaba Cloud ACR
Distributed Execution Infra	Scalability for millions of parallel jobs	MegaFlow, Elastic Compute Service

Environments are constructed by extracting candidate PRs, separating patch types, and invoking an autonomous agent to synthesize an evaluation.sh verification script. Two explicit repository states (“buggy,” “resolved”) are toggled using dedicated tools, and iterative script evaluation is performed for semantic correctness. In-Loop hacking detection restricts the validity of verifiers by excluding those relying on superficial output matching.

2. Building Agent and Model Architecture

The Autonomous Building Agent is realized by Qwen-Next-80A3, an 80 billion parameter Mixture-of-Experts (MoE) model utilizing hybrid local linear and global full attention mechanisms. Its learning objective follows standard cross-entropy over successful environment build trajectories with rejection sampling:

$$L(\theta) = -\sum_{t=1}^{T} \log p_{\theta}(a_t \mid s_{<t}, o_{\leq t})$$

The agent samples and filters trajectories to select successful, non-hacked builds. MoE routing enables per-token expert selection, balancing computational cost and model capacity per inference, while hybrid attention accelerates sequence processing, achieving a ~30–50% reduction in inference latency and cost versus dense 80B transformer models.

Environment-building is logic-driven, iterating up to MAX_TURNS (default 100):

for each PR in filtered_PRs: apply_test_patch() for turn in 1..MAX_TURNS: gen_script = building_agent.generate("Write evaluation.sh") if hacking_detector(gen_script) == HACKED: agent.receive_feedback("Script uses forbidden patterns") continue status_buggy = run_script(gen_script, mode=buggy) status_fixed = run_script(gen_script, mode=fixed) if status_buggy != 0 and status_fixed == 0: save_environment(PR, gen_script) break # success else: agent.receive_feedback("Verifier failed self-test") if no success after MAX_TURNS: mark_PR_as_failed()

3. Verification Procedures and Security Filters

Iterative self-verification ensures that only scripts able to distinguish fixed from buggy states are accepted. At each loop:

1. Agent proposes evaluation.sh
2. Repository is toggled to buggy; script execution expects non-zero exit
3. Repository toggled to fixed; execution expects zero exit
4. Failures prompt error return and agent adjustment (up to 100 turns)

In-Loop Hacking Detector scans each candidate script $S$ for forbidden patterns $$P = \{\text{grep}, \text{sed `s/…/…/`}, \text{awk}\}$$, computing anomaly score $S_{hack} = \sum_{p \in P} 1_{p \in S}$. Scripts with $S_{hack} > 0$ are rejected and flagged as “hacked.” This enforces dynamic evaluation over static output matching, essential for robust test generation.

4. Scalability and Corpus Composition

The pipeline processes 33.3 million PRs and filters down to approximately 1 million build candidates, resulting in 807,693 successfully validated environments spanning 52,960 repositories (average of 15.25 environments per repository). Language breakdown is as follows:

Language	Environment Count
Python	202,302
JS/TS	175,660
Go	121,062
Java	86,105
Rust	74,180
C/C++	37,228
C♯	24,387
Others	86,769

Yield profiles include an initial held-out success rate of 82.6%, post-self-verification and hacking-detection rates at 94% (small held-out), and large-scale non-hacked yield of 75.9%. Approximate build cost per environment is $0.25, deriving from build times ($\sim$5 min/environment) and compute rates ($\sim$$0.05$/min, ECS).

5. Multilingual Support, Fidelity, and Reproducibility

SWE-Universe achieves language-agnostic verification through a universal bash wrapper, with evaluation.sh dynamically invoking appropriate test frameworks (e.g., pytest, cargo test, go test, mvn test). Prompts embed language-agnostic instructions, optimizing tool invocation.

Reproducibility is enforced by iterative verifier loops and a Quality Judge agent, which attains 78.7% scoring accuracy versus human assessment for correctness and task alignment. Docker container registry and explicit environment file capture lock dependencies for reliable environment reconstruction.

6. Empirical Evaluation and Agentic Training

Automated build benchmarks (320 PRs) operationalize the following metrics:

• $$\text{SuccessRate}_{w/oHack} = \frac{\# \text{valid, non-hacked builds}}{320} \times 100\%$$
• $$\text{SuccessRate}_{w/Hack} = \frac{\# \text{builds distinguishing bug/fix}}{320} \times 100\%$$

Key model results include:

Model Name	SuccessRate (w/o hack)	SuccessRate (w/ hack)
Qwen-Next-80A3	78.44%	82.50%
Claude-Opus-4.5	77.81%	85.00%
Gemini-3-Pro	69.69%	72.50%

Additional empirical results span mid-training (500K agentic trajectories, 30B tokens) across five scaffolds, improving performance from 50.3% to 61.0% on SWE-Bench Verified and 31.0% to 46.2% on SWE-Bench Multilingual over 2,000 training steps. Reinforcement learning—using asynchronous RL frameworks with 128k context and 200-turn caps—yields peak test rates: Qwen3-30B-A3B reaches 42.0% on SWE-Bench Multilingual, and Qwen3-Max-Thinking attains 75.3% on SWE-Bench Verified.

7. Contributions, Limitations, and Future Directions

SWE-Universe constitutes the first end-to-end, million-scale pipeline for real-world, multilingual, and verifiable SWE environment synthesis. Its deployment of Qwen-Next-80A3 with self-verifying and hacking-resistant build loops establishes a benchmark for dataset fidelity, releasing 807,000 tasks from 52,000 repositories. The utility for agentic training is demonstrated by absolute gains (+11% Python-only benchmark, +10% multilingual), underscoring applicability for next-generation coding agent research.

Outstanding challenges include enhancing task description clarity, closer alignment with issue text, expansion to interactive/performance-sensitive domains (UI automation, benchmarking), adaptive updates for system environments (OS/library patching), confidentiality safeguards for private codebases, and utilizing static analysis or symbolic execution for enriched verifier coverage.

This cohesive framework and dataset lay the foundation for next-generation coding agents with true cross-lingual and real-world problem-solving capability (Chen et al., 2 Feb 2026).