SWE-Next: Scalable Execution-Grounded SWE Framework

• SWE-Next is an execution-grounded framework that generates scalable, verifiable software tasks from real repository histories.
• It filters commit pairs by ensuring at least one test improves without any regressions, thus guaranteeing high-quality supervision.
• By reusing repository-quarter profiles and containerized environments, SWE-Next reduces system costs and enhances reproducibility.

SWE-Next is an execution-grounded framework for scalable software engineering task and trajectory collection from real repositories. It was introduced to address two scale barriers in training repository-level SWE agents: only a small fraction of real repository changes yield verifiable, high-signal instances, and naively building repository-specific environments quickly becomes the dominant systems cost. The framework mines real merged pull requests, forms base/merged commit pairs, executes the same test command on both states, retains only commit pairs with strict test improvements and no regressions, applies strict submission gating so collected trajectories remain evidence-driven rather than speculative, and reuses environments through repo-quarter profiles while keeping task runs isolated and reproducible (Liang et al., 21 Mar 2026).

1. Problem setting and design objectives

SWE-Next targets executable, verifiable tasks for training repository-level SWE agents. Its motivating claim is that data quality and systems cost are the two main blockers to scale. On the data side, weak filtering and speculative rollouts dilute supervision because only a small fraction of real repository changes produce verifiable, high-signal instances. On the systems side, per-repo or per-commit environment construction dominates time and storage, while environment drift and fragility further raise costs (Liang et al., 21 Mar 2026).

The framework’s central design choice is to treat real repository history as the source of tasks, but only after strict execution-based filtering. Real merged pull requests are mined, converted into commit pairs, and evaluated under a conservative criterion: at least one test must improve and no previously passing test may regress. This yields self-verifying instances by construction. The same execution-grounded philosophy carries over to trajectory collection, where submission gating requires observable evidence rather than speculative completion.

The current run focuses on curated Python repositories with pytest-style harnesses and containerizable dependencies. Seed repositories are selected from top PyPI categories mapped to GitHub, with duplicates removed and preference given to active maintenance and stable test commands. This scope is intentionally narrow: the paper presents extension to other languages as future work rather than claiming language-agnostic coverage (Liang et al., 21 Mar 2026).

2. End-to-end data construction pipeline

The SWE-Next pipeline is organized as a sequence of execution-aware stages.

1. Repository ingestion: seed selection uses curated Python repositories filtered for active maintenance, standard test harness, and containerizable builds.
2. Candidate mining: for each merged PR, SWE-Next derives a commit pair $(c_{\text{base}}, c_{\text{merged}})$ corresponding to the target branch state immediately before merge and after merge, including merge, squash, and rebase cases.
3. Lightweight pre-filtering: documentation-only or irrelevant edits are dropped; Python-source changes are kept; patch size and complexity are bounded to favor localizable, execution-worthy changes and to control cost.
4. Environment bring-up: the system resolves a repo-quarter profile, mounts the specific commit snapshot read-only, copies it into a writable workspace, and links the shared environment.
5. Execution-grounded filtering: the same test command is run on both commits, per-test outcomes are parsed, and only overlapping test identifiers are compared, with a fallback to file-level grouping under refactors.
6. Instance packaging: retained examples are emitted as SWE-Bench-like rows with base commit, unified diffs split into code versus test changes, FAIL_TO_PASS and PASS_TO_PASS lists, and reproducible Docker artifacts. A concise problem statement may be generated from diff and test evidence without leaking explicit curated test lists to downstream agents.
7. Trajectory collection: a fixed, non-leaky agent scaffold is used, with strict submission gating and storage of tool calls, diffs, logs, and outcomes for supervised fine-tuning (Liang et al., 21 Mar 2026).

This pipeline couples repository mining to runtime validation. The result is not merely a corpus of patches, but a corpus of task instances whose supervisory signal is grounded in differential execution between two commits. A plausible implication is that SWE-Next is better viewed as an overview-and-verification pipeline than as a static dataset alone.

3. Self-verification criterion and trajectory gating

The formal core of SWE-Next is the NewCommitBetter criterion. Let $R(c)$ map a test identifier to its status on commit $c$. SWE-Next compares only

$$\mathrm{CommonTests} = \mathrm{keys}(R(c_{\text{base}})) \cap \mathrm{keys}(R(c_{\text{merged}})).$$

If no overlapping test identifiers exist, the system falls back to file-level grouping to preserve comparability after refactors. It then defines

$$\mathrm{Improved} = \{ t \in \mathrm{CommonTests} \mid R(c_{\text{base}})[t] \neq \mathrm{PASSED} \ \text{and} \ R(c_{\text{merged}})[t] = \mathrm{PASSED} \},$$

and

$$\mathrm{Regressed} = \{ t \in \mathrm{CommonTests} \mid R(c_{\text{base}})[t] = \mathrm{PASSED} \ \text{and} \ R(c_{\text{merged}})[t] \neq \mathrm{PASSED} \}.$$

A candidate is retained iff $|\mathrm{Improved}| > 0$ and $|\mathrm{Regressed}| = 0$ (Liang et al., 21 Mar 2026).

Operationally, “strict test improvements without regressions” means that at least one failing or non-passing test becomes PASSED, no previously passing test becomes non-passing, and setup failures, test-run failures, unparseable logs, or the absence of comparable tests lead to rejection. The treatment is conservative: instability and proto-flakiness are handled by dropping instances rather than attempting to rescue them.

Trajectory collection is subject to equally strict gating. Prompts are non-leaky: curated FAIL_TO_PASS and PASS_TO_PASS lists are not exposed to the agent. A submission is accepted only if the agent produces a non-empty repository diff and runs at least one test command. This requirement is meant to ensure evidence-driven behavior. The paper’s pseudocode makes this explicit through HasNonEmptyDiff(state) and HasRunAtLeastOneTest(state) checks before a rollout can be marked successful (Liang et al., 21 Mar 2026).

A concrete accepted instance is one where the base commit has test_mod::test_x FAILED, the merged commit has test_mod::test_x PASSED, and all previously passing overlapping tests remain PASSED. In that case, Improved = {test_mod::test_x}, Regressed = {}, and the pair is accepted as self-verifying. The packaged instance includes the non-test diff, the test-only diff, FAIL_TO_PASS, PASS_TO_PASS, logs, and replay artifacts.

4. Systems architecture: repo-quarter profiles and reproducibility

The main systems innovation in SWE-Next is the repo-quarter profile. A profile is a reusable dependency environment identified by

$$\text{profile} = \langle \text{repo\_name} \rangle\_\langle \text{year} \rangle Q \langle \text{quarter} \rangle,$$

computed deterministically from commit timestamps. Each profile encapsulates system packages via apt, a Python interpreter, and a cached virtualenv with pytest and inferred or pinned dependencies relevant to that repository and time window. Repository source is not baked into the image; source is mounted per run (Liang et al., 21 Mar 2026).

The motivation is temporal locality in dependencies. Dependencies evolve slower than commits, so quarter granularity captures many regime changes while enabling broad reuse. This decouples dependency setup from source snapshots and amortizes the dominant cost of building and storing environments. The execution model mounts the commit snapshot read-only, copies it into a fresh writable workspace at container start, and links the shared environment. The copy-on-start design preserves isolation because agent edits never mutate the host snapshot, while still allowing ordinary git and test tooling inside the container.

Deterministic selection of the profile ID ensures that base and merged commits for the same quarter reuse the same environment. If a quarter build fails or proves insufficient, SWE-Next falls back to a per-commit environment rather than dropping coverage. Images are tagged per profile, and per-instance sidecars include the Dockerfile, resolved image tag, and execution logs for replayable runs.

The systems impact is substantial. The paper reports that per-candidate images at roughly 300 MB each would imply about 30.8 TB for 102,582 candidates, whereas reusable profiles reduce the entire run to 639 GB for environments and artifacts. This suggests that the framework’s scalability depends as much on the environment abstraction as on the filtering criterion itself (Liang et al., 21 Mar 2026).

5. Dataset structure, operational footprint, and benchmark results

A released SWE-Next instance follows a SWE-Bench-like JSONL format with sidecar artifacts. Inputs include repository identity, base commit, and an optional problem statement. Evidence includes parsed execution results for base and merged commits, FAIL_TO_PASS and PASS_TO_PASS lists computed from strict comparison, unified diffs split into code patch versus test patch, and logs. Environment fields include a Dockerfile, resolved docker image tag, and environment_setup_commit for reproducibility. The expected output is the per-test status on the merged commit, and exec_type is NEW_COMMIT_BETTER for released rows (Liang et al., 21 Mar 2026).

The single run described in the paper processed 3,971 seeded Python repositories and 102,582 candidate commit pairs mined from real merged PRs, yielding 2,308 self-verifying instances spanning 311 repositories. End-to-end processing took 30 hours and used 639 GB of environment storage and artifacts. The yield is approximately 2.2%. Failure composition across candidates is reported as 74.5% NewCommitNotBetter, 2.5% setup failures, and 20.8% test-run failures. Appendix statistics report 1,273 repo-quarter profiles, 173 unique environment signatures, and coverage of 55 quarters from 2012-01-31 to 2025-12-25. A collection run produced 8,454 trajectories, of which 3,693 were selected for SFT; 72.6% of trajectories succeed, and among successful trajectories at least one test command appears in 97.6% and an explicit reproduction script in 90.3% (Liang et al., 21 Mar 2026).

Evaluation uses SWE-Bench-Lite and SWE-Bench-Verified, with pass@1 reported as the percentage of tasks resolved with a single attempt under the benchmark harness. Expert trajectory generators include GPT-5-mini and Claude 4.5 Sonnet, with Claude 3.5 Sonnet used in ablations; student models are Qwen-2.5-Coder-Instruct 7B and 14B. Full-parameter SFT is performed via LLaMA-Factory with learning rate $10^{-5}$, context length 32,768, batch size 8, up to 4 epochs, and 8× A100 GPUs. Both expert rollouts and student inference use a rollout budget of at most 40 tool-interaction steps and vLLM inference at temperature 0 (Liang et al., 21 Mar 2026).

Model / training setup	Verified pass@1	Lite pass@1
Qwen2.5-Coder-Instruct 7B baseline	1.8%	1.0%
7B + R2E-Gym SFT	14.4%	11.3%
7B + SWE-smith SFT	15.2%	11.7%
7B + SWE-Next SFT	17.4%	13.7%
Qwen2.5-Coder-Instruct 14B baseline	4.0%	2.7%
14B + R2E-Gym SFT	26.8%	20.7%
14B + SWE-Next SFT	30.0%	23.3%

The main ablation controls for trajectory generator quality by using the same generator, Claude 3.5 Sonnet, and matched trajectory counts. Under that control, SFT on SWE-Next trajectories still outperforms SFT on R2E-Gym trajectories: 8.2% versus 7.8% on Verified and 6.67% versus 5.0% on Lite. The paper’s conclusion is that gains stem from execution-grounded supervision, stricter gating, and task construction rather than from a stronger generator. This is reinforced by the broader claim that data quality dominates raw volume when trajectory counts are comparable (Liang et al., 21 Mar 2026).

6. Limitations, operational use, and relation to later training pipelines

SWE-Next’s limitations follow directly from its execution-grounded design. Language coverage is currently Python-only. Test quality and test coverage determine which merged PRs can become accepted instances, so weak or underspecified tests reduce yield. The pipeline errs on the side of rejection for flakiness and non-determinism, and no multi-run flakiness modeling is reported in the described run. Because the framework mines merged PRs that passed CI and have usable tests, it may under-represent hard, under-specified, or non-test changes. Repo-quarter reuse also introduces a residual drift risk for rare commits, though per-commit fallback mitigates this (Liang et al., 21 Mar 2026).

The paper provides a practical adoption recipe: select repositories with a stable Dockerized test command, implement repo-quarter profiles without baking source into images, mine merged PRs and pre-filter by patch size and file type, execute both commits inside the selected quarter image, accept only strict improvements with no regressions, package SWE-Bench-like rows, optionally collect non-leaky trajectories with submission gating, and evaluate fine-tuned models on SWE-Bench-Lite and SWE-Bench-Verified using pass@1. Suggested operational practices include dependency pinning, wheel caching, copy-on-start isolation, hard timeouts, batch-building quarter images for active repos, publishing profile-tagged images to a registry, and restricting privileges and network egress during tests.

Ethical and security considerations are explicit because the framework executes untrusted repository code. Recommended controls include containerization, privilege dropping, restricted egress, strict resource and time limits, dependency scanning, secret isolation, and sanitization of logs to avoid exposing credentials or personal data in trajectory artifacts (Liang et al., 21 Mar 2026).

Subsequent work on SWE-agent training suggests a complementary rather than competing role for SWE-Next. BugPilot introduces FeatAdd, a synthetic bug-generation method in which an agent adds a feature and unintentional test breakages are recorded as realistic bugs. BugPilot reports that these bugs are difficult, diverse, and data-efficient for training, and describes an “AllData SFT” setting as the best-performing “SWE-Next” recipe in its own experimental framing (Sonwane et al., 22 Oct 2025). A plausible implication is that execution-grounded mining from real merged PRs and synthetic generation of realistic regressions can be combined: the former provides self-verifying instances rooted in authentic repository history, while the latter broadens bug distributions and training mixtures under controlled generation.

Future directions named for SWE-Next include adaptive profile clustering beyond quarter boundaries, content-aware environment fingerprints, multi-language support, domain-specific harnesses such as CUDA or Android, richer supervision signals from coverage traces or static and dynamic analysis hooks, explicit flakiness checks through multi-run voting, structured error-taxonomy filtering, and stronger or cheaper rollout generators and verifiers. In that sense, SWE-Next defines a reproducible recipe—strict filtering plus environment reuse—for building scalable, execution-grounded SWE data infrastructure rather than a fixed terminal benchmark (Liang et al., 21 Mar 2026).