SWE-Mirror Pipeline Overview
- SWE-Mirror is a pipeline that mirrors real GitHub issues into existing Gym environments, decoupling semantic data from repository-specific setups.
- It employs a three-phase process—task collection, mirroring via GPT-4o abstraction, and rigorous execution-based verification—to ensure task validity.
- The method scales to 60,671 validated tasks across multiple languages, significantly reducing engineering costs and improving coding agent performance.
SWE-Mirror is a pipeline for constructing large-scale verifiable issue-resolving datasets by taking the semantic essence of a real GitHub issue from a source repository, mirroring it into a different target repository that already has a configured Gym environment, and re-animating it as a verifiable task. It is designed to break the one-to-one dependency between authentic issue history and repository-specific environment engineering that has constrained prior dataset construction for software issue resolution. Applied to 40 repositories across 4 languages, the pipeline yields 60,671 validated tasks, denoted SWE-Mirror-60K, and is used to post-train coding agents evaluated on SWE-Bench-Verified and Multi-SWE-Bench-Flash (Wang et al., 10 Sep 2025).
1. Problem formulation and scaling rationale
The motivating bottleneck is the cost of turning authentic GitHub issue histories into runnable, testable Gym environments. The paper describes a rigid one-to-one dependency between the task context and the Gym: each real repository or version typically needs its own engineered environment, dependency setup, and validation harness. Existing automated Gym-creation pipelines are described as suffering from low success rates, high engineering overhead, and enormous storage cost; the paper notes that each Gym can consume about 1 GB, so scaling to 100,000 environments would require roughly 100 TB. At the same time, synthetic-task approaches such as SWE-Smith or SWE-Synth can scale, but they generate artificial bugs rather than exploiting authentic GitHub issue and fix histories (Wang et al., 10 Sep 2025).
SWE-Mirror addresses this by decoupling the semantic source of a task from the executable environment that verifies it. Rather than requiring the issue, tests, and Gym to come from the same repository, it uses real issue history as the semantic source and an existing Gym as the executable destination. A common misconception is that the method is another environment-synthesis pipeline; the paper’s central claim is the opposite. It scales by reusing a smaller number of existing environments and multiplying the utility of each Gym across many mirrored tasks.
2. Three-phase pipeline and the mechanics of mirroring
The pipeline comprises three phases: task collection, task mirroring, and task verification. In task collection, GitHub is searched for repositories similar to the target codebase, and merged pull requests with linked issues are harvested when they appear high-quality and mirrorable. In task mirroring, LLMs abstract the source issue and then instantiate an analogous task inside the target repository. In task verification, the introduced bug, the generated tests, and the ground-truth fix are checked for exact behavioral validity (Wang et al., 10 Sep 2025).
The mirroring stage is the key technical mechanism. The paper states that GPT-4o is used to distill an issue into an abstract description capturing the functionality, core logic, current versus expected behavior, and visible symptom. This abstraction removes repository-specific details such as local filenames, variable names, and stack traces. A “Test Agent” then uses the abstract description to generate a new test case inside the target Gym’s existing test suite. That test patch is withheld from the coding agent and serves as the hidden verification harness. A separate “Mirror Agent” takes the abstract description together with structural priors from the test patch, including file paths and function names, and edits the target code so that the new test fails. The resulting patch is mirror.patch, defining the buggy starting point; its inverse is automatically created as fix.patch, which becomes the ground-truth solution.
The final task statement shown to agents is synthesized from the original issue description, the generated test.patch, the fix.patch, and few-shot examples from SWE-Gym so that the mirrored task reads like a native issue in the target repository. This design preserves authentic issue semantics while relocating them into an already executable context.
3. Verification logic and dataset construction
SWE-Mirror emphasizes execution-based verifiability. Before running tests, the pipeline performs patch-application sanity checks to ensure that mirror.patch, test.patch, and fix.patch compose correctly. Verification then uses a three-log logic: Run.log after mirror.patch, Test.log after mirror.patch plus test.patch, and Fix.log after all three patches. The paper summarizes the acceptance criteria as follows: the mirror patch must apply cleanly; the test patch and fix patch must apply after the mirror patch; the test patch must be effective without disturbing unrelated tests; the fix patch must demonstrate at least one fail-to-pass transition; no regressions are allowed; and flaky or ambiguous behavior is discarded (Wang et al., 10 Sep 2025).
The transition rules are explicit. An effective test patch should preserve PASSED→PASSED, FAILED→FAILED, SKIPPED→SKIPPED, or NONE→FAILED behavior for unaffected tests. An effective fix must exhibit at least one ANY→FAILED→PASSED transition across the three execution states. Regressions such as PASSED→PASSED→FAILED or SKIPPED→SKIPPED→FAILED are disallowed. The paper also applies an LM-based prefilter over candidate issues, trained and evaluated on a balanced set of 100 manually labeled issues, and reports 84.3% precision and 86.0% recall.
After collection, mirroring, validation, and deduplication, the final dataset contains 60,671 validated tasks. The language distribution reported for SWE-Mirror-60K is as follows.
| Language | Tasks |
|---|---|
| Python | 46,820 |
| Rust | 7,183 |
| Go | 4,056 |
| JavaScript | 2,612 |
The paper additionally reports statistics such as average number of fix-patch hunks and lines, and counts of tasks whose tests are patch-to-pass (P2P) versus fail-to-pass (F2P), although the summary emphasizes scale and validation criteria rather than those detailed aggregates.
4. Training interface, trajectory collection, and post-training setup
The dataset is used through OpenHands, described as an event-driven agent framework that can edit files, run shell commands, and browse within sandboxed containers. For non-Python languages, the paper uses MOpenHands, a multi-language version. Agent trajectories are collected from a 15k subset of SWE-Mirror-60K using Claude-3.7-Sonnet and Claude-4-Sonnet, with three trials per task, temperature 1.0, and up to 100 rounds. A trajectory is counted as successful only if the agent finishes and the submitted patch fixes at least the tests that the ground-truth patch fixes (Wang et al., 10 Sep 2025).
This process yields 6,431 successful trajectories. Combined with 6,025 trajectories from SWE-rebench, the final post-training corpus contains 12,456 trajectories. The base models are Qwen2.5-Coder-Instruct-7B and Qwen2.5-Coder-Instruct-32B, finetuned into SWE-Mirror-LM-7B and SWE-Mirror-LM-32B using AdamW, weight decay 0.01, a cosine learning-rate schedule with warmup 0.1, and a max learning rate of for the main experiments. Evaluation uses resolved rate as the main metric on SWE-Bench-Verified and Multi-SWE-Bench-Flash.
5. Empirical performance and ablation findings
The principal empirical claim is that mirrored issue-resolving data materially improves coding-agent performance. On SWE-Bench-Verified, SWE-Mirror-LM-32B reaches a 52.2% resolve rate; on Multi-SWE-Bench-Flash, it reaches 21.33%. The paper further reports that, on SWE-Bench-Verified, Qwen2.5-Coder-Instruct based models trained with SWE-Mirror data achieve relative improvements of +21.8% for the 7B model and +46.0% for the 32B model in resolve rate, and presents SWE-Mirror-LM-32B as a new state-of-the-art among Qwen2.5-Coder-Instruct-based models on the OpenHands framework (Wang et al., 10 Sep 2025).
Several ablations are central to the method’s interpretation. The paper compares three training strategies for demonstration trajectories: Response Only, Error Pruning, and Error Masking. Error Masking is preferred because it preserves the full trajectory context while masking loss on incorrect assistant turns, allowing the model to learn from the recovery context without directly imitating errors. Across both 7B and 32B models, Error Masking consistently outperforms the alternatives, and the advantage grows with more data. The scaling curves are described as showing a clear scaling law: more trajectories lead to better SWE-Bench-Verified resolve rates. In the ablation setting, the 32B model reaches 35.6% with 4,096 trajectories under Error Masking.
The paper also studies cross-lingual generalization. Training on non-Python data still improves performance on the Python-only SWE-Bench-Verified benchmark. A 7B model trained on Rust, Go, or JavaScript data all improves over the 1.0% base model, with Rust producing the largest gain to 11.3%. The paper interprets this as suggesting that the model learns transferable problem-solving patterns rather than only language-specific heuristics.
6. Limitations, fidelity, and methodological significance
The paper is explicit that semantic mirroring is approximate rather than exact. The overall mirroring yield is reported as 46.0%, with Python substantially easier at 68.0%, while Rust and Go are markedly harder, largely because of compile or syntax failures rather than purely semantic failures. This language asymmetry indicates that typed and compiled languages are especially brittle for LLM-based patch generation (Wang et al., 10 Sep 2025).
Human auditing indicates substantial, but not perfect, fidelity: 156 of 177 majority-classified tasks were judged high or moderate consistency. At the same time, some generated tasks are inconsistent or unclassifiable. Coverage is also limited by repository-similarity search and by the availability of high-quality issue and pull-request pairs. The method therefore does not provide universal environment synthesis; it scales by reusing a constrained set of existing seed environments, subject to time and memory limits.
The broader significance claimed for SWE-Mirror is a change in scaling strategy for issue-resolving datasets. Instead of paying the full cost of environment creation for every authentic issue, or relying on synthetic bugs disconnected from real software evolution, it reuses existing Gym environments to host a much larger set of semantically grounded tasks derived from GitHub history. A plausible implication is that the method redefines the unit of reuse in software-engineering benchmarks: the scarce object is no longer the issue history, but the executable environment capable of validating many mirrored variants.
