SWE-Gym: Autonomous SWE Agent Benchmark

• SWE-Gym is a reproducible research environment designed for evaluating language model-based software engineering agents with realistic, executable tasks in an OpenAI-Gym style setup.
• It offers a large-scale benchmark suite featuring Docker images, curated Python repositories, and synthetic task generation to support rigorous agent training and testing.
• The platform employs advanced verifier pipelines that combine execution-based and LLM-based validation, ensuring reliable benchmarking and improved agent performance.

SWE-Gym is a research environment and benchmark suite specifically designed for the training and evaluation of LLM-based software engineering (SWE) agents on real-world tasks. It provides a large-scale, reproducible, executable environment, supporting rigorous assessment and development of autonomous agents that fix bugs or implement small enhancements in open-source Python repositories. SWE-Gym, along with its variant R2E-Gym, represents a central infrastructure underpinning the rapid advancement of autonomous SWE-agent research, merging realistic task composition, scalable synthetic dataset generation, and advanced verification strategies (Pan et al., 2024, Jain et al., 9 Apr 2025).

1. Environment Design and Dataset Composition

SWE-Gym delivers executable, OpenAI-Gym-style task environments for software engineering agents. Each of the 2,438 core SWE-Gym tasks consists of:

• A codebase snapshot at a specific semantic version, including all non-test source files
• A pre-built Docker image providing an executable runtime with all dependencies (∼2.6 GB per image, totaling 6 TB)
• A human-written natural-language issue description (mean length ≈ 240 words)
• A suite of unit tests for automated verification

The dataset enables manipulation and evaluation of agent actions (editing files, running commands, rerunning tests, etc.) in a controlled, reproducible environment. To facilitate rapid prototyping, SWE-Gym Lite offers a curated 230-instance subset of easier, self-contained tasks. Additionally, SWE-Gym Raw provides 66,894 GitHub issues from 358 repositories, without prebuilt environments, for future dataset synthesis. Task statistics reveal an average of 971.2 non-test files per repository, patches editing a mean of 2.5 files and 69.8 lines, and 10.0 failing-to-passing test transitions per task.

The environment architecture of R2E-Gym scales this approach, curating over 8,135 real-world Python tasks. These are predominantly bug-fix commits where tests transition from failing to passing, with a smaller fraction representing feature additions. Repository distribution is long-tailed: pandas constitutes approximately one-third of SWE-Gym tasks, while repositories like bokeh contribute less than 1%.

2. Procedural and Synthetic Task Generation

R2E-Gym extends SWE-Gym’s approach through the SYNGEN recipe, which supports scalable, procedural curation of real-world tasks. The pipeline includes:

• Test-generation: For each candidate commit, either extract existing fail→pass unit tests or invoke a trained “testing agent” to generate reproduction tests using the repository state and commit diff. Typically, each task includes ≈10 generated or extracted tests.
• Back-translation (issue synthesis): A LLM is prompted with commit metadata, test code, and execution logs to produce a GitHub-style natural language issue description mimicking authentic problem statements.

Filtering strategies (e.g., patch size heuristics, AST-level edit detection, LLM-based filtering) ensure diversity and technical quality. Coverage metrics such as repo diversity and AST node uniqueness formalize the breadth of the procedural dataset. The procedural recipe can be rerun indefinitely on new commit pools, reinforcing continual expansion and dataset freshness.

3. Agent Training Methodologies

SWE-Gym was used to train a family of Qwen-2.5-Coder-Instruct LLMs at 7B, 14B, and 32B parameter scales, leveraging both large-scale text-code pretraining and specialized fine-tuning. The core fine-tuning methodology is filtered behavior cloning:

• Agent–environment trajectories $\tau = [(o_1, a_1), \ldots, (o_n, a_n)]$ are sampled from high-quality teacher models (e.g., GPT-4o, Claude-3.5)
• Only trajectories that fully resolve the issue (i.e., pass all unit tests) are retained
• The student model is optimized via negative log-likelihood to maximize expert action probabilities, conditioned on context and issue text:

$$\mathcal{L}(\theta) = -\sum_{t=1}^{n} \log \pi_\theta(a_t \mid \tau_{<t}, \text{issue\_text})$$

• Hyperparameter configuration (OpenHands scaffold): learning rate $1 \times 10^{-4}$, batch size 8, epochs up to 5, and maximum context length 32,768 tokens
• LoRA fine-tuning is used for some verifier and MoatlessTools experiments (rank 64, learning rate $5 \times 10^{-4}$)

4. Verification and Inference-Time Scaling

The execution of multiple candidate solutions per task instance, coupled with verifier-based selection, is central to SWE-Gym’s performance.

• Outcome-Supervised Reward Model (ORM): A Qwen-32B LLM is fine-tuned to predict whether a trajectory $\tau$ is successful. The output is a scalar probability $p_{yes} \in [0,1]$, computed via the log-probabilities of "YES"/"NO" tokens.
• Inference-time pipeline:

1. Generate $K$ candidate trajectories using the agent, with a prescribed temperature schedule. 2. Score each via the ORM verifier. 3. Select the solution maximizing $p_{yes}$.

R2E-Gym integrates both execution-based and execution-free verifiers:

• Execution-based: Evaluates candidate patches by running all generated tests, forming a $TestScore_k$. Regression-test filtering restricts selection to candidates with maximal regression score. However, test distinguishability is low ($\lesssim 20\%$ distinguish rate).
• Execution-free: An LLM classifier attends to complete agent trajectories; performance saturates at $$\mathcal{L}(\theta) = -\sum_{t=1}^{n} \log \pi_\theta(a_t \mid \tau_{<t}, \text{issue\_text})$$0 best@K, with some attention bias toward agent "thoughts" rather than code edits.
• Hybrid verifier: Top $$\mathcal{L}(\theta) = -\sum_{t=1}^{n} \log \pi_\theta(a_t \mid \tau_{<t}, \text{issue\_text})$$1 candidates by execution-free score are rescored with execution-based metrics, offering additive gains and mitigating test-toxicity.

Performance metrics include resolve rate (fraction of tasks with at least one correct solution), Pass@K, and Best@K (verifier-selected success rate). In R2E-Gym, hybrid verifier pipelines achieve substantial improvement: from $$\mathcal{L}(\theta) = -\sum_{t=1}^{n} \log \pi_\theta(a_t \mid \tau_{<t}, \text{issue\_text})$$2 pass@1 to $$\mathcal{L}(\theta) = -\sum_{t=1}^{n} \log \pi_\theta(a_t \mid \tau_{<t}, \text{issue\_text})$$3 best@16 and $$\mathcal{L}(\theta) = -\sum_{t=1}^{n} \log \pi_\theta(a_t \mid \tau_{<t}, \text{issue\_text})$$4 best@26 (Jain et al., 9 Apr 2025).

5. Experimental Results and Benchmark Comparisons

Fine-tuning on SWE-Gym yields substantial improvements across standard benchmarks (SWE-Bench Lite, SWE-Bench Verified):

Model/Method	Lite (resolve rate)	Verified (resolve rate)	Details/Scaling
Zero-shot Qwen-32B	3.0%	7.0%	OpenHands scaffold
SWE-Gym fine-tuned	15.3%	20.6%
+ ORM verifier (K=16)	26.0%	32.0%	Open-weight SOTA
MoatlessTools 7B/32B	10.0%/19.7%	–	Online fine-tuning
+ verifier (K=8)	26.3% (32B)	–
R2E-Gym 32B (hybrid)	–	51.0% (best@26)
Proprietary O1 pipeline	–	48.0%
Claude-3.7 Sonnet+tools	–	62.3% (best@any)

Training-time resolve rates scale roughly logarithmically with the number of distinct successful trajectories, showing no saturation up to 491 samples. Inference-time scaling is approximately linear in $$\mathcal{L}(\theta) = -\sum_{t=1}^{n} \log \pi_\theta(a_t \mid \tau_{<t}, \text{issue\_text})$$5 (number of agent samples).

R2E-Gym’s hybrid approach is the first to match or surpass proprietary O1-based systems on SWE-Bench Verified and close more than half the gap to Claude-3.7 Sonnet+tools (Jain et al., 9 Apr 2025).

6. Open-Source Release and Research Utility

All SWE-Gym artifacts are publicly available, including:

• Full Docker images for all tasks (6 TB for main, 230 Lite images)
• SWE-Gym Raw (66,894 issues)
• Fine-tuned Qwen-2.5-Coder-Instruct models (7B/14B/32B) and verifiers
• All training and evaluation code, and sampled agent trajectories

Similarly, R2E-Gym’s deliverables include all curated tasks, synthetic issue descriptions, generated tests, trained agents and verifiers, and supporting scripts. Best practice recommendations highlight the importance of containerization, test-generation with a dedicated agent, commit-filter deployment, temperature tuning, and continuous refinement based on test distinguishability and toxicity rates.

7. Context, Impact, and Directions

SWE-Gym and its procedural derivatives such as R2E-Gym provide a rigorous foundation for SWE LLM agent research, enabling methodologically sound ablation studies, scaling analyses, and benchmarking. By democratizing access to executable SWE environments, synthetic task generation recipes, and model checkpoints, these resources accelerate reproducibility and innovation. The hybrid verifier paradigm, combining execution-based and LLM-based validation, has advanced open-weight agents to parity with proprietary systems on benchmarks like SWE-Bench Verified.

A plausible implication is that further integration of procedural task generation, verifiable runtime environments, and advanced hybrid selection will continue to close performance gaps, and serve as a robust platform for autonomous software engineering research (Pan et al., 2024, Jain et al., 9 Apr 2025).