R2E-Gym: Executable Gym for Bug Fixing

• R2E-Gym is an executable gym dataset that leverages synthetic curation to create realistic GitHub bug-fixing tasks.
• It uses a commit-centered, procedural environment construction pipeline with rule-based and LLM heuristics for scalable task generation.
• Its hybrid verifier framework, combining execution-based and execution-free strategies, significantly enhances patch selection and evaluation.

R2E-Gym is a procedurally curated executable gym for training and evaluating open-weights software engineering agents on real GitHub-style bug-fixing tasks. In "R2E-Gym: Procedural Environments and Hybrid Verifiers for Scaling Open-Weights SWE Agents," it appears alongside a synthetic environment-construction recipe, referred to in the paper as SweGen and in the accompanying description as SYNGEN, and a hybrid verifier framework for test-time scaling (Jain et al., 9 Apr 2025). The term "gym" places it in the lineage of standardized environment interfaces associated with OpenAI Gym, but its objects of interaction are executable repositories, tests, and patches rather than conventional control or game simulators (Brockman et al., 2016).

1. Problem setting and conceptual scope

The paper motivates R2E-Gym by identifying two constraints on open-source SWE agents: scalable curation of execution environments for training, and optimal scaling of test-time compute (Jain et al., 9 Apr 2025). Existing datasets are described as either non-executable, too small, or dependent on human-written issues or unit tests, which limits both training scale and evaluation realism. R2E-Gym is introduced as the environment-side response to that bottleneck.

The paper presents three linked ideas. The first is R2E-Gym, described as a procedurally curated executable gym dataset. The second is SweGen/SYNGEN, a synthetic data curation recipe that constructs executable environments from commits rather than from human-authored issue reports. The third is hybrid test-time scaling, which combines execution-based and execution-free verifiers for reranking candidate patches (Jain et al., 9 Apr 2025).

A terminological detail is notable. The abstract introduces AgentGym as "the largest procedurally-curated executable gym environment" and states that it consists of more than 8.7K tasks, whereas the detailed dataset accounting reports R2E-Gym (full): 8,135 tasks and R2E-Gym-Subset / R2E-Gym-lite: 4,578 tasks (Jain et al., 9 Apr 2025). The detailed description treats R2E-Gym as the central contribution and uses the subset for contamination-free training.

This framing places R2E-Gym at the intersection of benchmark design and agent training infrastructure. It is not only a test set, nor only a training corpus, but an executable environment collection intended to support both supervised trajectory collection and verifier-based test-time selection.

2. Dataset composition and synthetic environment construction

The detailed dataset summary reports the following composition for the principal splits (Jain et al., 9 Apr 2025):

Dataset	Tasks	Note
R2E-Gym (full)	8,135	Procedurally curated executable gym dataset
R2E-Gym-Subset / R2E-Gym-lite	4,578	Training subset
SWE-Gym	2,438	Comparison point in the paper

The subset is drawn from 10 repositories with no overlap with SWE-Bench test repositories, and is used for training to avoid contamination with SWE-Bench test repos (Jain et al., 9 Apr 2025). In the comparative table reproduced in the paper, R2E-Gym is positioned against APPS: 10,000, R2E: 246, SWE-Bench(train): 19,008 but not executable, SWE-Gym Raw: 66,894, SWE-Bench(test): 2,294, and SWE-Gym: 2,438. The stated implication is that R2E-Gym substantially expands executable SWE training data relative to prior executable SWE environments.

The environment construction pipeline is synthetic and commit-centered. Repository and commit curation uses SEART GitHub search to find Python repositories with many commits, extracts commit history and code changes, applies rule-based and LLM-based heuristics to select "interesting" commits, and collects build scripts semi-manually by searching dependency pins and setup files (Jain et al., 9 Apr 2025). Test validation and generation follows SWE-Bench style failure-to-pass logic, reusing existing tests when they yield Fail→Pass (F2P) cases and automatically generating tests when suitable tests do not already exist.

Issue creation is done by backtranslation from commits into synthetic problem statements. The prompt is said to include failing tests and execution traces, and the appendix states that each synthetic issue prompt may include commit hash and message, non-test patch, test-file changes, failing and passing execution results, extracted test functions, and assertion failures (Jain et al., 9 Apr 2025). This is central to the paper’s claim that large-scale training data need not depend on human-written GitHub issues.

The commit filtering rules are explicit and restrictive: max 5 non-test files modified, max 100 edited lines across non-test files, max patch length 2000 characters, no more than 1 deleted entity in non-test files, max 3 added entities, max 3 edited entities, no more than 10 statement-level changes, plus an LLM-as-judge filter (Jain et al., 9 Apr 2025). These heuristics are intended to preserve task focus and quality.

Reproducible historical environments are created through a Docker-based, search-based dependency resolution strategy with four stages: extract dependencies, identify conflicts, try multiple candidate dependency configurations, and test until a working setup is found (Jain et al., 9 Apr 2025). This makes executability a first-class property of the dataset rather than an external assumption.

3. Agent scaffold and training protocol

The training scaffold is described as a minimal OpenHands-based ReAct scaffold with exactly four tools: file_editor, search_tool, execute_bash, and submit (Jain et al., 9 Apr 2025). The setup provides no browser or internet access. This restriction is important because it fixes the tool surface available during training and evaluation.

Successful trajectories are collected using Sonnet-3.5-v2 on R2E-Gym tasks. The paper reports 3,321 successful trajectories from 2,048 unique task environments, with training performed on the contamination-free R2E-Gym-Subset / R2E-Gym-lite containing 4,578 environments (Jain et al., 9 Apr 2025). The trained base models are Qwen-2.5-Coder 7B, Qwen-2.5-Coder 14B, and Qwen-2.5-Coder 32B.

Supervised fine-tuning is performed via LLaMA-Factory with the following reported setup: 2 epochs, batch size 8, learning rate 1e-5, warmup ratio 0.1, max context length 20K, max trajectory length 40 steps, max trajectory tokens 32K, and trajectory collection temperature 0.2 (Jain et al., 9 Apr 2025). These parameters define the training regime used for the paper’s main scaling experiments.

The paper’s ablations make two claims about the representation of task context. First, thought traces matter: with thoughts: 34.4, without thoughts: 30.4 on SWE-Bench-Verified for the 32B setting (Jain et al., 9 Apr 2025). Second, synthetic issue statements are nearly as effective as real ones: real: 28.0, synthetic: 27.8. The paper interprets these results as evidence that synthetic curation can preserve useful task semantics while scaling beyond human issue-writing capacity.

4. Reported performance and scaling behavior

The core training result is that R2E-Gym-trained models outperform corresponding SWE-Gym-trained models on both SWE-Bench-Verified and SWE-Bench-Lite (Jain et al., 9 Apr 2025). The verified-set comparison is:

Model	SWE-Gym Verified	R2E-Gym Verified
7B	10.6	19.0
14B	16.4	26.8
32B	20.6	34.4

On SWE-Bench-Lite, the paper reports 7B: 10.0 → 11.0, 14B: 12.7 → 20.67, and 32B: 15.3 → 23.77 (Jain et al., 9 Apr 2025). The paper emphasizes that synthetic data curation scales more effectively with model size, especially on SWE-Bench-Verified.

The final reported numbers for the open-weight 32B R2E-Gym model on SWE-Bench-Verified are Pass@1: 34.4%, Best@16 w/ hybrid: 49.4%, and Best@26 w/ hybrid: 51.0% (Jain et al., 9 Apr 2025). These are the headline benchmark results of the paper.

The paper also provides a broader comparison on SWE-Bench-Verified: SWE-Gym-32B: 32.0%, SWE-RL-70B: 41.0%, Agentless / DeepSeek-R1: 49.2%, Claude 3.6 Sonnet + tools: 49.0%, OpenHands + Claude 3.6 Sonnet: 53.0%, Claude 3.7 Sonnet + tools: 62.3%, and Claude 3.7 Sonnet + tools (Best@Any): 70.3% (Jain et al., 9 Apr 2025). The paper characterizes the R2E-Gym result as a new open-weight SOTA and as competitive with some proprietary tool-using systems.

These results serve two distinct claims. One is a data-scaling claim: larger executable synthetic corpora improve supervised agent training. The other is a selection claim: test-time reranking contributes a further large gain beyond Pass@1.

5. Verifier architecture and hybrid test-time scaling

The paper’s second major contribution is a verifier framework that combines execution-based and execution-free reranking (Jain et al., 9 Apr 2025). The execution-based verifier has two components: a testing agent that generates reproduction tests and a regression-test filter that rejects patches harming existing functionality. The testing agent uses Qwen-Coder-32B as its base model, generates $M = 10$ diverse tests, and is trained on 2,203 test-generation trajectories. The reported training setup is max steps 40, max tokens 20K, timeout 5 min per trajectory, 60 sec per action, 2 epochs, batch size 8, lr 1e-5, and warmup 0.1. An in-context Django starter snippet is supplied to improve formatting and domain grounding.

The execution-based score is defined as

$$s^{EB}_k = \begin{cases} \mathrm{TestScore}_k, & \text{if } RS_k = \max\limits_{j \in [1,K]} RS_j,\ 0, & \text{otherwise}, \end{cases} \qquad \mathrm{TestScore}_k = \sum_i \mathrm{Pass}(\mathcal{P}_k, Test_i)$$

where $RS_k$ is the regression-test score of patch $k$ (Jain et al., 9 Apr 2025). Only patches with the best regression score are kept; among them, the score is the number of generated tests passed.

The execution-free verifier is a learned trajectory scorer built from Qwen2.5-Coder-14B, trained on 5,700 balanced positive and negative trajectories and consuming the full trajectory: issue, thoughts, actions, observations, patch (Jain et al., 9 Apr 2025). Its score is

$$s^{EF} = \mathrm{P}(YES)/(\mathrm{P}(YES) + \mathrm{P}(NO)).$$

The interpretation given in the paper is that the model predicts YES when the patch is correct and NO otherwise.

The paper analyzes complementary strengths and failure modes. Execution-based verifiers provide direct correctness signals when tests are good, but they suffer from low distinguishability and occasional toxic tests. The appendix defines

$\text{Distinguish}(t) = \mathbbm{1}\left[\max_{p \in P_i} \text{Pass}(p, t) \neq \max_{p \in P_c} \text{Pass}(p, t)\right]$

and

$$\text{DistinguishRate}(T) = \frac{1}{|T|}\sum_{t \in T} \text{Distinguish}(t),$$

together with

$\text{Toxic}(t) = \mathbbm{1}\left[\max_{p \in P_i} \text{Pass}(p, t) > \max_{p \in P_c} \text{Pass}(p, t)\right]$

and

$$\text{ToxicityRate}(T) = \frac{1}{|T|}\sum_{t \in T} \text{Toxic}(t).$$

Empirically, the paper states that for most problems less than 20% of tests are distinguishable, while toxic tests are rare overall but can reach up to 10% on some problems (Jain et al., 9 Apr 2025).

Execution-free verifiers have better reranking discrimination than raw test pass counts, but the paper finds that they can be biased by stylistic / heuristic cues, especially by the agent’s thoughts rather than only the final patch. The ablation is explicit: full trajectory: 42.8% Best@26, patch only: 37.6%, trajectory without thoughts: 41.4% (Jain et al., 9 Apr 2025). The accompanying attention visualizations are said to show attention to enthusiastic or self-confirming thoughts rather than to patch quality.

The hybrid score is defined as

$s_k^H = \mathrm{Top}_n(s_k^{EF}) + s_k^{EB},$

with

$$s^{EB}_k = \begin{cases} \mathrm{TestScore}_k, & \text{if } RS_k = \max\limits_{j \in [1,K]} RS_j,\ 0, & \text{otherwise}, \end{cases} \qquad \mathrm{TestScore}_k = \sum_i \mathrm{Pass}(\mathcal{P}_k, Test_i)$$0

The rationale is to use execution-free scoring for shortlist formation and execution-based feedback for grounding (Jain et al., 9 Apr 2025). Reported results are execution-based verifier alone: 43.7%, execution-free verifier alone: 42.8%, hybrid verifier: 51.0% Best@26, with hybrid Best@16: 49.4%. The paper also reports that regression tests alone: 47.4%, agent-generated tests + regression tests: 51.0%, Agentless tests in the same hybrid framework: 48.8%, and that top-n filtering improves from 49.8% to 51.0%.

A compute-efficiency point is also made: adding more test-agent rollouts can be more cost-effective than adding more editing-agent rollouts. The example given is editing rollouts 16 → 21: 47.6% → 48.4%, whereas adding 5 test rollouts can yield 49.3% (Jain et al., 9 Apr 2025).

6. Relation to Gym-style systems and broader significance

OpenAI Gym defined a toolkit for reinforcement learning research centered on a growing collection of benchmark problems, a common interface, and a website for sharing results and comparing algorithms, while standardizing the environment side of agent-environment interaction through operations such as reset() and step() (Brockman et al., 2016). That conceptual template is visible across several later systems: Gym-preCICE exposes active-flow-control simulations through a Gymnasium-compliant adapter and the tuple $$s^{EB}_k = \begin{cases} \mathrm{TestScore}_k, & \text{if } RS_k = \max\limits_{j \in [1,K]} RS_j,\ 0, & \text{otherwise}, \end{cases} \qquad \mathrm{TestScore}_k = \sum_i \mathrm{Pass}(\mathcal{P}_k, Test_i)$$1 (Shams et al., 2023); Gym-Ignition presents reproducible robotic environments through the OpenAI Gym interface while separating Environment, Task, Robot, and Runtime (Ferigo et al., 2019); and pyRDDLGym compiles declarative RDDL models into Gym-compatible environments, preserving explicit access to the underlying model (Taitler et al., 2022).

Within that broader family, R2E-Gym shifts the "gym" abstraction from continuous-control, robotics, or declarative MDP/POMDP domains to executable software engineering tasks (Jain et al., 9 Apr 2025). This suggests a widening of the Gym idea: the standardized environment need not be a simulator in the classical RL sense, but can also be a repository-centered executable task whose evaluation depends on patch generation, test execution, and verifier-mediated reranking. The paper’s contribution is therefore not merely a larger dataset; it is a specification of how software-maintenance problems can be turned into scalable executable environments.

The released-artifact discussion reinforces that interpretation. The paper indicates a project page at https://r2e-gym.github.io and references agentless official test artifacts, trajectories, generated tests, and environment examples in the appendix, an open-weights 32B model, and the R2E-Gym environment itself (Jain et al., 9 Apr 2025). In encyclopedic terms, R2E-Gym is best understood as both a benchmark corpus and an infrastructure layer for scaling open-weight SWE agents through synthetic environment curation and hybrid test-time verification.