SWE-Oriented RL Framework

• SWE-oriented RL frameworks are end-to-end systems that employ reinforcement learning to modify and repair real software repositories via reproducible test-suite validation.
• They utilize sandboxed execution environments and advanced policy optimization techniques to generate precise patches across multiple programming languages.
• These frameworks integrate SFT warm starts, self-play curricula, and scalable pipelines to enhance learning efficiency and policy robustness in practical software engineering tasks.

A SWE-oriented reinforcement learning framework is an end-to-end training and evaluation setup in which an agent modifies real repositories to fix bugs, repair tests, or implement features, and correctness is verified by running repository test suites or other verifiers that provide reproducible reward signals at scale (Badertdinov et al., 27 Feb 2026). In current work, the term spans execution-grounded pipelines built around Dockerized or container-free sandboxes, multi-turn tool-using agents, and policy optimization schemes such as KL-regularized on-policy RL, GRPO, GSPO, DAPO, and self-play; it also includes adjacent variants that replace or augment execution rewards with reward models, rubric process reward models, or proposal-selection rewards when direct execution is sparse, noisy, or expensive (Chen et al., 3 Aug 2025, Shum et al., 26 Dec 2025, Han et al., 16 Apr 2026, Tan et al., 30 Jan 2026).

1. Problem setting and task semantics

In repository-repair settings, issue resolving is defined as modifying a codebase to generate a patch that addresses a given issue (Zan et al., 3 Apr 2025). A typical episode starts when the repository is initialized and the issue is presented; the agent then localizes the fault or implementation target, edits code across one or more files, runs builds and tests, and submits a patch (Zan et al., 3 Apr 2025). Within SWE-rebench V2, this is formalized as an end-to-end training and evaluation setup in which an agent modifies real repositories to fix bugs, repair tests, and implement features, with correctness verified by running repository test suites (Badertdinov et al., 27 Feb 2026).

The task distribution is no longer limited to Python-only benchmarks. SWE-rebench V2 reports 32,079 containerized, executable tasks across 3,617 repositories and 20 programming languages, plus 120,000+ PR-derived tasks with installation instructions, fail-to-pass tests and rich metadata (Badertdinov et al., 27 Feb 2026). Multi-SWE-bench provides 1,632 high-quality instances across Java, TypeScript, JavaScript, Go, Rust, C, and C++, while Multi-SWE-RL releases 4,723 well-structured instances spanning seven programming languages for RL training (Zan et al., 3 Apr 2025). Repository changes are often small in median terms but heavy-tailed in difficulty: SWE-rebench V2 reports a median 3 files and 34 changed lines, with the 90th percentile at 9 files and 181 lines (Badertdinov et al., 27 Feb 2026).

Not all SWE-oriented RL frameworks operate at the level of repository execution. SWE-RL treats software evolution data as an offline RL substrate in which the environment is single-step and the reward is a patch-similarity score rather than test execution (Wei et al., 25 Feb 2025). SWE-Manager moves even further upstream: it frames proposal selection as a reasoning task over issue descriptions and candidate proposals, without executing code or running tests, yet still optimizes a scalar RL reward over selection correctness, reasoning, justification, and synthesis quality (Tan et al., 30 Jan 2026). This suggests that the label “SWE-oriented RL framework” now covers both execution-grounded repository repair and adjacent decision problems in the software engineering workflow.

2. Environment construction, sandboxing, and verifier design

A central component of these frameworks is the construction of verifiable environments. SWE-rebench V2 uses a language-agnostic automated pipeline with preliminary collection, setup synthesis, parser synthesis, execution-based validation, issue-clarity filtering, and metadata enrichment (Badertdinov et al., 27 Feb 2026). Each task includes a repository snapshot, a problem statement, a test patch, an install_config.json containing ordered shell commands and a full-suite test_cmd, and a repository-specific log parser (Badertdinov et al., 27 Feb 2026). Multi-SWE-bench standardizes environment evaluation around three canonical configurations—run.log, test.log, and fix.log—and tracks transitions among PASSED, FAILED, SKIPPED, and NONE to determine whether a bug is reproduced and then repaired without regressions (Zan et al., 3 Apr 2025).

RepoForge emphasizes autonomous environment generation from real GitHub commits. Its Foundry is described as a fully autonomous multi-agent ReAct pipeline that ingests real repositories and issues, infers build graphs, constructs isolated Docker environments, and validates them through end-to-end PASS_TO_PASS and FAIL_TO_PASS checks, producing 7,304 validated SWE instances generated with zero manual intervention (Chen et al., 3 Aug 2025). SWE-Universe pushes this construction paradigm further by synthesizing a per-task evaluation.sh verifier from pull requests, validating that it fails on the buggy state and succeeds on the resolved state, and rejecting hacked verifiers through in-loop hacking detection; the resulting corpus reaches 807,693 environments over 52,960 repositories (Chen et al., 2 Feb 2026).

Sandboxing has diversified. A common misconception is that scalable SWE RL requires per-task containers. SWE-MiniSandbox explicitly replaces per-task containers with isolated workspaces built from mount namespaces and chroot plus pre-cached Python virtual environments, reporting disk usage of approximately 5\% of container-based pipelines and environment preparation time of about 25\% of the container baseline (Yuan et al., 11 Feb 2026). By contrast, RepoForge remains container-based but reduces per-instance storage from 1.4 GB to 102 MB through image dependency pruning and minimal runtime environments (Chen et al., 3 Aug 2025). The field therefore contains both container-centric and container-free realizations of the same environment-manager abstraction.

Verifier design is increasingly treated as a first-class research problem rather than a passive benchmark artifact. SWE-Universe’s evaluation.sh is meant to be a strong binary oracle; Multi-SWE-bench uses explicit transition semantics; RepoForge and SWE-rebench V2 enforce PASS_TO_PASS and FAIL_TO_PASS; and reward-model-based work treats the verifier itself as a learned component (Chen et al., 2 Feb 2026, Zan et al., 3 Apr 2025, Chen et al., 3 Aug 2025, Shum et al., 26 Dec 2025). The shared requirement is reproducible, repository-level correctness signals.

3. States, actions, rewards, and optimization objectives

Most execution-grounded frameworks expose a tool-mediated state and action space. RepoForge defines states $s_t$ to include the repository state, test results, and tool outputs, while actions $a_t$ are execute_bash, str_replace_editor, and finish (Chen et al., 3 Aug 2025). SWE-Master uses a ReAct-style loop in which the model emits Thought and a structured tool invocation among bash_execute, file_editor, lsp_tool, and submit, with explicit budget awareness through the environment message Steps Remaining: STEP_REMAINING (Song et al., 3 Feb 2026). SWE-Universe adds switch-to-bug and switch-to-resolved as environment toggles, and OpenClaw-RL generalizes the notion of next state to tool returns, terminal state changes, test verdicts, diffs, linter messages, and coverage reports (Chen et al., 2 Feb 2026, Wang et al., 10 Mar 2026).

Reward design varies sharply across frameworks. RepoForge uses a terminal binary reward,

$$R(\tau)= \begin{cases} 1, & \text{if all designated tests pass after applying the patch} \ 0, & \text{otherwise,} \end{cases}$$

with no partial credit and no explicit penalties for non-compiling code, runtime errors, or timeouts; these simply yield 0 via failing tests (Chen et al., 3 Aug 2025). SWE-rebench V2 does not define a formal reward in the paper, but it proposes an RL-compatible formulation combining incremental tests-passed gain, regression penalties, timeout penalties, and a terminal full-success bonus (Badertdinov et al., 27 Feb 2026). SWE-Master retains a binary execution outcome but adds stop-reason shaping, using $\alpha=0.5$ when trajectories terminate through TIMEOUT, MAX_STEPS, or MAX_TOKENS rather than voluntary submission (Song et al., 3 Feb 2026).

Other frameworks pursue denser or more structured rewards. Satori-SWE trains a mutation policy on a local potential-difference objective,

$$\max_{\pi_{RL}} \mathbb{E}[R(x,y)-R(x,y')-\lambda F(y)],$$

where the reward model score difference supplies non-zero feedback at each refinement step and $F(y)$ is a formatting penalty (Zeng et al., 29 May 2025). SWE-RM uses a learned execution-free score ScoreEF and, in hybrid RL, assigns 1 + ScoreEF for resolved trajectories, 0.5 + ScoreEF for unfinished trajectories, and 0 + ScoreEF otherwise (Shum et al., 26 Dec 2025). OpenClaw-RL combines scalar process rewards from a PRM judge, with per-step outputs in $\{+1,-1,0\}$, and token-level directive supervision from Hindsight-Guided On-Policy Distillation, where the next state is mined for textual hints that define directional advantages (Wang et al., 10 Mar 2026). SWE-TRACE composes sparse execution rewards with rubric-conditioned process scores into a margin-separated trajectory reward, and Socratic-SWE uses a multi-component Solver reward that separately values full-suite success, partial repair of originally failing tests, and preservation of originally passing tests (Han et al., 16 Apr 2026, Xiao et al., 5 Jun 2026).

Optimization methods are likewise heterogeneous. RepoForge performs KL-regularized on-policy RL anchored to an SFT reference policy with $\lambda=0.001$ (Chen et al., 3 Aug 2025). SWE-Master uses a GRPO variant with leave-one-out advantage, clip-higher exploration, and no KL penalty (Song et al., 3 Feb 2026). SWE-RM adopts GSPO for MoE backbones, with group-normalized advantages (Shum et al., 26 Dec 2025). SWE-Manager optimizes a DAPO objective with no KL term for long-form selection-and-synthesis outputs (Tan et al., 30 Jan 2026). These choices reflect different trade-offs between policy stability, exploration, long-horizon credit assignment, and reward noise.

4. Training pipelines, curricula, and self-improvement regimes

A common pattern is SFT warm start followed by RL. RepoForge first filters tasks with SPICE, then uses rejection sampling with OpenHands-32B-Agent to obtain 1,202 high-quality trajectories, performs SFT for one epoch, and finally runs RL on a curated set of 160 instances for 40 steps over two epochs with temperature 0.5 and a cap of 35 iterations per trajectory (Chen et al., 3 Aug 2025). SWE-Master builds a much larger post-training pipeline: teacher-trajectory synthesis from SWE-Gym, SWE-rebench, R2E-Gym, and SWE-smith; difficulty-based filtering that excludes trivially solved and consistently failed instances; long-horizon SFT on ~60K instances for 5 epochs; and RL with real execution feedback on top of the SFT initialization (Song et al., 3 Feb 2026).

SWE-RL occupies a different point in the design space. It is offline RL over issue-context-oracle triplets derived from software evolution data, using GRPO and a reward based on difflib.SequenceMatcher similarity between the generated patch and the ground-truth patch (Wei et al., 25 Feb 2025). This formulation does not execute code during training. It demonstrates that repository history, pull requests, issues, and full-file contexts can serve as a large-scale reasoning substrate even when executable environments are unavailable or too costly (Wei et al., 25 Feb 2025).

Several frameworks replace static datasets with self-generated or self-evolving curricula. SSR trains a single LLM policy to both inject and repair bugs in real repositories with no human-labeled issues or tests, using a self-play setup in which each bug is formally specified by a test patch rather than a natural-language issue description (Wei et al., 21 Dec 2025). The bug-injection reward depends on the solver’s solve rate, penalizing both trivial and impossible bugs (Wei et al., 21 Dec 2025). Socratic-SWE closes the loop further: it distills historical solving traces into an Agent Skill Registry, uses those skills to generate targeted repair tasks in real repositories, filters them with execution-based validation, and prioritizes them with a solver-gradient alignment reward against a held-out validation gradient (Xiao et al., 5 Jun 2026). Satori-SWE, by contrast, treats generation as an evolutionary process in which selection and mutation iteratively shift the candidate distribution toward higher-scoring patches, and then uses RL so that the model learns to self-evolve without an external selector at inference time (Zeng et al., 29 May 2025).

A recurring lesson is that training efficacy depends on more than task count. RepoForge reports that random envs yield 3.0\% after SFT, whereas SPICE-filtered data reaches 12.7\%, and that increasing SFT trajectories from 1,000 to 2,000 can reduce small-model accuracy from 12.7\% to 10.7\% (Chen et al., 3 Aug 2025). SWE-Master reports that difficulty-based filtering improves SFT resolve rate from 54.2% to 57.8% with negligible changes in turns and tokens (Song et al., 3 Feb 2026). SWE-RM shows that verifier quality for RL depends not just on TTS performance but on AUC and ECE, because top-1 selection ability does not guarantee stable policy-gradient updates (Shum et al., 26 Dec 2025). This suggests that curriculum design, verifier calibration, and trajectory curation are structural elements of the framework, not preprocessing conveniences.

5. Systems infrastructure and representative framework families

SWE-oriented RL frameworks are also systems frameworks. They must manage storage, environment startup, asynchronous rollout generation, reward computation, and high-concurrency agent interaction. Representative systems and reported outcomes include the following (Chen et al., 3 Aug 2025, Song et al., 3 Feb 2026, Xiao et al., 5 Jun 2026, Chen et al., 2 Feb 2026).

Framework	Distinctive mechanism	Reported result
RepoForge	End-to-end pipeline unifying Foundry, SPICE, Harness, and bubble-free RL scaffold	17.4% on SWE-Bench-Verified for ≤8B non-thinking LLMs
SWE-Master	Long-horizon SFT, GRPO with real execution feedback, and TTS with SWE-World	61.4% Pass@1 and 70.8% at TTS@8 on SWE-bench Verified
Socratic-SWE	Trace-derived skills, gradient-aligned task selection, and closed-loop self-evolution	50.40% on SWE-bench Verified after three iterations
SWE-Universe	Autonomous verifier construction at multilingual million scale	75.3% on SWE-Bench Verified with Qwen3-Max-Thinking

RepoForge centers system efficiency around storage-efficient sandboxing, a Ray-powered distributed harness, and a bubble-free RL scaffold. It reports 10× faster builds (537 s → 58 s), evaluation times in the 17–75 s range, 96% of instances completing within two minutes, a ~3× end-to-end scaffold speedup over vanilla OpenHands, and up to 64 workers with remote sandbox servers running ~80–100 containers/host (Chen et al., 3 Aug 2025). SWE-MiniSandbox addresses the same bottleneck from the opposite direction by removing containers entirely; it integrates with SWE-Rex, SWE-agent, and SkyRL/Ray, and reports evaluation performance comparable to standard container-based pipelines while substantially reducing storage and environment preparation overhead (Yuan et al., 11 Feb 2026).

OpenClaw-RL represents a different systems family: the serving layer, environment server, PRM judge, OPD module, and trainer run in fully decoupled asynchronous loops with zero coordination overhead, allowing the model to serve live requests, the PRM to judge ongoing interactions, and the trainer to update the policy concurrently (Wang et al., 10 Mar 2026). At a more general RL-systems level, Parallel Actors and Learners introduces a scalable off-policy design with parallel actors, parallel learners, and a prioritized replay buffer implemented by a K-ary sum tree with lazy writing, demonstrating how actor/learner concurrency and contention-aware data structures can accelerate RL training on multi-core CPU+GPU platforms (Zhang et al., 2021). In SWE settings, later frameworks frequently substitute Ray actors, cached images, asynchronous I/O, or per-task sandboxes for the generic environment instances described in that earlier systems literature.

6. Benchmarks, controversies, and future directions

Benchmarking has expanded from Python-centric evaluation to broader multilingual and multi-domain regimes. SWE-bench Verified remains the most frequently reported headline benchmark, but recent work also evaluates on SWE-bench Lite, SWE-bench Pro, Terminal-Bench 2.0, Multi-SWE-bench, and the task collections underlying SWE-rebench V2 and SWE-Universe (Xiao et al., 5 Jun 2026, Zan et al., 3 Apr 2025, Badertdinov et al., 27 Feb 2026, Chen et al., 2 Feb 2026). Multi-SWE-bench reports limited generalization beyond Python, identifies a performance hierarchy by language domain, and concludes that fault localization is the principal bottleneck; it also notes that agents often exhaust a 50-round limit without submitting (Zan et al., 3 Apr 2025). SWE-rebench V2 adds instance-level confounder metadata B1–B7, including TEST_SUITE_COUPLING, IMPLICIT_NAMING, and EXTERNAL_DEPENDENCY, which makes the dataset usable არა only for evaluation but also for curriculum construction and failure analysis (Badertdinov et al., 27 Feb 2026).

Several active controversies are now explicit in the literature. One concerns execution-based versus execution-free feedback. SWE-RM argues that two verifiers can have nearly identical TTS performance yet induce very different RL behavior, because RL also depends on discriminative accuracy and calibration, measured by AUC and ECE (Shum et al., 26 Dec 2025). OpenClaw-RL argues that the next state carries not only evaluative information but also directive information, and that textual hints extracted from tool outputs can be converted into token-level supervision through OPD (Wang et al., 10 Mar 2026). SWE-TRACE responds to sparse terminal rewards by introducing a rubric-based PRM, a memory mechanism that preserves critical verbatim anchors, and heuristic-guided TTS that prunes action candidates step by step rather than only reranking full trajectories (Han et al., 16 Apr 2026). Another controversy concerns data origin: SSR minimizes human data assumptions through self-play bug injection and repair (Wei et al., 21 Dec 2025), whereas RepoForge, SWE-Master, and SWE-rebench V2 remain heavily grounded in curated or filtered trajectories and executable instances (Chen et al., 3 Aug 2025, Song et al., 3 Feb 2026, Badertdinov et al., 27 Feb 2026).

Future work is described in increasingly concrete terms. RepoForge lists multi-language support and broader build systems, continuous learning loops where deployed agents feed back validated data, more sophisticated long-horizon planning and multi-agent coordination, and richer reward signals (Chen et al., 3 Aug 2025). SWE-rebench V2 proposes multi-service tasks, richer non-functional rewards such as performance, latency, memory, and better naming or interface hints (Badertdinov et al., 27 Feb 2026). Socratic-SWE points toward richer, compositional skill representations, dynamic repository augmentation, cross-repo transfer learning, and multi-agent coordination under trace-derived skills (Xiao et al., 5 Jun 2026). Across these directions, the persistent research problem is unchanged: to align environment realism, verifier fidelity, reward density, and systems efficiency closely enough that LLM agents can improve reliably on real repositories rather than only on benchmark scaffolds.