CodeGym: Synthetic RL Coding Environment

• CodeGym is a synthetic reinforcement learning framework that converts static coding problems into interactive, multi-turn tool-use environments.
• It employs a two-stage process with Gym Synthesis and Gym Verification to extract atomic coding logic and ensure task solvability via unit tests.
• Empirical results show that CodeGym training enhances out-of-distribution performance by promoting structured tool invocation and error recovery.

Searching arXiv for CodeGym and closely related “gym”-style coding environments and benchmarks. {"query":"CodeGym arXiv synthetic reinforcement learning tool use coding interactive environment", "max_results": 10} {"query":"CodeGym tool-use RL synthetic coding environment arXiv", "max_results": 5} CodeGym is a synthetic reinforcement-learning environment framework for training LLM agents to use tools in interactive, multi-turn workflows. It rewrites static coding problems into Gym-style environments by extracting atomic functions or logic into callable tools, so that an agent solves a task through stepwise tool invocation and terminal answer submission rather than by emitting a complete program in one shot. Within current arXiv literature, CodeGym is most precisely the framework introduced for end-to-end tool-use RL, but related work also suggests a broader family of code-centric “gym” paradigms spanning gamified competition, iterative code generation, feedback-driven debugging, and memory-sensitive agent evaluation (Du et al., 22 Sep 2025).

1. Conceptual basis and motivation

CodeGym was introduced to address two limitations in prior agent training practice: supervised fine-tuning over static trajectories and reinforcement learning on narrow domains. The first tends to expose models to narrow, fixed interaction patterns; the second can improve performance within a domain such as debugging or web search without providing a broad enough training signal for general-purpose tool use. CodeGym’s response is to use coding problems as a source of diverse, verifiable, and controllable environments, on the premise that code execution already embodies multi-step decision making, function selection, dependence on intermediate observations, structured error handling, and compositional execution logic (Du et al., 22 Sep 2025).

The framework formalizes each environment as a partially observable Markov decision process,

$$\mathcal{E} = \langle \mathcal{S}, \mathcal{A}, T, R, \mathcal{O} \rangle,$$

where $\mathcal{S}$ contains task-specific variables, $\mathcal{A}$ includes generic actions such as Observe and Done together with task-specific tools, $T$ is implemented by executing the chosen tool, $R$ is sparse and given only at termination, and $\mathcal{O}$ returns textual feedback from tool execution. The environment starts from a unit-test input configuration and hides part of the state from the agent, which prevents trivial direct inspection and makes tool selection part of the task itself (Du et al., 22 Sep 2025).

This design makes CodeGym distinct from static code benchmarks. Its target capability is not merely code synthesis, but the acquisition of generalizable interaction policies: deciding when to call tools, which tool to call, how to adapt to partial observations, and when to terminate with Done(answer). The paper explicitly treats coding not simply as a domain, but as a source of workflow geometry that can be repurposed for agent training (Du et al., 22 Sep 2025).

2. Environment synthesis from static code problems

CodeGym constructs environments in two stages, beginning with Gym Synthesis. In the first step, the system collects competitive-programming-style problems from the KodCode dataset, specifically the “Coding Assessment Questions” subset. Because original solutions vary in structure, an LLM rewrites them into a unified format suitable for processing. The framework then extracts reusable and atomized code logic from the solution and turns that logic into callable tools. These tools may be standalone functions, calculation utilities, loop fragments, or other recurring logic pieces (Du et al., 22 Sep 2025).

The synthesis process does more than wrap existing functions. It restructures the original algorithm into stepwise actions that an agent can invoke interactively. For each task, the generator is prompted to produce a task description, a runnable Gym environment, and a clear definition of each action, including name, parameters, and functionality. The resulting environment exposes a standardized API with methods such as reset(options), step(action_json), Observe(), Done(answer), get_ref_answer(), and an optional oracle solve() function. Action input is a JSON string of the form:

$\mathcal{S}$4

The environment class maintains a func_mapping from action names to methods, and class names follow the {TaskName}Env convention. Partial observability is deliberate: for example, Observe() reveals only some task inputs, not the full hidden state. This forces strategic probing rather than direct solution recovery (Du et al., 22 Sep 2025).

The paper’s illustrative example is “find the number closest to $K$ in a sorted list.” The extracted tools are Observe, LookUpPos, and Done. The example is important because it demonstrates the intended abstraction level: not arbitrary code execution, but controlled tool-mediated interaction over a hidden task state (Du et al., 22 Sep 2025).

3. Verification, reward structure, and corpus properties

The second construction stage is Gym Verification, which addresses two failure modes: correctness error and solvability error. Correctness errors include compilation failures, timeouts, and OOM failures. Solvability errors arise when the exposed tool set is insufficient for any agent to solve the task. To verify environments, the framework synthesizes unit tests intended to cover normal, edge, boundary, and increasing-difficulty cases. Each environment is sampled twice with 15 tests each, for 30 tests total, and invalid tests are filtered by running the ground-truth solution. The original coding solution is then executed on the generated tests to produce reference outputs, and an LLM is prompted to write a solve(self) function that only uses the action API. Environments are marked solvable by pass@K verification with $K=10$ if any candidate solve() passes all unit tests (Du et al., 22 Sep 2025).

Reward is binary and terminal:

$$R = \begin{cases} 1 & \text{if submitted answer matches the unit-test reference output} \ 0 & \text{otherwise.} \end{cases}$$

This sparse-reward design is central to the framework’s claim of verifiability. The same construction also yields controllability through explicit task initialization via reset(options), explicit tool definitions, standardized action format, partial observability, and complexity-based filtering (Du et al., 22 Sep 2025).

The paper applies two explicit filtering criteria. First, environments are retained only when oracle execution requires at least 10 tool calls, at most 256 tool calls, and at least 4 distinct tools. Second, difficulty is estimated by evaluating each instance four times with Qwen2.5-32B-Instruct and retaining only tasks with accuracy at most 25%. The filtered training corpus contains more than 80k training instances. The appendix reports 13,116 environments and 86,165 task configurations, with an average of 6.52 tools per environment and 44.07 steps per task (Du et al., 22 Sep 2025).

These design choices are significant because they move CodeGym away from trivial environment generation. The retained tasks are intended to be neither too easy nor purely repetitive; instead, they emphasize nontrivial workflow structure and verifiable interaction.

4. Reinforcement-learning setup and empirical behavior

CodeGym is trained with GRPO using a distributed rollout framework composed of a CPU-bound environment server, separate service ports for rollout workers, centralized environment initialization, and real-time transmission of tool calls and observations. At the beginning of each epoch, the server receives the environment identifier and input configuration, loads the corresponding environment, assigns a service port, and returns terminal reward to the RL learner after rollout completion. To stabilize early training, the implementation uses a trial-then-overwrite mechanism: the environment state is serialized, the candidate tool call executes in a subprocess on that snapshot, and the state is committed only if the call succeeds; otherwise the original state is preserved and an error is returned (Du et al., 22 Sep 2025).

The reported GRPO hyperparameters include actor learning rate $1 \times 10^{-6}$, warm-up 5 steps, KL coefficient 0, max prompt length 5120, max response length 24,576, Adam with $\mathcal{S}$0 and $\mathcal{S}$1, weight decay 0.1, global batch $\mathcal{S}$2, clip ratio 0.2, gradient clip 1.0, and rollout temperature 1.0. Training proceeds until rewards saturate; smaller models plateau around 100 steps, whereas the 72B model saturates around 50 steps (Du et al., 22 Sep 2025).

Empirically, the central claim is out-of-distribution generalization. Models trained in CodeGym improve not only on held-in validation environments but also on unseen tool-use and multi-turn benchmarks. The abstract highlights that Qwen2.5-32B-Instruct achieves an absolute accuracy gain of 8.7 points on $\mathcal{S}$3-Bench. In the main table, its average OOD score rises from 42.3 to 49.6 after CodeGym training. Filtered training also outperforms unfiltered training, with the appendix reporting base-model average OOD 42.3, CodeGym-UF 46.2, and the filtered setting 49.6. The paper interprets these improvements as evidence that active interaction through RL transfers better than imitation of static trajectories (Du et al., 22 Sep 2025).

The training dynamics are also behaviorally informative. Average numbers of tool calls increase during training, and the gap to oracle tool-call counts narrows, which the paper reads as a sign that agents are learning more structured workflow behavior rather than shortcutting. At the same time, the 7B model exhibits repetitive retry loops with the same tool and arguments, indicating weaker error recovery. For long-chain-of-thought models, QwQ-32B improves on tool use and multi-turn interaction but may lose some performance on reasoning benchmarks, suggesting that CodeGym-style RL and pure reasoning optimization may be complementary rather than identical objectives (Du et al., 22 Sep 2025).

5. Position within the broader code-gym literature

Related systems show that “code gym” has become a useful organizing idea for several distinct research programs. Some systems treat coding as a social or pedagogical activity; others treat it as a substrate for benchmarking, agent training, memory evaluation, or reward design.

System	Primary unit	Distinguishing focus
JediCode	Gamified platform	Real-time leaderboards, synchronized challenges, random matchmaking
Coffee-Gym	RL environment	Natural-language feedback for erroneous code
CodeFlowBench	Benchmark	Multi-turn, iterative code generation
CatCode	Evaluation framework	Code-text mixtures and functional equivalence
MemGym	Benchmark wrapper	Memory-isolated coding evaluation
Accodemy	Learning platform	Personalization, motivation, and coding-anxiety support

JediCode frames competitive coding as a live, multiplayer game-like experience with real-time leaderboards, synchronized challenges, random matchmaking, WebSocket-based live updates, and Judge0-powered immediate feedback, thereby emphasizing engagement, community, and instant evaluation rather than RL-based tool use (Mishra et al., 2023). Coffee-Gym, by contrast, is an RL environment centered on natural-language feedback for code editing; it combines the Coffee dataset of human code edit traces with CoffeeEval, a reward function that measures feedback quality by the fraction of downstream unit tests passed by the revised code (Chae et al., 2024). CodeFlowBench shifts the focus again: it benchmarks iterative, dependency-aware code generation over 5,258 Codeforces problems and introduces structural metrics such as Pass Depth, Average Pass Depth, and Dependency Structure Complexity to quantify failure within multi-turn codeflow (Wang et al., 30 Apr 2025). CatCode is broader still, proposing a category-theoretic framework for evaluating debugging, transformation, translation, generation, explanation, and reproduction across mixtures of code and natural language, with automatic pairing and execution tests (Lin et al., 2024). MemGym places coding inside a memory-isolated benchmark regime through SWE-Gym and MEMGYM-CODEQA, explicitly separating memory effects from reasoning, retrieval, and tool-use confounders (Xu et al., 20 May 2026). Accodemy returns to the educational setting, combining MERN-stack implementation, RAG, personalized roadmaps, AI-generated challenging questions, mindfulness quotes, motivational tips, and a supportive chatbot for novice programmers who experience fear of coding (Aamina et al., 9 Mar 2025).

An earlier pedagogical line of work on automatic exercise generation also aligns with the code-gym idea, though without agentic interaction. That work proposes a modular, constraint-driven exercise generator with random exercise generation, test-set generation, student-specific seeding, external rendering, backtracking, and automatic hints based on AST or bytecode error patterns; it reports practical deployment of about 18,000 automatically generated exercises in a Python course (Sovietov, 2022).

Taken together, these systems suggest that CodeGym occupies one specific point in a larger design space. Its distinguishing feature is not gamification alone, nor benchmarking alone, nor learning support alone, but the conversion of static coding problems into interactive tool-use environments for end-to-end RL.

6. Limitations, misconceptions, and open research directions

A common misconception is that CodeGym is simply a coding benchmark. The framework is more specific: it is an environment-construction and RL-training methodology in which coding problems are rewritten into partially observable tool-use tasks with sparse verifiable reward. It is therefore closer to an agent-training substrate than to a conventional pass/fail benchmark (Du et al., 22 Sep 2025).

Another possible misconception is that CodeGym eliminates shortcutting. The paper explicitly notes that long-chain-of-thought models may still bypass intended workflows in some cases, which motivated difficulty augmentation through larger parameter magnitudes, more intricate tool dependencies, and increased tool-call limits for harder versions. The authors also identify open problems around stronger theoretical guarantees against shortcut solutions, improved error recovery for smaller models, and combinations of CodeGym-style RL with reasoning objectives (Du et al., 22 Sep 2025).

Related work makes these open questions sharper. CodeFlowBench shows that even strong models degrade substantially in multi-turn settings and that performance inversely correlates with dependency complexity, which suggests that interactive workflow competence remains fragile under deeper function dependencies (Wang et al., 30 Apr 2025). CatCode reports that explanation-plus-reproduction yields about a 30% lower Pass@1 than direct translation, indicating that preserving functionality across code-text-code cycles is substantially harder than preserving it across code-to-code mappings (Lin et al., 2024). MemGym finds memory roughly neutral on SWE-Gym overall while becoming far more informative in synthetic settings where the relevant facts are genuinely memory-only, suggesting that environment design must carefully separate what is recoverable from tools and repository state from what must be retained across long horizons (Xu et al., 20 May 2026).

A plausible implication is that future CodeGym-like systems will need to integrate several strands of this literature at once: workflow diversity from synthetic tool-use RL, structural analysis of multi-turn dependencies, memory-isolated evaluation, and richer feedback or pedagogical signals. The current literature does not yet unify these strands into a single framework, but it makes clear that code-centric environments have become a central experimental substrate for studying generalizable agent behavior.