ReasoningGYM: Procedural Reasoning Tasks

• ReasoningGYM is a library of procedurally generated reasoning environments that creates unlimited training tasks using algorithmic generators and verifiable rewards.
• It leverages modular task generators across over 100 domains, enabling curriculum learning and adaptive evaluation for reinforcement learning agents.
• Its dynamic difficulty scaling and automated verification mechanisms mitigate overfitting and promote robust, generalizable reasoning skills in language models.

ReasoningGYM is a library of procedurally generated reasoning environments designed for reinforcement learning with verifiable rewards. Its principal contribution is the provision of algorithmic task generators and automated verifiers spanning over 100 distinct reasoning domains—including algebra, arithmetic, computation, cognition, geometry, graph theory, logic, and various common games. ReasoningGYM departs from the prevailing paradigm of static dataset benchmarks by enabling the generation of unlimited training instances with dynamically adjustable complexity, supporting both model evaluation and curriculum-driven training. The central concept is to continuously scale the difficulty and variety of reasoning tasks in a way that outpaces rote memorization and supports robust, generalizable reasoning capacities in LLMs (Stojanovski et al., 30 May 2025).

1. System Architecture and Domains

ReasoningGYM is structured as a modular library comprising both data generators and automatic verifiers. Generators implement algorithmic schemes to sample task instances from parametrized families. For example, an algebra generator may output polynomial equations with randomized degree, coefficient bounds, and variable structure, while the geometry generator may sample triangle configurations for orthocenter calculation within a prescribed coordinate range.

Verifiers are tightly coupled to generators, ensuring each instance admits a unique, programmatically computed solution. This mechanism enables reinforcement learning agents to receive unambiguous scalar reward signals without human intervention, a property crucial for RL research.

The domains incorporated include:

• Symbolic algebra (equations, polynomials, factorization)
• Arithmetic (basic, intermediate, fractional operations)
• Algorithmic reasoning (matrix manipulation, string and sequence tasks)
• Geometry (spatial puzzles, coordinate computation)
• Logic (deductive puzzles, circuit logic, classic riddles)
• Cognition (pattern puzzles, arc tasks)
• Graph theory (traversals, connectivity, coloring)
• Code generation and verification (sorting, reversal, functional tests)
• Classic games (mini-sudoku, knights-and-knaves)

Each domain exposes structural, stylistic, and difficulty parameters, facilitating systematic scaling and variant creation.

2. Data Generation and Complexity Modulation

Data generation proceeds algorithmically, using sampling schemes controlled by:

• Difficulty parameters (e.g., polynomial degree, operand count, sequence length)
• Structural parameters (e.g., graph node counts, geometry dimensions)
• Stylistic parameters (e.g., variable naming, formatting protocols)

This procedural mechanism allows both continual training and adaptive evaluation: as models reach proficiency, generators dynamically scale up complexity. Importantly, no two instances need ever be repeated exactly, mitigating memorization effects endemic to fixed datasets.

Continuous evaluation is achieved by tracking model performance as task parameters evolve. The system supports curriculum learning regimes by incrementally advancing task complexity in correspondence with agent mastery levels.

3. Reward Structure and Automated Verification

Reward signals are derived algorithmically, with each generator-verifier pair yielding programmatic correctness and auxiliary formatting signals. The canonical reward formula is:

$$R_{\text{total}} = 1.0 \cdot (\text{correctness}) + 0.2 \cdot (\text{formatting quality})$$

Accuracy metrics are computed as:

$$\text{Accuracy (\%)} = \frac{\text{Number of Correct Answers}}{\text{Total Number of Tasks}} \times 100\%$$

Tasks such as geometry yield precise fractional and decimal answers (e.g., orthocenter $(7/23, -28/23) \approx (0.304, -1.217)$), allowing rigorous evaluation.

Automated verifiers obviate the need for subjective or manual answer checking, a frequent bottleneck in reasoning dataset curation. This approach also eliminates reward hacking and ensures that RL agents optimize for substantive correctness, not superficial features.

4. Experimental Evaluation and Findings

ReasoningGYM was used to train and evaluate RLVR-optimized LLMs, benchmarking both zero-shot and curriculum-trained agents. Experiments revealed:

• RLVR-focused models consistently outperform general-purpose LLMs on RG tasks, with sharper scaling advantages as difficulty rises.
• The "difficulty cliff" phenomenon: performance drops dramatically as task complexity increases, quantified as a –71.9% decline in accuracy for high-difficulty code tasks, and similarly severe drops in algorithmic and graph-based domains.
• Transfer learning: RLVR training on one class of reasoning task (e.g., algorithmic) benefits related domains (mathematics, geometry).
• Curriculum learning is facilitated by procedural generation, allowing performance tracking and challenge adaptation.
• Training on RG tasks induced improvements on external reasoning benchmarks (GSM8K, MATH), attesting to RG’s generalization capacity.

A summary of severity for performance drop as difficulty increases across RG tasks:

Domain	Maximum Performance Drop
Code tasks	–71.9%
Algorithmic puzzles	Severe
Graph theory	Severe

5. Applications and Utility in Reinforcement Learning Research

ReasoningGYM serves as a benchmark and a dynamic RL environment for cognitive and formal reasoning research. Its procedural generation enables open-ended task scaling, crucial for developing agents that transcend memorization and display true reasoning ability.

Key applications are:

• Reinforcement learning training with verifiable numerical rewards for reasoning models
• Curriculum learning via adaptive difficulty scheduling
• Robust evaluation of model reasoning capacity across a wide suite of formal and informal tasks
• Transfer learning from procedural environments to external static benchmarks

The use of automated reward signals ensures reproducibility and transparency in comparative evaluation regimes.

6. Comparative Perspective and Distinctive Features

Contrasted with web-derived or static benchmarks (e.g., NaturalReasoning (Yuan et al., 18 Feb 2025), AM-DeepSeek-R1-Distilled (Zhao et al., 25 Mar 2025), R-Bench (Guo et al., 4 May 2025)), ReasoningGYM emphasizes:

• Algorithmic, infinite task generation versus fixed or finite datasets
• Verifiable, non-subjective reward structures
• Adjustable complexity via explicit parameterization
• Suitability for RLVR, curriculum learning, and adaptive evaluation
• Focus on core mathematical, logical, and cognitive task classes rather than open-ended real-world scenarios

A plausible implication is that ReasoningGYM complements corpus-based datasets by ensuring that reasoning skills are acquired in a manner resilient to dataset memorization and suitable for open-ended agent development.

7. Limitations and Future Prospects

Much of ReasoningGYM’s content focuses on algorithmic and formal tasks, potentially limiting coverage of tasks requiring broad world knowledge, linguistic nuance, or multi-step discourse. The procedural generation paradigm, while powerful, is best suited to domains with clear structural definitions and programmatic verifiers.

Future work directions include:

• Extending domain coverage to embrace broader real-world and cross-disciplinary problems
• Integrating multimodal task generators and verifiers, as in R-Bench (Guo et al., 4 May 2025)
• Refining curriculum strategies for optimal agent progression
• Exploring the interplay between RG-based training and large-scale web benchmarks

This suggests that hybrid evaluation settings combining RG and corpus benchmarks may afford a richer view of reasoning capacity in advanced LLMs.

ReasoningGYM establishes a foundational, programmatically rigorous platform for reinforcement learning-based reasoning research and open-ended cognitive evaluation in artificial agents (Stojanovski et al., 30 May 2025).