---
title: 'SimpleRL-Zoo: Zero RL for Base Models'
url: https://www.emergentmind.com/topics/simplerl-zoo
type: topic
---

# SimpleRL-Zoo: Zero RL for Base Models

SimpleRL-Zoo is an open-source framework for investigating the paradigm of zero reinforcement learning (zero RL) applied to open base language models. Zero RL training denotes direct reinforcement learning (RL) optimization performed on a pretrained base model, excluding any intermediate supervised fine-tuning stages. The primary motivation behind SimpleRL-Zoo is to elucidate the generality and transferability of emergent chain-of-thought (CoT) reasoning and cognitive behaviors when using a simple rule-based reward structure, previously demonstrated in DeepSeek-R1 for very large models. The suite encompasses extensive empirical studies on a diverse "zoo" of ten open-source models from multiple families (Llama3, Mistral, DeepSeek, Qwen2.5) and a range of sizes (0.5B–32B parameters), employing a unified GRPO optimization methodology and controlled experimental protocol. The repository provides open-source code, checkpoints, and comprehensive analysis tools for further study [2503.18892].

## 1. Zero RL Paradigm and Chain-of-Thought Emergence

Zero RL is formally defined as RL optimization starting from a pretrained base language model with no prior supervised fine-tuning (SFT) on the target task. The key phenomenon under investigation is the spontaneous emergence of extended CoT reasoning and self-reflection capabilities ("aha moment") in model outputs when trained under a sparse, rule-based reward signal. For instance, DeepSeek-R1 showed that rewarding only the final correct answer (+1 for correct, 0 otherwise) allowed large models to develop long, multi-step reasoning sequences. SimpleRL-Zoo evaluates whether such behavior generalizes to smaller and more diverse base model architectures. A critical empirical insight is that large open models such as Qwen2.5 already possess strong instruction-following and self-reflection, potentially obscuring true zero RL effects. Tracking additional cognitive behaviors (verification, enumeration, backtracking) is essential for characterizing reasoning emergence.

## 2. Implementation Architecture and Optimization

SimpleRL-Zoo employs a length-rectified version of the Generalized Reinforcement Policy Optimization (GRPO) algorithm for policy updates. For each query $q$, $G$ rollouts $\{o_1, ..., o_G\}$ are sampled using the old policy $\pi_{\theta_{old}}$. The token-level importance ratio is defined as $r_{i,t}(\theta) = \pi_\theta(o_{i,t}|q,o_{i,<t}) / \pi_{\theta_{old}}(o_{i,t}|q,o_{i,<t})$. The standardized advantage for each rollout is $\, \hat{A}_i = (r_i - \mu_j r_j)/\sigma_j r_j \,$ where $r_i \in \{0,1\}$. The clipped GRPO objective is:

$$
J_{\text{GRPO}}(\theta) = \frac{1}{\sum_i |o_i|} \sum_{i=1}^G \sum_{t=1}^{|o_i|} \min \left[ r_{i,t}(\theta) \hat{A}_i,\, \text{clip}(r_{i,t}(\theta); 1-\epsilon, 1+\epsilon) \hat{A}_i \right] - \beta D_{\mathrm{KL}}[\pi_\theta \parallel \pi_{\text{ref}}]
$$

where $\epsilon=0.2$, and $\beta=1e-4$ for models $\leq$14B, $\beta=1e-3$ for larger models. Reward design is minimal: $R(o) = 1$ if the final answer in $o$ is correct, $R(o) = 0$ otherwise. The training system (verl/HybridFlow) utilizes batch size 256, prompt batch size 1,024, up to 8,192 tokens per rollout (4,096 for Qwen-Math-7B), and sampling temperature 1.0. Sampling size $G$ is set to 8 by default. No strict format rewards are imposed, and KL regularization is controlled by $\beta$.

## 3. Key Experimental Design Strategies

SimpleRL-Zoo demonstrates several critical strategies for successful zero RL training:

- **Correctness-Only Rewards**: Strict format-based penalties (e.g., enforcing \boxed{…}) suppress exploration and risk collapse in weaker models. Only correctness signals are used for reward.
- **Difficulty Alignment**: The training set is subdivided into Easy (GSM8K + MATH lv.1), Medium (MATH lv.1–4), and Hard (MATH lv.3–5) splits (~8K examples each). Model-specific difficulty alignment is essential: excessive difficulty causes learning collapse; insufficient difficulty fails to incentivize advanced reasoning.
- **Prompt Tailoring**: Weak instruction-followers (Llama3-8B, Mistral-7B, Qwen0.5B/1.5B) are trained with minimal "step-by-step" prompts, while more complex prompt structures are used for stronger models.
- **Exploration Hyperparameters**: Larger sampling sizes ($G$ up to 32) and increased temperature stabilize training and enhance accuracy by promoting broader exploration in the policy space.

## 4. Model Coverage, Benchmarks, and Evaluation Protocols

The suite evaluates zero RL across the following models: Llama-3.1-8B, DeepSeek-Math-7B, Mistral-v0.1-7B, Mistral-Small-24B-Base-2501, Qwen-2.5 (0.5B, 1.5B, 7B, 14B, 32B), and Qwen-2.5-Math-7B. Tasks and benchmarks are selected for both math reasoning and generalization:

- **Training Data**: GSM8K, MATH (total ~15.5K problems)
- **Evaluation**: GSM8K, MATH500, Minerva Math, OlympiadBench, AIME 2024, AMC 2023 (math); IFEVAL (instruction), MMLU (knowledge), GPQA-Diamond (science QA)

All models use matched hyperparameters for comparability.

## 5. Quantitative Results: Reasoning Gains and Training Dynamics

Substantial improvements are observed after zero RL, as measured by Pass@1 (percentage of correct first attempts) and generalization metrics:

| Model             | Base Avg (%) | SimpleRL-Zoo Avg (%) | Δ      |
|-------------------|-------------|----------------------|--------|
| Llama-8B          | 10.6        | 22.0                 | +11.4  |
| DeepSeek-7B       | 11.3        | 29.2                 | +17.9  |
| Mistral-7B        | 5.3         | 18.6                 | +13.3  |
| Mistral-24B       | 27.6        | 49.6                 | +22.0  |
| Qwen-0.5B         | 12.1        | 20.9                 | +8.8   |
| Qwen-1.5B         | 18.5        | 36.1                 | +17.6  |
| Qwen-7B           | 40.3        | 55.2                 | +14.9  |
| Qwen-Math-7B      | 37.2        | 59.5                 | +22.3  |
| Qwen-14B          | 43.2        | 56.8                 | +13.6  |
| Qwen-32B          | 45.9        | 61.9                 | +16.0  |

On generalization benchmarks: Mistral-24B improves from 25.0% to 55.3% (+30.3), Llama-8B from 23.6% to 32.6% (+9.0), DeepSeek-7B from 18.7% to 34.1% (+15.4), Qwen-32B from 46.7% to 60.6% (+13.9).

Notably, the pass@k gap widens post-RL, with Mistral-24B's base pass@8 ≈ 60% versus post-RL pass@1 ≈ 65%, consistent with genuine capability gains rather than reranking. Average response length increases significantly (e.g., DeepSeek-7B from ~300 to ~1,200 tokens), yet this does not always correlate with cognitive behavior emergence. Clip ratio is maintained <5% except in unstable models.

## 6. Cognitive Behavior Tracking and "Aha Moment" Emergence

SimpleRL-Zoo performs explicit cognitive behavior analysis using GPT-4o and the "Cognitive Behaviors" framework, labeling responses for subgoal setting, enumeration, verification, and backtracking. Model-specific patterns are revealed:

- Qwen-2.5 (7B/32B): Already strong reasoning, minimal change in verification/backtracking.
- Mistral-24B, Llama-8B, DeepSeek-7B: Verification & backtracking rise from ~0% to ≈50% of responses, indicating bona fide "aha moments."
- Qwen-0.5B/1.5B: Marked increases in subgoal setting and enumeration.
- Mistral-v0.1-7B: Unstable training, low behavior rates, collapse.

For example, after RL training, Mistral-24B evolves from producing straight-line CoT to exhibiting systematic root solving, constraint checking, and solution revision—demonstrating reflective reasoning.

## 7. Best Practices, Implications, and Open Resources

Principal findings are as follows:

1. Increased response length does not equate to advanced reasoning; behaviors must be tracked directly.
2. Rigid format rewards constrain exploration; correctness-only reward fosters emergent cognition.
3. Data difficulty must be matched to model capability to prevent collapse or stagnation.
4. Applying zero RL directly yields improved reasoning; SFT on short CoT data prior to RL impedes exploration.
5. Large sampling sizes ($G \geq 8$) and higher exploration temperature enable stable learning.
6. Zero RL reliably boosts pass@k by 10–30 points, with evidence for genuine capability enhancement over superficial reranking.

SimpleRL-Zoo provides open-source code and resources (https://github.com/hkust-nlp/simpleRL-reason), including verl/HybridFlow training scripts, all model checkpoints, and cognitive behavior analysis notebooks to facilitate reproducibility and further study [2503.18892].

Source: https://www.emergentmind.com/topics/simplerl-zoo