VCRL: Variance-based Curriculum Reinforcement Learning for Large Language Models (2509.19803v1)

Abstract: Policy-based reinforcement learning currently plays an important role in improving LLMs on mathematical reasoning tasks. However, existing rollout-based reinforcement learning methods (GRPO, DAPO, GSPO, etc.) fail to explicitly consider LLMs' learning ability for samples of different difficulty levels, which is contrary to the human cognitive process of mathematical reasoning tasks from easy to difficult. Intuitively, we find that the variance of the rollout group's reward in RLVR partly reflects the difficulty of the current sample for LLMs. Samples that are too easy or too difficult have a lower variance, while samples with moderate difficulty have a higher variance. Based on this, we propose VCRL, a curriculum reinforcement learning framework that dynamically controls the difficulty of training samples based on the variance of group rewards. Experiments on five mathematical benchmarks and two models reveal the advantages of VCRL over the current LLM RL baselines.

• The paper introduces VCRL, a variance-based curriculum RL method that dynamically selects training samples based on reward variance to target the model’s learning frontier.
• It employs dynamic sampling with a memory bank replay mechanism that sustains high-value, moderately difficult samples for stable, efficient training.
• Empirical results on mathematical reasoning benchmarks demonstrate that VCRL outperforms standard RL methods, achieving faster convergence and reduced gradient variance.

Variance-based Curriculum Reinforcement Learning for LLMs: A Technical Analysis

Motivation and Background

Reinforcement learning (RL) has become a central paradigm for post-training LLMs on complex reasoning tasks, particularly in mathematics. While rollout-based RL methods such as Group Relative Policy Optimization (GRPO), DAPO, and GSPO have demonstrated empirical success, they do not explicitly account for the alignment between sample difficulty and the model's current capabilities. This is in contrast to curriculum learning (CL) principles, which advocate for a progression from easy to hard samples to maximize learning efficiency. The paper introduces Variance-based Curriculum Reinforcement Learning (VCRL), a framework that dynamically selects training samples based on the variance of group rewards, thereby targeting samples at the frontier of the model's competence.

Methodology

Variance-based Dynamic Sampling

VCRL leverages the variance in rewards across multiple rollouts for a given query as a proxy for sample difficulty. For a binary reward system, the variance $\sigma^2$ for $G$ rollouts with $k$ successes is:

$\sigma^2 = \frac{k(G-k)}{G(G-1)}$

The normalized difficulty metric is defined as $p = \frac{\sigma^2}{\sigma^2_{\max}}$, where $\sigma^2_{\max}$ is the maximum possible variance for the group size. Samples with $p$ near 1 are at the model's decision boundary—neither too easy nor too hard—making them most informative for learning. VCRL filters training samples by a threshold $\kappa$ on $p$, dynamically adjusting the curriculum as the model evolves.

Figure 1: Schematic of the VCRL pipeline, showing variance-based filtering and memory bank replay in the RL training loop.

Replay Learning with Memory Bank

To address the computational cost of repeatedly sampling and evaluating $p$ for all queries, VCRL maintains a memory bank $\mathcal{M}$ of high-$p$ samples. The memory bank is implemented as a priority queue, with priorities updated via a momentum mechanism:

$$P(x_j) \leftarrow \alpha P(x_j) + (1-\alpha) \beta(x_j)$$

where $\beta(x_j)$ tracks the recency of use and $\alpha$ is a momentum constant. During each training step, samples with $p < \kappa$ are replaced by high-priority samples from $\mathcal{M}$, ensuring that the training batch is consistently composed of high-value, moderately difficult samples.

Policy Optimization Objective

VCRL builds on the GRPO objective, but restricts policy updates to samples with $p \geq \kappa$:

$$\mathcal{J}_\text{VCRL}(\theta) = \mathbb{E}_{x \sim \mathcal{D} \cup \mathcal{M}, \{y_i\}_{i=1}^G} \left[ \frac{1}{G} \sum_{i=1}^G \mathbb{I}(p_i \geq \kappa) \cdot \text{GRPO\_loss}(y_i) \right]$$

This selective update mechanism theoretically reduces the variance of policy gradients, leading to more stable and efficient training.

Empirical Results

Main Results

VCRL was evaluated on five mathematical reasoning benchmarks (AIME-2024, AIME-2025, MATH500, OlympiadBench, AMC23) using Qwen3-4B-Base and Qwen3-8B-Base models. Across all benchmarks and both model sizes, VCRL outperformed GRPO, DAPO, and GSPO by substantial margins. For Qwen3-8B-Base, VCRL achieved an average score of 57.76, exceeding the best baseline (GSPO) by 4.67 points and the base model by 24.8 points. The gains were especially pronounced on the most challenging datasets, indicating that VCRL is particularly effective at advancing the model's reasoning frontier.

Figure 2: Performance curves for Qwen3-4B-Base across five benchmarks, showing VCRL's superior learning trajectory compared to baseline RL methods.

Figure 3: Performance curves for Qwen3-8B-Base, further demonstrating VCRL's consistent advantage in both learning speed and final performance.

Training Dynamics

VCRL not only accelerates early-stage learning but also maintains higher reward scores, longer response lengths, and higher entropy throughout training compared to GRPO. This indicates that VCRL encourages exploration and avoids premature convergence to deterministic policies.

Figure 4: The normalized variance $p$ as a function of the number of successful rollouts $k$ for $G=16$, illustrating the U-shaped relationship between variance and sample difficulty.

Ablation Study

Incremental ablations confirmed that both variance-based dynamic sampling and replay learning contribute positively to performance. The largest marginal gain was observed when replay learning was added, validating the importance of maintaining a diverse set of high-difficulty samples.

Theoretical Analysis

The paper provides a formal analysis showing that VCRL's selective update mechanism leads to a lower expected policy gradient norm compared to GRPO. This reduction in gradient variance translates to more stable optimization and improved convergence properties. Empirical gradient norm curves corroborate this theoretical result, with VCRL exhibiting smoother and lower-magnitude updates throughout training.

Implementation Considerations

• Computational Overhead: Calculating reward variance for each sample requires multiple rollouts per query, increasing GPU utilization. However, the memory bank and dynamic sampling mitigate redundant computation by reusing high-value samples.
• Hyperparameters: The $p$-threshold $\kappa$ and momentum constant $\alpha$ are critical for balancing exploration and exploitation. The paper uses $\kappa=0.3$ initially, increasing to $0.8$ after 20 steps, and $\alpha=0.9$.
• Scalability: VCRL was demonstrated on 4B and 8B parameter models with batch sizes of 128 and 16 rollouts per query, indicating feasibility for large-scale LLMs.
• Integration: VCRL is implemented atop the verl RLHF framework and is compatible with any rollout-based RL method that provides per-sample rewards.

Implications and Future Directions

VCRL provides a principled approach to curriculum RL for LLMs, directly targeting the model's learning boundary at each training step. This dynamic curriculum is more adaptive than static difficulty sorting and is robust to the evolving capabilities of the model. The strong empirical results on mathematical reasoning benchmarks suggest that variance-based curricula could generalize to other domains requiring complex, multi-step reasoning.

Potential future directions include:

• Extending VCRL to non-binary or continuous reward settings.
• Investigating alternative difficulty metrics, such as uncertainty estimates from Bayesian models.
• Applying VCRL to multi-modal or agentic LLMs, where task difficulty is even more dynamic.
• Exploring automated threshold scheduling for $\kappa$ to further optimize learning efficiency.

Conclusion

VCRL introduces a variance-driven curriculum mechanism for RL-based LLM alignment, dynamically focusing training on samples at the model's current decision boundary. By combining variance-based dynamic sampling with replay learning, VCRL achieves superior performance, faster convergence, and greater training stability compared to existing rollout-based RL methods. The approach is theoretically justified, empirically validated, and readily applicable to large-scale LLM training pipelines.