Learning at the Right Pace: Adaptive Data Scheduling Improves LLM Reinforcement Learning

Abstract: LLMs achieve remarkable reasoning capabilities through reinforcement learning (RL) post-training. However, existing RL post-training commonly relies on uniform data sampling, which ignores the semantic structure of the training data and the changing capability of the training policy. To address these limitations, we propose Adaptive Data Scheduling (ADS), a dual-level data scheduling framework for pacing RL post-training that replaces uniform sampling with an adaptive distribution over semantic clusters and policy-boundary sample selection. At the cluster level, ADS organizes samples according to semantic patterns and maintains an adaptive inter-cluster distribution to solidify current training progress. At the sample level, ADS performs intra-cluster scheduling to continuously sample policy-boundary samples, which provides informative relative advantages. Experimental results across three LLMs and seven reasoning benchmarks demonstrate that ADS improves average accuracy by 5.2% over Group Relative Policy Optimization (GRPO). Notably, ADS consistently improves RL methods with different objective designs, highlighting its potential as a general data scheduling strategy for LLM RL post-training. The source code is available at: https://github.com/Richard-zrx/ADS.

• The paper introduces a dual-level Adaptive Data Scheduling (ADS) framework that aligns semantic clusters and policy competence to enhance LLM reinforcement learning.
• The paper validates ADS across multiple LLMs and reasoning benchmarks, achieving an average accuracy improvement of 5.2% over baseline methods.
• The paper demonstrates that focusing training on policy-boundary samples and dynamic cluster sampling boosts learning efficiency and generalization to out-of-distribution tasks.

Adaptive Data Scheduling for Enhanced LLM Reinforcement Learning

Introduction

The paper "Learning at the Right Pace: Adaptive Data Scheduling Improves LLM Reinforcement Learning" (2606.22305) presents Adaptive Data Scheduling (ADS), a principled framework for non-uniformly scheduling training data during reinforcement learning (RL) post-training of LLMs. The motivation is rooted in the observed limitations of the canonical uniform sampling protocol, which discards the semantic structure of prompts and disregards the evolving policy boundary during RL training. This neglect leads to inefficiencies: models repeatedly sample prompts that are either too easy or too hard relative to their current capability. ADS addresses these challenges by decomposing the scheduling into two levels, dynamically aligning data selection with both the semantic decomposition of the dataset and the instantaneous policy competence.

Figure 1: The overall framework of ADS.

Dual-Level Adaptive Scheduling: Semantics and Policy Boundary

ADS designates explicit semantic clusters over the training dataset by projecting prompt-solution pairs into the representation space of the current base model and partitioning these representations using $K$-means clustering. This clustering reveals domains of semantic coherence, such as particular mathematical topics or reasoning patterns (see Figure 2 for qualitative examples).

Figure 2: Word cloud visualization of 16 representative semantic clusters constructed from Qwen2.5-Math-1.5B embeddings, showing coherent domains such as calculus, geometry, sequences, counting, age reasoning, and calendar reasoning.

Cluster-Level Scheduling:

Training is paced across these semantic clusters based on policy competency. At each optimization step, a running estimate of success rate within each cluster is maintained, and clusters are sampled in proportion to their recent success rates (with exponential smoothing for stability). This inter-cluster adaptation explicitly focuses learning on clusters where the current policy is accumulating competence, facilitating consolidation of learning before transitioning to harder clusters. It avoids unproductive training on clusters far beyond the current policy's capacity, which yield little advantage signal for gradient computations.

Intra-Cluster Scheduling:

Within a selected cluster, ADS dynamically maintains a mini-cluster of policy-boundary samples. These are prompts whose current empirical rollout success rates are close to 50%, ensuring that sampled response groups contain both correct and incorrect outputs. Such prompts generate maximal within-group advantage for RL objectives such as GRPO. The policy-boundary set is adapted online as the policy improves, using not only online performance but also precomputed difficulty scores (e.g., negative log-likelihood of the reference solution under the current policy) to guide the movement of samples toward or away from the mini-cluster.

Empirical Evaluation

ADS is evaluated across three representative LLMs—Qwen2.5-Math-1.5B, Qwen3-4B-Base, and Olmo3-7B-SFT—using seven diverse mathematical and scientific reasoning benchmarks.

Figure 3: Qwen2.5-Math-1.5B.

Superior Quantitative Performance

ADS achieves consistent, robust performance improvements over established baselines, including GRPO and DOTS+RR, across all tasks and model scales. The average accuracy improvement over GRPO is 5.2%. This advantage reflects both higher mean accuracy (Mean@16) and increased pass rates (Pass@16) on challenging reasoning datasets. The gains extend to out-of-distribution domains (e.g., GPQA-D, MMLU-Pro), confirming that ADS does not merely overfit to the training set geometry but discovers transferable, general reasoning improvements. The improvements persist from 1.5B through 7B parameter scales, underscoring the generality of the scheduling paradigm.

Objective-Agnostic Gains

ADS provides consistent gains regardless of the specific RL objective employed for post-training. Integrating ADS with orthogonal improvements such as On-Policy Distillation, DAPO, and GSPO synergistically boosts their efficacy. The data scheduling mechanism operates strictly at the sampling protocol level and does not require any adaptation or reweighting of the underlying RL objective—policy updates are strictly more informative due to better sample selection, independent of the loss function used.

Robustness and Practicality

Cross-policy clustering experiments establish that clusters computed with one policy can be efficiently transferred to other policies with negligible loss in performance. This indicates that the learned semantic domains are robust and capture generic structure usable across multiple LLM architectures.

Ablation studies demonstrate that both inter-cluster and intra-cluster scheduling components contribute critically to final accuracy, and omissions at either level substantially degrade performance. The offline preprocessing cost for clustering and difficulty estimation is shown to be minor relative to total RL training cost.

Implications and Future Directions

ADS introduces a systematic, domain- and policy-aware approach to data scheduling in LLM RL post-training. This framework complements ongoing advances in RL loss design (e.g., token-level vs. sequence-level importance, reward shaping), suggesting that additional improvements can be realized by parallel advances in training data selection—especially as model sizes and reasoning task complexities increase.

ADS is directly tailored for settings with verifiable, outcome-based reward signals (math/logic, multi-step inference, program synthesis), but extending the scheduling principles to open-ended instruction, human preference optimization, or multi-modal/multi-turn tasks is an open direction. Future work should generalize the notion of policy-boundary samples to non-binary rewards and investigate more sophisticated online estimators of prompt informativeness. Combining adaptive scheduling with model-based curriculum learning, bandit-driven exploration, or automated cluster granularity selection are likely to deliver further improvements in learning efficiency and generalization.

Conclusion

Adaptive Data Scheduling introduces a dual-level, semantically grounded, and policy-aware sample selection protocol for RL post-training of LLMs. Empirical results demonstrate that dynamically aligning the training distribution with the evolving competence frontier of the policy across semantic domains yields substantial improvement over uniform sampling, leading to more efficient and robust acquisition of reasoning capabilities. This work reframes data scheduling as a central component of RL-driven LLM optimization and opens new avenues for synergistic improvements across sample selection and objective design.