APRIL: Active Partial Rollouts in Reinforcement Learning to tame long-tail generation (2509.18521v1)

Abstract: Reinforcement learning (RL) has become a cornerstone in advancing large-scale pre-trained LLMs. Successive generations, including GPT-o series, DeepSeek-R1, Kimi-K1.5, Grok 4, and GLM-4.5, have relied on large-scale RL training to enhance reasoning and coding capabilities. To meet the community's growing RL needs, numerous RL frameworks have been proposed. Most of these frameworks primarily rely on inference engines for rollout generation and training engines for policy updates. However, RL training remains computationally expensive, with rollout generation accounting for more than 90% of total runtime. In addition, its efficiency is often constrained by the long-tail distribution of rollout response lengths, where a few lengthy responses stall entire batches, leaving GPUs idle and underutilized. As model and rollout sizes continue to grow, this bottleneck increasingly limits scalability. To address this challenge, we propose Active Partial Rollouts in Reinforcement Learning (APRIL), which mitigates long-tail inefficiency. In the rollout phase, APRIL over-provisions rollout requests, terminates once the target number of responses is reached, and recycles incomplete responses for continuation in future steps. This strategy ensures that no rollouts are discarded while substantially reducing GPU idle time. Experiments show that APRIL improves rollout throughput by at most 44% across commonly used RL algorithms (GRPO, DAPO, GSPO), accelerates convergence, and achieves at most 8% higher final accuracy across tasks. Moreover, APRIL is both framework and hardware agnostic, already integrated into the slime RL framework, and deployable on NVIDIA and AMD GPUs alike. Taken together, this work unifies system-level and algorithmic considerations in proposing APRIL, with the aim of advancing RL training efficiency and inspiring further optimizations in RL systems.

• The paper demonstrates APRIL’s active partial rollout mechanism to mitigate long-tail delays, resulting in substantial throughput gains and improved training stability.
• It implements over-provisioned rollout requests with early termination and buffering, optimizing GPU utilization and accelerating convergence.
• Empirical evaluations on multiple RL algorithms and hardware platforms confirm enhanced accuracy and efficiency, underscoring APRIL's practical impact on LLM training.

APRIL: Active Partial Rollouts in Reinforcement Learning to Tame Long-Tail Generation

Introduction

Reinforcement learning (RL) is a critical component in the post-training alignment and capability enhancement of LLMs. However, the rollout phase—where the model generates responses for policy optimization—remains a dominant computational bottleneck, often accounting for over 90% of RL training time. This inefficiency is exacerbated by the long-tail distribution of rollout response lengths: a small fraction of extremely long responses stalls entire batches, leading to significant GPU underutilization. The paper introduces APRIL (Active Partial Rollouts in Reinforcement Learning), a system-level scheduling mechanism that addresses this bottleneck by over-provisioning rollout requests, terminating once the target number of responses is reached, and recycling incomplete rollouts for future continuation. This essay provides a technical analysis of APRIL, its implementation, empirical results, and implications for scalable RL training.

The Long-Tail Problem in RL Rollouts

The rollout phase in RL for LLMs is characterized by high variance in response lengths across input instances. Empirical analysis reveals a pronounced long-tail distribution: while most responses terminate quickly, a minority approach the maximum allowed length, dominating batch completion time and causing substantial GPU idle periods.

Figure 2: Distribution of rollout response lengths reveals a pronounced long-tail peak.

In standard synchronous RL, all rollouts in a batch must complete before policy updates, resulting in suboptimal hardware utilization.

Figure 3: (Above) In the standard synchronous RL training paradigm, GPU utilization is often suboptimal. (Below) The APRIL mechanism mitigates this issue by reducing the bubble during RL.

The APRIL Mechanism

APRIL introduces a preemptive, over-provisioned rollout scheduling strategy within the synchronous RL paradigm. The core steps are:

1. Over-provisioned Generation: At each RL iteration, more rollout requests ($N'$) than the required batch size ($N$) are issued to the inference engine.
2. Early Termination: Once $N$ rollouts are completed, the remaining unfinished rollouts are aborted.
3. Buffering and Resumption: Aborted partial rollouts are stored in a buffer and resumed in subsequent iterations, ensuring no computation is wasted.
4. Policy Update: Completed rollouts are used for policy optimization, while continued rollouts may span multiple policy versions, introducing mild off-policy data.

   Figure 1: APRIL over-provisions instances, stops once the target number of instances finishes their rollouts, and updates the policy model. Incomplete rollouts are stored in the data buffer and resumed in later steps, thereby reducing idle time and improving training efficiency.

This mechanism is implemented at the RL system scheduling layer and is agnostic to the underlying inference engine or hardware platform.

Empirical Evaluation

Throughput and Efficiency

APRIL demonstrates substantial improvements in rollout throughput across multiple RL algorithms (GRPO, DAPO) and LLMs (Qwen3-4B, Qwen3-8B). On AMD MI300 and NVIDIA H100 platforms, throughput gains range from 24% to 44% for GRPO and 8% to 11% for DAPO, depending on the dataset and model size.

Figure 4: Comparison of rollout throughput between the baseline (non-partial rollout) and APRIL. APRIL consistently achieves higher throughput across datasets and models.

Convergence and Accuracy

Despite introducing partial off-policy rollouts, APRIL does not destabilize training. In fact, it often accelerates convergence and achieves higher final accuracy—up to 8% improvement on the AIME-2024 math reasoning benchmark. The learning curves for APRIL and the baseline are nearly identical, with APRIL sometimes exhibiting greater robustness against pathological long-sequence generation.

Figure 5: Convergence and accuracy comparison on Qwen3-4B model. APRIL achieves faster convergence and higher final accuracy.

Off-Policy Data Analysis

APRIL's partial rollouts introduce a significant proportion of hybrid-policy data (i.e., rollouts generated by multiple policy versions). Analysis shows that up to 40% of tokens in a batch may originate from continued rollouts. Despite this, training remains stable, and the diversity introduced by hybrid-policy rollouts may act as a regularizer.

Figure 6: Proportion of hybrid-policy rollouts in each RL training step.

Rollout Length Variance

APRIL reduces the standard deviation of rollout lengths at both the batch and instance levels, mitigating the long-tail effect without reintroducing intra-group variance.

Figure 7: $\sigma_{batch-level}$: standard deviation of response length per iteration at the batch level.

Figure 8: $\sigma_{instance-level}$: standard deviation of response length per iteration at the instance level.

Hardware Agnosticism

APRIL is integrated into the slime RL framework and is compatible with both NVIDIA and AMD GPUs. Modifications to memory management enable efficient operation on AMD MI300, and throughput improvements are consistent across hardware.

System and Algorithmic Implications

APRIL occupies a unique position between synchronous and asynchronous RL architectures. While fully asynchronous RL frameworks maximize hardware utilization at the cost of increased rollout staleness and potential instability, APRIL introduces a controlled degree of asynchrony within the synchronous paradigm. This enables significant efficiency gains without the complexity or instability associated with fully off-policy training.

APRIL is orthogonal to inference-level optimizations such as speculative decoding and continuous batching. It can be combined with these techniques for further efficiency improvements.

Limitations and Future Directions

While APRIL demonstrates strong empirical performance, several open questions remain:

• Excessive Off-Policy Influence: If rollouts are continued across too many policy versions, the risk of distributional shift increases. Empirically, this was not observed within five policy versions, but further paper is warranted.
• Adaptive Scheduling: Dynamic adjustment of over-provisioning ratios and buffer management could further optimize efficiency.
• Integration with Asynchronous Frameworks: Extending APRIL's principles to fully asynchronous or hybrid RL systems may yield additional benefits.

Conclusion

APRIL provides a practical, system-level solution to the long-tail inefficiency in RL training for LLMs. By actively managing partial rollouts, it achieves substantial improvements in throughput, convergence speed, and final accuracy, all while maintaining training stability. Its hardware-agnostic design and compatibility with existing RL frameworks make it readily deployable in large-scale LLM training pipelines. The approach highlights the importance of joint system-algorithm co-design and paves the way for further research on adaptive rollout strategies and efficient RL system architectures.