APRIL: Active Partial Rollouts in RL

• APRIL is a reinforcement learning scheduling method that over-provisions rollouts and recycles partial trajectories to mitigate long-tail latency.
• It addresses computational inefficiencies by terminating extra rollouts early and reusing incomplete responses, thus optimizing GPU utilization.
• Empirical results show 20–44% throughput gains and up to 8% accuracy improvements, with seamless integration into existing RL pipelines.

Active Partial Rollouts in Reinforcement Learning (APRIL) are system-level and algorithmic innovations aimed at dramatically improving efficiency in reinforcement learning (RL) for LLMs, specifically by addressing the computational bottleneck caused by the long-tail distribution of rollout response lengths. APRIL leverages over-provisioned, preemptable rollout generation and systematic recycling of incomplete responses to maximize GPU utilization, reduce rollout-phase wall-clock time, and maintain strict data efficiency. Experiments demonstrate 20–44% throughput gains and up to 8% final accuracy improvement across multiple RL objectives and model sizes, all within a hardware- and framework-agnostic design—allowing seamless integration into existing RL pipelines for LLM training (Zhou et al., 23 Sep 2025).

1. Long-Tail Effects in RL Training for LLMs

In standard on-policy RL for LLMs, a batch of $N$ prompts is processed synchronously by an inference engine, producing auto-regressive trajectories whose lengths follow a heavy-tailed distribution. Empirically, most sequences terminate quickly, but a minority—the "long tail"—approach the maximal generation length, causing overall batch completion to be gated by the slowest trajectories.

Formally, if $X_1, ..., X_N \overset{\text{i.i.d.}}{\sim} F_X$ are rollout generation times, the standard batch completion time is

$T_{\text{std}} = \max_{i=1...N} X_i = X_{(N)},$

and expected per-batch idle time is

$$\Delta_{\text{std}} = \mathbb{E}[X_{(N)}] - \mathbb{E}[X].$$

For long-tailed $F_X$ (e.g., Pareto or log-normal), $\mathbb{E}[X_{(N)}]$ scales rapidly with $N$, leading to $\Delta_{\text{std}}$ values that exceed 30–40% of wall-clock time. This results in severe underutilization of GPU resources during synchronous batch rollout (Zhou et al., 23 Sep 2025).

2. APRIL Algorithm: Over-Provisioned and Partial Rollout Scheduling

APRIL reduces tail-induced inefficiency by over-provisioning rollout requests, terminating trajectories as soon as the batch quota is met, and recycling partial rollouts for continuation. The methodology is captured by the following workflow:

1. Over-provisioned rollouts: Initiate $N'$ trajectories where $N' = rN$ ($r > 1$).
2. Active collection: Accept the first $N$ full completions as the batch, abort all remaining rollouts upon reaching this quota.
3. Partial recycling: Store all incomplete (aborted) trajectories in a FIFO buffer $\mathcal{B}$.
4. Continuation: In subsequent RL steps, resume rollout of partials from $\mathcal{B}$ under the current policy.
5. Policy update: Use only completed rollouts for policy optimization.

Pseudocode (LaTeX-style) exemplifies the precise steps and buffer management. The design ensures that no rollout computation is wasted—partial rollouts are deterministically resumed, and the system guarantees strict data utilization without discarding tokens (Zhou et al., 23 Sep 2025).

3. Theoretical and Empirical Throughput Gains

Let $X_{(k:n)}$ denote the $k$-th order statistic of $n$ i.i.d. samples from $F_X$. In APRIL, batch processing terminates at $T_{\text{APRIL}} = X_{(N : rN)}$. The expected speedup is thus

$$S(r) = \frac{\mathbb{E}[X_{(N:N)}]}{\mathbb{E}[X_{(N:rN)}]},$$

which, for large $N$, asymptotically approximates $F_X^{-1}(1)/F_X^{-1}(1/r)$ under a continuous model. For heavy-tailed $F_X$, even moderate $r$ (e.g., $r=2$) delivers significant speedup: experimental results show 20–44% improvement in rollout throughput across algorithms (GRPO, DAPO, GSPO) and tasks (Zhou et al., 23 Sep 2025).

Empirical evaluation provides the following observed throughput and accuracy gains:

Model	Algorithm	Dataset	Throughput Gain	Accuracy Gain
Qwen3-8B	GRPO	DeepMath-103K	+44%	+7.5%
Qwen3-8B	DAPO	DeepMath-103K	+10%	+3.0%
Qwen3-4B	GRPO	DeepMath-103K	+35%	+7.2%

APRIL also accelerates convergence (fewer RL steps to target reward) and achieves up to 8% higher final accuracy, attributed in part to the diversity introduced by partial trajectory continuation (Zhou et al., 23 Sep 2025).

4. Compatibility and Integration in RL Infrastructure

APRIL operates as a scheduler enhancement within "two-engine" RL pipelines, which couple inference engines (e.g., vLLM, SGLang) for rollout generation and training engines (e.g., FSDP, Megatron-LM) for policy optimization. APRIL modifies only the inference scheduling layer:

• Over-provisioned batch requests to the inference engine.
• Abortion and recycling of slow rollouts.
• All protocol, model, and hardware interfaces remain unchanged.

This design has been validated on both NVIDIA (H100/H200) and AMD (MI300) clusters, and integrated into the slime RL framework. No framework-specific or hardware-specialized code changes are necessary, ensuring deployment compatibility with existing RLHF and open RL pipelines (Zhou et al., 23 Sep 2025).

5. Relation to Selective Rollout Filtering

A complementary strategy is GRESO (GRPO with Efficient Selective Rollout), which realizes a different form of "active partial rollout" by predicting and omitting uninformative prompts before rollout. GRESO employs a lightweight, online filtering mechanism using reward-dynamics traces:

• Each prompt maintains a reward-variance trace.
• If a prompt is repeatedly zero-variance (produces identical rewards across all responses), it is skipped with a probability increasing in the length of the zero-variance streak.
• The skip rate is controlled adaptively to maintain a target proportion of effective prompts.

GRESO yields up to 2.4× rollout-phase speedup and 2.0× total-training speedup on large math reasoning benchmarks with no significant accuracy loss (Zheng et al., 2 Jun 2025). This approach is orthogonal to APRIL: APRIL targets system-level batch scheduling, while GRESO filters uninformative data at the algorithmic level. A plausible implication is that their combination could provide cumulative gains by reducing both computation and idle time.

6. Hyperparameters, Limitations, and Future Work

APRIL's principal parameter is the oversampling ratio $r$; $r=2$ was found to be robust across tasks. Higher $r$ increases buffer pressure and the staleness of partials, with diminishing returns and possible risks to the on-policy nature of training if partials are resumed too many steps later. APRIL requires storing approximately 40% of tokens from the previous iteration for recycling, but this fits within existing rollout KV caches without additional GPU RAM overhead.

Potential directions for future development include:

• Adaptive staleness bounds or importance weighting for resumed partials to maintain on-policy guarantees.
• Dynamic $r_t$ selection or reinforcement-learned scheduling responsive to current rollout statistics.
• Integration with inference-level optimizations such as continuous batching or speculative decoding for compound efficiency improvements.

A plausible implication is that further system-algorithm co-design, building on ideas from both APRIL and active filtering approaches like GRESO, will be central to achieving scalable RL training for the next generation of LLMs (Zhou et al., 23 Sep 2025, Zheng et al., 2 Jun 2025).