APRIL: Active Partial Rollouts in RL

• APRIL is a system-level scheduler that mitigates the long-tail rollout bottleneck in on-policy reinforcement learning for large language models.
• It employs over-provisioning, early termination, and recycling of incomplete responses to optimize GPU utilization without discarding work.
• Experimental results report throughput improvements up to 44% and final accuracy gains of 7–8% across multiple hardware platforms.

APRIL, short for Active Partial Rollouts in Reinforcement Learning, is a lightweight, system-level scheduler for reinforcement learning training of LLMs that targets the long-tail distribution of rollout response lengths. It was introduced to address a central systems bottleneck in on-policy RL: rollout generation accounts for more than 90% of total runtime, and a small fraction of very long responses can stall an otherwise synchronous batch, leaving GPUs idle and underutilized. APRIL responds by over-provisioning rollout requests, terminating once the target number of responses is reached, and recycling incomplete responses for continuation in future steps, with the stated aim of reducing GPU idle time without discarding work (Zhou et al., 23 Sep 2025).

1. Problem setting and motivation

Reinforcement learning has become a cornerstone in advancing large-scale pre-trained LLMs, and successive generations including GPT-o series, DeepSeek-R1, Kimi-K1.5, Grok 4, and GLM-4.5 have relied on large-scale RL training to enhance reasoning and coding capabilities (Zhou et al., 23 Sep 2025). In this setting, most RL frameworks primarily rely on inference engines for rollout generation and training engines for policy updates.

The difficulty addressed by APRIL is the long-tail rollout bottleneck. Each training step issues hundreds of autoregressive inference requests, each producing a variable-length natural-language response. Empirically, response lengths follow a pronounced long tail: most rollouts terminate within a few thousand tokens, while a small fraction approach the maximal context window, such as 16 K or 32 K tokens. In a conventional synchronous rollout engine, all workers in the batch must finish before the batch can advance. The result is that a few slow sequences hold up the entire GPU fleet, creating “bubbles” of idle time on faster workers and capping attainable throughput (Zhou et al., 23 Sep 2025).

As model sizes, sample counts $\mathcal{N}$, and maximum length $\mathcal{L}$ grow, the paper characterizes this long-tail phenomenon as the single-largest barrier to scaling on-policy RL training. APRIL is positioned as a response to that systems-level constraint rather than as a change to the underlying LLM or CUDA kernels.

2. Scheduler design and rollout lifecycle

APRIL sits above any LLM inference engine and organizes rollout generation around four design elements (Zhou et al., 23 Sep 2025):

• Over-provisioning: at each RL iteration $k$, APRIL submits $N' > N$ concurrent queries instead of requesting exactly $N$ completed rollouts.
• Early Termination: once $N$ of the $N'$ requests finish, APRIL immediately aborts the remaining $N'-N$ inflight generations.
• Recycling: incomplete tokens are cached in a continuation buffer and resumed in the next rollout phase before issuing any brand new samples.
• On-Policy Integrity: completed rollouts, whether finished within iteration $k$ or resumed from earlier steps, enter the training engine.

Operationally, the mechanism is simple. APRIL first draws from the continuation buffer, then fills the remaining request slots with new prompts until it reaches $N'$. It launches all $\mathcal{L}$0 generations concurrently, waits until any $\mathcal{L}$1 complete, aborts the rest, stores unfinished trajectories back into the buffer, and returns the completed rollouts for policy update (Zhou et al., 23 Sep 2025).

A common misconception is that APRIL improves throughput by throwing away slow generations. The method is defined to do the opposite: incomplete responses are not discarded, but recycled. The paper further states that the resulting mixture of slightly off-policy trajectories does not destabilize learning and often improves generalization through greater data diversity (Zhou et al., 23 Sep 2025). This suggests that APRIL trades strict per-iteration synchrony for a controlled degree of continuation across iterations.

3. Efficiency analysis

The efficiency argument for APRIL is formulated through order statistics of rollout times. Let $\mathcal{L}$2 denote the time to generate rollout $\mathcal{L}$3, with $\mathcal{L}$4 assumed i.i.d. from a long-tailed distribution with CDF $\mathcal{L}$5. In vanilla synchronous batching of size $\mathcal{L}$6, the expected batch time is (Zhou et al., 23 Sep 2025)

$\mathcal{L}$7

APRIL issues $\mathcal{L}$8 requests and stops when the $\mathcal{L}$9th fastest finishes. If $k$0 denotes the $k$1th order statistic, the expected batch time becomes

$k$2

The idle-time reduction is then defined as the difference between the synchronous maximum and the APRIL stopping time. In the paper’s discussion, when $k$3 and rollout times follow common heavy-tail models such as a Pareto distribution with shape $k$4, the resulting idle-time reduction is substantial, on the order of 20–40% of batch time (Zhou et al., 23 Sep 2025).

The significance of this formulation is that APRIL is not presented merely as a heuristic. Its benefit is tied directly to the gap between the maximum of $k$5 long-tailed rollout times and the $k$6th order statistic among $k$7 concurrent requests. That framing makes the method especially relevant when the tail of the response-length distribution thickens with larger context windows and larger sample counts.

4. System integration and hardware scope

APRIL is implemented as a scheduler layer within the slime RL framework (Zhou et al., 23 Sep 2025). In that framework, inference and training are already decoupled: inference uses vLLM or SGLang backends, while training uses PyTorch FSDP or Megatron-LM on collocated GPUs. APRIL hooks into slime’s request dispatcher, adds early-stop signals, and maintains a lightweight continuation buffer in host memory.

The implementation is explicitly described as requiring no changes to CUDA kernels or LLM internals (Zhou et al., 23 Sep 2025). That design choice is central to the claim of framework agnosticism: APRIL acts at the scheduling layer rather than at the model architecture or kernel level.

The reported hardware scope includes validation on NVIDIA H100, H200, and AMD MI300 platforms. On AMD, deployment uses slime’s existing multi-vendor device abstractions together with a small patch to the torch_memory_saver plugin for ROCm. The paper reports identical percentage improvements across platforms, and characterizes APRIL as both framework and hardware agnostic (Zhou et al., 23 Sep 2025). A plausible implication is that the method is portable because its core intervention is concurrency control and continuation management rather than vendor-specific inference optimization.

5. Experimental evaluation

The reported experiments use Qwen3-4B and Qwen3-8B; training is conducted on the mathematical reasoning datasets DAPO-Math-17k, DeepScaleR, and DeepMath-103K; evaluation is on AIME-2024; RL algorithms include GRPO, DAPO, and GSPO; and hardware consists of a single node with 8× AMD MI300 and 8× NVIDIA H100/H200 GPUs (Zhou et al., 23 Sep 2025). The listed hyperparameters are rollout_batch_size=32, n_samples_per_prompt=8 for 256 total samples, over_sampling_batch_size=64 for 512 issued requests, and rollout_max_response_len=16384.

Figure 1 of the paper compares tokens/sec across 10 K training steps and reports the following throughput gains for APRIL over the synchronous baseline (Zhou et al., 23 Sep 2025):

Configuration	Throughput gain on the three datasets
Qwen3-4B + GRPO	+24% / +31% / +35%
Qwen3-4B + DAPO	+8% / +11% / +10%
Qwen3-8B + GRPO	+26% / +35% / +44%
Qwen3-8B + DAPO	+8% / +8% / +10%

Across the study, APRIL is reported to improve rollout throughput by at most 44% and to achieve at most 8% higher final accuracy (Zhou et al., 23 Sep 2025). In the convergence plots against AIME-2024 accuracy, APRIL matches or slightly exceeds baseline convergence speed and attains up to +7–8% higher final accuracy. The paper also notes that occasional baseline training “explosions,” defined as rollouts saturating max length, are wholly avoided under APRIL, and suggests that its off-policy diversity regularizes the policy (Zhou et al., 23 Sep 2025).

Section 5.3 further analyzes the composition of APRIL-generated data. It reports that up to 40% of tokens in a given iteration derive from resumed partial sequences. At the same time, instance-level analysis finds that rollouts for the same prompt have far lower variance in length, with $k$8 tokens, than the batch-level variance, with $k$9–4{,}500$. The stated conclusion is that APRIL does not introduce fresh intra-group long-tails (Zhou et al., 23 Sep 2025).

Finally, on NVIDIA H200, APRIL delivers nearly the same percentage throughput and evaluation gains as on AMD MI300, which the paper presents as evidence of cross-vendor portability (Zhou et al., 23 Sep 2025).

6. Interpretation, limitations, and prospective extensions

APRIL is presented as a method that unifies system-level and algorithmic considerations in RL training. Its central claim is not only that scheduling can recover lost utilization, but that mild off-policy bias introduced by continuation and resumption can be tolerated in practice and may correlate with improved generalization (Zhou et al., 23 Sep 2025). This is important because long-tail inefficiency is usually treated as a systems issue, whereas APRIL explicitly couples it to learning behavior.

A natural point of controversy is the status of on-policy learning. APRIL preserves the usual training loop only partially: completed rollouts may include trajectories resumed from earlier iterations. The paper’s position is that this mixture of slightly off-policy trajectories does not destabilize learning. It also reports improved final accuracy and the absence of baseline training explosions under APRIL (Zhou et al., 23 Sep 2025). This does not eliminate the conceptual distinction between strictly on-policy and mildly off-policy data collection, but it places the trade-off in a measurable throughput–stability regime.

The future extensions proposed in the paper are concrete (Zhou et al., 23 Sep 2025). They include Adaptive Oversampling, in which $N' > N$0 is dynamically tuned based on current rollout-length statistics; Integration with Speculative Decoding, intended to compound inference speedups; Fine-grained Cost Models, using per-instance length predictors to inform oversample ratios per prompt; and Broader RL Domains, including interactive environments such as embodied agents and very long horizon tasks in video and control.

Within that framing, APRIL occupies a specific place in the RL systems landscape: it is neither a new policy-gradient objective nor a new inference kernel, but a scheduler for active partial rollouts whose purpose is to tame long-tail generation, improve GPU utilization, and increase overall RL training throughput at scale (Zhou et al., 23 Sep 2025).