---
title: Partial Rollout Buffer in RL for LLMs
url: https://www.emergentmind.com/topics/partial-rollout-buffer
type: topic
---

# Partial Rollout Buffer in RL for LLMs

A partial rollout buffer is a data structure and scheduling mechanism used in reinforcement learning (RL) for large language models (LLMs) that enables the retention, management, and efficient utilization of incomplete (partial) trajectories. Its central purpose is to address challenges posed by the long-tail distribution of rollout lengths, GPU idle time, and sample efficiency during RL training, particularly under large-scale RLHF or RLVR regimens. Recent practices have adopted partial rollout buffers to maximize hardware throughput, accelerate convergence, and, in some variants, transform sparse terminal rewards into dense, prefix-level guidance across RL formalisms including APRIL, Contextual Rollout Bandits, and WS-GRPO [2509.18521][2602.08499][2602.17025].

## 1. Motivation and Design Rationale

The long-tail distribution of LLM-generated rollout lengths results in batch-level synchronization inefficiencies, where a small fraction of extended rollouts can stall all batch members, causing substantial underutilization of GPU resources. This inefficiency is amplified as models and rollout sizes increase, driving RL practitioners to seek system-level remedies that avoid discarding partial work and that decouple sample collection from strict batch synchronization [2509.18521]. Furthermore, managing partial and historical rollouts in a buffer allows for improved sample reuse, adaptive rollout selection, and denser reward propagation strategies, particularly when only sparse correctness signals are available [2602.08499][2602.17025].

## 2. Architectures and Operational Principles

Partial rollout buffer implementations differ according to their contextual use:

- **APRIL PartialRolloutBuffer**: Buffers incomplete rollouts when over-provisioned requests are aborted upon achieving the target number of completed rollouts ($N$). Each buffer entry retains a unique rollout ID, token prefix, model KV-cache, initiation step index, and policy version. At the subsequent RL step, entries are resumed from their saved state until rollouts are completed or again pre-empted, ensuring eventual completion and eliminating rollout waste [2509.18521].

- **Replay Buffers with Truncation (CBS)**: Contextual Rollout Bandits store both complete and truncated rollouts (flagged via a binary feature). Each is annotated with multi-dimensional arm features (e.g., normalized length, usage count, sample age, policy-perception metrics). The scheduler adaptively selects high-value historical and partial rollouts for policy improvements. Feature tracking enables noise-aware, frequency-adaptive replay prioritization exposed to an online neural scheduler [2602.08499].

- **In-Batch Prefix Buffers (WS-GRPO)**: WS-GRPO constructs a per-optimization-step buffer consisting of all consecutive prefixes from sampled rollouts. There is no large replay memory, but each step collects and evaluates all prefixes in the minibatch. These prefixes are used to propagate dense pseudo-reward signals, mediating between sparse terminal correctness and per-token/local guidance [2602.17025].

## 3. Algorithmic Procedures

A representative procedure from APRIL is summarized as follows [2509.18521]:

1. **Over-provision rollout requests**: Issue $N' = \alpha N$ requests, with $\alpha \in [1.5,2]$.
2. **Early termination**: Upon $N$ completions, abort all remaining in-flight rollouts.
3. **Partial rollout buffering**: Extract states of aborted rollouts—including token sequence, KV-cache, and metadata—and enqueue in PartialRolloutBuffer.
4. **Recycling**: At each subsequent step, resume up to $M$ buffered partials before issuing new rollouts.
5. **Policy update**: Use only the $N$ completed rollouts for policy improvement.
6. **Memory management**: Typical buffers sustain up to $(\alpha-1)N$ entries per step.

For the neural rollout scheduler in CBS, each round updates the buffer, computes features (length, truncation), trains the scheduler, and selects rollouts to maximize a reward function tied to policy improvement. Truncated or incomplete trajectories are reweighted or downweighted according to their flagged status, enabling informed sample selection under a contextual bandit paradigm [2602.08499].

WS-GRPO, in turn, generates for each prompt a new buffer containing all prefix pairs, which are scored by a weakly-supervised preference model to compute prefix-level rewards. These guide policy optimization within each minibatch, sidestepping reliance on terminal-only rewards [2602.17025].

## 4. Mathematical Models and Theoretical Guarantees

Table 1: Mathematical Forms and Empirical Gains

| Buffer Variant       | Key Metric                                   | Empirical Impact                    |
|----------------------|----------------------------------------------|-------------------------------------|
| APRIL                | Throughput $T$                               | $\Delta T\sim$24–44%                |
|                      | Convergence step reduction                   | 15–25% fewer steps                  |
|                      | Final accuracy gain                          | +2–8 percentage points              |
| CBS                  | Regret $R_T$ (bandit formalism)              | $R_T = \tilde{O}(\sqrt{T})$         |
|                      | Performance bound increase with buffer size  | $\varphi(B_1)\geq\varphi(B_2)$      |
| WS-GRPO              | Generalization bound vs buffer/complexity    | $\tilde{O}(\sqrt{\frac{d_{\max}+\lambda^2 BT_{\max}^2}{n}})$ |

APRIL models the expected improvement in rollout throughput as $\Delta T\% = \frac{T_\text{april}}{T_\text{baseline}}-1$ and shows empirical gains of 24–44% on representative language reasoning tasks. GPU idle fraction, modeled as $f_\text{idle} = \frac{t_\text{slow} - t_\text{mean}}{t_\text{slow}}$, decreases in proportion to throughput gains. Convergence steps to reach a fixed evaluation target are reduced by 15–25%, with a +2–8% absolute gain in final accuracy [2509.18521].

CBS guarantees sublinear regret under standard neural-bandit assumptions. The achievable performance upper bound scales monotonically with buffer size: larger buffers (and thus longer rollout histories, including partials) improve optimization opportunities [2602.08499].

WS-GRPO provides theoretical consistency bounds for prefix-level reward propagation, robustness to preference model error, and generalization guarantees matching $\tilde{O}(1/\sqrt{n})$ rates, with low additional variance from prefix rewards [2602.17025].

## 5. System-Level and Implementation Considerations

APRIL’s buffer requires memory management tuned to store up to $(\alpha-1)N$ in-flight partials per RL step. Each entry contains $O(Ld)$ bytes for token and KV-cache states. Lock-free queues or ring buffers support high-throughput operation. Strict on-policy constraints necessitate discarding or pruning entries older than $M$ steps [2509.18521].

Hardware-agnostic integration requires only the ability to abort GPU-side inference and extract KV-cache state. On NVIDIA, CUDA streams and vLLM abort APIs suffice; on AMD, similar functionality is achieved via HIP streams and custom memory snapshotting (see torch_memory_saver PR #43). Minimal changes are needed for frameworks like slime, with a PartialRolloutBuffer class handling all buffer manipulations.

CBS and related bandit-scheduling buffers employ a FIFO design with capacity $B \approx L \cdot |\mathcal{R}|$ and per-rollout arm features. Periodic featurization incurs $<4\%$ extra training cost. WS-GRPO’s buffer is in-batch and transient, rebuilt each optimization step and does not require persistent replay memory; it scales with the aggregate number of token- or sentence-level prefixes per batch [2602.17025].

## 6. Empirical Results and Comparative Performance

On math reasoning benchmarks with Qwen3-4B and 8 AMD MI300 GPUs, APRIL achieves the following improvements over strong RLHF baselines [2509.18521]:

| Algorithm         | Throughput (Kt/s) Baseline | APRIL | Accuracy Baseline (%) | APRIL (%) | Throughput Gain | Accuracy Gain |
|-------------------|---------------------------|-------|----------------------|-----------|----------------|--------------|
| GRPO (D17k)       | 50                        | 67    | 62                   | 68        | +34%           | +6           |
| GRPO (DS)         | 48                        | 63    | 59                   | 65        | +31%           | +6           |
| DAPO (D17k)       | 52                        | 56    | 61                   | 65        | +8%            | +4           |
| GSPO (D17k)       | 51                        | 70    | 60                   | 66        | +37%           | +6           |

WS-GRPO, on ARC/GSM8K/CommonsenseQA/DeepMath, achieves 50–90% reduction in average rollout lengths (tokens or reasoning steps)—with at most 2–3 percentage points of accuracy lost, and sometimes no loss at all (e.g., GSM8K steady at 92.4%) [2602.17025].

CBS, across six reasoning benchmarks, improves performance and efficiency by continually reusing (and adaptively deprioritizing) partial and out-of-sample rollouts, with monotonic gain in attainable policy performance as buffer size increases [2602.08499].

## 7. Applications and Implications

Partial rollout buffers are now standard in system-level RL optimizations for LLMs, permitting scalable, efficient, and accurate training at industrial scale [2509.18521]. Their use in contemporary frameworks (e.g., slime) is trivial to port across CUDA/HIP devices. Adaptive sampling schedulers (CBS-style) further boost efficiency by leveraging historical and incomplete rollouts rather than relying solely on freshly generated samples [2602.08499]. When paired with reward propagation via weak supervision (WS-GRPO), partial buffers enable dense feedback even under sparse, delayed objectives, thus reducing redundancy and overthinking without significant loss in accuracy [2602.17025].

The underlying mechanisms—over-provisioning, early abort and retention, feature-rich buffer storage, adaptive scheduler utilization, and prefix-level reward propagation—now form the backbone of RL pipelines for reasoning, coding, and alignment tasks in LLMs. They enable substantial improvements in both training dynamics and sample efficiency, while maintaining compatibility with diverse hardware and policy optimization schemes.

Source: https://www.emergentmind.com/topics/partial-rollout-buffer