---
title: 'RolloutPipe: Disaggregated RL Training'
url: https://www.emergentmind.com/topics/rolloutpipe
type: topic
---

# RolloutPipe: Disaggregated RL Training

Searching arXiv for the primary RolloutPipe paper and closely related rollout-scheduling work.
RolloutPipe is a post-training framework for disaggregated reinforcement learning with verifiable rewards (RLVR) that overlaps rollout generation and policy training for on-policy GRPO without introducing stale data. It operates in settings where rollout generation, reward computation, and optimization are split across independent GPU pools, and it replaces a round-level synchronization barrier with a group-level pipeline based on complete-group pipelining (CGP) and frontier-group dispatch (FGD) [2606.26997].

## 1. Operational setting and design objective

RolloutPipe is defined for on-policy RL post-training of large language models in the RLVR regime, where rewards are computed or verified algorithmically on mathematical, logical, and scientific tasks. The immediate algorithmic context is GRPO, a PPO-style method in which each prompt is sampled multiple times and rewards are normalized within the resulting group rather than through a learned value critic [2606.26997].

The framework assumes a disaggregated architecture with a rollout GPU pool, a trainer GPU pool, and a control or coordination node. In such systems, rollout generation and optimization are decoupled for flexible allocation of heterogeneous hardware, but synchronous on-policy execution introduces a utilization bottleneck. In the Slime baseline, an entire rollout round of \(R\) prompt groups is generated with fixed weights before any training begins, even though the first trainable group may materialize far earlier than the last. In the reported LSAT-AR example with Qwen3-1.7B and \(R=96, K=8\), the first group materializes at approximately 200 s, the last at approximately 400 s, and trainer GPUs remain idle throughout that staggered-completion window, yielding a trainer waiting ratio of 47–52% of end-to-end time [2606.26997].

The architectural problem is therefore not reward computation or optimizer throughput in isolation, but the existence of a fixed rollout-completion barrier in synchronous GRPO. Asynchronous RL pipelines overlap rollout and training, yet do so by training on data produced by older policy snapshots. RolloutPipe is explicitly designed to recover the idle interval while preserving strict on-policy semantics within each fixed-weight round [2606.26997].

## 2. Groupwise trainability and on-policy invariants

RolloutPipe is organized around the fact that, under GRPO, the minimal trainable unit is not an individual sample but a complete prompt group. For a prompt with \(K\) responses and rewards \(r_1,\dots,r_K\), GRPO computes per-response advantages as
\[
A_i = \frac{r_i - \mu_{\text{group}}}{\sigma_{\text{group}} + \epsilon_{\mathrm{std}}},
\]
with
\[
\mu_{\text{group}} = \frac{1}{K}\sum_{j=1}^{K} r_j,
\qquad
\sigma_{\text{group}} = \sqrt{\frac{1}{K}\sum_{j=1}^{K}(r_j-\mu_{\text{group}})^2}.
\]
No response in the group is trainable until all \(K\) responses, rewards, and group statistics are available [2606.26997].

RolloutPipe therefore distinguishes several units. A **group** is the \(K\)-sample set associated with one prompt. A **complete group** is a group for which all responses have finished decoding, rewards and verifier outputs have been computed, and group statistics have been materialized. A **trainable complete group** is a complete group that has been converted into the GRPO training format, including advantages and masks. If the global sample-level batch size is \(B\), then one logical GRPO update consumes
\[
U = \frac{B}{K}
\]
complete groups. In the reported setup, \(K=8\), \(B=16\), and hence \(U=2\) groups per logical update [2606.26997].

A recurrent misconception is that RolloutPipe is an asynchronous or partially off-policy system because it starts training before rollout completion. The framework is formulated to avoid exactly that outcome. All groups in a rollout round are generated by the same fixed rollout weights, the optimizer step and weight publication occur only after all \(R\) groups in the round have been consumed by the trainer, and there is no policy update between generation and training of any group within the round. The scheduling changes concern handoff timing and admission order, not sample provenance or update boundaries [2606.26997].

## 3. Complete-group pipelining

CGP is RolloutPipe’s training-side mechanism. Its purpose is to remove the unnecessary dependency between “last group materialized” and “first training update started.” As soon as a group is materialized on the control side, it is appended FIFO to the **Pending Complete Groups** queue. Once at least \(U\) trainable groups are available, the earliest \(U\) groups are dispatched to the trainer as a logical update batch [2606.26997].

If \(t_{\text{first}}^{(U)}\) denotes the earliest time at which the first \(U\) groups have all finished materialization, and \(t_{\text{complete}}\) denotes the materialization time of the last group in the rollout, then Slime begins training at
\[
t_{\text{start}}^{\text{Slime}} = t_{\text{complete}},
\]
whereas CGP begins training at
\[
t_{\text{start}}^{\text{CGP}} = t_{\text{first}}^{(U)}.
\]
The recovered overlap window is
\[
\Delta t_{\text{CGP}} = t_{\text{complete}} - t_{\text{first}}^{(U)}.
\]
This quantity is the headroom that would otherwise be lost to trainer idleness [2606.26997].

RolloutPipe also defines a trainer waiting ratio,
\[
w_{\text{train}} =
\frac{t_{\text{start}} - t_{\text{rollout\_start}}}
{t_{\text{train\_end}} - t_{\text{rollout\_start}}},
\]
which decreases when CGP advances the first dispatch time. A second trainer-side definition used in plots is
\[
\text{waiting ratio} =
\frac{\text{trainer wait}}{\text{trainer wait}+\text{trainer compute}}.
\]
Both emphasize the same systems effect: training work is unchanged, but idle time before and between updates is reduced [2606.26997].

The control path includes a **Feasible Batch Selector** that chooses the largest prefix of pending groups whose total token count fits the per-GPU token budget while always respecting group boundaries. If a single group is too large, it is taken alone and handled by dynamic micro-batching. This detail is central to maintaining the complete-group invariant in variable-length GRPO workloads [2606.26997].

## 4. Frontier-group dispatch

CGP alone advances training start, but it does not control the order in which groups become ready. In the default rollout engine, request-level FIFO is group-agnostic, so the groups needed for the next logical update may finish with large and irregular gaps. FGD is the rollout-side admission policy that regularizes this supply [2606.26997].

Let \(O=\{g_1,\dots,g_R\}\) be the set of groups that have arrived but are not yet serving-complete, and let \(\mathrm{order}(g)\) denote submission order. FGD maintains a frontier set \(\mathcal{F}\) of at most \(F_w\) groups,
\[
\mathcal{F} = \arg\min^{\,F_w}_{g\in O}\mathrm{order}(g),
\]
that is, the \(F_w\) lowest-order groups among those not yet completed. A request \(q\) is admitted iff
\[
\mathrm{admit}(q) \iff g(q)\in\mathcal{F}.
\]
All other requests are deferred until frontier capacity opens [2606.26997].

In the reported experiments, \(F_w=U=2\). This aligns the rollout frontier with the trainer’s logical batch requirement: exactly the number of groups needed for one GRPO update are privileged at any moment. When a frontier group completes all \(K\) responses, it is removed from \(\mathcal{F}\), the next lowest-order deferred group is admitted, and rollout capacity remains concentrated on the next trainable batch rather than spread over many partially completed groups [2606.26997].

| Mechanism | Locus | Primary effect |
|---|---|---|
| CGP | Control/training side | Starts training once \(U\) groups materialize |
| FGD | Rollout side | Makes frontier groups arrive earlier and more steadily |

This organization also clarifies a second misconception: FGD is not a change to the RL objective or GRPO statistics. It is strictly an admission policy over rollout requests. The groupwise reward normalization, logical update size, and weight-publication schedule remain those of the synchronous baseline [2606.26997].

## 5. System architecture and implementation

RolloutPipe is implemented on top of Slime, Megatron-LM, SGLang, and Ray, and is deployed across three node roles: rollout, control, and training. The rollout node maintains the Prompt Group Buffer, applies FGD, and passes admitted requests to the SGLang serving engine, whose Token Scheduler and Decode Workers execute generation. When all \(K\) requests of a group complete, the rollout node marks the group serving-complete and notifies the control node [2606.26997].

The control node performs **group materialization**. It collects the \(K\) responses, computes rewards and verifier outputs, converts samples into trainable records, computes \(\mu_{\text{group}}\) and \(\sigma_{\text{group}}\), attaches advantages and loss masks, and appends the result to Pending Complete Groups. It then invokes the Feasible Batch Selector and forwards the selected prefix to the trainer’s **U-group Ready Queue** [2606.26997].

The training node consumes groups from that queue, executes forward and backward passes, and counts groups through a Gradient Accumulator. Every \(U\) groups constitute one logical GRPO update. Once all \(R\) groups from the current rollout have been consumed, the Weight Publisher exports refreshed policy weights to SGLang for the next rollout round [2606.26997].

| Node | Principal functions | Key queues or buffers |
|---|---|---|
| Rollout | FGD, serving, completion notification | Prompt Group Buffer, Deferred Groups |
| Control | Materialization, verifier, CGP handoff | Pending Complete Groups |
| Training | Pipelined train RPC, accumulation, publish | U-group Ready Queue |

The paper states a control overhead of \(\mathcal{O}(R\cdot K)\) for group tracking, admission, and handoff, and a trainer computation cost of \(\mathcal{O}(R\cdot K\cdot L^2\cdot d)\) per rollout, identical to Slime because the same number of samples are processed. Communication overhead is linear in \(R\) and is overlapped with rollout and training. Backpressure is handled by the pending FIFO and ready queue if the trainer lags; if rollout lags, the trainer eventually waits, but CGP and FGD are intended to minimize that condition [2606.26997].

## 6. Empirical behavior, comparative position, and limitations

Evaluation is reported on Qwen3-1.7B across four reasoning and science workloads: LSAT-AR from AGIEval, Sci-XW and Sci-JL from SciBench, and OlyPhys from OlympiadBench. The trainer uses \(8\times\) RTX 4090 24GB with \(TP=4, DP=2\), the rollout node uses \(2\times\) A100 40GB PCIe with \(TP=2\), rollout sizes are \(R\in\{32,64,96\}\), \(K=8\), \(B=16\), \(U=2\), and \(F_w=2\). Reported results are averaged over four rounds [2606.26997].

Across all twelve settings, RolloutPipe shortens rollout-to-train-end time by 30.7–42.3% relative to Slime and lowers the trainer waiting ratio by 37–76%. The benefit grows with rollout size: approximately 30.7–32.9% at \(R=32\) and 39.8–42.3% at \(R=96\). CGP accounts for 71–96% of the total speedup, while FGD contributes an additional 2.5–11.4% over CGP alone [2606.26997].

Dispatch timing illustrates the scheduling effect sharply. At \(R=96\), the first logical batch is dispatched at 509–543 s under Slime, at 46–90 s under CGP, and at 52–61 s under CGP+FGD. The narrowing under FGD is significant because it indicates not only an earlier first update but a steadier arrival process for subsequent updates. The trainer-side waiting ratio falls from 47–52% in Slime to 14–33% under CGP+FGD, with approximately 13.8–15.0% reported for the \(R=96\) workloads [2606.26997].

A further misconception is that these gains arise from reducing training work or shortening responses. The evaluation explicitly reports near-equality of trainer compute time and response lengths across Slime, CGP, and CGP+FGD. For example, on LSAT-AR with \(R=96\), trainer compute time is 587.4 s for Slime, 589.0 s for CGP, and 586.4 s for CGP+FGD. The optimization is therefore a scheduling optimization, not a change in computational load or objective function [2606.26997].

In relation to adjacent systems, RolloutPipe is closest to synchronous disaggregated RLVR frameworks such as Slime and is explicitly contrasted with asynchronous systems such as AReaL and AsyncFlow, which improve utilization through stale data. A nearby but distinct synchronous strategy is RollPacker, which mitigates GPU bubbles caused by long-tail response lengths through tail batching, dynamic resource allocation for reward, and stream-based training while preserving synchronous semantics; its gains are driven by response-length imbalance rather than complete-group scheduling [2509.21009]. This suggests that RolloutPipe and tail-batching systems address different synchronization bottlenecks within on-policy LLM RL.

The reported limitations are also specific. Experiments are confined to Qwen3-1.7B and a modest cluster; the design assumes GRPO-style fixed-size groups with a well-defined materialization step; rounds still use a single fixed weight snapshot; and FGD uses a static low-order frontier policy with \(F_w\) rather than an adaptive scheduler. Suggested future directions include extending the pipelining ideas to other cognitive computing systems, including multimodal RAG and multi-agent collaborative settings where on-policy constraints remain important [2606.26997].

Source: https://www.emergentmind.com/topics/rolloutpipe