RolloutPipe: Disaggregated RL Training

• RolloutPipe is a post-training framework for on-policy reinforcement learning that overlaps rollout generation with policy training, preserving strict on-policy invariants.
• It employs complete-group pipelining (CGP) to dispatch early trainable groups and uses frontier-group dispatch (FGD) to regularize group admissions for efficient GPU utilization.
• Empirical evaluations on benchmarks like LSAT-AR show a 30–42% reduction in rollout-to-training end time and a significant drop in trainer idle time.

Searching arXiv for the primary RolloutPipe paper and closely related rollout-scheduling work. RolloutPipe is a post-training framework for disaggregated reinforcement learning with verifiable rewards (RLVR) that overlaps rollout generation and policy training for on-policy GRPO without introducing stale data. It operates in settings where rollout generation, reward computation, and optimization are split across independent GPU pools, and it replaces a round-level synchronization barrier with a group-level pipeline based on complete-group pipelining (CGP) and frontier-group dispatch (FGD) (Chen et al., 25 Jun 2026).

1. Operational setting and design objective

RolloutPipe is defined for on-policy RL post-training of LLMs in the RLVR regime, where rewards are computed or verified algorithmically on mathematical, logical, and scientific tasks. The immediate algorithmic context is GRPO, a PPO-style method in which each prompt is sampled multiple times and rewards are normalized within the resulting group rather than through a learned value critic (Chen et al., 25 Jun 2026).

The framework assumes a disaggregated architecture with a rollout GPU pool, a trainer GPU pool, and a control or coordination node. In such systems, rollout generation and optimization are decoupled for flexible allocation of heterogeneous hardware, but synchronous on-policy execution introduces a utilization bottleneck. In the Slime baseline, an entire rollout round of $R$ prompt groups is generated with fixed weights before any training begins, even though the first trainable group may materialize far earlier than the last. In the reported LSAT-AR example with Qwen3-1.7B and $R=96, K=8$, the first group materializes at approximately 200 s, the last at approximately 400 s, and trainer GPUs remain idle throughout that staggered-completion window, yielding a trainer waiting ratio of 47–52% of end-to-end time (Chen et al., 25 Jun 2026).

The architectural problem is therefore not reward computation or optimizer throughput in isolation, but the existence of a fixed rollout-completion barrier in synchronous GRPO. Asynchronous RL pipelines overlap rollout and training, yet do so by training on data produced by older policy snapshots. RolloutPipe is explicitly designed to recover the idle interval while preserving strict on-policy semantics within each fixed-weight round (Chen et al., 25 Jun 2026).

2. Groupwise trainability and on-policy invariants

RolloutPipe is organized around the fact that, under GRPO, the minimal trainable unit is not an individual sample but a complete prompt group. For a prompt with $K$ responses and rewards $r_1,\dots,r_K$, GRPO computes per-response advantages as

$$A_i = \frac{r_i - \mu_{\text{group}}}{\sigma_{\text{group}} + \epsilon_{\mathrm{std}}},$$

with

$$\mu_{\text{group}} = \frac{1}{K}\sum_{j=1}^{K} r_j, \qquad \sigma_{\text{group}} = \sqrt{\frac{1}{K}\sum_{j=1}^{K}(r_j-\mu_{\text{group}})^2}.$$

No response in the group is trainable until all $K$ responses, rewards, and group statistics are available (Chen et al., 25 Jun 2026).

RolloutPipe therefore distinguishes several units. A group is the $K$-sample set associated with one prompt. A complete group is a group for which all responses have finished decoding, rewards and verifier outputs have been computed, and group statistics have been materialized. A trainable complete group is a complete group that has been converted into the GRPO training format, including advantages and masks. If the global sample-level batch size is $B$, then one logical GRPO update consumes

$U = \frac{B}{K}$

complete groups. In the reported setup, $R=96, K=8$0, $R=96, K=8$1, and hence $R=96, K=8$2 groups per logical update (Chen et al., 25 Jun 2026).

A recurrent misconception is that RolloutPipe is an asynchronous or partially off-policy system because it starts training before rollout completion. The framework is formulated to avoid exactly that outcome. All groups in a rollout round are generated by the same fixed rollout weights, the optimizer step and weight publication occur only after all $R=96, K=8$3 groups in the round have been consumed by the trainer, and there is no policy update between generation and training of any group within the round. The scheduling changes concern handoff timing and admission order, not sample provenance or update boundaries (Chen et al., 25 Jun 2026).

3. Complete-group pipelining

CGP is RolloutPipe’s training-side mechanism. Its purpose is to remove the unnecessary dependency between “last group materialized” and “first training update started.” As soon as a group is materialized on the control side, it is appended FIFO to the Pending Complete Groups queue. Once at least $R=96, K=8$4 trainable groups are available, the earliest $R=96, K=8$5 groups are dispatched to the trainer as a logical update batch (Chen et al., 25 Jun 2026).

If $R=96, K=8$6 denotes the earliest time at which the first $R=96, K=8$7 groups have all finished materialization, and $R=96, K=8$8 denotes the materialization time of the last group in the rollout, then Slime begins training at

$R=96, K=8$9

whereas CGP begins training at

$K$0

The recovered overlap window is

$K$1

This quantity is the headroom that would otherwise be lost to trainer idleness (Chen et al., 25 Jun 2026).

RolloutPipe also defines a trainer waiting ratio,

$K$2

which decreases when CGP advances the first dispatch time. A second trainer-side definition used in plots is

$K$3

Both emphasize the same systems effect: training work is unchanged, but idle time before and between updates is reduced (Chen et al., 25 Jun 2026).

The control path includes a Feasible Batch Selector that chooses the largest prefix of pending groups whose total token count fits the per-GPU token budget while always respecting group boundaries. If a single group is too large, it is taken alone and handled by dynamic micro-batching. This detail is central to maintaining the complete-group invariant in variable-length GRPO workloads (Chen et al., 25 Jun 2026).

4. Frontier-group dispatch

CGP alone advances training start, but it does not control the order in which groups become ready. In the default rollout engine, request-level FIFO is group-agnostic, so the groups needed for the next logical update may finish with large and irregular gaps. FGD is the rollout-side admission policy that regularizes this supply (Chen et al., 25 Jun 2026).

Let $K$4 be the set of groups that have arrived but are not yet serving-complete, and let $K$5 denote submission order. FGD maintains a frontier set $K$6 of at most $K$7 groups,

$K$8

that is, the $K$9 lowest-order groups among those not yet completed. A request $r_1,\dots,r_K$0 is admitted iff

$r_1,\dots,r_K$1

All other requests are deferred until frontier capacity opens (Chen et al., 25 Jun 2026).

In the reported experiments, $r_1,\dots,r_K$2. This aligns the rollout frontier with the trainer’s logical batch requirement: exactly the number of groups needed for one GRPO update are privileged at any moment. When a frontier group completes all $r_1,\dots,r_K$3 responses, it is removed from $r_1,\dots,r_K$4, the next lowest-order deferred group is admitted, and rollout capacity remains concentrated on the next trainable batch rather than spread over many partially completed groups (Chen et al., 25 Jun 2026).

Mechanism	Locus	Primary effect
CGP	Control/training side	Starts training once $r_1,\dots,r_K$5 groups materialize
FGD	Rollout side	Makes frontier groups arrive earlier and more steadily

This organization also clarifies a second misconception: FGD is not a change to the RL objective or GRPO statistics. It is strictly an admission policy over rollout requests. The groupwise reward normalization, logical update size, and weight-publication schedule remain those of the synchronous baseline (Chen et al., 25 Jun 2026).

5. System architecture and implementation

RolloutPipe is implemented on top of Slime, Megatron-LM, SGLang, and Ray, and is deployed across three node roles: rollout, control, and training. The rollout node maintains the Prompt Group Buffer, applies FGD, and passes admitted requests to the SGLang serving engine, whose Token Scheduler and Decode Workers execute generation. When all $r_1,\dots,r_K$6 requests of a group complete, the rollout node marks the group serving-complete and notifies the control node (Chen et al., 25 Jun 2026).

The control node performs group materialization. It collects the $r_1,\dots,r_K$7 responses, computes rewards and verifier outputs, converts samples into trainable records, computes $r_1,\dots,r_K$8 and $r_1,\dots,r_K$9, attaches advantages and loss masks, and appends the result to Pending Complete Groups. It then invokes the Feasible Batch Selector and forwards the selected prefix to the trainer’s U-group Ready Queue (Chen et al., 25 Jun 2026).

The training node consumes groups from that queue, executes forward and backward passes, and counts groups through a Gradient Accumulator. Every $$A_i = \frac{r_i - \mu_{\text{group}}}{\sigma_{\text{group}} + \epsilon_{\mathrm{std}}},$$0 groups constitute one logical GRPO update. Once all $$A_i = \frac{r_i - \mu_{\text{group}}}{\sigma_{\text{group}} + \epsilon_{\mathrm{std}}},$$1 groups from the current rollout have been consumed, the Weight Publisher exports refreshed policy weights to SGLang for the next rollout round (Chen et al., 25 Jun 2026).

Node	Principal functions	Key queues or buffers
Rollout	FGD, serving, completion notification	Prompt Group Buffer, Deferred Groups
Control	Materialization, verifier, CGP handoff	Pending Complete Groups
Training	Pipelined train RPC, accumulation, publish	U-group Ready Queue

The paper states a control overhead of $$A_i = \frac{r_i - \mu_{\text{group}}}{\sigma_{\text{group}} + \epsilon_{\mathrm{std}}},$$2 for group tracking, admission, and handoff, and a trainer computation cost of $$A_i = \frac{r_i - \mu_{\text{group}}}{\sigma_{\text{group}} + \epsilon_{\mathrm{std}}},$$3 per rollout, identical to Slime because the same number of samples are processed. Communication overhead is linear in $$A_i = \frac{r_i - \mu_{\text{group}}}{\sigma_{\text{group}} + \epsilon_{\mathrm{std}}},$$4 and is overlapped with rollout and training. Backpressure is handled by the pending FIFO and ready queue if the trainer lags; if rollout lags, the trainer eventually waits, but CGP and FGD are intended to minimize that condition (Chen et al., 25 Jun 2026).

6. Empirical behavior, comparative position, and limitations

Evaluation is reported on Qwen3-1.7B across four reasoning and science workloads: LSAT-AR from AGIEval, Sci-XW and Sci-JL from SciBench, and OlyPhys from OlympiadBench. The trainer uses $$A_i = \frac{r_i - \mu_{\text{group}}}{\sigma_{\text{group}} + \epsilon_{\mathrm{std}}},$$5 RTX 4090 24GB with $$A_i = \frac{r_i - \mu_{\text{group}}}{\sigma_{\text{group}} + \epsilon_{\mathrm{std}}},$$6, the rollout node uses $$A_i = \frac{r_i - \mu_{\text{group}}}{\sigma_{\text{group}} + \epsilon_{\mathrm{std}}},$$7 A100 40GB PCIe with $$A_i = \frac{r_i - \mu_{\text{group}}}{\sigma_{\text{group}} + \epsilon_{\mathrm{std}}},$$8, rollout sizes are $$A_i = \frac{r_i - \mu_{\text{group}}}{\sigma_{\text{group}} + \epsilon_{\mathrm{std}}},$$9, $$\mu_{\text{group}} = \frac{1}{K}\sum_{j=1}^{K} r_j, \qquad \sigma_{\text{group}} = \sqrt{\frac{1}{K}\sum_{j=1}^{K}(r_j-\mu_{\text{group}})^2}.$$0, $$\mu_{\text{group}} = \frac{1}{K}\sum_{j=1}^{K} r_j, \qquad \sigma_{\text{group}} = \sqrt{\frac{1}{K}\sum_{j=1}^{K}(r_j-\mu_{\text{group}})^2}.$$1, $$\mu_{\text{group}} = \frac{1}{K}\sum_{j=1}^{K} r_j, \qquad \sigma_{\text{group}} = \sqrt{\frac{1}{K}\sum_{j=1}^{K}(r_j-\mu_{\text{group}})^2}.$$2, and $$\mu_{\text{group}} = \frac{1}{K}\sum_{j=1}^{K} r_j, \qquad \sigma_{\text{group}} = \sqrt{\frac{1}{K}\sum_{j=1}^{K}(r_j-\mu_{\text{group}})^2}.$$3. Reported results are averaged over four rounds (Chen et al., 25 Jun 2026).

Across all twelve settings, RolloutPipe shortens rollout-to-train-end time by 30.7–42.3% relative to Slime and lowers the trainer waiting ratio by 37–76%. The benefit grows with rollout size: approximately 30.7–32.9% at $$\mu_{\text{group}} = \frac{1}{K}\sum_{j=1}^{K} r_j, \qquad \sigma_{\text{group}} = \sqrt{\frac{1}{K}\sum_{j=1}^{K}(r_j-\mu_{\text{group}})^2}.$$4 and 39.8–42.3% at $$\mu_{\text{group}} = \frac{1}{K}\sum_{j=1}^{K} r_j, \qquad \sigma_{\text{group}} = \sqrt{\frac{1}{K}\sum_{j=1}^{K}(r_j-\mu_{\text{group}})^2}.$$5. CGP accounts for 71–96% of the total speedup, while FGD contributes an additional 2.5–11.4% over CGP alone (Chen et al., 25 Jun 2026).

Dispatch timing illustrates the scheduling effect sharply. At $$\mu_{\text{group}} = \frac{1}{K}\sum_{j=1}^{K} r_j, \qquad \sigma_{\text{group}} = \sqrt{\frac{1}{K}\sum_{j=1}^{K}(r_j-\mu_{\text{group}})^2}.$$6, the first logical batch is dispatched at 509–543 s under Slime, at 46–90 s under CGP, and at 52–61 s under CGP+FGD. The narrowing under FGD is significant because it indicates not only an earlier first update but a steadier arrival process for subsequent updates. The trainer-side waiting ratio falls from 47–52% in Slime to 14–33% under CGP+FGD, with approximately 13.8–15.0% reported for the $$\mu_{\text{group}} = \frac{1}{K}\sum_{j=1}^{K} r_j, \qquad \sigma_{\text{group}} = \sqrt{\frac{1}{K}\sum_{j=1}^{K}(r_j-\mu_{\text{group}})^2}.$$7 workloads (Chen et al., 25 Jun 2026).

A further misconception is that these gains arise from reducing training work or shortening responses. The evaluation explicitly reports near-equality of trainer compute time and response lengths across Slime, CGP, and CGP+FGD. For example, on LSAT-AR with $$\mu_{\text{group}} = \frac{1}{K}\sum_{j=1}^{K} r_j, \qquad \sigma_{\text{group}} = \sqrt{\frac{1}{K}\sum_{j=1}^{K}(r_j-\mu_{\text{group}})^2}.$$8, trainer compute time is 587.4 s for Slime, 589.0 s for CGP, and 586.4 s for CGP+FGD. The optimization is therefore a scheduling optimization, not a change in computational load or objective function (Chen et al., 25 Jun 2026).

In relation to adjacent systems, RolloutPipe is closest to synchronous disaggregated RLVR frameworks such as Slime and is explicitly contrasted with asynchronous systems such as AReaL and AsyncFlow, which improve utilization through stale data. A nearby but distinct synchronous strategy is RollPacker, which mitigates GPU bubbles caused by long-tail response lengths through tail batching, dynamic resource allocation for reward, and stream-based training while preserving synchronous semantics; its gains are driven by response-length imbalance rather than complete-group scheduling (Gao et al., 25 Sep 2025). This suggests that RolloutPipe and tail-batching systems address different synchronization bottlenecks within on-policy LLM RL.

The reported limitations are also specific. Experiments are confined to Qwen3-1.7B and a modest cluster; the design assumes GRPO-style fixed-size groups with a well-defined materialization step; rounds still use a single fixed weight snapshot; and FGD uses a static low-order frontier policy with $$\mu_{\text{group}} = \frac{1}{K}\sum_{j=1}^{K} r_j, \qquad \sigma_{\text{group}} = \sqrt{\frac{1}{K}\sum_{j=1}^{K}(r_j-\mu_{\text{group}})^2}.$$9 rather than an adaptive scheduler. Suggested future directions include extending the pipelining ideas to other cognitive computing systems, including multimodal RAG and multi-agent collaborative settings where on-policy constraints remain important (Chen et al., 25 Jun 2026).