RolloutPipe: Overlapping Pipelined Rollout and Training in Disaggregated On-Policy LLM Reinforcement Learning

Abstract: LLM post-training for reasoning increasingly relies on reinforcement learning with verifiable rewards (RLVR), where models learn from ground-truth feedback on mathematical, logical, and scientific tasks. To enable flexible resource allocation and support heterogeneous training setups, modern RLVR systems adopt disaggregated architectures that decouple rollout generation and policy training across independent GPU pools. However, existing synchronous on-policy GRPO (Group Relative Policy Optimization) RLVR systems finish an entire rollout before starting training, leaving the trainer GPU pool idle while rollout is still ongoing. Asynchronous RL pipelines overlap the two stages, but at the cost of training on stale data. To address these challenges, we propose RolloutPipe, a post-training framework for disaggregated RLVR systems, which turns the fixed-weight rollout into a complete-group pipeline where trainable groups move to the trainer while later groups are still being generated. RolloutPipe achieves this through two techniques including complete-group pipelining (CGP) and frontier-group dispatch (FGD). CGP dispatches each trainable complete group to the trainer FIFO as soon as group materialization finishes, and FGD is an admission policy on the Rollout node that first admits requests for the frontier groups needed to form the next training batch, so that trainer-ready groups arrive earlier and more steadily. The design starts training before the rollout completes while maintaining on-policy correctness. Evaluated on Qwen3-1.7B across four reasoning and science benchmarks and twelve rollout settings, RolloutPipe shortens the rollout-to-train-end time by 30.7%-42.3%, and lowers the trainer waiting ratio by 37%-76% compared to Slime, a state-of-the-art rollout and training system.

• The paper introduces overlapping pipelined rollout and training strategies (CGP and FGD) that cut trainer idle time by up to 42.3%.
• It employs complete-group pipelining to hand off groups immediately upon completion, improving throughput in disaggregated RL systems.
• Empirical results with Qwen3-1.7B demonstrate significant reductions in waiting ratios while preserving on-policy update correctness and computation integrity.

RolloutPipe: Pipelined Rollout and Training for Disaggregated On-Policy LLM RLVR

Motivation and Architectural Limitations

The post-training phase for LLMs, particularly for reasoning-centric tasks, increasingly utilizes RLVR (Reinforcement Learning with Verifiable Rewards) due to its reliance on objective, ground-truth feedback [deepseekr1]. Modern RLVR systems leverage disaggregated architectures for flexible allocation of rollout and training resources (distinct GPU pools). However, serial execution dominates current frameworks such as Slime: rollout completes for all prompt groups before training commences, causing acute trainer GPU idle time as early-completed groups are blocked from entering training (Figure 1).

Figure 1: Slime's architecture and serial rollout-then-train execution, highlighting trainable groups waiting outside the trainer FIFO and leaving the trainer idle.

This inefficiency is magnified in GRPO (Group Relative Policy Optimization), where advantages are normalized across complete groups rather than single samples. Thus, training cannot start at any sample's completion but must wait for group materialization—computation of mean and standard deviation rewards across $K$ responses.

The second inefficiency arises from naive FIFO rollout scheduling (oblivious to group boundaries), causing unordered group completion and staggered arrival of frontier groups—those needed for batching the next trainer update. This further expands the trainer idle window.

RolloutPipe Design: Complete-Group Pipelining and Frontier-Group Dispatch

RolloutPipe fundamentally shifts the synchronization granularity from rollout-round level to complete-group level, exploiting two scheduling mechanisms:

• Complete-Group Pipelining (CGP): Trainable groups are delivered to the trainer FIFO immediately upon materialization, so training can begin before full rollout completion. The trainer-side logic accumulates $U = B/K$ groups, after which the training update proceeds.
• Frontier-Group Dispatch (FGD): The Rollout node maintains a frontier set $\mathcal{F}$ of up to $F_w = U$ groups. Only requests belonging to these frontier groups are admitted into the generation engine, focusing GPU resources and ensuring that groups forming the next training batch complete earliest and arrive steadily.

  Figure 2: RolloutPipe system architecture with insertion points for FGD (admission scheduling) and CGP (complete-group handoff).

  

  Figure 3: RolloutPipe timeline comparison. CGP delivers $U$ groups immediately upon materialization; CGP+FGD shifts earliest batch dispatch further forward by prioritizing frontier groups.

All groups in a round share fixed rollout weights, preserving on-policy update semantics. Optimizer step and weight publishing boundaries strictly follow logical batch consumption ($U$ groups at a time), not sample-level readiness.

Algorithmic Flow and System Implementation

RolloutPipe realizes scheduling policies across three event-driven nodes:

• Rollout Node: Implements frontier admission and generation. Deferred groups are held until a frontier slot opens; admitted requests proceed through SGLang scheduling and decoding.
• Control Node: Upon group completion, executes reward verification and materialization, then enqueues the trainable group for batch selection and handoff.
• Training Node: Consumes ready batches (each of $U$ groups), performs forward/backward passes and gradient accumulation, triggers optimizer step, and finally publishes refreshed weights.

The control/space/communication complexity overhead is linear in $R \cdot K$; per-rollout compute remains identical to serial baselines.

Empirical Evaluation

Experiments use Qwen3-1.7B across four benchmarks: LSAT-AR, Sci-XW, Sci-JL, and OlyPhys, spanning reasoning and science domains [agieval, scibench, olympiadbench]. Training and rollout GPU pools are separately provisioned. Primary metrics are rollout-to-train-end time (main window), trainer waiting ratio, response length, and trainer compute time.

CGP+FGD consistently shortens end-to-end training time relative to Slime by 30.7%–42.3% across workloads and group counts (Figure 4). The impact grows with the rollout size ($R$), as more early-completed groups are trapped behind the serial barrier in Slime. Notably, CGP accounts for 71%–96% of the speedup, with FGD contributing steady supply that further reduces trainer idle periods by 2.5%–11.4%.

Figure 4: Rollout-to-train-end time. CGP+FGD reduces Slime's wall-clock by up to 42.3%.

The reduction in trainer waiting ratio is significant: Slime's idle ratio (47%–52%) drops to 14%–33% in CGP+FGD configurations (Figure 5). Critically, raw trainer compute time and response length remain consistent, confirming that pipelining shifts existing workload placement rather than decreasing computational effort.

Figure 5: Mean response length and trainer waiting ratio across workloads and group counts. CGP+FGD achieves lower idle ratio with equivalent workload.

Practical and Theoretical Implications

RolloutPipe's pipelining is a system-level intervention for synchronous on-policy RLVR with disaggregated resources. Its mechanisms preserve on-policy correctness—groups are generated under fixed weights, and optimizer steps strictly follow logical batch consumption—unlike asynchronous pipelines that risk stale updates [areal, asyncflow].

The practical implication is a direct reduction in hardware cost for RLVR post-training, especially as model, group, and rollout scales grow. For real-time, cognitively-inspired reasoning tasks, the minimized trainer latency directly enables faster iteration and more aggressive fine-tuning.

Theoretically, RolloutPipe exposes the latency bottleneck created by group-level dependencies in reward normalization (GRPO), highlighting the need for granularity-aware scheduling in cognitive computing systems. This paradigm can extend to other decoupled learning/generation architectures (e.g., multimodal RAG, multi-agent coordination), wherever logical training units cannot be processed at sample granularity.

Future developments may pursue more dynamic pipeline adaptation to varying group sizes, response lengths, and workloads, as well as further interaction with system-level token schedulers and memory optimization strategies.

Conclusion

RolloutPipe advances on-policy RLVR by overlapping rollout and training at the complete-group level, thereby reclaiming a substantial fraction of trainer idle time and pipeline stalls without sacrificing correctness or workload integrity. The dual scheduling of CGP and FGD transforms the throughput window and enables scalable post-training for large, resource-intensive LLMs, with direct implications for accelerating reasoning-driven fine-tuning in cognitive AI systems. Generalization to broader cognitive architectures and adaptive scheduling is a prospective avenue for future exploration.