---
title: Two-phase RL Pipelines Overview
url: https://www.emergentmind.com/topics/two-phase-rl-pipelines
type: topic
---

# Two-phase RL Pipelines Overview

A two-phase reinforcement learning (RL) pipeline refers to any RL system in which the training loop is structurally divided into two distinct, sequentially executed phases, most canonically a rollout (data generation) phase and a training (policy update) phase. This pattern has become the de facto standard for RL in large-scale language models, multitool-augmented vision-language settings, interpretable agents, and hybrid RL/control applications. Two-phase architectures enable explicit resource specialization, modularity, and improved scaling but also introduce new efficiency challenges. The following sections synthesize the recent research landscape, rigorously characterizing two-phase RL pipeline structure, scheduling, optimization, and empirical performance.

## 1. Architectural Definition and Motivation

The canonical two-phase RL pipeline alternates between:

- **Rollout phase:** The acting policy (e.g., an LLM, agent, or actor network) interacts with the environment, generating trajectories under the current policy (often called "on-policy" rollouts). For LLM-based RL, this corresponds to token-stream generation; for control, it can mean state-action sequence sampling.
- **Training phase:** The collected rollouts are processed—scored by a reward model, compared to references, or evaluated by value/advantage estimators—and then used to compute policy or value updates. This phase is compute-intensive, often requiring significant memory and optimized parallelism.

The two-phase split is architecturally motivated by the heterogeneous computational demands of inference (rollout) versus training (backpropagation), as well as the need to maintain modularity and scalability across large clusters or multi-agent systems. Strict synchronization between phases, however, creates dependency "bubbles"—periods when some resources idle while awaiting phase completion in other components [2512.11306, 2504.15930].

## 2. Disaggregation and Scheduling: Overcoming Dependency Bubbles

Disaggregated architectures physically separate the clusters or hardware pools used for rollout (typically inference-optimized GPUs) and for training (compute-optimized GPUs). This maximizes theoretical hardware efficiency but exposes the pipeline to inter-phase dependency bubbles caused by strict on-policy synchronization.

Recent frameworks (e.g., RollMux [2512.11306], StreamRL [2504.15930], SeamlessFlow [2508.11553]) introduce scheduling, spatiotemporal multiplexing, and asynchrony at the phase and job level to reclaim idle resources:

- **Co-execution group abstraction:** Jobs are grouped, statically assigned to fixed nodes, and their phases executed in a round-robin schedule. Group residency constraints ensure that massive model states remain in host DRAM for rapid context switching [2512.11306].
- **Two-tier scheduling:** Inter-group schedulers perform worst-case stochastic planning for job placement, while intra-group schedulers run a provably optimal cyclic schedule [2512.11306].
- **Streaming and asynchronous generation:** Dynamic mini-batch pipelining and overlapped generation/training eliminate pipeline bubbles; a length ranker and skewness-aware dispatching address heavy-tailed trajectory runtimes [2504.15930].
- **Tag scheduling and capability abstraction:** Resources are dynamically tagged according to capability (rollout, training, etc.) and retagged to ensure maximal utilization, with fine-grained pause/resume implemented via a central trajectory/data plane [2508.11553].

These approaches collectively enable near-100% hardware utilization and strict service-level objective (SLO) guarantees in production-scale clusters.

## 3. RL Algorithmic Structure in Two-Phase Pipelines

Within each phase, the RL algorithm is tightly coupled to pipeline control:

- **Rollout phase:** Typically, the latest policy is used to generate a predetermined or adaptive batch of trajectories. In LLM settings, memory bandwidth or batch size limits are critical; in robotics or vision tasks, tool-augmented interaction or environment resetting dominates [2512.04069, 2506.00555].
- **Training phase:** Policy or value updates are computed, most often via policy-gradient methods (PPO, GRPO, DAPO), with critic and reward models running on collected rollouts. For on-policy RL, the rollout and training phases must use matched models and often matched numerical precision [2601.14243].
- **Precision flow synchronization:** Errors introduced by mismatched rollout/training precisions (e.g., FP8 rollout with BF16 training) break the on-policy assumption and can destabilize optimization; unified precision propagation through both phases is necessary for stable large-scale training [2601.14243].

For multi-agent or multi-stage reasoning, the two-phase structure may appear in either the interaction protocol (e.g., triage specialist pipeline [2506.00555]) or within recurrent alternations (parallel thinking in competitive programming [2604.01302]).

## 4. Specializations and Pipeline Innovations

Recent works extend the classical two-phase architecture with domain-specific adaptations:

- **Parallel/Multithreaded Rollouts:** In competitive programming, the "parallel thinking" pipeline combines multiple concurrent and sequential solution attempts, verification, and refinement, with aggregate token budgets as high as 7.6M per problem [2604.01302].
- **Tool-Augmented Phases:** For multimodal/vision-language spatial reasoning, a two-phase double-interactive RL pipeline combines a supervised/curriculum "teaching" phase and an interactive RL "exploration" phase, enabling efficient tool coordination [2512.04069].
- **Interpretable Pipelines:** Two-phase evolutionarily optimized, glass-box RL pipelines for vision tasks use a feature extraction phase (interpretable kernel convolution) followed by a reasoning phase (decision tree), co-evolved for performance and interpretability [2202.04943].
- **Hybrid RL+MPC Pipelines:** In control, an offline robust, goal-conditioned RL value function is learned in Phase 1, then deployed as a terminal cost in online scenario-based MPC in Phase 2, effectively combining RL's exploration power with MPC's constraint satisfaction [2502.06996].
- **RL-then-SFT and Cooperative SFT-RL:** In multimodal reasoning and large language model training, two-phase paradigms also structure learning itself—either as explicit RL followed by expert-assisted SFT enhancement (Metis-RISE [2506.13056]), or as strictly decoupled SFT→RL (with known forgetting/exploration limitations), or as joint bilevel cooperative optimization (BRIDGE [2509.06948]).

## 5. Empirical Results and Efficiency Trade-Offs

Empirical evaluations consistently demonstrate the necessity and impact of two-phase pipeline structure, as well as the superiority of recent scheduling and architectural innovations:

| System        | Idle Elimination/Utilization                | Cost Efficiency        | Notable Gains and Benchmarks                |
|---------------|--------------------------------------------|-----------------------|---------------------------------------------|
| RollMux [2512.11306]      | >2x reduction in idle/bubble time, 100% SLO    | 1.84x over naive disagg, 1.38x over co-loc   | H800 usage down 2.16x, jobs 3B–32B, 4k–32k tokens |
| StreamRL [2504.15930]     | Utilization up to 90+%           | 1.31–1.33x in cross-DC | 1.30–2.66x throughput vs. SoTA, large clusters   |
| SeamlessFlow [2508.11553] | Empirical GPU utilization >95%    | N/A                  | 2x sample/sec vs. VERL, near-linear scaling      |
| Jet-RL [2601.14243]       | Stability at ultra-long rollout, no collapse | 16% end-to-end speedup | Less than 3% acc. loss, up to 1.8x inference/training |
| DIRL [2512.04069]         | N/A                              | N/A                  | +12–16% absolute gains, SOTA spatial reasoning  |
| Metis-RISE [2506.13056]   | N/A                              | N/A                  | +4.8% avg (RL only) +2.4% (SFT) vs. strong SFT baseline |
| BRIDGE [2509.06948]       | 44% faster vs. classic SFT→RL    | N/A                  | +13% avg. accuracy over cold-start                |

Across these systems, best practices include: fine-grained scheduling and residency enforcement, conservative admission control, warm-start context switching, topology-aware model sync, and round-robin time-multiplexed scheduling within small execution groups.

## 6. Interpretability, Generalization, and Limitations

Two-phase pipelines admit both black-box and glass-box realizations. The interpretable pipeline by Virga et al. [2202.04943] demonstrates that glass-box, two-phase RL architectures (convolutional high-level feature extractors plus decision trees) can match deep network performance in deterministic Atari, though performance degrades under stochasticity. In reasoning LLMs, explicit two-phase RL–SFT or bilevel SFT–RL approaches address known exploration and catastrophic forgetting problems, though full sample efficiency and generalization across domains or tools remain active research areas.

Known limitations include:

- Sensitivity to straggler/long-tail trajectory completion in rollout;
- Increased scheduling and cluster provisioning complexity;
- The need for large DRAM capacities for state residency;
- Stability issues when rollout and training graphs diverge in numerical precision or architectural detail, particularly at extreme sequence lengths or model sizes.

## 7. Outlook and Best Practices

As two-phase RL pipelines are increasingly deployed at scale, best practices are converging:

1. Partition clusters into small co-execution groups to ensure state residency.
2. Employ conservative job admission and worst-case phase duration estimates.
3. Use provably optimal intra-group schedules (e.g., round-robin) unless additional job interaction models warrant adaptation.
4. Exploit structural dependency bubbles for multiplexing and apply hierarchical, topology-aware model sync.
5. Monitor and relabel straggler and tail jobs for batch migration, maintaining strict SLO adherence.
6. For glass-box or interpretable settings, evolve modular phase-specific components (feature extractors, decision logic) with joint fitness evaluation [2202.04943].

This two-phase paradigm, in varied instantiations, extends beyond LLM post-training to tool-augmented spatial VLMs, curriculum-based multi-agent medical agents, hybrid RL/control systems, and interpretable agents. Its efficiency, modularity, and extensibility underpin much of the recent empirical progress in large-scale, high-reliability RL deployments.

Source: https://www.emergentmind.com/topics/two-phase-rl-pipelines