Bidirectional Resource Scheduling for Disaggregated and Asynchronous RL Post-Training

Abstract: It is well established that the reasoning capabilities of LLMs can be improved by applying reinforcement learning (RL) in a post-training stage. In a standard RL iteration, the current model (the policy) generates experience through rollouts, and the resulting data is then used to update the policy during training. High-performance RL frameworks such as StreamRL and AReaL employ a disaggregated architecture and asynchronous rollouts to better exploit both rollout and training resources, thereby increasing overall system throughput. Nonetheless, across varying RL setups (e.g., hardware configurations, model scales, staleness levels, and hyperparameters) and under changing workloads, it remains common for both rollout and training resources to experience idle periods. In this paper, we present BiDiRL, a hybrid time-space multiplexing architecture for asynchronous, disaggregated RL designed to reduce resource idleness. First, we develop a hot-switch runtime that enables rapid switching between rollout and training resources with negligible overhead. Second, we propose a static, scheduling-aware planner based on time-performance modeling that chooses a hot-switch-friendly resource partition, so that rollout and training durations are roughly balanced at a coarse level. Third, at execution time, we introduce a bidirectional scheduler that further exploits runtime bubbles through fine-grained resource switching, allowing the bottleneck stage to temporarily borrow idle resources from the other pool. Across a wide range of workloads, datasets, and models on two 32-GPU testbeds, BiDiRL increases RL training throughput by up to 1.94x compared with RL systems including veRL, AReaL, and ROLL, without affecting convergence behavior.

• The paper introduces BiDiRL, a hybrid planner and runtime scheduler that hot-switches idle rollout and training resources to reclaim structural and execution-time bubbles in asynchronous RL post-training.
• BiDiRL combines compatibility-aware static partitioning with overhead-conscious bidirectional borrowing, achieving up to 1.94× speedup over strong baselines and 1.02×–1.31× over opportunistic borrowing across diverse hardware and workloads.
• The system preserves RL convergence while reducing idle time, although its gains depend on bubble size, switching overhead, workload statistics, and scalability beyond the evaluated 32-GPU clusters and 8B models.

Problem: residual idle bubbles in disaggregated RL

Reinforcement learning (RL) post-training of LLMs alternates between a rollout stage, in which the actor generates responses via an inference engine such as vLLM, and a training stage, in which policy losses under objectives like GRPO are computed and the actor is updated with engines such as PyTorch FSDP. Modern systems disaggregate these stages onto separate committed resource pools and use asynchronous, off-policy execution governed by a staleness bound $s$ to overlap them. The paper's central observation is that overlap alone does not eliminate idleness: even well-tuned disaggregated allocations leave resource bubbles at two timescales.

Structural bubbles are predictable before execution: rollout and training exhibit different scaling behavior with respect to response length and GPU count—rollout is more sensitive to long-tail response lengths and gains less from additional resources than training—so a fixed partition can create persistent rate mismatch. Residual bubbles arise during execution from response-length drift as the policy evolves, model-parallelism constraints that limit partition granularity, and staleness constraints that cap rollout run-ahead or force trainers to wait for fresh samples. Notably, these bubbles are two-sided: long-reasoning workloads leave trainers waiting; short-output multimodal workloads can leave rollouters idle; strict staleness bounds ($s=0$ or fractional $0

BiDiRL design

BiDiRL is a hybrid time-space multiplexing architecture built on veRL (8.3K lines of Python), using vLLM for rollout, FSDP for training, and Ray for orchestration. It keeps disaggregated committed pools (RollPoll and TrainPoll) but allows either pool to temporarily execute the other stage's role through three coordinated mechanisms.

Hot-switch runtime. Auxiliary workers are initialized passively at startup; when borrowing is admitted, primary workers are released and auxiliary workers wake, with a preempt-and-yield rule ensuring a pool never hosts both roles simultaneously. Replica-layout compatibility selected by the planner guarantees that hot switching requires no process restarts or resharding. Measured switch costs are modest but non-negligible—for Qwen3VL-4B, switch-in/switch-out costs range roughly 3.4–7.7 seconds depending on direction, plus 0.66–1.21 seconds for gradient synchronization when Trainer-on-RollPoll covers an actor update.

Scheduling-aware static planner. Before execution, the planner selects minimal feasible replica layouts and enumerates legal partitions $(g_r, g_t)$ subject to a hot-switch compatibility constraint: each pool must be divisible into an integer number of replicas for every role it may host, and primary/auxiliary workers of a stage must share layouts so weights transfer without resharding. Among compatible partitions it maximizes the steady-state pipeline rate $$\lambda_{\mathrm{pipe}} = \min(\lambda_r, \lambda_t)$$ using fitted stage-time models. The search is $O(|\mathcal{S}_\Pi|)$, i.e., linear in candidate partitions, and runs once.

Bidirectional scheduler. At runtime, an online profiler emits per-unit records that calibrate the stage-time models and reports idle windows, rollout deficits, remaining trainer workload, and measured switching costs. Each borrowing opportunity is an admission-control problem: borrowing proceeds only when the predicted benefit exceeds the measured overhead. For Rollouter-on-TrainPoll, admission requires $$\mathcal{M}_r(Q_r, d_p) - \mathcal{M}_r(Q_r, d_p + d_a) > C_{\mathrm{in}} + C_{\mathrm{out}}$$, after which the rollout deficit is dispatched proportionally to data-parallel capacity—an explicit choice to avoid relying on pre-generation length prediction. For Trainer-on-RollPoll, trainer work is decomposed into chunks executed through an asynchronous enqueue-run-dequeue pipeline with lazy pulling; a startup drain removes work primary trainers finish during auxiliary preparation, and a tail split searches over chunk assignments to minimize the larger predicted completion time of the two sides. Chunk state classes (queued, running, finished, cancelled) define recovery semantics: queued chunks return to the pending queue on lease revocation, running chunks complete, and outputs merge in original order, preserving sequential-trainer semantics.

Evaluation results

The evaluation spans two 32-GPU testbeds (A6000-48G and H100-80G nodes), models from 2B to 8B (Qwen3 family, including Qwen3VL), text (GSM8K) and multimodal (Geo3K) datasets, max response lengths of 1K–4K, staleness bounds 0–4 including fractional values, and budgets of 8–32 GPUs, against veRL v0.7.1, AReaL v1.0.3, and ROLL v0.2.1.

Setting	Speedup over strongest baseline
A6000 end-to-end sweeps	1.27×–1.68×
A6000 scale-up (32 GPUs, batch 128)	up to 1.94×
H100 default / scale-up	1.23×–1.47× / 1.05×–1.53×
H100 resource-partition studies	1.11×–1.41×

Ablations isolate the scheduling contribution: BiDiRL improves over no borrowing by 1.12×–1.71× and over opportunistic borrowing (borrow whenever the opposite pool is idle, without cost modeling or workload splitting) by 1.02×–1.31×. Both one-direction variants fall short of full bidirectional scheduling (up to 1.68× vs. w/o T-on-R), supporting the claim that residual bubbles are inherently two-sided. Static planning validation shows measured throughput varies by 1.40×–2.09× across machine-aligned partitions for the same budget, and the planner selects the measured-best partition in the displayed sweeps; the stage-time models achieve median errors of 3.12% (rollout) and 2.92% (trainer). Convergence is preserved: over the first 60 steps, reward gaps versus veRL are +0.017/+0.000 at staleness 1/2, consistent with the design claim that scheduling changes computation placement without altering the logical GRPO dataflow.

Two caveats qualify these numbers. First, relative speedups shrink on H100 because faster devices and larger batches shorten the residual windows available for harvesting—the benefit is contingent on bubble magnitude rather than universal. Second, the main end-to-end runs deliberately use node-aligned partitions matching the baselines for comparability, so the planner's own partition selection is validated separately rather than folded into headline speedups.

Limitations and open questions

The paper concedes several boundaries. Rollouter-on-TrainPoll dispatches prompts by capacity proportionally because final response lengths are unknown before generation; interrupted auxiliary windows therefore return partially generated groups to primary rollouters, and the authors note that prompt-feature or early-decoding signals could improve placement but leave this unimplemented. Trainer-on-RollPoll trades chunk efficiency for tail balance; the current design uses large lazy-pull chunks with smaller remainder chunks only at the tail, and the observation that auxiliary trainers hold no optimizer states suggests—but does not yet realize—more aggressive adaptive chunk sizing. The planner is explicitly not intended to predict absolute throughput accurately, only to rank partitions, and its accuracy depends on representative workload statistics gathered before execution. Finally, evaluation is limited to 32-GPU clusters and models up to 8B parameters; whether the admission-control economics (switching costs versus window lengths) remain favorable at substantially larger scale is not established by this paper.

Conclusion

BiDiRL frames idle resources in disaggregated LLM RL as a two-timescale scheduling problem and addresses both scales: a static planner selects a throughput-balanced, hot-switch-compatible resource envelope, while a model-guided bidirectional scheduler reclaims residual bubbles by lending idle pool resources to the bottleneck stage whenever predicted gain exceeds measured switching cost. Across diverse workloads, staleness settings, models, datasets, and hardware, the system delivers throughput improvements up to 1.94× over state-of-the-art frameworks while preserving convergence behavior, demonstrating that bidirectional, overhead-aware borrowing is a practical complement to asynchronous disaggregation.