Dynamic ORchestration for Asynchronous Rollout

• Dynamic ORchestration for Asynchronous Rollout (DORA) is an advanced framework that decouples rollout, reward computation, and training in reinforcement learning using dedicated resource pools.
• It leverages micro-batching, versioned policy updates, and dynamic load balancing to minimize idle time and maintain strict on-policy consistency.
• Empirical studies show DORA achieves 2–7× throughput gains and near-zero device idleness compared to traditional synchronous RL systems.

Dynamic ORchestration for Asynchronous Rollout (DORA) is an advanced coordination framework that enables efficient, scalable, and consistent rollout management in large-scale reinforcement learning (RL) systems, particularly those involving long-context or multi-agent tasks. DORA’s fundamental objective is to resolve the tension between high-throughput, low-latency rollout execution and the strict consistency guarantees required for stable RL convergence. It accomplishes this by orchestrating asynchronous rollout, experience gathering, and training phases through novel architectural and algorithmic innovations.

1. Architectural Principles and System Decomposition

DORA’s architecture instantiates a fully disaggregated RL workflow, wherein the rollout (data sampling), reward computation, and training (policy optimization) phases execute on physically or logically separated resource pools. In contrast to traditional colocated models in which rollout and training contend for the same accelerators, DORA-equipped systems:

• Deploy dedicated inference clusters (for parallel rollout generation) and training clusters (for model updates), bridged by an orchestration control plane (Jiang et al., 10 Feb 2026, Team et al., 23 Sep 2025, Li et al., 19 Jan 2026).
• Insert specialized middleware servers—trajectory servers (TS) and parameter servers (PS)—to mediate streaming data and parameter versioning across the workflow (Li et al., 19 Jan 2026).

Key orchestration components include:

Component	Role	Reference
Experience Store	Relational or buffer-based sample repository per agent/domain	(Jiang et al., 10 Feb 2026, Team et al., 23 Sep 2025)
Asynchronous Dispatcher	Micro-batch allocation and distributed gradient scheduling	(Jiang et al., 10 Feb 2026)
Staleness Manager	Policy version bounding and consistency enforcement	(Li et al., 19 Jan 2026, Team et al., 23 Sep 2025)
Load Balancing/Migration	Dynamic instance/workload redistribution	(Jiang et al., 10 Feb 2026, Li et al., 19 Jan 2026)

This demultiplexed design enables independent scaling and elastic resource (re)allocation, while providing mechanisms for policy version control, consistency guarantees, and long-tail latency hiding.

2. Core Algorithmics: Asynchronous Micro-Batching and Versioned Consistency

DORA achieves fine-grained asynchrony through micro-batch–driven pipelines and version-tracking:

• Micro-Batch Orchestration: Rather than waiting for all samples in a global batch to complete (as in synchronous RL), the asynchronous dispatcher accumulates fixed-size micro-batches per agent or domain, schedules their gradient computation in parallel, and caches gradients locally (Jiang et al., 10 Feb 2026).
• Global All-Reduce and Policy Update: When all micro-batches reach completion, model updates are synchronized (e.g., via all-reduce), policy versions incremented, and new weights broadcast atomically to all rollout entities (Jiang et al., 10 Feb 2026).
• Version Tagging and On-Policy Guarantee: Every experience is explicitly tagged with the policy version used for its generation, and DORA ensures that training batches only aggregate experiences with matching versions. No new rollouts for version $V+1$ are permitted until all data and updates for version $V$ have been committed, resulting in strict on-policy RL semantics (Jiang et al., 10 Feb 2026, Team et al., 23 Sep 2025).

This protocol removes global barriers, overlaps rollout and training, and eliminates device idleness associated with traditional blocking.

3. Dynamic Load Balancing, Scheduling, and Migration

Handling workload skew and long-tail latency is central to DORA’s effectiveness. Its dynamic scheduling primitives—refined in multi-agent and single-agent contexts—are typified by:

• Hierarchical Load Balancing: Across agents, an optimization problem is solved to minimize the maximum queueing time by adaptively assigning rollout/inference slots, subject to global GPU/NPU budget and per-agent load (Jiang et al., 10 Feb 2026). Within each agent, requests are greedily assigned to the least-loaded inference instance (local min-heap approach).
• Trajectory-Centric Scheduling: In step-intensive RL with heterogeneously long trajectories, progressive priority or longest-processing-time-first schemes are employed. A runtime predictor assesses remaining trajectory length; long-tail trajectories are prioritized to reduce makespan and queueing delay (Zhang et al., 30 Mar 2026).
• State Migration: In cases of load skew or resource contention, live migration of in-progress trajectories—including KV cache—is performed over high-speed links (RDMA), allowing mid-rollout transfer without re-prefilling (Zhang et al., 30 Mar 2026).

Algorithmic strategies ensure that system throughput is governed by the mean, not the maximum, per-sample generation time, yielding empirical speedups well over 2–3x versus synchronous or naively pipelined systems (Team et al., 23 Sep 2025, Li et al., 19 Jan 2026, Zhang et al., 30 Mar 2026).

4. Staleness Control and Global Consistency Protocols

DORA explicitly controls policy staleness and data consistency through a suite of mechanisms:

• Staleness Bound Protocols: Each rollout sample is tagged with the model version used (V_traj), and the system prohibits the staleness gap (V_buf − V_traj) from exceeding a coordinator-enforced maximum η (Li et al., 19 Jan 2026, Team et al., 23 Sep 2025). Virtual buffer slots are reserved, occupied, and consumed on this basis, ensuring that asynchronous execution does not compromise RL convergence.
• Routing, Migration, and Synchronization:
  • Staleness-aware routes direct trajectories to compatible accelerators.
  • Synchronization decisions trigger policy pulls/updates on lagging workers only when routing cannot resolve staleness.
  • Proactive migration and queue throttling maintain system balance under skewed trajectory distributions. Experimental ablations confirm that DORA’s combination of these tactics is essential for joint staleness-skewness mitigation (Li et al., 19 Jan 2026).

Feedback from convergence and throughput metrics guides dynamic adaptation, allowing the full Slack (η) to be exploited without divergence.

5. End-to-End System Integration and Domain Parallelism

DORA is integrated in large-scale RL systems such as FlexMARL (Jiang et al., 10 Feb 2026), LongCat-Flash-Thinking (Team et al., 23 Sep 2025), and StaleFlow (Li et al., 19 Jan 2026). Distinctive system-level features include:

• Elastic Grouping: Pools of devices switch roles (e.g., generator, reference actor/critic, trainer) on demand, driving near-zero idle time (Team et al., 23 Sep 2025).
• Domain-Parallel Training: For models targeting multiple domains (e.g., STEM, coding, agentic reasoning), independent RL streams proceed in parallel, sharing execution clusters but managing domain-specific queues, rewards, and curriculum (Team et al., 23 Sep 2025).
• Policy Synchronization: Layer-wise or model-wide weight broadcasts, coupled with group-key management and compression for cluster scalability.
• Experimental Evaluation: Scaling tests on up to 20,000+ GPUs confirm DORA-enabled systems achieve linear throughput scaling up to hardware and network bounds, with empirical 2–3.2x improvements in samples/sec and marked reductions in device idle fractions (Team et al., 23 Sep 2025, Jiang et al., 10 Feb 2026, Li et al., 19 Jan 2026).

6. Quantitative Performance and Trade-Offs

Multiple empirical studies illustrate DORA’s impact:

System	Relative Throughput Gain	Hardware Utilization Gain	Additional Effects	Reference
FlexMARL (DORA)	up to 7.3× (vs MAS-RL)	up to 2× (vs MARTI)	Preserves strict on-policy RL	(Jiang et al., 10 Feb 2026)
LongCat-Flash	3.2× (vs synchronous)	↑ from 70% to 99% (train)	<0.5% idle, linear scaling	(Team et al., 23 Sep 2025)
StaleFlow (DORA)	up to 2.68× (vs VeRL)	Maintains convergence	Ablations: each strategy 10–20%+	(Li et al., 19 Jan 2026)
Heddle-DORA	up to 2.5× (vs SOTA)	Queue delay ↓ 40%	1.1–1.3× via adaptive resources	(Zhang et al., 30 Mar 2026)

Ablation studies consistently demonstrate sensitivity to both staleness bounds and load balancing/micro-batching strategy. Too loose staleness (high η) precipitates RL divergence; omitting load adaptation collapses throughput (Li et al., 19 Jan 2026, Jiang et al., 10 Feb 2026). These results underscore the need for careful joint algorithmic design.

7. Extensions, Limitations, and Generality

DORA’s framework is agnostic to model types (dense, MoE), RL setting (single-agent, multi-agent, self-play), and supported inference accelerations (quantization, speculative decoding). It can extend beyond text models, to robotics or game AI, wherever asynchronous RL pipeline bottlenecks exist (Team et al., 23 Sep 2025).

Limitations include non-trivial orchestration layer complexity, heightened memory management overhead for KV-cache under extreme skew, and the need for careful staleness parameter tuning to avoid off-policy drift (Team et al., 23 Sep 2025, Li et al., 19 Jan 2026).

Future research directions include cloud-agnostic deployments leveraging elastic and preemptible resources, tighter integration with pretraining workflows, and generalized support for multi-modal and open-ended RL tasks (Team et al., 23 Sep 2025, Li et al., 19 Jan 2026, Zhang et al., 30 Mar 2026).

---

References:

(Jiang et al., 10 Feb 2026, Zhang et al., 30 Mar 2026, Team et al., 23 Sep 2025, Li et al., 19 Jan 2026)