Part II: ROLL Flash -- Accelerating RLVR and Agentic Training with Asynchrony (2510.11345v1)

Abstract: Synchronous Reinforcement Learning (RL) post-training has emerged as a crucial step for enhancing LLMs with diverse capabilities. However, many systems designed to accelerate RL post-training still suffer from low resource utilization and limited scalability. We present ROLL Flash, a system that extends ROLL with native support for asynchronous RL post-training. ROLL Flash is built upon two core design principles: fine-grained parallelism and rollout-train decoupling. Guided by these principles, ROLL Flash provides flexible programming interfaces that enable a fully asynchronous training architecture and support efficient rollout mechanisms, including queue scheduling and environment-level asynchronous execution. Through comprehensive theoretical analysis and extensive experiments, we demonstrate that ROLL Flash significantly improves resource utilization and scalability over synchronous RL post-training. ROLL Flash achieves up to 2.24x speedup on RLVR tasks and 2.72x on agentic tasks, using the same GPU budget as synchronous baselines. Furthermore, we implement several popular off-policy algorithms and verify that asynchronous training can achieve performance on par with synchronous training.

• The paper introduces violet, a system that decouples rollout and training for asynchronous reinforcement learning post-training of large language models.
• It employs innovations like queue scheduling and prompt replication to achieve significant speedups, reducing average generation time by up to 3.4×.
• The system integrates robust off-policy correction methods to maintain training stability and scales efficiently across numerous GPUs.

Accelerating RLVR and Agentic Training with Asynchrony: The violet System

Introduction

This paper presents violet, a system that extends the ROLL framework to support fully asynchronous reinforcement learning (RL) post-training for LLMs. The motivation arises from the inefficiencies and poor scalability of synchronous RL post-training, where resource bubbles and long-tail rollouts lead to substantial GPU underutilization. violet is designed around two core principles: fine-grained parallelism and rollout–train decoupling. These enable flexible, high-throughput RL post-training, supporting both RL via reward (RLVR) and agentic tasks. The system introduces novel mechanisms such as queue scheduling, prompt replication, environment-level asynchronous rollout, and redundant environment rollout, and integrates off-policy algorithms to maintain training stability under asynchrony.

Figure 1: Overview of ROLL-Sync and Async Framework. The async architecture decouples rollout and training, enabling parallel execution and resource saturation.

Theoretical Foundations of Asynchronous RL Post-Training

The paper provides a rigorous theoretical analysis of asynchronous RL post-training. In synchronous systems, strict barriers between rollout and training stages cause resource bubbles, especially due to long-tail response generation times. The analysis formalizes the completion time bounds for both synchronous and asynchronous settings, showing that asynchrony can strictly improve throughput. Specifically, the average per-sample completion time in the async setting converges to $\mu_{\text{gen}} / K$ as the asynchrony ratio $\alpha \to \infty$, where $\mu_{\text{gen}}$ is the mean generation time and $K$ is the number of workers. The maximum theoretical speedup is $$(L_{\text{gen}} + \mu_{\text{gen}}) / \mu_{\text{gen}}$$, with $L_{\text{gen}}$ the maximum generation time.

Resource partitioning between training and inference is also analyzed. The optimal allocation ratio $\beta^*$ is derived to balance generation and training, minimizing end-to-end completion time. The async setting yields a strictly tighter bound than sync, with speedup increasing as the asynchrony ratio and resource count grow.

System Architecture and Design Principles

violet is architected to realize fine-grained parallelism and rollout–train decoupling. The system comprises four main components:

• LLMProxy: Orchestrates LLM inference across backend workers, maximizing GPU utilization via non-blocking, step-wise inference and immediate post-processing.
• EnvManager: Manages environment interactions, enabling sample-level parallel rollouts and overlapping LLM generation, environment steps, and reward computation.
• SampleBuffer: Stores generated trajectories, supporting asynchronous consumption by the training stage.
• AsyncController: Coordinates weight synchronization and training, enforcing per-sample freshness via the asynchronous ratio $\alpha$.

  Figure 2: Asynchronous Execution Workflow of violet for RLVR and Agentic Post-Training. Components orchestrate parallel rollout and training with fine-grained control.

The asynchronous ratio $\alpha$ bounds the staleness of samples, ensuring that no sample is generated from a policy older than $n-\alpha$ if the current policy is at version $n$. This per-sample constraint maintains training stability while maximizing resource utilization.

RLVR Pipeline: Queue Scheduling and Prompt Replication

Queue Scheduling

Traditional batch rollouts suffer from straggler effects, where the longest response in a batch gates progress, causing GPU idle time. violet implements queue scheduling, treating each prompt as an independent task. Responses are dispatched to reward workers immediately upon generation, overlapping reward computation with ongoing generation and eliminating pipeline bubbles.

Figure 3: Comparison of Batch Rollout and Queue Scheduling Rollout. Queue scheduling maintains high GPU utilization and accelerates sample collection.

Empirical results show that queue scheduling, especially with redundant prompts, reduces average per-step generation time by up to $3.4\times$ in high-redundancy configurations.

Figure 4: Efficiency comparison of generation time across different batch sizetimes8 configurations. Queue scheduling achieves substantial speedups over synchronous batch rollout.

Prompt Replication

Prompt replication further alleviates bottlenecks in multi-candidate decoding. Instead of synchronously generating multiple responses per prompt on a single worker, violet expands each prompt into independent rollout tasks, allowing candidates to be scheduled across GPUs. This decoupling mitigates long-tail effects and improves intra-rollout parallelism.

Figure 5: Efficiency of using prompt replication across different rollout configurations. Prompt replication achieves up to $1.84\times$ speedup under large-batch or large-response settings.

Agentic Pipeline: Environment-Level Asynchrony and Redundancy

Agentic tasks involve complex, multi-turn interactions with external environments, where latency variance and failures are common. violet introduces two mechanisms:

Environment-Level Asynchronous Rollout

Trajectories are decomposed into fine-grained environment interaction units. Pending trajectories are dispatched to available LLM workers as soon as environments are ready, sustaining high throughput even under high latency variance.

Empirical simulations show that speedup increases with environment latency variance, reaching $2.46\times$ under high-variance conditions. Real-environment evaluations confirm reductions in training time by $1.23\times$ (SWE) and $1.58\times$ (ALFWorld).

Redundant Environment Rollout

To mitigate fail-slow and fail-stop environments, violet allows increasing the number of concurrent environment groups and candidate trajectories per group. This redundancy prevents slow environments from bottlenecking the system.

Figure 6: Heatmap of speedup across group size and env group count. Increasing group count yields stronger resilience and higher throughput.

Real-environment results show additional $7\%$–$16\%$ throughput improvements when combining environment-level asynchrony and redundancy.

Off-Policy Algorithm Integration and Training Stability

Asynchronous training introduces policy staleness, necessitating off-policy correction. violet integrates several algorithms:

• Decoupled PPO: Regulates updates via a proximal policy.
• Truncated IS: Clips importance sampling ratios to stabilize gradients.
• CISPO and TOPR: Apply asymmetric clipping and trajectory partitioning for robust off-policy learning.

Empirical evaluation demonstrates that, under async ratios of 2 and 8, all methods—including GRPO—achieve performance on par with synchronous training, with negligible degradation. Weighted TOPR further enhances stability.

Figure 7: Off-Policy Algorithm Performance Comparison under Async Ratio 2 and 8. Async variants match or exceed sync baseline performance across benchmarks.

Resource Scalability and Utilization

violet exhibits strong scalability with increasing GPU resources. Asynchronous architectures achieve near-linear throughput scaling, reaching $2.24\times$ speedup on RLVR tasks and $2.72\times$ on agentic tasks at scale. The async ratio hyperparameter is shown to be robust, with values as low as 2 sufficing for maximal acceleration without significant off-policy penalties.

Figure 8: An illustration of Training Acceleration with violet. Throughput scales efficiently with GPU count, especially under asynchronous execution.

Implementation Details

violet is implemented atop SGLang and vLLM for generation, and Megatron for distributed training. The system supports flexible configuration of async ratio, resource allocation, and off-policy algorithm selection. Importance sampling correction is applied to account for discrepancies between inference and training engines. The system is evaluated on DAPO-Math-18K, SWE-Bench, ALFWorld, and ShopSimulator datasets, with comprehensive ablation studies across model sizes, sequence lengths, batch sizes, and resource allocations.

Implications and Future Directions

The violet system demonstrates that asynchronous RL post-training can achieve high resource utilization, scalability, and training stability for LLMs in both RLVR and agentic settings. The integration of fine-grained parallelism, rollout–train decoupling, and robust off-policy algorithms enables efficient scaling to hundreds of GPUs. The design principles and mechanisms introduced—queue scheduling, prompt replication, environment-level asynchrony, and redundancy—are broadly applicable to future RLHF and agentic training pipelines.

Potential future directions include extending asynchronous architectures to multi-agent and multi-environment scenarios, further optimizing off-policy correction for extreme asynchrony, and integrating more flexible sampling strategies in the inference engine. The architectural interface between generation and training engines remains an area for improvement, particularly for supporting diverse sampling hyperparameters without compromising fidelity.

Conclusion

violet advances the state of RL post-training for LLMs by providing a principled, efficient, and scalable asynchronous training system. Theoretical analysis and extensive empirical validation confirm that asynchrony, when combined with fine-grained parallelism and robust off-policy algorithms, delivers substantial speedups and maintains training fidelity. The system's design and results have significant implications for the development of next-generation RLHF and agentic training frameworks.