MindSpeed RL: Distributed RL Training

• MindSpeed RL is a distributed RL training system that reframes LLM alignment as a dataflow problem with parallel worker states.
• It introduces a distributed transfer dock to manage sample flows, significantly reducing cross-node dispatch overhead and memory duplication.
• The allgather–swap resharding strategy reclaims device memory during state transitions, achieving throughput gains of up to 3.97× over baselines.

Searching arXiv for the cited paper and closely related systems to ground the article in current literature. {"query":"MindSpeed RL Distributed Dataflow for Scalable and Efficient RL Training on Ascend NPU Cluster (Feng et al., 25 Jul 2025) related VeRL OpenRLHF GRPO speculative decoding rollout scheduling", "max_results": 10} {"query":"(Feng et al., 25 Jul 2025)", "max_results": 5} MindSpeed RL is a distributed RL training system for LLM alignment on Ascend NPU clusters that reframes reinforcement learning as a distributed dataflow problem and directly targets two dominant bottlenecks in large-scale RL: dispatch overhead from cross-worker sample movement and redundant device memory consumption during state transitions between training and generation. Its central mechanisms are a distributed transfer dock for sample flow and an allgather–swap resharding strategy for weight flow, combined with parallelization strategies and Ascend-specific acceleration techniques. Large-scale experiments on Qwen2.5-Dense-7B/32B, Qwen3-MoE-30B, and DeepSeek-R1-MoE-671B report throughput gains of $1.42 \sim 3.97$ times over state-of-the-art baselines on a 384-NPU Ascend super pod (Feng et al., 25 Jul 2025).

1. Problem formulation as distributed dataflow

MindSpeed RL begins from the observation that mainstream RL for LLM alignment, including GRPO, PPO, PF-PPO, and DAPO, involves multiple distributed workers—actor, reference, reward, and optimizer—that repeatedly cycle between generation, inference, and update. These workers exchange prompts, responses, logits, rewards, metadata, and updated weights with heavy cross-node dependencies. In conventional deployments, those dependencies produce poor cluster scalability, because centralized queuing and serialization amplify dispatch overhead, and low memory utilization, because transitions between update and generation partitions retain duplicate buffers on device (Feng et al., 25 Jul 2025).

The system formalizes RL training as a directed multi-worker dataflow graph $G = (V, E)$, where $V$ denotes worker states and $E$ denotes dataflows among states. Two edge sets are distinguished. $S_{flow} \subset E$ represents sample flow edges that move prompts, responses, logits, rewards, and metadata among actor, reference, reward, and optimizer states. $R_{flow} \subset E$ represents resharding flow edges that move updated actor weights from the update state to the generation state under different parallel partitions such as DP, TP, PP, EP, and CP. An adjacency relation $A$ encodes data dependencies.

End-to-end throughput is measured as

$$T = \frac{G \times N \times (PL + SL)}{ND \times ETE},$$

where $G$ is the global prompt batch size, $N$ is responses per prompt, $G = (V, E)$0 and $G = (V, E)$1 are maximum prompt and response lengths, $G = (V, E)$2 is the number of devices, and $G = (V, E)$3 is end-to-end iteration time. For sample flow under GRPO with a centralized replay buffer, the communication volume per iteration is modeled as

$G = (V, E)$4

and

$G = (V, E)$5

where $G = (V, E)$6 is bytes per element, $G = (V, E)$7 counts items proportional to response length, and $G = (V, E)$8 is the scalar count. In practical clusters, centralized dispatch can dominate iteration time at scale, reaching hundreds to thousands of seconds for large $G = (V, E)$9, $V$0, and $V$1 under 100–1000 MB/s inter-server links, especially with Ray tensor serialization.

For resharding flow, the paper isolates redundant device memory created when update and generation use different partitions. With naive gather-and-re-slice, redundant memory is

$V$2

where $V$3 and $V$4 are sizes of tensor-parallel and expert weights, $V$5 is update TP size, and $V$6 and $V$7 are generation DP and EP sizes. For Qwen3-MoE-30B, this redundancy exceeds 60 GB per device.

2. Distributed transfer dock for sample flow

MindSpeed RL replaces centralized dispatch with a distributed transfer dock (TD) built atop, but decomposed from, the replay buffer. The design introduces two roles. TD warehouses are sharded, distributed storage nodes for sample tensors indexed by global sample IDs; $V$8 warehouses, typically equal to the cluster node count, partition the global batch $V$9 and share data ingress and egress load. TD controllers are per-worker-state metadata managers, one per state, with total $E$0–$E$1 depending on the algorithm; they track sample indices, warehouse locations, and state flags, and are colocated with their workers to minimize cross-node control traffic (Feng et al., 25 Jul 2025).

The TD workflow has three stages. First, a worker state requests the next sample IDs and warehouse locations from its colocated controller; the controller returns a compact TensorDict containing scalars such as sample_id, warehouse_id, and stage flags. Second, the worker directly pulls the actual tensors—prompts, responses, logits, or rewards—from the corresponding warehouse via Ray, while producers push tensors to warehouses in the same way. Third, the warehouse updates item states, such as generated, scored, or ready_to_update, and broadcasts updated metadata to all controllers, which then update their local views.

This organization reduces cross-node contention because heavy tensors move only between workers and designated warehouses, while controllers circulate only lightweight metadata. The per-warehouse communication volume is given by

$E$2

Since $E$3 is small, approximately $E$4–$E$5, and metadata movement is divided across $E$6 warehouses, the transfer dock substantially cuts dispatch overhead. The system further uses TensorDict with Ray to accelerate serialization and deserialization. In the reported cluster, with inter-server bandwidth of 300 MB/s, the paper states that TD enables scaling where centralized dispatch would otherwise take more than 1000 seconds per iteration for large $E$7, $E$8, and $E$9.

The TD also includes explicit flow control. Controllers throttle next_ready_ids by stage quotas, warehouses track per-stage queue depths, and workers pull only when stage queues exceed thresholds. Controllers can order sample IDs by latency, length, or score; the implementation described in the paper uses simple FIFO plus stage gating consistent with GRPO.

3. Allgather–swap resharding and memory reclamation

The second core mechanism addresses the transition of actor weights between update and generation states when those states use different parallel partitions. MindSpeed RL adopts an allgather–swap sequence intended to eliminate duplicate device buffers during resharding (Feng et al., 25 Jul 2025).

The sequence has four steps. A temporary allgather buffer is allocated on device and used to materialize the full weight slices required for the new partition. Each device then selects the exact slices needed by its generation partition and installs them as generation weights. The original update-state weight buffers are subsequently swapped from device to host by D2H transfer, fully freeing device memory used by update-state partitions. Finally, the temporary allgather buffer is released and generation resumes.

This procedure is designed to avoid two redundant states present in naive strategies: tensor-parallel weights that share buffer space with common weights and therefore cannot be freed, and EP slice buffers in which unused experts remain resident. By swapping update weights to host, device memory is reclaimed during generation; before the next update, weights are swapped back by H2D transfer. The paper reports Ascend host–device bandwidth of at least 50 GB/s, with D2H and H2D swaps completing in seconds and overlap possible, including H2D prefetch during inference.

The memory model is written as

$S_{flow} \subset E$0

for naive resharding and

$S_{flow} \subset E$1

for allgather–swap, with savings ratio

$S_{flow} \subset E$2

For a single device, the redundant component matches the earlier expression for $S_{flow} \subset E$3. In Qwen2.5-Dense-32B resharding from TP8DP2 to TP4DP4, allgather–swap released approximately 8 GB per device for KV cache during generation, directly improving generation efficiency.

Communication cost for the allgather collective is modeled as

$S_{flow} \subset E$4

where $S_{flow} \subset E$5 is the number of participants, $S_{flow} \subset E$6 is message size per rank, and $S_{flow} \subset E$7 and $S_{flow} \subset E$8 reflect latency and bandwidth constants of the HCCL collective on Ascend interconnect.

4. System architecture, parallelism, and Ascend integration

MindSpeed RL is organized in four layers, coupling distributed orchestration with hardware-specific optimization (Feng et al., 25 Jul 2025).

Layer	Function
Resource pool	Ascend NPU cluster managed and allocated via Ray actors
Computing engines	vLLM-Ascend for generation; MindSpeed for training
Dataflow layer	TD controllers and warehouses for sample flow; allgather–swap for resharding
Algorithm layer	Trainers for PPO, GRPO, PF-PPO, DAPO, DPO, etc.

The experimental deployment used 48 nodes with 8 NPUs per node, for a total of 384 NPUs, each with 128 GB. Workers interact with controllers for metadata and with warehouses for tensors; controllers expose stage-gated next_ready_ids, and warehouses support put and get by sample_id. Ray supervises distributed actors and resources, while throughput metrics are averaged over five iterations and training behavior is summarized with reward curves.

The system composes multiple parallel paradigms: data parallel, tensor parallel, pipeline parallel, expert parallel, and context or sequence parallel. Variants listed in the paper include ZeRO, virtual pipeline, RingAttention, and Ulysses. Dense models use balanced combinations of DP, TP, and PP. MoE models use EP with efficient all-to-all communication through HCCL collectives, often reducing the need for TP and PP. For DeepSeek-R1-MoE-671B, the update stage uses TP4PP6EP16DP2 and the generation stage uses TP2PP1EP64DP6; partition changes are handled by allgather–swap.

Ascend-specific optimization is a major aspect of the implementation. vLLM-Ascend and MindSpeed leverage fused kernels including FlashAttention, RMSNorm, RoPE, SwiGLU, MatmulAdd, and GMM. The system also applies stage fusion, partial rollout, overlap of H2D swaps with inference, improved buffer management for KV cache sizing, and prefetching to minimize resharding stalls. HCCL provides allgather and all-to-all communication, and the hardware characteristics explicitly used in system design are host–device bandwidth of about 50 GB/s and inter-server bandwidth of about 300 MB/s.

5. Empirical performance, scaling behavior, and comparative results

The reported experiments use GRPO with rule rewards and DeepScaleR prompts on an Ascend super pod with 48 nodes and 384 NPUs. The throughput metric is the previously defined

$S_{flow} \subset E$9

and reported values are averaged over five iterations (Feng et al., 25 Jul 2025).

Under a typical GRPO configuration on 16 NPUs, with $R_{flow} \subset E$0, $R_{flow} \subset E$1, $R_{flow} \subset E$2, and $R_{flow} \subset E$3, MindSpeed RL exceeds VeRL and OpenRLHF by $R_{flow} \subset E$4–$R_{flow} \subset E$5, depending on model and settings. For DeepSeek-R1-MoE-671B on 384 NPUs, with $R_{flow} \subset E$6, $R_{flow} \subset E$7, $R_{flow} \subset E$8, $R_{flow} \subset E$9, and partitions TP4PP6EP16DP2 $A$0 TP2PP1EP64DP6, throughput fluctuates between 200 and 250 TPS over 100 iterations with steadily increasing reward. The paper adds that few open-source systems can train such a large MoE model end-to-end.

Scaling efficiency is examined through transfer dock linearity. With 64 prompts per node and increasing node count, MindSpeed RL achieves 81.1% linearity at 192 NPUs, compared with 40.4% for VeRL and 61.9% for MSRLB, the replay-buffer variant. Dispatch-overhead modeling further shows that centralized replay buffers can consume thousands of seconds per iteration for large $A$1, $A$2, and $A$3, whereas TD reduces per-warehouse $A$4 by approximately $A$5 when metadata $A$6 is small.

Training stability is reported on Qwen2.5-Dense-32B with $A$7 and $A$8. Reward curves are described as stable, and benchmark scores at 600 and 1000 iterations are as follows.

Benchmark	MSRL	VeRL
MATH500 (Pass@1)	85.7 / 85.0	84.7 / 83.2
AIME24 (Avg@24)	23.7 / 23.8	23.3 / 21.9
GPQA (Avg@4)	47.4 / 48.1	45.9 / 47.6

The comparison to centralized RL systems is explicit. Centralized replay-buffer designs such as K1.5 and OpenRLHF incur heavy actor and optimizer requests to a central store, amplifying contention and serialization. VeRL improves resharding granularity but retains centralized sample movement and does not remove redundant device buffers across states. MindSpeed RL’s distributed TD shifts tensor traffic to per-warehouse data movement and controller metadata exchange, while allgather–swap eliminates device-resident duplicates across TP and EP slices during state transitions. The paper associates these changes with improved on-cluster scalability, better memory utilization, larger KV caches, and faster generation.

6. Limitations, reproducibility, and relation to adjacent RL systems work

The paper states three principal limitations. First, gains diminish for small clusters or for short-sequence, small-batch regimes in which centralized dispatch is not dominant and resharding buffers fit comfortably on device. Second, the system is optimized for Ascend through vLLM-Ascend, HCCL, and fused kernels; although SGLang and FSDP2 are being integrated, the paper does not claim full portability or equivalently optimized performance on non-Ascend hardware. Third, future work includes extending transfer dock policies through priority scheduling and dynamic sharding, broader algorithm coverage, and deeper overlap and prefetch strategies across diverse workloads (Feng et al., 25 Jul 2025).

Reproducibility information is unusually concrete. The repository is hosted at https://gitee.com/ascend/MindSpeed-RL. The cluster configuration used in the paper consists of 48 nodes $A$9 8 Ascend NPUs, each with 128 GB; H2D and D2H bandwidth is approximately 50 GB/s, and inter-server bandwidth is approximately 300 MB/s. A minimal run outline specifies Ray cluster setup, installation of vLLM-Ascend and MindSpeed, configuration of parallel partitions for update and generation, batch sizes and sequence lengths, TD parameters $$T = \frac{G \times N \times (PL + SL)}{ND \times ETE},$$0 and $$T = \frac{G \times N \times (PL + SL)}{ND \times ETE},$$1, and enabling allgather–swap together with stage fusion and partial rollout. The repository provides scripts and configuration files, while environment variables and HCCL settings are delegated to Ascend deployment guides.

MindSpeed RL also sits within a broader line of RL systems work that targets adjacent bottlenecks. ReSpec studies speculative decoding in RL systems and reports up to $$T = \frac{G \times N \times (PL + SL)}{ND \times ETE},$$2 end-to-end speedup while preserving reward convergence and training stability on Qwen models from 3B to 14B (Chen et al., 30 Oct 2025). EfficientRollout addresses rollout generation through system-aware self-speculative decoding and reports up to 19.6% rollout-latency reduction and 12.7% end-to-end training-step reduction over an accelerated autoregressive baseline (Kim et al., 17 Jun 2026). SortedRL focuses on online length-aware scheduling, reducing bubble ratio from 74% to 5.81% in fully on-policy mode or 3.37% in partial mode while reporting 3.9% to 18.4% superior performance over baseline given the same amount of data (Zhang et al., 24 Mar 2026). RollMux studies phase-level multiplexing for disaggregated rollout and training clusters and reports 1.84x cost efficiency over standard disaggregation with 100% SLO attainment on a production-scale testbed (Wu et al., 12 Dec 2025). Taken together, these systems indicate that large-scale RL post-training is increasingly treated as a systems problem spanning dataflow, rollout generation, scheduling, and cluster orchestration, while MindSpeed RL’s defining contribution remains its distributed treatment of sample flow and resharding flow on Ascend NPU clusters.