TensorHub: Scalable and Elastic Weight Transfer for LLM RL Training

Abstract: Modern LLM reinforcement learning (RL) workloads require a highly efficient weight transfer system to scale training across heterogeneous computational resources. However, existing weight transfer approaches either fail to provide flexibility for dynamically scaling clusters or incur fundamental data movement overhead, resulting in poor performance. We introduce Reference-Oriented Storage (ROS), a new storage abstraction for RL weight transfer that exploits the highly replicated model weights in place. ROS presents the illusion that certain versions of the model weights are stored and can be fetched on demand. Underneath, ROS does not physically store any copies of the weights; instead, it tracks the workers that hold these weights on GPUs for inference. Upon request, ROS directly uses them to serve reads. We build TensorHub, a production-quality system that extends the ROS idea with topology-optimized transfer, strong consistency, and fault tolerance. Evaluation shows that TensorHub fully saturates RDMA bandwidth and adapts to three distinct rollout workloads with minimal engineering effort. Specifically, TensorHub reduces total GPU stall time by up to 6.7x for standalone rollouts, accelerates weight update for elastic rollout by 4.8x, and cuts cross-datacenter rollout stall time by 19x. TensorHub has been deployed in production to support cutting-edge RL training.

• The paper introduces a reference-oriented storage abstraction (ROS) that decouples weight ownership from storage to optimize weight transfer in LLM RL training.
• It demonstrates significant performance gains with up to 6.7× reduction in GPU stall time, 4.8× faster elastic rollouts, and 19× improvement in cross-datacenter scenarios.
• It advances fault tolerance and consistency by utilizing transactional semantics and topology-aware pipeline replication across heterogeneous, elastic clusters.

TensorHub: Scalable and Elastic Weight Transfer for LLM RL Training

Motivation and Limitations of Existing Weight Transfer Methods

Large-scale reinforcement learning (RL) for LLMs mandates rapid, frequent, and reliable transfer of massive model weights—often terabytes in size—across thousands of heterogeneous computational nodes, which may span datacenters and include preemptible instances. The classical approaches to distributed weight transfer—collective communication (e.g., NCCL), point-to-point communication (e.g., UCX), and distributed storage (e.g., parameter servers, Ray Plasma)—display fundamental trade-offs:

• Collective communication saturates bandwidth but requires static topology and synchronizes all participants, amplifying stragglers and failing under churn.
• Point-to-point communication supports elasticity but offers poor fan-out and is bottlenecked under high concurrency.
• Distributed storage decouples producers and consumers but duplicates data movement and incurs high storage/memory overhead, becoming infeasible at TB scale.

None achieves simultaneous topology-awareness, elasticity, minimal coordination, minimal data movement, and negligible space overhead.

Figure 1: RL workloads entail streaming prompt/response pairs and model weights between trainers and rollout workers, with weights dominating transfer cost.

The Reference-Oriented Storage (ROS) Abstraction

The key technical insight underlying TensorHub is exploiting the natural, short-lived, highly-replicated, and immutable nature of model weights in RL. Instead of storing or moving heavyweight weight tensors through a central server, ROS maintains lightweight references to live replicas residing on workers' GPUs. Upon a fetch request, ROS orchestrates direct peer-to-peer transfer from any available source. This divorces data ownership from the storage abstraction, abolishing unnecessary data hops and storage duplication.

In ROS, model weight versions are not centrally owned but are tracked as references to their in-situ locations. Workers advertise available replicas, publish/unpublish as they mutate buffers, and declare explicit retention contracts for needed weight versions. Availability is preserved via protocols that safeguard required versions, offloading them to CPU memory if the last GPU-based replica disappears.

Figure 2: The ROS workflow: the server handles only metadata/references, while data transfer occurs directly between clients, eliminating storage overhead.

TensorHub: System Design and Implementation

TensorHub operationalizes ROS with robust sharding support, transactional consistency for SPMD worker groups, network/topology-aware scheduling, and fault tolerance tailored for preemptible and multi-datacenter deployments.

Naming, API, and Sharding Semantics

TensorHub organizes state hierarchically: models → versions → replicas (per model-parallel process group) → shards (per GPU). All APIs (publish, replicate, update, retain, unpublish, query) operate at the granularity of shards.

Pipeline replication, enabled by tracking replication progress at workers, transforms replication into a bandwidth-amplifying DAG rather than a static fan-out. Partial replicas can serve subsequent replicas, maximizing parallelism and minimizing contention.

Figure 4: Pipeline replication enables new workers to read from partially-filled peers, increasing scalability under bursty loads.

Topology-Aware Replication and Cross-Datacenter Optimization

TensorHub's centralized reference server schedules source selection and replication based on real-time load and network topology, always favoring intra-datacenter links and balancing load among source replicas.

For multi-datacenter deployments, initial seeding is performed over slower TCP transport, after which fast intra-datacenter RDMA is exploited for local replicas. Smart skipping during seeding prevents spurious contention and further minimizes stall times.

Consistency and Fault Tolerance

A core challenge with sharded, SPMD-parallel rollouts is maintaining a strict consistent view of available weights across all workers in a group. TensorHub enforces transactional semantics: all requests from a group within a transaction observe a serialized, group-consistent system state, precluding divergence and deadlock.

Figure 6: Without transactional semantics, group members could diverge on weight versions; TensorHub's transactional protocol ensures consistency.

Fault tolerance is fully integrated: failed clients are detected by missed heartbeats; incomplete replicas or failed sources are invalidated and promptly replaced. Critically, spot (preemptible) nodes are discounted for retention, ensuring progress despite churn.

Empirical Results and Quantitative Analysis

TensorHub is evaluated on hundreds to over a thousand GPUs using RL workloads spanning standard, elastic, and cross-datacenter rollouts. Strong quantitative results include:

• Standalone rollouts: Up to 6.7× reduction in total GPU stall time compared to NCCL, achieved by localizing stalls and fully saturating RDMA bandwidth.
• Elastic rollouts: With preemptible resources, TensorHub yields 4.8× faster weight update/replication time than UCX, due to decentralized, pipeline topology-aware distribution.
• Cross-datacenter rollouts: 19× reduction in rollout stall time, as initial cross-datacenter seeds are amortized and subsequent transfers proceed at local RDMA speed.

Figure 3: TensorHub saturates RDMA bandwidth and achieves lowest latency for large weight shards across all compared approaches.

Figure 5: Combined with pipeline replication, TensorHub minimizes and localizes stall times, even in the presence of heavy burst replication.

Figure 7: GPU stall time scales efficiently even as GPU count increases in elastic rollout scenarios.

Figure 11: Cross-datacenter rollouts: TensorHub avoids redundant TCP flows and concentrates cross-datacenter traffic to the minimum, resulting in dramatic stall time reduction.

Implications, Limitations, and Prospective Advances

The introduction of ROS and its realization in TensorHub offers a new pathway for efficiently orchestrating large RL systems, particularly as weight scales continue to grow and cluster heterogeneity increases. Its ownership-free architectural paradigm bridges the gap between the flexibility of distributed storage and the efficiency of direct communication.

Practical Implications

• Extreme scalability: The system is validated at the 1T parameter scale.
• Elastic resource usage: Transient (spot) and remote GPUs can be harnessed without complex framework coordination.
• Minimal engineering overhead: Integration with frameworks such as veRL requires minimal code changes (~40 lines), vastly less than the alternatives.

Theoretical Impact

ROS's abstraction generalizes classical storage concepts to the highly mutable and ephemeral regime of modern distributed AI. Future directions include scaling the reference server via partitioning, extending retention/consistency semantics, and further exploring the integration with asynchronous and off-policy RL architectures.

Conclusion

TensorHub advances the state of distributed systems for large-scale RL in LLMs by introducing a reference-oriented, ownership-free storage abstraction that decisively resolves the inefficiency-versus-flexibility dichotomy in weight transfer. Empirical results demonstrate its capacity to fully saturate bandwidth across topologies, maintain fault tolerance under extreme churn, and support high-throughput, heterogeneous clustering with minimal engineering burden. As LLM RL workloads become more distributed, elastic, and scalable, paradigms decoupling data movement from ownership, as exemplified by TensorHub, are poised to become foundational.