TransferEngine: Portable RDMA for LLM Systems

• TransferEngine is a high-performance, vendor-agnostic RDMA library for LLM systems, enabling efficient one-sided and two-sided data transfers without vendor lock-in.
• It leverages robust notification mechanisms like ImmCounter to ensure low-latency, scalable communication across heterogeneous NICs such as NVIDIA ConnectX and AWS EFA.
• Designed for disaggregated inference, MoE routing, and reinforcement learning, TransferEngine auto-shards transfers over multiple NICs to maximize throughput and performance.

TransferEngine refers to a high-performance, vendor-agnostic Remote Direct Memory Access (RDMA) point-to-point communication library designed for LLM systems and related distributed AI workloads. Its core function is to support portable, efficient, and reliable one-sided and two-sided RDMA transfers over heterogeneous network interface controllers (NICs), notably NVIDIA ConnectX and AWS Elastic Fabric Adapter (EFA), without requiring vendor lock-in or specific ordering guarantees. TransferEngine is architected to accommodate evolving LLM system patterns, such as disaggregated inference, Mixture-of-Experts (MoE) routing, and asynchronous reinforcement learning (RL) fine-tuning, where flexible, high-throughput, low-latency, and dynamic point-to-point communication is essential (Licker et al., 31 Oct 2025).

1. Motivations and System Design Requirements

Recent LLM system designs impose fundamentally new requirements on inter-node communication patterns, motivated by the following scenarios:

• Disaggregated inference: Prefill and decode computation stages are separated and elastically scheduled across independent pools of GPUs. This necessitates dynamic, per-request peer-to-peer data transfers with no fixed group membership.
• Mixture-of-Experts (MoE): Sparse model architectures demand arbitrary dispatch and aggregation of tensors between a variable subset of nodes, beyond what collective communication primitives can express efficiently.
• Asynchronous reinforcement learning (RL): Fast rollout and weight update cycles for trillion-parameter models require direct, cluster-scale weight distribution in minimal time.
• Cloud/on-premise heterogeneity: LLM researchers and providers increasingly require operation both on-prem hardware (e.g., ConnectX) and cloud infrastructure (e.g., AWS with EFA), whose network stacks differ—especially regarding operation ordering guarantees and GPU-initiated capabilities.

Traditional network libraries are tightly coupled to specific NICs and RDMA semantics, with collective libraries providing no satisfactory solution for highly dynamic, sparse, peer-to-peer traffic. TransferEngine addresses this gap by normalizing the RDMA abstraction across vendor and platform boundaries.

2. Core Architecture and Abstraction

TransferEngine exposes a uniform Rust-based API for RDMA memory registration, point-to-point data transfer, notification, and management of multiple NICs per GPU. Its main architectural elements are:

• DomainGroup: A logical group encapsulating all NICs attached to a single GPU. Each GPU is serviced by a dedicated per-GPU worker thread that aggregates and load-balances transfers across its available NICs.
• Domain: Represents a NIC-specific implementation, internally managing work requests (QPs, CQs), device memory registration, and peer communication handling.
• NetAddr and PeerGroupHandle: Abstract addresses and groups of peers for peer discovery, connection management, and collective-style multicast operations (scatter, barrier).
• Multi-NIC sharding: Transfers are automatically partitioned and striped across all NICs attached to the GPU, allowing aggregation up to system bandwidth limits (e.g., 400 Gbps across four EFAs).

The library supports both one-sided (e.g., WriteImm) and two-sided (Send/Recv) RDMA primitives, with a focus on providing a minimum abstraction matching features available across supported NICs.

3. Notification and Completion Handling: WriteImm and ImmCounter

A fundamental challenge in heterogeneous RDMA environments is reliable transfer completion notification without assuming ordered delivery:

• WriteImm: A one-sided Write request augmented with a 32-bit immediate value atomically delivered to the target’s completion queue. Both ConnectX and AWS EFA support this primitive, but only the former supports network-level ordering.
• ImmCounter: Editor's term for the TransferEngine abstraction that provides robust, vendor-neutral notification. The receiver exposes a counter per-tensor/transfer; upon observing the expected number of immediate values (regardless of network arrival order), completion is detected, and the transfer is marked as ready—typically via callback or atomic flag. This paradigm avoids dependence on in-order delivery (which EFA’s SRD transport does not provide) and enables asynchronous, lock-free notification.
• UVM Watcher: For pinned host–GPU memory synchronization, allowing CPU to respond to specific memory address changes, e.g., for pipelined streaming or GPU-involved communication.

This notification scheme underpins all asynchronous and streaming data movement in TransferEngine-based systems, including paged KV-cache delivery and MoE tensor dispatch.

4. Multi-NIC Aggregation and Vendor Portability

TransferEngine is deliberately designed to exploit all NICs attached to a GPU, maximizing bandwidth in platforms with multi-NIC networking (e.g., AWS p5: four EFAs per GPU). It enforces peer discovery symmetry and memory registration compatibility for cross-NIC data movement.

The library’s portable core restricts itself to:

• Send/Recv, WriteImm: Universally supported by both ConnectX and EFA.
• No reliance on in-order transport: Unlike solutions relying on InfiniBand General Device Assignment (IBGDA) or GPUDirect RDMA Write With Immediate, TransferEngine does not require ordering at the NIC layer, making it operational on EFA's unordered SRD transport.
• No dependence on GPU-initiated writes: GPU-initiated RDMA is only available on ConnectX; TransferEngine instead maps notifications via host-side proxy and GDRCopy window management.

This ensures that the same high-level interface and semantics function identically across hardware and cloud, removing vendor lock-in. Peer group APIs further facilitate groupwise notification (barrier, scatter), although these remain point-to-point in implementation.

5. Microbenchmark and Production Performance

TransferEngine achieves performance at or near the line rate of modern network hardware, with explicit evaluation on ConnectX-7 and AWS EFA:

• Bulk transfer throughput: 400 Gbps sustained for large transfers (all-NIC aggregation) on both platforms.
• Typical dispatch workloads:
  • 256 KiB Write (MoE routing): 54 Gbps (EFA), 116 Gbps (ConnectX-7).
  • 64 KiB paged Write (KV-cache): 364 Gbps (EFA), 370 Gbps (ConnectX-7).
• Production system latencies:
  • Disaggregated inference KVCache transfer: Full pipelined streaming, decoupled synchonization via ImmCounter, scalable from prefill to decode nodes.
  • Trillion-parameter RL weight synchronization: 256 training GPUs to 128 inference GPUs in 1.3 seconds (over 100× faster than prior Reinforcement Learning parameter synchronization routines).

TransferEngine outperforms host-based alternatives, and on ConnectX matches or surpasses GPU-initiated RDMA kernels. On AWS EFA, it provides the first viable portable backend for MoE latency-sensitive decoding.

6. Integration with LLM System Patterns

TransferEngine is used to underpin several advanced LLM system designs:

• Disaggregated Inference: Enables dynamic assignment and elastic scaling of prefill/decoder stages across distinct server pools, with KVCache sharded by layers and asynchronously transferred from prefillers to decoders.
• Reinforcement Learning (RL) Weights Update: sharply reduces the time required for large-scale inference fleet weight rollouts, removing the bottleneck of single-leader aggregation or synchronization.
• Mixture-of-Experts (MoE) Dispatch/Combine: Efficient scattered dispatch and aggregation (via submit_scatter, barrier) matches or exceeds decode latency of optimized ConnectX-specific implementations (e.g., DeepEP), while uniquely providing viable low-latency operation on EFA.

For all these system patterns, TransferEngine’s point-to-point model complements, but does not replace, collective libraries (e.g., NCCL)—the latter remain optimal for static, dense, barriered parallel traffic, while TransferEngine enables dynamic, per-request, and sparse data movement.

7. Summary Table: RDMA API Feature Coverage

Feature	ConnectX-7	AWS EFA	TransferEngine
Send/Recv (two-sided)	Yes	Yes	Yes
WriteImm (one-sided + imm)	Yes	Yes	Yes
Notification (w/o order)	Yes	Yes	Yes (via ImmCounter)
Ordering required	Yes	No	No
GPU-initiated write	Yes	No	Not required
Multi-NIC/port	Yes	Yes	Yes (auto-sharded)
Host-GPU proxy	Yes	Yes	Yes

8. Impact and Future Development Directions

TransferEngine sets a new standard for arbitrary, scalable, and portable RDMA-based point-to-point communication in LLM infrastructure. Its principal impact lies in:

• Breaking vendor lock-in: By only requiring the lowest-common-denominator RDMA feature set, it allows system architects to target both on-prem and public cloud infrastructure without rewriting communication logic.
• Performance portability: Delivers near-peak hardware utilization whether on ConnectX or EFA, automatically exploiting all available NICs/ports.
• Building block for future LLM and AI systems: As models and system architectures become more heterogeneous and dynamic, the library’s abstraction model is positioned to remain relevant; it also complements but does not supersede collective communication primitives.
• Potential generalization to other AI workloads: While developed and benchmarked for LLM-specific patterns (disaggregated inference, MoE, RL), TransferEngine's abstraction could be applied to other domains requiring similar elasticity and flexibility in point-to-point data movement, provided the workload fits within the RDMA model.

A plausible implication is that as model and workload diversity grow, a TransferEngine-style abstraction may serve as a required foundation for LLM-serving systems across vendor-neutral, hybrid on-prem/cloud environments. No invented performance metrics or anticipated roadmap is stated in the source; all claims, API, architectural details, microbenchmarks, and use cases appear verbatim in (Licker et al., 31 Oct 2025).