NVRAR: NVSHMEM-based Recursive All-reduce

• The paper introduces NVRAR, a hierarchical NVSHMEM-based recursive all-reduce algorithm that mitigates latency bottlenecks in tensor-parallel LLM inference.
• It combines intra-node NCCL-based reduce-scatter and all-gather with inter-node recursive-doubling to achieve O(log N) startup costs for 128 KB–2 MB messages.
• Empirical results demonstrate up to 3.6x lower latency than NCCL and notable end-to-end speedups when deployed in the YALIS research inference engine.

Searching arXiv for the NVRAR paper and closely related distributed LLM inference / all-reduce work. NVRAR is a hierarchical all-reduce algorithm for multi-node GPU inference, introduced in the context of distributed LLM inference as an NVSHMEM-based recursive-doubling design that targets the small-message, latency-bound regime characteristic of tensor-parallel decode workloads. In the source study, NVRAR is presented as a mitigation for a common bottleneck in multi-node model-parallel inference: per-layer all-reduce of hidden-state fragments of size $B \times H$, especially in the 128 KB–2 MB range. Its design combines intra-node collectives with an inter-node recursive-doubling phase, and it is integrated into the YALIS research inference engine to preserve GPU-initiated execution and CUDA-graph compatibility while reducing end-to-end latency (Singhania et al., 12 Nov 2025).

1. Definition and problem setting

NVRAR, expanded in the source as “NVSHMEM-based Recursive All-reduce,” is a hierarchical all-reduce algorithm based on recursive doubling with NVSHMEM (Singhania et al., 12 Nov 2025). It is motivated by the observation that, when running large-language-model inference in tensor-parallel mode across multiple nodes, the per-layer all-reduce of hidden-state fragments becomes the dominant latency contributor in the decode phase (Singhania et al., 12 Nov 2025).

The algorithm is positioned against two established families of collective communication. Standard NCCL ring all-reduce incurs an $O(N)$ inter-node startup cost, reported as $2(N \cdot G-1)\cdot \alpha_{\mathrm{inter}}$ for $N$ nodes and $G$ GPUs per node, while tree all-reduce has $O(\log N)$ startup cost but higher constant factors in each step (Singhania et al., 12 Nov 2025). The source also notes that MPI implementations based on recursive doubling often outperform NCCL on small messages, but lack GPU-initiated communication, CUDA-graph support, or optimized NVLink paths (Singhania et al., 12 Nov 2025). NVRAR is designed to bridge that gap.

Its stated goals are threefold: achieve $O(\log N)$ startup latency proportional to $\alpha \log N$ rather than $O(N)$, optimize the small-message regime from 128 KB to 2 MB, and expose a GPU-initiated, CUDA-graph–friendly kernel (Singhania et al., 12 Nov 2025). This suggests that NVRAR is best understood not as a general replacement for all all-reduce implementations, but as a specialization for decode-heavy distributed inference where message sizes are modest and startup costs dominate.

2. Hierarchical three-phase structure

NVRAR decomposes the all-reduce of a tensor $M$ across $O(N)$0 nodes with $O(N)$1 GPUs per node into three phases (Singhania et al., 12 Nov 2025).

First, an intra-node reduce-scatter is performed using nvshmemx_*_sum_reducescatter, described as internally NCCL-based. This reduces the $O(N)$2 fragments of $O(N)$3 on each node into $O(N)$4 local chunks of size $O(N)$5 (Singhania et al., 12 Nov 2025).

Second, an inter-node recursive-doubling phase exchanges data among GPUs of the same local rank across nodes. For $O(N)$6, each GPU index $O(N)$7 on node $O(N)$8 exchanges its current chunk with peer $O(N)$9 using non-blocking put_nbi of fused data-plus-flag words of 8 B. Upon receipt, the remote chunk is immediately added into the local buffer. Synchronization is handled via sequence numbers carried in the fused payload, thereby avoiding explicit NVSHMEM fence and quiet calls (Singhania et al., 12 Nov 2025).

Third, an intra-node all-gather reconstructs the full tensor on each GPU using nvshmemx_*_all_gather, again described as NCCL-based (Singhania et al., 12 Nov 2025).

This organization yields a mixed communication topology. Within each node, NVRAR uses a ring-based reduce-scatter and all-gather. Across nodes, it uses a flat recursive-doubling tree among GPUs of the same local rank, which the source states enables concurrent NIC usage when multiple NICs are available (Singhania et al., 12 Nov 2025). A plausible implication is that the hierarchy is intended to preserve fast intra-node paths while replacing the inter-node portion of the collective with a lower-startup alternative.

3. Algorithmic mechanics and synchronization model

The source presents NVRAR in LaTeX-style pseudocode, centered on an initial reduce-scatter, a sequence-number increment, inter-node synchronization with peers, packing into fused buffers, recursive doubling across chunks, unpacking, and a final all-gather (Singhania et al., 12 Nov 2025).

At the top level, the algorithm performs:

• REDUCE_SCATTER_intra(M,G) to produce $2(N \cdot G-1)\cdot \alpha_{\mathrm{inter}}$0,
• a sequence-number increment,
• peer synchronization over $2(N \cdot G-1)\cdot \alpha_{\mathrm{inter}}$1 rounds,
• Pack(M', seq),
• RD_inter(...),
• Unpack(...),
• and ALL_GATHER_intra(M', G) (Singhania et al., 12 Nov 2025).

Within RD_inter, the buffer is divided into $2(N \cdot G-1)\cdot \alpha_{\mathrm{inter}}$2 chunks. For each recursive-doubling level $2(N \cdot G-1)\cdot \alpha_{\mathrm{inter}}$3, the algorithm computes the peer $2(N \cdot G-1)\cdot \alpha_{\mathrm{inter}}$4, issues NonBlockingPut operations for each chunk, waits on the corresponding flag with WaitFlag, and then forms the next-stage send buffer by reduction:

$2(N \cdot G-1)\cdot \alpha_{\mathrm{inter}}$5

This formulation makes the dataflow explicit: the algorithm progresses by pairwise exchange-and-reduce over successively larger logical groups of nodes (Singhania et al., 12 Nov 2025).

A notable design choice is the use of fused data-plus-flag words and sequence numbers rather than explicit NVSHMEM fence/quiet synchronization (Singhania et al., 12 Nov 2025). The paper attributes this to avoiding higher synchronization overhead. It also notes that NVSHMEM’s put_with_signal was avoided in favor of fused flags, while suggesting that future NVSHMEM libfabric improvements could reduce overhead further (Singhania et al., 12 Nov 2025).

4. Latency model and asymptotic properties

The source analyzes NVRAR using the $2(N \cdot G-1)\cdot \alpha_{\mathrm{inter}}$6–$2(N \cdot G-1)\cdot \alpha_{\mathrm{inter}}$7 model, with startup cost $2(N \cdot G-1)\cdot \alpha_{\mathrm{inter}}$8 and per-byte cost $2(N \cdot G-1)\cdot \alpha_{\mathrm{inter}}$9 (Singhania et al., 12 Nov 2025). For the intra-node ring reduce-scatter and all-gather, the reported costs are

$N$0

$N$1

For the inter-node recursive-doubling phase, with $N$2 steps, message size $N$3, and an inflation factor $N$4 due to fused headers, the reported latency is

$N$5

Summing the three phases gives the total all-reduce time:

$N$6

For small, latency-bound messages, the source states that the bandwidth terms drop out, yielding the approximation

$N$7

This is contrasted with NCCL’s ring, characterized as $N$8, and NCCL’s tree, characterized as $N$9 but with $G$0 steps and higher constants (Singhania et al., 12 Nov 2025).

The significance of this model lies in its fit to decode-heavy inference. The source specifically targets hidden-state all-reduces in the 128 KB–2 MB regime, and the theoretical discussion treats this as a startup-dominated operating point rather than a bandwidth-dominated one (Singhania et al., 12 Nov 2025).

5. Empirical performance and integration into YALIS

The paper reports standalone microbenchmark results on Perlmutter, described as A100/Slingshot-11, and Vista, described as GH200/InfiniBand (Singhania et al., 12 Nov 2025). For message sizes in the 128 KB–2 MB regime, NVRAR achieves 1.9x–3.6x lower latency than NCCL on HPE Slingshot and InfiniBand interconnects (Singhania et al., 12 Nov 2025). A tabulated Perlmutter result gives typical speedups of 1.3x at 128 KB, 1.7x at 256 KB, 2.4x at 512 KB, and 3.1x at 1 MB (Singhania et al., 12 Nov 2025).

More detailed examples are also provided. On Perlmutter at 256 KB, NVRAR is reported as 1.06–1.44x faster beyond 8 GPUs, and up to 1.92x at 1024 KB. On Vista, for 256 KB–1 MB, the reported speedups are 1.08–3.5x beyond 4 GPUs (Singhania et al., 12 Nov 2025).

The algorithm is integrated into YALIS, a research-oriented prototype inference engine used for controlled experimentation (Singhania et al., 12 Nov 2025). In this setting, the NCCL all-reduce in YALIS’s tensor-parallel pipeline is replaced with NVRAR while maintaining identical CUDA-graph capture (Singhania et al., 12 Nov 2025). The workload is Llama 3.1 405B inference under decode-heavy settings with prompt length 1426, decode length 3072, and batch sizes $G$1 (Singhania et al., 12 Nov 2025).

The reported end-to-end gains are substantial for the target regime. For the 405B model on 32 A100s, the source reports a relative speedup of 1.17x at $G$2 and 1.72x at $G$3 (Singhania et al., 12 Nov 2025). On Vista, it reports up to 1.92x for the 70B model with batch 32 on 16 GH200s (Singhania et al., 12 Nov 2025). The paper summarizes this result as “up to a 1.72x reduction in end-to-end batch latency for the Llama 3.1 405B model in multi-node decode-heavy workloads using tensor parallelism” (Singhania et al., 12 Nov 2025).

Setting	Reported result	Source
Small-message all-reduce latency	1.9x–3.6x lower latency than NCCL	(Singhania et al., 12 Nov 2025)
405B, batch $G$4	1.17x end-to-end speedup	(Singhania et al., 12 Nov 2025)
405B, batch $G$5	1.72x end-to-end speedup	(Singhania et al., 12 Nov 2025)
70B on Vista, batch 32	up to 1.92x end-to-end speedup	(Singhania et al., 12 Nov 2025)

These measurements indicate that the gains observed in isolated collectives carry over, at least partially, to full inference pipelines when the workload is decode-heavy and tensor-parallel.

6. Scope, limitations, and relation to broader inference systems

The paper identifies several limitations. For very small messages, specifically 64 KB–128 KB, NVRAR microbenchmarks can be slower than NCCL because of intra-node all-gather kernel launch overheads and sequence-number waiting, although the source adds that these overheads diminish in real-workload CUDA graphs (Singhania et al., 12 Nov 2025). A marginal increase in GPU idle time is also reported when replacing NCCL with NVRAR, attributed in part to residual synchronization points (Singhania et al., 12 Nov 2025).

NVRAR currently targets flat node counts that are powers of two; non-power-of-two support is described as requiring more complex peer mapping (Singhania et al., 12 Nov 2025). The source also frames the present implementation as a point in a larger design space, listing possible extensions that include optimizing the intra-node all-gather path, extending to non-power-of-two node counts and heterogeneous topologies, integrating communication compression or quantization before recursive doubling, overlapping communication and computation by pipelining multi-layer all-reduces, and generalizing to other collectives such as broadcast and all-gather across nodes (Singhania et al., 12 Nov 2025).

Within the broader literature represented in the accompanying corpus, NVRAR belongs to a class of systems work that treats latency as a first-order constraint in immersive or interactive computation. The supplied materials include, for example, foveated NeRF rendering for virtual reality, which is formulated around an overview latency budget of approximately 24 ms and reports 20 ms stereo rendering under full stereo foveation (Deng et al., 2021), and on-device NeRF acceleration for AR/VR that targets reconstruction time below 5 seconds and reports 1.6 seconds per scene at 1.9 W (Li et al., 2023). This suggests a common systems perspective: interactive AI workloads are often bottlenecked not by peak throughput alone but by latency-critical subroutines that must be co-designed with the execution substrate. In NVRAR, that subroutine is all-reduce rather than rendering or reconstruction.

A common misconception would be to read NVRAR as a general claim that recursive doubling is universally superior to NCCL. The source does not make that claim. Its empirical and analytical case is explicitly tied to multi-node distributed inference, small messages, and decode-heavy tensor-parallel workloads (Singhania et al., 12 Nov 2025). Another possible confusion arises from the acronym itself: in the provided corpus, “NVRAR” also appears as shorthand for an NVRAM atomicity-and-recoverability context associated with persistent-stack execution and recoverable linearizability (Aksenov et al., 2021). In the distributed-inference literature, however, NVRAR refers specifically to the NVSHMEM-based recursive all-reduce algorithm of the 2025 study (Singhania et al., 12 Nov 2025).