---
title: 'NVRAR: NVSHMEM-based Recursive All-reduce'
url: https://www.emergentmind.com/topics/nvrar
type: topic
---

# NVRAR: NVSHMEM-based Recursive All-reduce

Searching arXiv for the NVRAR paper and closely related distributed LLM inference / all-reduce work.
NVRAR is a hierarchical all-reduce algorithm for multi-node GPU inference, introduced in the context of distributed large language model inference as an NVSHMEM-based recursive-doubling design that targets the small-message, latency-bound regime characteristic of tensor-parallel decode workloads. In the source study, NVRAR is presented as a mitigation for a common bottleneck in multi-node model-parallel inference: per-layer all-reduce of hidden-state fragments of size $B \times H$, especially in the 128 KB–2 MB range. Its design combines intra-node collectives with an inter-node recursive-doubling phase, and it is integrated into the YALIS research inference engine to preserve GPU-initiated execution and CUDA-graph compatibility while reducing end-to-end latency [2511.09557].

## 1. Definition and problem setting

NVRAR, expanded in the source as “NVSHMEM-based Recursive All-reduce,” is a hierarchical all-reduce algorithm based on recursive doubling with NVSHMEM [2511.09557]. It is motivated by the observation that, when running large-language-model inference in tensor-parallel mode across multiple nodes, the per-layer all-reduce of hidden-state fragments becomes the dominant latency contributor in the decode phase [2511.09557].

The algorithm is positioned against two established families of collective communication. Standard NCCL ring all-reduce incurs an $O(N)$ inter-node startup cost, reported as $2(N \cdot G-1)\cdot \alpha_{\mathrm{inter}}$ for $N$ nodes and $G$ GPUs per node, while tree all-reduce has $O(\log N)$ startup cost but higher constant factors in each step [2511.09557]. The source also notes that MPI implementations based on recursive doubling often outperform NCCL on small messages, but lack GPU-initiated communication, CUDA-graph support, or optimized NVLink paths [2511.09557]. NVRAR is designed to bridge that gap.

Its stated goals are threefold: achieve $O(\log N)$ startup latency proportional to $\alpha \log N$ rather than $O(N)$, optimize the small-message regime from 128 KB to 2 MB, and expose a GPU-initiated, CUDA-graph–friendly kernel [2511.09557]. This suggests that NVRAR is best understood not as a general replacement for all all-reduce implementations, but as a specialization for decode-heavy distributed inference where message sizes are modest and startup costs dominate.

## 2. Hierarchical three-phase structure

NVRAR decomposes the all-reduce of a tensor $M$ across $N$ nodes with $G$ GPUs per node into three phases [2511.09557].

First, an intra-node reduce-scatter is performed using `nvshmemx_*_sum_reducescatter`, described as internally NCCL-based. This reduces the $G$ fragments of $M$ on each node into $G$ local chunks of size $|M|/G$ [2511.09557].

Second, an inter-node recursive-doubling phase exchanges data among GPUs of the same local rank across nodes. For $i = 0 \ldots \log_2 N - 1$, each GPU index $r_g$ on node $r_n$ exchanges its current chunk with peer $(r_n \oplus 2^i,\; r_g)$ using non-blocking `put_nbi` of fused data-plus-flag words of 8 B. Upon receipt, the remote chunk is immediately added into the local buffer. Synchronization is handled via sequence numbers carried in the fused payload, thereby avoiding explicit NVSHMEM `fence` and `quiet` calls [2511.09557].

Third, an intra-node all-gather reconstructs the full tensor on each GPU using `nvshmemx_*_all_gather`, again described as NCCL-based [2511.09557].

This organization yields a mixed communication topology. Within each node, NVRAR uses a ring-based reduce-scatter and all-gather. Across nodes, it uses a flat recursive-doubling tree among GPUs of the same local rank, which the source states enables concurrent NIC usage when multiple NICs are available [2511.09557]. A plausible implication is that the hierarchy is intended to preserve fast intra-node paths while replacing the inter-node portion of the collective with a lower-startup alternative.

## 3. Algorithmic mechanics and synchronization model

The source presents NVRAR in LaTeX-style pseudocode, centered on an initial reduce-scatter, a sequence-number increment, inter-node synchronization with peers, packing into fused buffers, recursive doubling across chunks, unpacking, and a final all-gather [2511.09557].

At the top level, the algorithm performs:
- `REDUCE_SCATTER_intra(M,G)` to produce $M'$,
- a sequence-number increment,
- peer synchronization over $\log_2 N$ rounds,
- `Pack(M', seq)`,
- `RD_inter(...)`,
- `Unpack(...)`,
- and `ALL_GATHER_intra(M', G)` [2511.09557].

Within `RD_inter`, the buffer is divided into $Q=\lceil |B_{\text{send}}|/C \rceil$ chunks. For each recursive-doubling level $\ell$, the algorithm computes the peer $(r_n \oplus 2^\ell,\; r_g)$, issues `NonBlockingPut` operations for each chunk, waits on the corresponding flag with `WaitFlag`, and then forms the next-stage send buffer by reduction:
$$
B_{\text{send}[\ell+1][q]} \gets B_{\text{recv}[\ell][q]} + B_{\text{send}[\ell][q]}.
$$
This formulation makes the dataflow explicit: the algorithm progresses by pairwise exchange-and-reduce over successively larger logical groups of nodes [2511.09557].

A notable design choice is the use of fused data-plus-flag words and sequence numbers rather than explicit NVSHMEM fence/quiet synchronization [2511.09557]. The paper attributes this to avoiding higher synchronization overhead. It also notes that NVSHMEM’s `put_with_signal` was avoided in favor of fused flags, while suggesting that future NVSHMEM libfabric improvements could reduce overhead further [2511.09557].

## 4. Latency model and asymptotic properties

The source analyzes NVRAR using the $\alpha$–$\beta$ model, with startup cost $\alpha$ and per-byte cost $1/\beta$ [2511.09557]. For the intra-node ring reduce-scatter and all-gather, the reported costs are
$$
T_{\mathrm{RS}} = (G-1)\,\alpha_{\mathrm{intra}} + \frac{G-1}{G}\frac{|M|}{\beta_{\mathrm{intra}}},
$$
$$
T_{\mathrm{AG}} = (G-1)\,\alpha_{\mathrm{intra}} + \frac{G-1}{G}\frac{|M|}{\beta_{\mathrm{intra}}}.
$$

For the inter-node recursive-doubling phase, with $\log_2 N$ steps, message size $|M|/G$, and an inflation factor $\eta \in (1,2)$ due to fused headers, the reported latency is
$$
T_{\mathrm{RD}}
= \log_2(N)\,\alpha_{\mathrm{inter}}
+ \frac{N-1}{N}\,\frac{\eta\,|M|}{G\,\beta_{\mathrm{inter}}}.
$$

Summing the three phases gives the total all-reduce time:
$$
\begin{aligned}
T_{\mathrm{NVRAR}}
&= 2(G-1)\,\alpha_{\mathrm{intra}}
+ \log_2(N)\,\alpha_{\mathrm{inter}} \\
&\quad + \frac{|M|}{G}\Bigl[\tfrac{2(G-1)}{\beta_{\mathrm{intra}}}
+ \tfrac{(N-1)\,\eta}{N\,\beta_{\mathrm{inter}}}\Bigr].
\end{aligned}
$$

For small, latency-bound messages, the source states that the bandwidth terms drop out, yielding the approximation
$$
T_{\mathrm{NVRAR}} \approx 2(G-1)\,\alpha_{\mathrm{intra}} + \log_2(N)\,\alpha_{\mathrm{inter}}.
$$
This is contrasted with NCCL’s ring, characterized as $O(NG\,\alpha_{\mathrm{inter}})$, and NCCL’s tree, characterized as $O(\log N)$ but with $2\log_2(N)$ steps and higher constants [2511.09557].

The significance of this model lies in its fit to decode-heavy inference. The source specifically targets hidden-state all-reduces in the 128 KB–2 MB regime, and the theoretical discussion treats this as a startup-dominated operating point rather than a bandwidth-dominated one [2511.09557].

## 5. Empirical performance and integration into YALIS

The paper reports standalone microbenchmark results on Perlmutter, described as A100/Slingshot-11, and Vista, described as GH200/InfiniBand [2511.09557]. For message sizes in the 128 KB–2 MB regime, NVRAR achieves 1.9x–3.6x lower latency than NCCL on HPE Slingshot and InfiniBand interconnects [2511.09557]. A tabulated Perlmutter result gives typical speedups of 1.3x at 128 KB, 1.7x at 256 KB, 2.4x at 512 KB, and 3.1x at 1 MB [2511.09557].

More detailed examples are also provided. On Perlmutter at 256 KB, NVRAR is reported as 1.06–1.44x faster beyond 8 GPUs, and up to 1.92x at 1024 KB. On Vista, for 256 KB–1 MB, the reported speedups are 1.08–3.5x beyond 4 GPUs [2511.09557].

The algorithm is integrated into YALIS, a research-oriented prototype inference engine used for controlled experimentation [2511.09557]. In this setting, the NCCL all-reduce in YALIS’s tensor-parallel pipeline is replaced with NVRAR while maintaining identical CUDA-graph capture [2511.09557]. The workload is Llama 3.1 405B inference under decode-heavy settings with prompt length 1426, decode length 3072, and batch sizes $\#P=8/32$ [2511.09557].

The reported end-to-end gains are substantial for the target regime. For the 405B model on 32 A100s, the source reports a relative speedup of 1.17x at $\#P=8$ and 1.72x at $\#P=32$ [2511.09557]. On Vista, it reports up to 1.92x for the 70B model with batch 32 on 16 GH200s [2511.09557]. The paper summarizes this result as “up to a 1.72x reduction in end-to-end batch latency for the Llama 3.1 405B model in multi-node decode-heavy workloads using tensor parallelism” [2511.09557].

| Setting | Reported result | Source |
|---|---:|---|
| Small-message all-reduce latency | 1.9x–3.6x lower latency than NCCL | [2511.09557] |
| 405B, batch $\#P=8$ | 1.17x end-to-end speedup | [2511.09557] |
| 405B, batch $\#P=32$ | 1.72x end-to-end speedup | [2511.09557] |
| 70B on Vista, batch 32 | up to 1.92x end-to-end speedup | [2511.09557] |

These measurements indicate that the gains observed in isolated collectives carry over, at least partially, to full inference pipelines when the workload is decode-heavy and tensor-parallel.

## 6. Scope, limitations, and relation to broader inference systems

The paper identifies several limitations. For very small messages, specifically 64 KB–128 KB, NVRAR microbenchmarks can be slower than NCCL because of intra-node all-gather kernel launch overheads and sequence-number waiting, although the source adds that these overheads diminish in real-workload CUDA graphs [2511.09557]. A marginal increase in GPU idle time is also reported when replacing NCCL with NVRAR, attributed in part to residual synchronization points [2511.09557].

NVRAR currently targets flat node counts that are powers of two; non-power-of-two support is described as requiring more complex peer mapping [2511.09557]. The source also frames the present implementation as a point in a larger design space, listing possible extensions that include optimizing the intra-node all-gather path, extending to non-power-of-two node counts and heterogeneous topologies, integrating communication compression or quantization before recursive doubling, overlapping communication and computation by pipelining multi-layer all-reduces, and generalizing to other collectives such as broadcast and all-gather across nodes [2511.09557].

Within the broader literature represented in the accompanying corpus, NVRAR belongs to a class of systems work that treats latency as a first-order constraint in immersive or interactive computation. The supplied materials include, for example, foveated NeRF rendering for virtual reality, which is formulated around a synthesis latency budget of approximately 24 ms and reports 20 ms stereo rendering under full stereo foveation [2103.16365], and on-device NeRF acceleration for AR/VR that targets reconstruction time below 5 seconds and reports 1.6 seconds per scene at 1.9 W [2304.12467]. This suggests a common systems perspective: interactive AI workloads are often bottlenecked not by peak throughput alone but by latency-critical subroutines that must be co-designed with the execution substrate. In NVRAR, that subroutine is all-reduce rather than rendering or reconstruction.

A common misconception would be to read NVRAR as a general claim that recursive doubling is universally superior to NCCL. The source does not make that claim. Its empirical and analytical case is explicitly tied to multi-node distributed inference, small messages, and decode-heavy tensor-parallel workloads [2511.09557]. Another possible confusion arises from the acronym itself: in the provided corpus, “NVRAR” also appears as shorthand for an NVRAM atomicity-and-recoverability context associated with persistent-stack execution and recoverable linearizability [2105.11932]. In the distributed-inference literature, however, NVRAR refers specifically to the NVSHMEM-based recursive all-reduce algorithm of the 2025 study [2511.09557].

Source: https://www.emergentmind.com/topics/nvrar