Ring AllReduce: Distributed Gradient Sync

• Ring AllReduce is a master-less collective algorithm that aggregates gradients via a logical ring using reduce-scatter and all-gather phases.
• It minimizes bandwidth bottlenecks by enabling each node to communicate only with its immediate neighbors, thus avoiding centralized congestion.
• Its performance is sensitive to slow or imbalanced nodes and topology, with advanced variants addressing synchronization delays and straggler impacts.

Ring AllReduce is a master-less collective communication algorithm for exact global aggregation in which participants are arranged in a logical ring and obtain the full reduced result through a reduce-scatter followed by an all-gather. In distributed learning it is treated as a standard bandwidth-efficient primitive for synchronous gradient synchronization because it avoids concentrating traffic at a parameter server, while the same literature also emphasizes its strict dependence on synchronized progress and its sensitivity to slow, delayed, or degraded participants (Reisizadeh et al., 2019, Liu et al., 2020, Devraj et al., 29 May 2025).

1. Canonical algorithmic form

In the synchronous distributed gradient-descent setting, the aggregate computed at iteration $t$ is the full gradient

$$g = \sum_{x \in D} \nabla \ell \left(\theta^{(t)}; x \right),$$

which is then used in the model update

$$\theta^{(t+1)} = h_R \left( \theta^{(t)}, g \right).$$

When the dataset $D$ is too large for one machine, it is distributed across workers. In Ring AllReduce, the system is master-less: there is no parameter server or master collecting all gradients. Instead, the dataset is uniformly distributed over $N$ workers $\{W_1,\dots,W_N\}$, and the workers coordinate among themselves so that every worker ends with the aggregate $g$ (Reisizadeh et al., 2019).

The standard procedure is a logical-ring implementation. Each worker first computes its local partial gradient and partitions it into $N$ segments,

$v_{1,i}, \ldots, v_{N,i}.$

During the reduce-scatter phase, workers repeatedly send one segment to the next neighbor, receive the corresponding segment from the previous neighbor, and accumulate it into the matching local segment. After $N-1$ rounds, each worker holds one unique reduced segment of the full gradient. An all-gather phase then circulates the reduced segments so that every worker reconstructs the full aggregate. In the standard cost model used for Ring in GPU clusters, the collective consists of a ReduceScatter followed by an AllGather and requires $$g = \sum_{x \in D} \nabla \ell \left(\theta^{(t)}; x \right),$$0 rounds total (Reisizadeh et al., 2019, Devraj et al., 29 May 2025).

This two-phase decomposition is also the form used in standard DNN-training descriptions of ring all-reduce: with $$g = \sum_{x \in D} \nabla \ell \left(\theta^{(t)}; x \right),$$1 workers and tensor size $$g = \sum_{x \in D} \nabla \ell \left(\theta^{(t)}; x \right),$$2, each node splits the data into $$g = \sum_{x \in D} \nabla \ell \left(\theta^{(t)}; x \right),$$3 pieces, sends one chunk of size $$g = \sum_{x \in D} \nabla \ell \left(\theta^{(t)}; x \right),$$4 to a neighbor at each step, and after the scatter-reduce and all-gather phases every worker has the full reduced tensor (Liu et al., 2020).

2. Communication model and efficiency claims

The central systems motivation for Ring AllReduce is the elimination of centralized communication concentration. In a classical master-worker design, all workers send their results to a single master, creating what one paper calls “bandwidth congestion at the master due to concurrent communications from the workers to the master.” Ring AllReduce avoids that concentration because each worker communicates only with its neighbors on the ring; the literature therefore describes it as avoiding “bandwidth bottleneck at any particular node” and alleviating communication congestion at busy nodes (Reisizadeh et al., 2019).

Under the standard $$g = \sum_{x \in D} \nabla \ell \left(\theta^{(t)}; x \right),$$5-$$g = \sum_{x \in D} \nabla \ell \left(\theta^{(t)}; x \right),$$6 communication model, Ring is commonly written as

$$g = \sum_{x \in D} \nabla \ell \left(\theta^{(t)}; x \right),$$7

where sending $$g = \sum_{x \in D} \nabla \ell \left(\theta^{(t)}; x \right),$$8 bytes costs $$g = \sum_{x \in D} \nabla \ell \left(\theta^{(t)}; x \right),$$9. The bandwidth term approaches $$\theta^{(t+1)} = h_R \left( \theta^{(t)}, g \right).$$0 as $$\theta^{(t+1)} = h_R \left( \theta^{(t)}, g \right).$$1 grows, so each rank sends only about $$\theta^{(t+1)} = h_R \left( \theta^{(t)}, g \right).$$2 sequential bytes in the large-$$\theta^{(t+1)} = h_R \left( \theta^{(t)}, g \right).$$3 limit even though the schedule has many rounds. This is the sense in which the algorithm is treated as bandwidth-efficient for large buffers (Devraj et al., 29 May 2025).

A closely related formulation in distributed DNN training writes the communication time as

$$\theta^{(t+1)} = h_R \left( \theta^{(t)}, g \right).$$4

with $$\theta^{(t+1)} = h_R \left( \theta^{(t)}, g \right).$$5 workers, message size $$\theta^{(t+1)} = h_R \left( \theta^{(t)}, g \right).$$6, per-message latency $$\theta^{(t+1)} = h_R \left( \theta^{(t)}, g \right).$$7, and bandwidth $$\theta^{(t+1)} = h_R \left( \theta^{(t)}, g \right).$$8. In this model, each node transmits $$\theta^{(t+1)} = h_R \left( \theta^{(t)}, g \right).$$9, which tends to almost $D$0 as $D$1 becomes large. The same source characterizes ring all-reduce as bandwidth-optimal and capable of contention-free communication, which explains why it became, in that account, the “most popular algorithm” after Baidu introduced it (Liu et al., 2020).

Order-wise comparisons in distributed-learning work make the same point in a different language. One summary assigns Ring AllReduce straggler resiliency $D$2 and communication parallelization gain $D$3, describing that communication gain as optimal. In that framing, Ring AllReduce is the communication-optimal point among the compared synchronous aggregation schemes (Reisizadeh et al., 2019).

3. Synchronization, stragglers, and imbalanced arrivals

The canonical weakness of Ring AllReduce is its sensitivity to slow participants. One source states this directly: Ring AllReduce “cannot tolerate any straggling nodes since the communications are carried out over a ring and each node requires its neighbor’s result to proceed in the ring,” and therefore assigns it straggler resiliency

$D$4

A later study of persistent stragglers sharpens the same criticism: bulk-synchronous AllReduce libraries impose a barrier before the collective starts, so a GPU that repeatedly arrives late causes all non-stragglers to idle before the ring begins. In that setting, Ring is efficient once the collective starts, but the barrier makes it unable to exploit the slack that exists before the straggler reaches synchronization (Reisizadeh et al., 2019, Devraj et al., 29 May 2025).

This dependence on synchronized arrival had already motivated process-arrival-aware variants. For imbalanced process arrival patterns, the “pre-reduced ring” algorithm performs reducing pre-steps between faster processes before regular ring processing begins. In the reported experiments, the proposed methods become up to $D$5 faster than regular ring for sufficiently imbalanced arrivals, and in a CIFAR-10 training example the pre-reduced ring reduced average all-reduce elapsed time by $D$6 and total training time by $D$7 relative to ring (Proficz, 2018).

The same synchronization fragility appears inside the network. Symphony studies ring-based collectives under network jitter and congestion and identifies step misalignment—loss of lockstep progression across pipeline stages—as a first-class bottleneck. Its switch-based mechanism tracks per-job step progress and selectively ECN-throttles outpacing flows; in simulation it reports up to $D$8 improvement in job/collective communication time (Jin et al., 18 Apr 2026). A related line of work studies asymmetric bandwidth after NIC or network failures: modern libraries can reroute traffic through surviving NICs, but a degraded server then remains on the critical path of the standard ring. For up to $D$9 bandwidth loss, OptCC completes AllReduce within $N$0–$N$1 of NCCL’s fault-free ring performance, whereas the cited state of the art incurs up to $N$2 overhead (Chen et al., 1 Jun 2026).

4. Formal generalizations and non-ring alternatives

Ring AllReduce has also been formalized as one point in a larger algorithmic design space. A permutation-group treatment describes communication patterns as elements of a transitive abelian subgroup acting on the process set and shows that Ring is the cyclic special case: repeated application of a single generator yields the usual two-phase schedule in $N$3 steps, which the paper identifies as “the exact definition of the Ring algorithm.” The same framework recovers Recursive Halving and Recursive Doubling as different group choices and supports arbitrary-$N$4 algorithms with step counts between $N$5 and $N$6 (Kolmakov et al., 2020).

A more recent alternative replaces ring reduce-scatter with a non-pipelined circulant-graph algorithm. For reduce-scatter it completes in $N$7 communication rounds while each processor still sends, receives, and reduces exactly $N$8 blocks; when combined with a matching allgather, the resulting allreduce uses $N$9 rounds and still sends and receives exactly $\{W_1,\dots,W_N\}$0 blocks per process. This formulation assumes that the reduction operator is commutative (Träff, 2024).

Ring nonetheless remains the reference large-message algorithm in other collective frameworks. In GASPI-based work on eventually consistent collectives, the “classic/consistent” implementation chosen for large messages is explicitly a segmented pipelined ring Allreduce built from scatter-reduce and allgather over one-sided writes and notifications, while the relaxed SSP variant switches to a hypercube-style design rather than modifying the ring itself (Iakymchuk et al., 2022).

5. Topology-aware reinterpretations

Whether Ring AllReduce is preferable depends strongly on the physical topology and the performance model used. On ring-based GPU-to-GPU interconnects, “Short-circuiting Rings for Low-Latency AllReduce” argues that the usual rule of thumb—Recursive Doubling for small messages, Ring for large ones—can fail once realistic propagation delays and link-capacity constraints are modeled. In that analysis, cumulative propagation delay on a static ring is essentially the same for Ring and Recursive Doubling, while Recursive Doubling suffers substantially higher congestion; under those assumptions, Ring can remain optimal even for short messages (Hammer et al., 3 Oct 2025).

On torus and torus-like networks, the picture changes. Swing argues that standard Ring has ideal congestion behavior but suffers from a very large latency deficiency because it needs $\{W_1,\dots,W_N\}$1 steps, and reports up to $\{W_1,\dots,W_N\}$2 improvement over existing allreduce algorithms for vectors ranging from $\{W_1,\dots,W_N\}$3B to $\{W_1,\dots,W_N\}$4MiB by “short-cutting” the ring on torus topologies (Sensi et al., 2024). Trivance makes a complementary point for bidirectional rings and multidimensional tori: it completes AllReduce in $\{W_1,\dots,W_N\}$5 steps on a bidirectional ring, reduces congestion compared to Bruck’s algorithm by a factor of three, preserves bandwidth-optimality, and improves state-of-the-art approaches by $\{W_1,\dots,W_N\}$6–$\{W_1,\dots,W_N\}$7 in the reported evaluations (Juerss et al., 19 Feb 2026).

In optical reconfigurable networks, Bridge argues that Ring’s bandwidth optimality is incomplete as a completion-time criterion once reconfiguration delay, hop count, and congestion are explicit parts of the model. By reusing optical subrings across multiple future steps, Bridge reports up to $\{W_1,\dots,W_N\}$8 speedup for AllReduce and exceeds the bandwidth-optimal Ring algorithm by $\{W_1,\dots,W_N\}$9 to $g$0 on low to moderate-sized workloads (Juerss et al., 12 May 2026).

6. In-network, hierarchical, and cloud-oriented adaptations

Several systems papers keep Ring AllReduce as the baseline but alter the network substrate around it. NetReduce contrasts in-network reduction with ring all-reduce, which it describes as the most popular bandwidth-optimal and contention-free baseline. In the single-GPU-per-machine model, it compares

$g$1

with

$g$2

and reports training acceleration up to $g$3 for CNN-based CV tasks and $g$4 for transformer-based NLP tasks, with simulations indicating superior scalability to state-of-the-art ring all-reduce (Liu et al., 2020).

Rina takes the opposite direction: instead of replacing ring with a parameter-server-like in-network design, it incorporates in-network aggregation into Ring-AllReduce itself. Its agent-worker mechanism makes all workers in a rack appear as one abstracted worker when an INA-capable ToR switch is deployed, thereby shortening the dependency chain of the ring. The paper reports that, compared with the state-of-the-art PS-based INA method ATP, Rina can achieve more than $g$5 throughput with the same hardware cost, and compared with traditional RAR and PS it can improve throughput by up to $g$6 (Chen et al., 2024).

Cloud-oriented work departs further from exact ring semantics. OptiReduce presents a bounded-time approximate AllReduce for shared cloud environments that uses Transpose AllReduce, unreliable bounded transport with adaptive timeout, and Hadamard Transform to tolerate approximate or lost gradients. Its abstract reports $g$7 and $g$8 faster time-to-accuracy, on average, compared to Gloo and NCCL, respectively, in shared cloud environments (Warraich et al., 2023).

7. Implementations, applications, and emerging directions

In practical distributed-learning experiments, Ring AllReduce often appears through standard communication libraries. In convex optimization experiments over Amazon EC2, one work implements the ring baseline with MPI_Allreduce() in a Python/mpi4py environment and repeatedly treats it as the standard bandwidth-efficient comparator for coded and uncoded aggregation schemes (Reisizadeh et al., 2019). In HPC communication middleware, GASPI-based segmented pipelined ring Allreduce is explicitly proposed for large vectors, and for $g$9 elements it reports speedups of $N$0 versus Shumilin’s ring and $N$1 versus ring; at $N$2 elements it reports $N$3 versus MPI ring (Iakymchuk et al., 2022).

Ring-specific compression techniques modify the payload rather than the schedule. “Bandwidth Reduction using Importance Weighted Pruning on Ring AllReduce” argues that naïve sparse top-$N$4 communication becomes denser as sparse supports merge around the ring, and instead uses shared masks based on $N$5. The reported compression factors are $N$6 on AlexNet and $N$7 on ResNet50 on ImageNet while preserving the reported training accuracy (Cheng et al., 2019).

The ring abstraction has also been reused outside conventional DNN gradient synchronization. In distributed quantum Monte Carlo, a GPU-RDMA ring communication algorithm circulates rank-local buffers so that each rank updates only its own shard of a large tensor, reducing the allocation size of the most memory-intensive data structure per GPU to $N$8 of the original size, where $N$9 is the number of GPUs in the ring communicator (Wei et al., 2021). At the opposite extreme, “Quantum ring all-reduce” proposes a quantum version that keeps the ring schedule but replaces classical transmission with entanglement-assisted communication: with pre-shared entanglement and superdense coding it reduces per-link online communication by a provably optimal factor of two, and with verified GHZ states it adds composable $v_{1,i}, \ldots, v_{N,i}.$0-secure aggregation at a $v_{1,i}, \ldots, v_{N,i}.$1 overhead in GHZ copies (Garcés et al., 18 Jun 2026).

Across these literatures, Ring AllReduce remains the canonical bandwidth-oriented reference point: a decentralized exact collective whose strengths are balanced communication and simple nearest-neighbor structure, and whose weaknesses are synchronization rigidity, topology sensitivity, and vulnerability to tails unless the surrounding system compensates for them.