Papers
Topics
Authors
Recent
Search
2000 character limit reached

ScatterReduce in Distributed Systems

Updated 12 July 2026
  • ScatterReduce is a distributed reduce-scatter mechanism that partitions state among processors to perform local reductions and scatter partial results efficiently.
  • It supports multiple implementations including circulant algorithms, persistent MPI routines, serverless gradient aggregation, and sparse GPU adaptations.
  • Key challenges include ensuring commutativity, managing buffer constraints, and balancing load to minimize communication rounds and overhead.

ScatterReduce denotes a class of reduce-scatter mechanisms in which distributed state is partitioned, partially exchanged, reduced, and left scattered across participants rather than fully replicated. In the canonical collective-communication formulation, pp processors hold vectors VrV_r partitioned into pp blocks and compute the global reduction W=⨁r=0p−1VrW=\bigoplus_{r=0}^{p-1} V_r such that processor rr finally holds only W[r]=⨁i=0p−1Vi[r]W[r]=\bigoplus_{i=0}^{p-1} V_i[r] (Träff, 2024). The name also appears in recent literature for a serverless distributed-training protocol that scatters gradient chunks through shared storage and for optimized or sparse reduce_scatter implementations in MPI and GPU communication stacks (Barrak et al., 18 Sep 2025, Jocksch et al., 2020, Hough et al., 6 Jul 2026). Across these usages, the central design question is how to minimize rounds, transferred volume, synchronization stalls, or format-conversion overhead while preserving the reduction semantics.

1. Semantic core and scope

The reduce-scatter operation is important both in its own right and as building block for other collective operations. With processors numbered 0,1,…,p−10,1,\dots,p-1, each processor rr starts with a vector VrV_r of mm elements partitioned into pp disjoint blocks of equal size m/pm/p:

Vr=[Vr[0],Vr[1],…,Vr[p−1]].V_r = [V_r[0], V_r[1], \dots, V_r[p-1]].

The operation applies an associative, commutative binary operator ⊕\oplus blockwise, but stores block W[i]W[i] only on processor ii rather than materializing the full reduced vector everywhere (Träff, 2024).

The literature uses “ScatterReduce” in more than one sense. In message-passing systems it can denote an optimized reduce_scatter routine, including persistent MPI implementations based on recursive multiply/divide and cyclic shift, or a simple non-pipelined circulant schedule that is optimal in both rounds and communicated volume (Jocksch et al., 2020, Träff, 2024). In distributed ML it denotes a protocol in which each worker partitions its gradient vector into equally sized chunks, writes nonlocal chunks to shared storage, reduces one assigned chunk, and then gathers the aggregated chunks for the model update (Barrak et al., 18 Sep 2025). On GPU platforms, sparse reduce-scatter variants exploit unstructured sparsity by compressing data and adaptively switching between dense and sparse representations during ring-style communication (Hough et al., 6 Jul 2026).

Usage Core mechanism Reported property
Reduce-scatter collective Partition VrV_r into pp blocks and compute only the local reduced block ⌈log⁡2p⌉\lceil \log_2 p\rceil-round optimal circulant schedule (Träff, 2024)
Persistent MPI ScatterReduce Recursive multiply/divide or cyclic shift with multi-port tuning and rank reordering Factors of $2$–$6$ over standard MPI for small to medium blocks (Jocksch et al., 2020)
Serverless ScatterReduce Fetch, compute-and-scatter, reduce, gather-update via Redis or S3 Roughly 3×3\times gradient traffic per iteration (Barrak et al., 18 Sep 2025)
Sparse GPU reduce-scatter Adaptive ring using Pici bitvector format Up to 2.5×2.5\times over NCCL at 99%99\% sparsity (Hough et al., 6 Jul 2026)
PAT reduce-scatter Truncated reversed-dimension binomial tree plus pipeline Logarithmic latency when the intermediate buffer is large enough (Jeaugey, 25 Jun 2025)

This breadth of usage is not terminological drift alone; it reflects the fact that the same semantic primitive is instantiated under markedly different systems assumptions.

2. Non-pipelined circulant algorithms and optimality

A particularly compact formulation is the circulant reduce-scatter algorithm of Träff. It defines q=⌈log⁡2p⌉q=\lceil \log_2 p\rceil and a skip sequence

s0=p,sk+1=⌈sk/2⌉,k=0,…,q−1,s_0=p,\qquad s_{k+1}=\lceil s_k/2\rceil,\qquad k=0,\dots,q-1,

with sq=1s_q=1. In round kk, processor rr sends to t=(r+sk+1) mod pt=(r+s_{k+1})\bmod p, receives from f=(r−sk+1+p) mod pf=(r-s_{k+1}+p)\bmod p, transmits the consecutive run of blocks R[sk+1…sk−1]R[s_{k+1}\dots s_k-1], receives equally many blocks, and locally reduces the received data into R[0…sk−sk+1−1]R[0\dots s_k-s_{k+1}-1] (Träff, 2024).

The local state is initialized as rotated partial sums,

R[i]←V(r+i) mod p[r],i=0,…,p−1,R[i]\leftarrow V_{(r+i)\bmod p}[r],\qquad i=0,\dots,p-1,

so that R[i]R[i] is the running partial sum that eventually contributes to W[(r+i) mod p]W[(r+i)\bmod p]. After qq rounds, each processor retains R[0]=W[r]R[0]=W[r]. The communication pattern is a simple, ⌈log⁡2p⌉\lceil\log_2 p\rceil-regular, circulant graph also used elsewhere (Träff, 2024).

The cost profile is explicit. The number of rounds is q=⌈log⁡2p⌉q=\lceil\log_2 p\rceil. In round kk each processor sends and receives exactly sk−sk+1s_k-s_{k+1} blocks, and summing over all rounds yields

∑k=0q−1(sk−sk+1)=s0−sq=p−1.\sum_{k=0}^{q-1}(s_k-s_{k+1}) = s_0-s_q = p-1.

Hence each processor sends p−1p-1 blocks, receives p−1p-1 blocks, and applies ⊕\oplus exactly p−1p-1 times. In the linear-cost model with latency α\alpha and bandwidth term β\beta per block of size m/pm/p,

Tcomm=α⋅⌈log⁡2p⌉+β⋅((p−1)⋅m/p).T_{\mathrm{comm}}=\alpha\cdot\lceil\log_2 p\rceil+\beta\cdot((p-1)\cdot m/p).

A classic information-flow argument shows that reducing pp values to one requires at least ⌈log⁡2p⌉\lceil\log_2 p\rceil steps of pairwise combination, so the algorithm is optimal both in rounds and volume within that model (Träff, 2024).

The same construction yields allreduce by composing reduce-scatter with a standard ⌈log⁡2p⌉\lceil\log_2 p\rceil-round allgather. The resulting allreduce runs in 2⌈log⁡2p⌉2\lceil\log_2 p\rceil rounds, sends $2(p-1)$ blocks per processor, and has cost

Tallreduce=2α⋅⌈log⁡2p⌉+2β⋅((p−1)⋅m/p)+γ⋅((p−1)⋅m/p),T_{\mathrm{allreduce}}= 2\alpha\cdot\lceil\log_2 p\rceil +2\beta\cdot((p-1)\cdot m/p) +\gamma\cdot((p-1)\cdot m/p),

where γ\gamma is the per-block reduction cost (Träff, 2024). The same circulant pattern can also serve as a template for round-optimal all-to-all communication by taking “reduction” to be block concatenation.

A central caveat is the commutativity assumption. Because blocks are reduced in the order induced by the circulant skipping pattern rather than in rank order, correctness relies on ⊕\oplus being commutative as well as associative. For non-commutative operators, enforcing the same global sequence of inputs on each reduction tree is described as complex or expensive to arrange (Träff, 2024).

3. Persistent MPI ScatterReduce and installation-time tuning

In MPI libraries, optimized ScatterReduce has been formulated as a persistent communication routine built from two classical building blocks: recursive multiply/divide (“doubling/halving”) and cyclic-shift (Bruck’s algorithm). The optimization adds three features: multi-port execution with step-wise tuning, rank-reordering to balance non-uniform block sizes, and a bytecode-driven initialization/execution separation for persistent calls (Jocksch et al., 2020).

For recursive multiply/divide, the number of nodes pp is factored into radixes r1,…,rsr_1,\dots,r_s with ∏iri=p\prod_i r_i=p. In step ii each node exchanges (ri−1)(r_i-1) equal-sized subblocks in parallel with (ri−1)(r_i-1) partners and performs a local reduction. With equal radixes rr and equal blocks of size n/pn/p, the model is

Tcomm=α⋅s+(β+γ)⋅p−1r−1⋅(n/p).T_{\mathrm{comm}}=\alpha\cdot s+(\beta+\gamma)\cdot\frac{p-1}{r-1}\cdot(n/p).

For cyclic shift, the cost is identical up to a small local re-shuffle overhead (Jocksch et al., 2020).

Non-equal message sizes are handled by a rank-reordering heuristic. The implementation sorts blocks by size, pairs the largest remaining block with the smallest, the second largest with the second smallest, and so on, records a permutation π\pi, and applies π\pi to sources and destinations in every step. The stated goal is to equalize the communicated MiM_i in each step and on each port. The one-time setup cost is

Treorder=δ⋅p⋅(log⁡2p)2,T_{\mathrm{reorder}}=\delta\cdot p\cdot(\log_2 p)^2,

with δ\delta the per-comparison cost (Jocksch et al., 2020).

The complete routine has three phases: intra-node gather, inter-node ScatterReduce across pp “virtual” nodes, and intra-node scatter. During initialization, the library measures on-node α\alpha, β\beta, and γ\gamma, chooses optimal radixes and ports per step, computes the permutation π\pi, and emits bytecode consisting of (step,port,offset,length,reduce-flag)(\text{step},\text{port},\text{offset},\text{length},\text{reduce-flag}) instructions. Execution interprets this bytecode via MPI nonblocking point-to-point operations such as MPI_Irecv, MPI_Isend, and MPI_Waitall, followed by local_reduce or memcpy on received buffers (Jocksch et al., 2020).

Measured results show lower times than standard MPI baselines on both a 160-node KNL Cray XC40 and a 17-node Infiniband cluster. On the Cray XC40 at 160 nodes and 9600 tasks, for 8 B per final block the optimized routine is approximately 10 μs10\,\mu s versus approximately 60 μs60\,\mu s for Cray MPI, a ×6\times 6 speedup; for 4 KB per block it is approximately 150 μs150\,\mu s versus approximately 280 μs280\,\mu s, a ×1.8\times 1.8 improvement. On the Infiniband cluster at 17 nodes and 408 tasks, 8 B per block gives approximately 12 μs12\,\mu s versus approximately 25 μs25\,\mu s for MVAPICH, and 4 KB per block gives approximately 160 μs160\,\mu s versus approximately 230 μs230\,\mu s (Jocksch et al., 2020). In ORB5, where reduce_scatter and allgatherv have non-uniform block sizes, the pairing heuristic gives a further ∼20%\sim 20\% reduction in overall filter time versus no reordering, while the Cray MPI baseline is up to 3×3\times slower.

The implementation also emphasizes deterministic reduction order for bit-wise reproducibility. Reported limitations are uniform core-count per node and lack of support for non-contiguous datatypes or dynamic inter-node communicators (Jocksch et al., 2020).

4. ScatterReduce as a serverless gradient-aggregation protocol

In serverless distributed training, ScatterReduce is a four-stage iteration protocol: fetch, compute-and-scatter, reduce, and gather-update. Each of WW workers loads a minibatch and the current parameters into its local Lambda function, computes a local gradient vector g∈RDg\in\mathbb{R}^D, partitions it into WW contiguous chunks of size D/WD/W, retains chunk ii locally, and writes each g(j)g^{(j)} for j≠ij\neq i to shared storage under a key encoding iteration, chunk index, and source (Barrak et al., 18 Sep 2025).

In the reduce phase, worker kk waits until it can fetch exactly WW values g1(k),…,gW(k)g^{(k)}_1,\dots,g^{(k)}_W, sums them element-wise, and writes back the partial aggregate

h(k)=∑i=1Wgi(k).h^{(k)}=\sum_{i=1}^W g^{(k)}_i.

In the gather-update phase, every worker downloads all aggregated chunks, concatenates them into

H=[h(1)∥h(2)∥⋯∥h(W)],H=[h^{(1)}\parallel h^{(2)}\parallel \dots \parallel h^{(W)}],

and updates the model by

θ←θ−η⋅H.\theta \leftarrow \theta-\eta\cdot H.

This distributes the aggregation load evenly across workers rather than concentrating it in a single master (Barrak et al., 18 Sep 2025).

The communication volume is substantially higher than the logical gradient size. If G=D⋅sizeof(float)G=D\cdot\text{sizeof(float)} is the full gradient size in bytes, then per worker

Ccomm=G⋅(1−1/W)+G⋅(1+1/W)+G=3G+O(G/W).C_{\mathrm{comm}} = G\cdot(1-1/W) +G\cdot(1+1/W) +G = 3G+O(G/W).

Aggregated across all workers, the total network traffic is O(WG)O(WG). Under the α\alpha–β\beta model, the iteration wall-clock overhead is on the order of

Titer=O(W⋅α+β⋅G),T_{\mathrm{iter}}=O(W\cdot\alpha+\beta\cdot G),

and the extra CPU overhead beyond gradient computation is O(D)O(D) per iteration (Barrak et al., 18 Sep 2025).

The reported evaluation uses CIFAR-10 with four parallel Lambda workers, each launching 24 concurrent invocations per epoch. For MobileNet with batch 512, the average per-function execution time is 14.343 s14.343\,s using 2 048 MB2\,048\,MB RAM, yielding approximately 0.000442 USD0.000442\,USD per invocation, 0.0106 USD0.0106\,USD per worker across 24 functions, and 0.0422 USD0.0422\,USD total per epoch. For ResNet-18, each function runs 27.17 s27.17\,s on 2.88 GB2.88\,GB, yielding 0.001302 USD0.001302\,USD per invocation and 0.1249 USD0.1249\,USD per epoch overall (Barrak et al., 18 Sep 2025).

The same study reports a split performance profile. Under increasing worker counts on ResNet-50, AllReduce’s synchronization time grows to 21.88 s21.88\,s, whereas ScatterReduce peaks at 8.36 s8.36\,s. For the smaller MobileNet, however, AllReduce becomes faster beyond 8–16 workers, with 4.77 s4.77\,s versus ScatterReduce’s 6.47 s6.47\,s at 16 workers. End-to-end convergence is slower: reaching 80%80\% accuracy requires roughly 1 6521\,652 minutes, and final accuracy plateaus at 82.1%82.1\%, compared with 1 3671\,367 minutes and 85.05%85.05\% for AllReduce, $84.96$ minutes and 83.2%83.2\% for SPIRT, $189.68$ minutes and 83.48%83.48\% for MLLess, and $70.33$ minutes and 84.5%84.5\% for the GPU baseline (Barrak et al., 18 Sep 2025).

Its principal advantage is balanced aggregation. Its principal limitation is that each iteration incurs roughly three full-gradient-sized transfers. Balanced aggregation also does not remove synchronization sensitivity: a single slow or faulty worker stalls both the scatter barrier and the reduce barrier, and no extra mechanism is introduced beyond storage-layer idempotency and simple barrier synchronization (Barrak et al., 18 Sep 2025).

5. Sparse GPU ScatterReduce and adaptive representation switching

On GPU platforms, sparse reduce-scatter has been developed around Pici, a bitvector-based sparse format. Pici stores an NN-element tensor as an NN-bit bitvector BB, an index array II of length ⌈N/4096⌉\lceil N/4096\rceil, and a values array VV of length nnz\text{nnz}. Its total storage in bits is

SPici(n,ρ)=nρb+n+n4096i,S_{\mathrm{Pici}}(n,\rho)=n\rho b+n+\frac{n}{4096}i,

or in bytes

SPici(n,ρ)=nρ⋅(b/8)+n/8+(n/4096)⋅(i/8),S_{\mathrm{Pici}}(n,\rho)=n\rho\cdot(b/8)+n/8+(n/4096)\cdot(i/8),

where ρ\rho is the nonzero fraction, bb the value bit-width, and ii the index width. Relative to dense storage, the compression ratio is

R(ρ)=ρ+1b+i4096b.R(\rho)=\rho+\frac{1}{b}+\frac{i}{4096b}.

For fp32 with b=32b=32 and i=32i=32, this gives R(ρ)=ρ+1/32+1/4096≃ρ+0.0315R(\rho)=\rho+1/32+1/4096\simeq \rho+0.0315 (Hough et al., 6 Jul 2026).

The adaptive ScatterReduce algorithm is based on NCCLX’s ring algorithm. It starts with an initial representation choice, then executes p−1p-1 ring steps. At each step it posts a send to (rank+1) mod p(\text{rank}+1)\bmod p, receives from (rank−1+p) mod p(\text{rank}-1+p)\bmod p, decompresses if necessary, performs DenseReduceInplace, recomputes the current sparsity, and decides whether the next step should use the dense or Pici format. Separate thresholds are used for intra-node and inter-node sends, and the implementation extends NCCLX’s Simple protocol with a 48-byte header containing format bits and message lengths (Hough et al., 6 Jul 2026).

The communication model compares dense and sparse traffic per rank. For sparse Pici data,

Vsparse(ρ)≃n8p⋅(ρb+1+i/4096),V_{\mathrm{sparse}}(\rho)\simeq \frac{n}{8p}\cdot(\rho b+1+i/4096),

which to first order becomes

Vsparse(ρ)≃[nb8p]⋅(ρ+1/b).V_{\mathrm{sparse}}(\rho)\simeq \left[\frac{nb}{8p}\right]\cdot(\rho+1/b).

The resulting speedup factor is

S(ρ)≃1ρ+1/b.S(\rho)\simeq \frac{1}{\rho+1/b}.

For fp32, this yields approximately 24.2×24.2\times at ρ=0.01\rho=0.01, approximately 7.6×7.6\times at ρ=0.10\rho=0.10, and approximately 1.07×1.07\times at ρ=0.90\rho=0.90, ignoring compression and decompression overheads and the effect of fill-in over multiple steps (Hough et al., 6 Jul 2026).

Empirically, the reported speedups over NCCL at 99%99\% input sparsity are up to 5.25×5.25\times for all-gather, 2.5×2.5\times for reduce-scatter, and 2.66×2.66\times for all-reduce. For end-to-end pruned DDP training of 1.5 B and 3.3 B LLMs, the reported iteration-time reductions versus dense NCCL are 16%16\% on 40 GB A100s with 32 GPUs and 26%26\% on 80 GB A100s with 64 GPUs. The recommended thresholds are approximately inter_thresh≈0.5, intra_thresh≈0.6, and ag_thresh≈0.1, and sparse collectives often benefit from channel counts up to 64 rather than NCCLX’s default of at most 16 (Hough et al., 6 Jul 2026).

The gains are not uniform. Densification erodes benefits at lower sparsities or higher process counts, and for very small messages, defined here as less than 50 MiB, latency can dominate and sparse overheads may outweigh bandwidth savings (Hough et al., 6 Jul 2026).

6. Tree–pipeline alternatives and recurrent design constraints

PAT, or Parallel Aggregated Trees, is a separate reduce-scatter algorithm designed for scale and aimed at improving NCCL when the ring algorithm’s linear latency is inefficient for small sizes and/or at scale. PAT uses a truncated, reversed-dimension binomial tree followed, when necessary, by a linear pipeline. If C=N/PC=N/P is the chunk size and BB the pre-registered intermediate buffer, then

A=⌊B/C⌋A=\lfloor B/C\rfloor

determines the number of parallel sub-trees. When A≥PA\ge P, PAT becomes a full reversed-dimension binomial-tree reduce-scatter and completes in ⌈log⁡2P⌉\lceil \log_2 P\rceil steps; when A<PA<P, the second phase adds exactly P/A−1P/A-1 sends/receives (Jeaugey, 25 Jun 2025).

Its round complexity is

SPAT={Dif t=P,D+(P/A−1)if t<P,S_{\mathrm{PAT}}= \begin{cases} D & \text{if } t=P,\\ D+(P/A-1) & \text{if } t<P, \end{cases}

with t=min⁡(A,P)t=\min(A,P) and D=⌈log⁡2t⌉D=\lceil \log_2 t\rceil, while each round sends exactly m=t⋅Cm=t\cdot C bytes. In the degenerate case A=1A=1, PAT reduces to the same cost as ring:

T=(P−1)α+(N(P−1)/P)β.T=(P-1)\alpha+(N(P-1)/P)\beta.

Measured one-to-all reduce-scatter latencies on an 8-node NVSwitch network show, for P=8P=8, 48 μs48\,\mu s for PAT versus 320 μs320\,\mu s for ring at 256 B, 60 μs60\,\mu s versus 550 μs550\,\mu s at 4 KB, 80 μs80\,\mu s versus 1 200 μs1\,200\,\mu s at 64 KB, and 1 200 μs1\,200\,\mu s versus 4 800 μs4\,800\,\mu s at 1 MB. For fixed N=4N=4 KB per rank, the reported PAT latency is 22 μs22\,\mu s at 4 ranks, 60 μs60\,\mu s at 8 ranks, 75 μs75\,\mu s at 16 ranks, and 100 μs100\,\mu s at 32 ranks, all lower than ring (Jeaugey, 25 Jun 2025).

Across these designs, several recurrent constraints dominate. Reduction order matters: the circulant round-optimal algorithm explicitly requires commutativity, whereas the persistent MPI implementation emphasizes deterministic reduction order for bit-wise reproducibility (Träff, 2024, Jocksch et al., 2020). Buffer availability matters: PAT trades intermediate memory for logarithmic latency, while sparse GPU implementations trade extra format-management work for lower communication volume (Jeaugey, 25 Jun 2025, Hough et al., 6 Jul 2026). Decentralization also has different meanings in different systems. In serverless training it means distributing chunk reductions across workers through shared storage, but that same choice preserves barrier sensitivity and retry cascades under failures (Barrak et al., 18 Sep 2025). In collective communication libraries it instead denotes a schedule over direct point-to-point exchanges or ring/tree neighbors.

ScatterReduce is therefore best understood not as a single algorithm but as a recurring decomposition pattern for reduce-scatter: partition the state, route disjoint pieces, perform local reductions on assigned pieces, and only then reconstruct or disseminate if a higher-level collective requires it. The literature differs on what is being optimized—rounds, byte volume, storage-tier balance, persistence overhead, sparsity exploitation, or buffer-bounded latency—but the common substrate is the same collective primitive.

Topic to Video (Beta)

No one has generated a video about this topic yet.

Whiteboard

No one has generated a whiteboard explanation for this topic yet.

Follow Topic

Get notified by email when new papers are published related to ScatterReduce.