--- title: ScatterReduce in Distributed Systems url: https://www.emergentmind.com/topics/scatterreduce type: topic --- # ScatterReduce in Distributed Systems ScatterReduce denotes a class of reduce-scatter mechanisms in which distributed state is partitioned, partially exchanged, reduced, and left scattered across participants rather than fully replicated. In the canonical collective-communication formulation, $p$ processors hold vectors $V_r$ partitioned into $p$ blocks and compute the global reduction $W=\bigoplus_{r=0}^{p-1} V_r$ such that processor $r$ finally holds only $W[r]=\bigoplus_{i=0}^{p-1} V_i[r]$ [2410.14234]. The name also appears in recent literature for a serverless distributed-training protocol that scatters gradient chunks through shared storage and for optimized or sparse reduce_scatter implementations in MPI and GPU communication stacks [2509.14920] [2006.13112] [2607.04676]. Across these usages, the central design question is how to minimize rounds, transferred volume, synchronization stalls, or format-conversion overhead while preserving the reduction semantics. ## 1. Semantic core and scope The reduce-scatter operation is important both in its own right and as building block for other collective operations. With processors numbered $0,1,\dots,p-1$, each processor $r$ starts with a vector $V_r$ of $m$ elements partitioned into $p$ disjoint blocks of equal size $m/p$: $$ V_r = [V_r[0], V_r[1], \dots, V_r[p-1]]. $$ The operation applies an associative, commutative binary operator $\oplus$ blockwise, but stores block $W[i]$ only on processor $i$ rather than materializing the full reduced vector everywhere [2410.14234]. The literature uses “ScatterReduce” in more than one sense. In message-passing systems it can denote an optimized reduce_scatter routine, including persistent MPI implementations based on recursive multiply/divide and cyclic shift, or a simple non-pipelined circulant schedule that is optimal in both rounds and communicated volume [2006.13112] [2410.14234]. In distributed ML it denotes a protocol in which each worker partitions its gradient vector into equally sized chunks, writes nonlocal chunks to shared storage, reduces one assigned chunk, and then gathers the aggregated chunks for the model update [2509.14920]. On GPU platforms, sparse reduce-scatter variants exploit unstructured sparsity by compressing data and adaptively switching between dense and sparse representations during ring-style communication [2607.04676]. | Usage | Core mechanism | Reported property | |---|---|---| | Reduce-scatter collective | Partition $V_r$ into $p$ blocks and compute only the local reduced block | $\lceil \log_2 p\rceil$-round optimal circulant schedule [2410.14234] | | Persistent MPI ScatterReduce | Recursive multiply/divide or cyclic shift with multi-port tuning and rank reordering | Factors of $2$–$6$ over standard MPI for small to medium blocks [2006.13112] | | Serverless ScatterReduce | Fetch, compute-and-scatter, reduce, gather-update via Redis or S3 | Roughly $3\times$ gradient traffic per iteration [2509.14920] | | Sparse GPU reduce-scatter | Adaptive ring using Pici bitvector format | Up to $2.5\times$ over NCCL at $99\%$ sparsity [2607.04676] | | PAT reduce-scatter | Truncated reversed-dimension binomial tree plus pipeline | Logarithmic latency when the intermediate buffer is large enough [2506.20252] | This breadth of usage is not terminological drift alone; it reflects the fact that the same semantic primitive is instantiated under markedly different systems assumptions. ## 2. Non-pipelined circulant algorithms and optimality A particularly compact formulation is the circulant reduce-scatter algorithm of Träff. It defines $q=\lceil \log_2 p\rceil$ and a skip sequence $$ s_0=p,\qquad s_{k+1}=\lceil s_k/2\rceil,\qquad k=0,\dots,q-1, $$ with $s_q=1$. In round $k$, processor $r$ sends to $t=(r+s_{k+1})\bmod p$, receives from $f=(r-s_{k+1}+p)\bmod p$, transmits the consecutive run of blocks $R[s_{k+1}\dots s_k-1]$, receives equally many blocks, and locally reduces the received data into $R[0\dots s_k-s_{k+1}-1]$ [2410.14234]. The local state is initialized as rotated partial sums, $$ R[i]\leftarrow V_{(r+i)\bmod p}[r],\qquad i=0,\dots,p-1, $$ so that $R[i]$ is the running partial sum that eventually contributes to $W[(r+i)\bmod p]$. After $q$ rounds, each processor retains $R[0]=W[r]$. The communication pattern is a simple, $\lceil\log_2 p\rceil$-regular, circulant graph also used elsewhere [2410.14234]. The cost profile is explicit. The number of rounds is $q=\lceil\log_2 p\rceil$. In round $k$ each processor sends and receives exactly $s_k-s_{k+1}$ blocks, and summing over all rounds yields $$ \sum_{k=0}^{q-1}(s_k-s_{k+1}) = s_0-s_q = p-1. $$ Hence each processor sends $p-1$ blocks, receives $p-1$ blocks, and applies $\oplus$ exactly $p-1$ times. In the linear-cost model with latency $\alpha$ and bandwidth term $\beta$ per block of size $m/p$, $$ T_{\mathrm{comm}}=\alpha\cdot\lceil\log_2 p\rceil+\beta\cdot((p-1)\cdot m/p). $$ A classic information-flow argument shows that reducing $p$ values to one requires at least $\lceil\log_2 p\rceil$ steps of pairwise combination, so the algorithm is optimal both in rounds and volume within that model [2410.14234]. The same construction yields allreduce by composing reduce-scatter with a standard $\lceil\log_2 p\rceil$-round allgather. The resulting allreduce runs in $2\lceil\log_2 p\rceil$ rounds, sends $2(p-1)$ blocks per processor, and has cost $$ T_{\mathrm{allreduce}}= 2\alpha\cdot\lceil\log_2 p\rceil +2\beta\cdot((p-1)\cdot m/p) +\gamma\cdot((p-1)\cdot m/p), $$ where $\gamma$ is the per-block reduction cost [2410.14234]. The same circulant pattern can also serve as a template for round-optimal all-to-all communication by taking “reduction” to be block concatenation. A central caveat is the commutativity assumption. Because blocks are reduced in the order induced by the circulant skipping pattern rather than in rank order, correctness relies on $\oplus$ being commutative as well as associative. For non-commutative operators, enforcing the same global sequence of inputs on each reduction tree is described as complex or expensive to arrange [2410.14234]. ## 3. Persistent MPI ScatterReduce and installation-time tuning In MPI libraries, optimized ScatterReduce has been formulated as a persistent communication routine built from two classical building blocks: recursive multiply/divide (“doubling/halving”) and cyclic-shift (Bruck’s algorithm). The optimization adds three features: multi-port execution with step-wise tuning, rank-reordering to balance non-uniform block sizes, and a bytecode-driven initialization/execution separation for persistent calls [2006.13112]. For recursive multiply/divide, the number of nodes $p$ is factored into radixes $r_1,\dots,r_s$ with $\prod_i r_i=p$. In step $i$ each node exchanges $(r_i-1)$ equal-sized subblocks in parallel with $(r_i-1)$ partners and performs a local reduction. With equal radixes $r$ and equal blocks of size $n/p$, the model is $$ T_{\mathrm{comm}}=\alpha\cdot s+(\beta+\gamma)\cdot\frac{p-1}{r-1}\cdot(n/p). $$ For cyclic shift, the cost is identical up to a small local re-shuffle overhead [2006.13112]. Non-equal message sizes are handled by a rank-reordering heuristic. The implementation sorts blocks by size, pairs the largest remaining block with the smallest, the second largest with the second smallest, and so on, records a permutation $\pi$, and applies $\pi$ to sources and destinations in every step. The stated goal is to equalize the communicated $M_i$ in each step and on each port. The one-time setup cost is $$ T_{\mathrm{reorder}}=\delta\cdot p\cdot(\log_2 p)^2, $$ with $\delta$ the per-comparison cost [2006.13112]. The complete routine has three phases: intra-node gather, inter-node ScatterReduce across $p$ “virtual” nodes, and intra-node scatter. During initialization, the library measures on-node $\alpha$, $\beta$, and $\gamma$, chooses optimal radixes and ports per step, computes the permutation $\pi$, and emits bytecode consisting of $(\text{step},\text{port},\text{offset},\text{length},\text{reduce-flag})$ instructions. Execution interprets this bytecode via MPI nonblocking point-to-point operations such as `MPI_Irecv`, `MPI_Isend`, and `MPI_Waitall`, followed by `local_reduce` or `memcpy` on received buffers [2006.13112]. Measured results show lower times than standard MPI baselines on both a 160-node KNL Cray XC40 and a 17-node Infiniband cluster. On the Cray XC40 at 160 nodes and 9600 tasks, for 8 B per final block the optimized routine is approximately $10\,\mu s$ versus approximately $60\,\mu s$ for Cray MPI, a $\times 6$ speedup; for 4 KB per block it is approximately $150\,\mu s$ versus approximately $280\,\mu s$, a $\times 1.8$ improvement. On the Infiniband cluster at 17 nodes and 408 tasks, 8 B per block gives approximately $12\,\mu s$ versus approximately $25\,\mu s$ for MVAPICH, and 4 KB per block gives approximately $160\,\mu s$ versus approximately $230\,\mu s$ [2006.13112]. In ORB5, where reduce_scatter and allgatherv have non-uniform block sizes, the pairing heuristic gives a further $\sim 20\%$ reduction in overall filter time versus no reordering, while the Cray MPI baseline is up to $3\times$ slower. The implementation also emphasizes deterministic reduction order for bit-wise reproducibility. Reported limitations are uniform core-count per node and lack of support for non-contiguous datatypes or dynamic inter-node communicators [2006.13112]. ## 4. ScatterReduce as a serverless gradient-aggregation protocol In serverless distributed training, ScatterReduce is a four-stage iteration protocol: fetch, compute-and-scatter, reduce, and gather-update. Each of $W$ workers loads a minibatch and the current parameters into its local Lambda function, computes a local gradient vector $g\in\mathbb{R}^D$, partitions it into $W$ contiguous chunks of size $D/W$, retains chunk $i$ locally, and writes each $g^{(j)}$ for $j\neq i$ to shared storage under a key encoding iteration, chunk index, and source [2509.14920]. In the reduce phase, worker $k$ waits until it can fetch exactly $W$ values $g^{(k)}_1,\dots,g^{(k)}_W$, sums them element-wise, and writes back the partial aggregate $$ h^{(k)}=\sum_{i=1}^W g^{(k)}_i. $$ In the gather-update phase, every worker downloads all aggregated chunks, concatenates them into $$ H=[h^{(1)}\parallel h^{(2)}\parallel \dots \parallel h^{(W)}], $$ and updates the model by $$ \theta \leftarrow \theta-\eta\cdot H. $$ This distributes the aggregation load evenly across workers rather than concentrating it in a single master [2509.14920]. The communication volume is substantially higher than the logical gradient size. If $G=D\cdot\text{sizeof(float)}$ is the full gradient size in bytes, then per worker $$ C_{\mathrm{comm}} = G\cdot(1-1/W) +G\cdot(1+1/W) +G = 3G+O(G/W). $$ Aggregated across all workers, the total network traffic is $O(WG)$. Under the $\alpha$–$\beta$ model, the iteration wall-clock overhead is on the order of $$ T_{\mathrm{iter}}=O(W\cdot\alpha+\beta\cdot G), $$ and the extra CPU overhead beyond gradient computation is $O(D)$ per iteration [2509.14920]. The reported evaluation uses CIFAR-10 with four parallel Lambda workers, each launching 24 concurrent invocations per epoch. For MobileNet with batch 512, the average per-function execution time is $14.343\,s$ using $2\,048\,MB$ RAM, yielding approximately $0.000442\,USD$ per invocation, $0.0106\,USD$ per worker across 24 functions, and $0.0422\,USD$ total per epoch. For ResNet-18, each function runs $27.17\,s$ on $2.88\,GB$, yielding $0.001302\,USD$ per invocation and $0.1249\,USD$ per epoch overall [2509.14920]. The same study reports a split performance profile. Under increasing worker counts on ResNet-50, AllReduce’s synchronization time grows to $21.88\,s$, whereas ScatterReduce peaks at $8.36\,s$. For the smaller MobileNet, however, AllReduce becomes faster beyond 8–16 workers, with $4.77\,s$ versus ScatterReduce’s $6.47\,s$ at 16 workers. End-to-end convergence is slower: reaching $80\%$ accuracy requires roughly $1\,652$ minutes, and final accuracy plateaus at $82.1\%$, compared with $1\,367$ minutes and $85.05\%$ for AllReduce, $84.96$ minutes and $83.2\%$ for SPIRT, $189.68$ minutes and $83.48\%$ for MLLess, and $70.33$ minutes and $84.5\%$ for the GPU baseline [2509.14920]. Its principal advantage is balanced aggregation. Its principal limitation is that each iteration incurs roughly three full-gradient-sized transfers. Balanced aggregation also does not remove synchronization sensitivity: a single slow or faulty worker stalls both the scatter barrier and the reduce barrier, and no extra mechanism is introduced beyond storage-layer idempotency and simple barrier synchronization [2509.14920]. ## 5. Sparse GPU ScatterReduce and adaptive representation switching On GPU platforms, sparse reduce-scatter has been developed around Pici, a bitvector-based sparse format. Pici stores an $N$-element tensor as an $N$-bit bitvector $B$, an index array $I$ of length $\lceil N/4096\rceil$, and a values array $V$ of length $\text{nnz}$. Its total storage in bits is $$ S_{\mathrm{Pici}}(n,\rho)=n\rho b+n+\frac{n}{4096}i, $$ or in bytes $$ S_{\mathrm{Pici}}(n,\rho)=n\rho\cdot(b/8)+n/8+(n/4096)\cdot(i/8), $$ where $\rho$ is the nonzero fraction, $b$ the value bit-width, and $i$ the index width. Relative to dense storage, the compression ratio is $$ R(\rho)=\rho+\frac{1}{b}+\frac{i}{4096b}. $$ For fp32 with $b=32$ and $i=32$, this gives $R(\rho)=\rho+1/32+1/4096\simeq \rho+0.0315$ [2607.04676]. The adaptive ScatterReduce algorithm is based on NCCLX’s ring algorithm. It starts with an initial representation choice, then executes $p-1$ ring steps. At each step it posts a send to $(\text{rank}+1)\bmod p$, receives from $(\text{rank}-1+p)\bmod p$, decompresses if necessary, performs `DenseReduceInplace`, recomputes the current sparsity, and decides whether the next step should use the dense or Pici format. Separate thresholds are used for intra-node and inter-node sends, and the implementation extends NCCLX’s Simple protocol with a 48-byte header containing format bits and message lengths [2607.04676]. The communication model compares dense and sparse traffic per rank. For sparse Pici data, $$ V_{\mathrm{sparse}}(\rho)\simeq \frac{n}{8p}\cdot(\rho b+1+i/4096), $$ which to first order becomes $$ V_{\mathrm{sparse}}(\rho)\simeq \left[\frac{nb}{8p}\right]\cdot(\rho+1/b). $$ The resulting speedup factor is $$ S(\rho)\simeq \frac{1}{\rho+1/b}. $$ For fp32, this yields approximately $24.2\times$ at $\rho=0.01$, approximately $7.6\times$ at $\rho=0.10$, and approximately $1.07\times$ at $\rho=0.90$, ignoring compression and decompression overheads and the effect of fill-in over multiple steps [2607.04676]. Empirically, the reported speedups over NCCL at $99\%$ input sparsity are up to $5.25\times$ for all-gather, $2.5\times$ for reduce-scatter, and $2.66\times$ for all-reduce. For end-to-end pruned DDP training of 1.5 B and 3.3 B LLMs, the reported iteration-time reductions versus dense NCCL are $16\%$ on 40 GB A100s with 32 GPUs and $26\%$ on 80 GB A100s with 64 GPUs. The recommended thresholds are approximately `inter_thresh≈0.5`, `intra_thresh≈0.6`, and `ag_thresh≈0.1`, and sparse collectives often benefit from channel counts up to 64 rather than NCCLX’s default of at most 16 [2607.04676]. The gains are not uniform. Densification erodes benefits at lower sparsities or higher process counts, and for very small messages, defined here as less than 50 MiB, latency can dominate and sparse overheads may outweigh bandwidth savings [2607.04676]. ## 6. Tree–pipeline alternatives and recurrent design constraints PAT, or Parallel Aggregated Trees, is a separate reduce-scatter algorithm designed for scale and aimed at improving NCCL when the ring algorithm’s linear latency is inefficient for small sizes and/or at scale. PAT uses a truncated, reversed-dimension binomial tree followed, when necessary, by a linear pipeline. If $C=N/P$ is the chunk size and $B$ the pre-registered intermediate buffer, then $$ A=\lfloor B/C\rfloor $$ determines the number of parallel sub-trees. When $A\ge P$, PAT becomes a full reversed-dimension binomial-tree reduce-scatter and completes in $\lceil \log_2 P\rceil$ steps; when $A