ClusterGather: Locality-Aware Aggregation Mechanisms

• ClusterGather is a family of locality-aware approaches that include hierarchical all-gather operations in MPI/GPU contexts and adaptive clustering in gravitational-wave searches.
• In high-performance computing, it reduces redundant data transfers by leveraging shared-memory windows, leader coordinations, and DSMEM-based block exchanges.
• In continuous-wave searches, it implements seed-centered adaptive clustering to improve false alarm rejection and efficiency in candidate signal detection.

ClusterGather is a non-unified research term used for several locality-aware gather or clustering mechanisms. In high-performance communication, it denotes hierarchical all-gather designs that exploit shared memory either across MPI ranks on the same node or across thread blocks within a Hopper GPU cluster. In continuous gravitational-wave postprocessing, it denotes an adaptive clustering procedure that forms seed-centered candidate clusters in frequency–spin-down and sky coordinates. Related literatures on $r$-gather clustering and multi-agent gathering address adjacent mathematical problems—minimum-size clustering in metric spaces and rendezvous of mobile agents—but they do not define the same primitive or workflow (Zhou et al., 2020, Luo et al., 26 Aug 2025, Singh et al., 2017, Epasto et al., 2021, Barel et al., 2019).

1. Terminological scope

The term is best understood as a family of domain-specific constructions rather than a standardized abstraction. Across the cited works, “ClusterGather” refers either to an all-gather operation confined to a locality with explicit synchronization and shared storage, or to an adaptive clustering rule that collects nearby objects around a seed only when the local density profile supports doing so.

Context	Object called ClusterGather	Core mechanism
Multi-core clusters	Hybrid MPI+MPI allgather	Shared-memory window per node plus leader-only inter-node MPI_Allgatherv
Hopper GPU LLM inference	Cluster-level all-gather primitive	DSMEM block-to-block exchange in $\log_2 N$ rounds
Continuous-wave searches	Adaptive clustering procedure	Seed-centered over-density detection in F-space and S-space

A related but distinct usage of “gather” appears in the $r$-gather problem, where the objective is to partition points into clusters of size at least $r$ while minimizing the maximum radius. Another related usage appears in geometric consensus, where “gathering” means that all pairwise distances tend to zero. These neighboring formulations clarify the broader semantic field around ClusterGather, but they are not interchangeable with the communication primitives or adaptive clustering procedure described above (Epasto et al., 2021, Barel et al., 2019).

2. Hybrid MPI+MPI ClusterGather on multi-core clusters

In the MPI setting, ClusterGather is a hierarchical allgather designed for hybrid MPI+MPI codes that use MPI-3 shared-memory extensions for on-node parallelism. The construction begins by splitting MPI_COMM_WORLD into one shared-memory subcommunicator per node with MPI_Comm_split_type(..., MPI_COMM_TYPE_SHARED, ...), selecting one leader per node, and then forming a bridge communicator containing exactly one rank per node with MPI_Comm_split. On the shared-memory communicator, all ranks call MPI_Win_allocate_shared; the leader allocates P_total·m elements for an allgather of $m$-element messages, while non-leaders allocate zero bytes and then recover the same base pointer via MPI_Win_shared_query. Each rank $i$ sets mybuf = baseptr + i·m, writes its own $m$ elements directly into that segment, and thereby avoids intra-node peer-to-peer copies (Zhou et al., 2020).

The inter-node phase is executed only by leaders. They call MPI_Allgatherv(mybuf, m, …, baseptr, recvcounts[], displs[], …) on the bridge communicator, with recvcounts[k] = m·(size of node k) and displs[k] = (starting [offset](https://www.emergentmind.com/topics/generalized-inner-outer-offset-degeneracy) of node k in baseptr). A barrier on the shared-memory communicator before the inter-node exchange ensures that every leader’s local block is ready; a second barrier after MPI_Allgatherv ensures that all on-node ranks observe the fully updated shared buffer. Because every on-node rank points into the same window, there is exactly one copy of each block of the result per node, and no on-node broadcast or local copy loop is invoked (Zhou et al., 2020).

The cost model makes the intended savings explicit. For pure MPI allgather, memory per node is

$M_{\text{pure}} = p \cdot (P \cdot m),$

where $P$ is the total number of MPI ranks and $p$ is the number of ranks per node. For the hybrid MPI+MPI method, memory per node is

$\log_2 N$0

so the memory reduction factor is approximately $\log_2 N$1. The communication-time models are summarized as

$\log_2 N$2

and

$\log_2 N$3

where $\log_2 N$4 is network latency, $\log_2 N$5 is per-byte transmission time, and $\log_2 N$6 is a small shared-memory synchronization cost. The comparison given in the paper is that, if $\log_2 N$7, the total $\log_2 N$8-bytes term in $\log_2 N$9 is smaller by approximately $r$0, while the added overhead is two barriers of order $r$1 (Zhou et al., 2020).

The reported measurements validate these trade-offs on a Cray XC40 system using Cray MPI and on an NEC cluster using OpenMPI. In single-node tests with $r$2 up to $r$3, pure MPI latency grows linearly with $r$4, whereas the hybrid method remains at a few microseconds; at $r$5 elements, the reported example is approximately $r$6 for hybrid versus approximately $r$7 for pure MPI. In the mixed case on $r$8 nodes with $r$9, hybrid overtakes pure once $r$0 on OpenMPI at small $r$1, and always on Cray MPI for $r$2. In an irregular case with $r$3 nodes with $r$4 ranks and one node with $r$5 ranks, hybrid remains at approximately $r$6 while pure rises to approximately $r$7. Application-level validation shows that hybrid SUMMA can be up to $r$8 faster at small block size $r$9 when all ranks lie on one node, and that hybrid BPMF reduces total runtime by up to $m$0 at $m$1 cores while cumulative allgather time drops by approximately $m$2, barriers included (Zhou et al., 2020).

3. ClusterGather as a Hopper on-chip collective primitive

In the LLM-inference setting, ClusterGather is a cluster-level all-gather primitive that runs entirely on-chip within a Hopper GPU thread-block cluster. The primitive is defined for up to $m$3 blocks, with $m$4 required to be a power of two. Each block owns a shared-memory buffer $m$5 of length $m$6, with D_b[0:size] initially holding that block’s local segment. ClusterGather then replicates each block’s local data segment to every other block through the SM-to-SM interconnect, DSMEM, without off-chip global-memory round-trips (Luo et al., 26 Aug 2025).

The protocol is a binary-tree all-gather executed in $m$7 rounds. At round $m$8, with stride = 2^{r-1}, block $m$9 computes

$i$0

Each block sends the prefix it has gathered so far, of length $i$1, to send_to, placing it at offset stride * size in the peer’s buffer; symmetrically, it receives the next chunk of the same size from recv_from into its own buffer at the same offset. A barrier-like wait guarantees visibility before the next round, and the stride doubles until it reaches $i$2. The semantics are strict: shared memory per block must accommodate $i$3, all $i$4 blocks must participate, and no block may exit early (Luo et al., 26 Aug 2025).

The on-chip traffic formula given for all-gather is

$i$5

For comparison, the paired primitive ClusterReduce has

$i$6

The paper uses these formulas to guide which dimensions should be gathered and which reduced in the fused execution plan. It also distinguishes ClusterGather from standard CUDA intrinsics: warp intrinsics operate within a warp, __syncthreads and grid-level synchronization coordinate threads within a block, whereas ClusterGather leverages DSMEM for direct on-chip block-to-block transfer that is not exposed by standard CUDA C++ APIs (Luo et al., 26 Aug 2025).

Within ClusterFusion, ClusterGather is invoked after each block computes its local Q, K, and V segments: $i$7 After that call, every block has the full Q, K, and V for its attention head, allowing the kernel to continue with per-block partial attention, ClusterReduce for softmax statistics, and the final $i$8 projection while keeping intermediates on-chip. In the collective microbenchmark reported in Table 5, for a data size of $i$9, off-chip gather takes $m$0 and on-chip ClusterGather takes $m$1, a $m$2 speedup; for $m$3, the times are $m$4 and $m$5, a $m$6 speedup. The ablation in Figure 1 reports that disabling DSMEM increases end-to-end TPOT by up to $m$7. At the framework level, ClusterFusion reports an average $m$8 reduction in end-to-end latency on H100 GPUs across different models and configurations (Luo et al., 26 Aug 2025).

4. ClusterGather as adaptive candidate clustering in continuous-wave searches

In continuous gravitational-wave searches, ClusterGather denotes an adaptive clustering procedure for post-processing stage-1 candidates. The input is

$m$9

where $M_{\text{pure}} = p \cdot (P \cdot m),$0 are template parameters and $M_{\text{pure}} = p \cdot (P \cdot m),$1 is the detection statistic. Clustering proceeds iteratively: select the loudest candidate $M_{\text{pure}} = p \cdot (P \cdot m),$2 with $M_{\text{pure}} = p \cdot (P \cdot m),$3, form a cluster $M_{\text{pure}} = p \cdot (P \cdot m),$4 around that seed, remove $M_{\text{pure}} = p \cdot (P \cdot m),$5 from the current candidate set, and stop when no remaining candidate exceeds the seed threshold $M_{\text{pure}} = p \cdot (P \cdot m),$6. Typical thresholds quoted in the procedure are $M_{\text{pure}} = p \cdot (P \cdot m),$7 and $M_{\text{pure}} = p \cdot (P \cdot m),$8 for $M_{\text{pure}} = p \cdot (P \cdot m),$9, or $P$0 and $P$1 for $P$2 (Singh et al., 2017).

The method is explicitly two-space. In frequency–spin-down space, distances from the seed are measured by

$P$3

so contours of constant $P$4 are ellipses in normalized $P$5 coordinates. Distances are binned into equal-area annuli with edges $P$6, where

$P$7

The histogram $P$8 is smoothed by first fitting a superposition $P$9 of Gaussians and sinusoids and then re-fitting the result by a single Gaussian $p$0. The first local minimum of $p$1 near $p$2 yields the F-space cluster radius $p$3. The procedure then evaluates three hill parameters,

$p$4

and accepts the F-cluster only if $p$5, $p$6, and $p$7. Example values given are $p$8, $p$9, and $\log_2 N$00. If the test fails, the radius is reset to $\log_2 N$01, which restricts the shortlist to the first bin (Singh et al., 2017).

In sky space, the procedure transforms $\log_2 N$02 to a uniform plane $\log_2 N$03 and defines

$\log_2 N$04

Sky annuli have edges $\log_2 N$05, with

$\log_2 N$06

where $\log_2 N$07 is the $\log_2 N$08 containment diameter and $\log_2 N$09 is a tuning term that may increase with seed loudness. If the first sky bin is the most populated, the smallest $\log_2 N$10 satisfying

$\log_2 N$11

with $\log_2 N$12 determines $\log_2 N$13; otherwise $\log_2 N$14, yielding a single-occupant sky cluster. The final cluster is

$\log_2 N$15

The stated rationale is that real CW signals produce a strong local over-density both in F-space and in sky, whereas noise disturbances often do not (Singh et al., 2017).

The computational trade-off differs sharply from the communication uses of ClusterGather. Per seed, AdCl performs $\log_2 N$16 distance computations, $\log_2 N$17 histogram binning, two-stage fitting, and hill checks, whereas fixed-volume clustering only requires distance computations and a simple radius cut. However, the adaptive method typically produces at most half as many clusters as fixed clustering, reducing downstream follow-up cost. In the Einstein@Home O1 results quoted in the summary, for the high-significance $\log_2 N$18 search, AdCl has noise rejection approximately $\log_2 N$19 versus old clustering at at most $\log_2 N$20, with efficiency approximately $\log_2 N$21 versus $\log_2 N$22. In the sub-threshold $\log_2 N$23 search, AdCl has noise rejection approximately $\log_2 N$24 versus old clustering at at most $\log_2 N$25 at efficiency approximately $\log_2 N$26 (Singh et al., 2017).

5. Related gather and clustering formalisms

A neighboring formalism is the $\log_2 N$27-gather problem. Given a metric space $\log_2 N$28 and a point set $\log_2 N$29, an $\log_2 N$30-gather clustering is a partition

$\log_2 N$31

with centers $\log_2 N$32 such that $\log_2 N$33 for all $\log_2 N$34, and with objective

$\log_2 N$35

minimized. The paper states that $\log_2 N$36-gather is NP-hard and has a tight $\log_2 N$37-approximation via Gonzalez–Shalita–Zwick. It then gives two algorithmic regimes beyond the usual offline setting: an MPC algorithm for Euclidean input points that computes an $\log_2 N$38-approximation in $\log_2 N$39 rounds with total space $\log_2 N$40, and a fully dynamic algorithm in doubling metrics with amortized update time $\log_2 N$41 and query time $\log_2 N$42. These are minimum-size clustering results rather than ClusterGather collectives, but they formalize the “gather” motif as a radius-minimization problem under cardinality constraints (Epasto et al., 2021).

A second adjacent body of work studies gathering as geometric consensus of mobile agents. In that literature, the gathering objective is

$\log_2 N$43

or its discrete-time analogue. The survey classifies eight systems by visibility range, sensing modality, and continuous versus discrete time. The listed results include exponential asymptotic gathering to the initial centroid for continuous infinite-visibility position sensing, asymptotic convergence in discrete time when $\log_2 N$44, finite-time gathering under infinite-visibility bearing-only dynamics, finite-time gathering to a single point for several limited-visibility models, and finite expected-time clustering into an $\log_2 N$45-disk for a randomized semi-synchronous bearing-only model. These results concern anonymous, oblivious agents in $\log_2 N$46, not shared-memory collectives or seed-based postprocessing, but they supply a formal notion of “gathering” that is conceptually adjacent to the uses above (Barel et al., 2019).

This broader landscape helps prevent a common conflation. ClusterGather in MPI or Hopper GPU papers is a communication primitive; ClusterGather in CW searches is a density-adaptive clustering rule; $\log_2 N$47-gather and multi-agent gathering are formal optimization or dynamical-system problems with different state spaces, guarantees, and cost models.

6. Comparative themes, constraints, and common misconceptions

The three direct ClusterGather usages share a common locality principle, but they operationalize it differently. In hybrid MPI+MPI, locality is the node: data are written once into a shared-memory window, leaders perform the inter-node exchange, and barriers protect data integrity. In Hopper GPUs, locality is the thread-block cluster: all blocks participate in a power-of-two binary-tree all-gather over DSMEM, with shared-memory buffers sized to hold the full gathered result. In CW postprocessing, locality is the seed neighborhood in two parameter spaces, and clustering is allowed to grow only while histogram shape and hill-parameter checks remain consistent with a signal-like over-density (Zhou et al., 2020, Luo et al., 26 Aug 2025, Singh et al., 2017).

These usages also embody different optimization criteria. The MPI and GPU variants are explicitly bandwidth- and memory-traffic-oriented: both remove unnecessary copies, and both replace repeated off-locality movement with a structured local exchange. The CW variant is explicitly false-alarm- and follow-up-budget-oriented: it spends more work per seed on density estimation and consistency testing in order to reduce the number of downstream seeds. This suggests a shared design pattern—local aggregation with strict participation and validation rules—but not a shared objective function.

A frequent misconception is that ClusterGather denotes a single standardized API or algorithmic schema. The cited papers do not support that view. Another misconception is that “gather” always means an all-gather collective. The CW usage is not a communication primitive at all; it is an adaptive clustering procedure over candidate sets. Conversely, the $\log_2 N$48-gather and mobile-agent literatures should not be read as implementations of ClusterGather, even though they articulate mathematically precise gather or clustering objectives. A plausible implication is that the term is best treated as a family resemblance across locality-constrained aggregation problems rather than as a canonically defined operator.

From a systems perspective, the cited works converge on one practical lesson. When the relevant locality is exploited explicitly—whether by a shared-memory communicator, a bridge communicator of leaders, a DSMEM-resident block cluster, or a signal-consistent seed neighborhood—the resulting design can reduce redundant replication, suppress unnecessary movement, or narrow downstream search. The exact mechanism, however, remains domain-specific, and the term “ClusterGather” acquires its technical meaning only within that domain’s communication model, geometry, or statistical postprocessing pipeline.