ClusterReduce: On-chip Reduction for LLMs

• ClusterReduce is a cluster-level collective communication primitive that performs fast on-chip reduction across thread blocks using DSMEM.
• It implements a binary-tree reduction pattern across 2^k thread blocks, enabling fusion of QKV Projection, Attention, and Output Projection in LLM inference.
• Empirical results show significant latency reduction and scalable performance on NVIDIA Hopper GPUs compared to conventional off-chip reduction methods.

ClusterReduce denotes a cluster-level collective communication primitive introduced for LLM inference on NVIDIA Hopper thread-block clusters. In its exact usage, it abstracts on-chip reduction across thread blocks using distributed shared memory (DSMEM) and low-latency SM-to-SM communication, so that partial results can be combined with associative operators such as sum or max without materializing intermediates in global memory. Within ClusterFusion, this primitive is part of a larger execution framework that fuses QKV Projection, Attention, and Output Projection into a single cluster-centric kernel schedule (Luo et al., 26 Aug 2025).

1. Definition and execution model

ClusterReduce operates on a cluster of $N=2^k$ thread blocks, with $k\le 4$. Each block $b$ holds a shared-memory buffer $\mathbf{D}_b$, and the primitive applies an associative reduction operator $\oplus$ across the per-block buffers. The implementation pattern is a binary-tree-style exchange over $\log_2 N$ rounds: at each round, a block sends its current partial result to a partner at a prescribed stride, receives a partner buffer into a temporary shared-memory buffer $\mathbf{B}_b$, waits for arrival, and updates its local state by

$$\mathbf{D}_b \gets \mathbf{D}_b \oplus \mathbf{B}_b.$$

After $\log_2 N$ rounds, every block holds the fully reduced result. The communication remains on chip because Hopper exposes thread-block clusters, DSMEM, and a low-latency intra-cluster interconnect; ClusterReduce supplies the collective semantics missing from the underlying low-level PTX-style data-movement interface (Luo et al., 26 Aug 2025).

The primitive is explicitly distinguished from both conventional global-memory reductions and ordinary shared-memory reductions within a single thread block. In the former case, partial results must be written to global memory and later re-read by a separate kernel; in the latter case, communication scope is restricted to one block. ClusterReduce fills the intermediate scope: reduction across multiple thread blocks inside one cluster. Its DSMEM traffic model is

$$Traffic_{Reduce}(size, N) = size \times \log_2 N \times N,$$

which reflects constant message size per round, linear scaling in data size and block count, and logarithmic scaling in cluster size (Luo et al., 26 Aug 2025).

2. Role inside fused LLM decoding

Within ClusterFusion, ClusterReduce is used precisely where different blocks compute partial results that must be aggregated before downstream computation can continue. In the fused Llama-style execution path, blocks first compute local QKV projection tiles and local attention partials; ClusterReduce then performs two reductions inside attention. First, it reduces softmax statistics,

$k\le 4$0

so that online softmax can be computed across partitioned KV segments. Second, after local rescaling, it reduces the partial attention outputs,

$k\le 4$1

The reduced attention result then remains on chip and can be consumed immediately by Output Projection in the same fused kernel (Luo et al., 26 Aug 2025).

The same pattern appears in the fused MLA path. There, ClusterReduce is again applied to $k\le 4$2, $k\le 4$3, the attention output $k\le 4$4, and the Down Projection output. This indicates that the primitive is not a one-off optimization for a single operator, but a reusable reduction abstraction for multi-stage decoding dataflows. A plausible implication is that ClusterReduce matters less as an isolated primitive than as the reduction-shaped dependency resolver that makes broader operator fusion possible.

3. Empirical characteristics, performance, and limits

Microbenchmarks in ClusterFusion compare an off-chip implementation without DSMEM to on-chip ClusterReduce over DSMEM. The reported reduction latencies are:

Data Size	Off-chip	On-chip
32 KB	8.03 $k\le 4$5	6.77 $k\le 4$6
64 KB	9.01 $k\le 4$7	6.61 $k\le 4$8
128 KB	14.95 $k\le 4$9	7.42 $b$0
256 KB	22.44 $b$1	9.17 $b$2

These measurements show increasing benefit with larger reduction payloads, reaching $b$3 at 256 KB. The same paper reports average SM-to-SM DSMEM access latency of about 190 cycles at cluster size 2, versus global memory latency exceeding 470 cycles, which is consistent with the primitive’s motivation: avoid off-chip reduction paths whenever inter-block dependencies remain within a Hopper cluster (Luo et al., 26 Aug 2025).

End-to-end gains are reported for the full ClusterFusion framework rather than ClusterReduce in isolation, but the primitive is structurally necessary for those gains. On H100 GPUs, ClusterFusion reports $b$4 average end-to-end latency improvement across models and configurations, and disabling DSMEM increases TPOT by up to $b$5. The primitive is, however, constrained by Hopper’s cluster model: $b$6, $b$7, hence $b$8. Performance is also sensitive to cluster size. The reported results indicate that cluster size 4 is best for many 32-head and 64-head settings, cluster size 2 is better when head count is 128, and cluster sizes 8 and 16 degrade because of higher interconnect latency, bandwidth contention, and reduced active SM count. The supported operators are only associative reductions, explicitly sum and max (Luo et al., 26 Aug 2025).

4. Related reduction ideas in statistical learning

In a broader methodological sense, ClusterReduce belongs to a family of procedures that remove or compress cluster-structured dependence before applying a downstream algorithm. A prominent example is the Clustering Removal Algorithm (CRA) for sparse regression with highly correlated predictors. CRA assumes clustered columns in a design matrix $b$9, estimates cluster directions $\mathbf{D}_b$0, forms

$\mathbf{D}_b$1

and removes the cluster-induced subspace by projection: $\mathbf{D}_b$2 The transformed regression

$\mathbf{D}_b$3

preserves support information because $\mathbf{D}_b$4 is only a diagonal rescaling of $\mathbf{D}_b$5. Under the paper’s spherical-cap model, the projected and normalized columns become uniformly distributed on

$\mathbf{D}_b$6

and $\mathbf{D}_b$7 satisfies RIP with high probability once the cluster subspace is removed (Ghorbani et al., 2015).

This use of “cluster reduction” differs sharply from the Hopper primitive. CRA does not reduce thread-block communication or fuse GPU operators; rather, it reduces cluster-induced correlation structure in a statistical design matrix. The connection is therefore conceptual rather than terminological: both procedures isolate a low-dimensional cluster effect and remove it so that a later computation becomes better conditioned.

5. Representative-point and distributed forms of cluster reduction

The broader literature also uses cluster reduction to mean replacing large local structures with smaller representative objects before global clustering. The reduction target differs substantially across domains.

Method	Reduced object	Immediate purpose
Distributed density-based clustering	Boundary points, optional internal representatives, local parameters	Reduce communication while preserving shape
SBSC	Sub-clusters built from a random sample and full-data neighbors	Replace point clustering by clustering sub-clusters
Distributed coreset clustering	Weighted global coreset built from local samples and local centers	Reduce communication over general topologies

In distributed density-based clustering, the reduction mechanism is explicitly shape-aware. Each local cluster $\mathbf{D}_b$8 is summarized by its boundary $\mathbf{D}_b$9, optional internal representatives $\oplus$0, and local parameters $\oplus$1; the global model is then built by merging local boundaries and regenerating points inside the merged shapes. Boundary detection uses a balance vector

$\oplus$2

with the final hyper-cone predicate

$\oplus$3

The paper’s central claim is that sending compact boundary structure reduces communication overhead while preserving enough shape information to recover higher-quality global clusters (Le-Khac et al., 2017).

In subspace clustering, SBSC reduces the original $\oplus$4-point problem to clustering a much smaller set of representative sub-clusters. A random subset $\oplus$5 of size $\oplus$6 is sampled; each sampled point is expanded into a local sub-cluster $\oplus$7 by selecting the $\oplus$8 most correlated points in the full dataset; pairwise sub-cluster distances are then computed by a symmetric ridge-regression residual, and spectral clustering is run only on the resulting $\oplus$9 graph. The paper states that if $\log_2 N$0, $\log_2 N$1, and $\log_2 N$2 are linear in $\log_2 N$3, the complexity is $\log_2 N$4 (Li et al., 2018).

For center-based clustering on distributed data, a different reduction appears in global coreset construction over arbitrary communication graphs. Each node computes a local constant-factor clustering $\log_2 N$5, shares only its local approximation cost $\log_2 N$6, samples local points proportionally to their local contribution $\log_2 N$7, and adds the local centers with residual weights. The union becomes a single global $\log_2 N$8-coreset rather than a recursive union of local coresets. For $\log_2 N$9-median, the resulting coreset size is

$\mathbf{B}_b$0

and the graph-communication cost scales as $\mathbf{B}_b$1 on an arbitrary connected graph (Balcan et al., 2013).

6. Terminological boundaries and common confusions

The exact term ClusterReduce belongs to the ClusterFusion vocabulary: it is the reduction-side sibling of ClusterGather, where ClusterReduce performs element-wise associative reductions across thread blocks and ClusterGather replicates local shards to all blocks in the cluster (Luo et al., 26 Aug 2025). Many earlier papers are close in spirit but use different names and target entirely different objects: cluster-induced correlation in sparse regression, boundary summaries in distributed DBSCAN-style pipelines, representative sub-clusters in subspace clustering, or weighted coresets in distributed $\mathbf{B}_b$2-median and $\mathbf{B}_b$3-means. This suggests that “ClusterReduce” is most precise when reserved for the Hopper cluster-scoped primitive, while “cluster reduction” is the broader methodological family.

A specific bibliographic confusion also exists. One record sometimes associated with a parallel clustering method, (Sastry et al., 2014), is in fact the documentation for the LaTeX package algorithmicx. It contains no proposed clustering method, no subclustering schemes, no representative-point generation procedure, and no clustering evaluation. It therefore does not define a ClusterReduce technique in either the GPU or the statistical sense (Sastry et al., 2014).

Taken narrowly, ClusterReduce is an on-chip collective reduction primitive for Hopper-class LLM inference. Taken broadly, it exemplifies a recurrent research pattern: identify a cluster-structured intermediate object, compress or remove it in a controlled way, and thereby make the downstream computation cheaper, more stable, or more scalable.