Clustering-Based Activation Pattern Compression

• The framework organizes high-dimensional neural activations into clusters and compresses them using operations like averaging, low-rank factorization, or codebook assignment.
• It is applied to compress various representational objects, including on-device memory tokens for LLMs, CNN feature maps for imaging, and sparse FFN activation masks, reducing latency and storage.
• Empirical results demonstrate significant improvements in efficiency, balancing the trade-off between reduced computation (FLOPs, latency) and maintained model performance across diverse settings.

A clustering-based activation pattern compression framework is a class of compression methods in which high-dimensional neural representations are first organized into clusters and then compressed through cluster-conditioned operations such as averaging, centroid substitution, low-rank factorization, or codebook assignment. Across recent work, the compressed objects range from learned memory tokens for on-device LLM personalization, to feed-forward activation masks in LLMs, to spatio-channel activation structures in convolutional networks. The common objective is to reduce context length, storage, FLOPs, latency, or prediction cost while preserving the task-relevant structure of the underlying representations, especially when global averaging or uniform layer-wise compression would blur heterogeneous patterns or destroy local functional structure (Bohdal et al., 24 Jan 2026, Hamlomo et al., 25 Apr 2026, Dhar et al., 11 Jul 2025).

1. Formal scope and representational objects

The framework is defined by the nature of the object being clustered and the budget under which compression is performed. In personalized on-device language modeling, each supporting document is compressed into a learned memory $M_i \in \mathbb{R}^{D_m \times D_e}$, where $D_m$ is the number of memory tokens and $D_e$ is the embedding dimension. Compression then operates over the set of document-specific memories $\{M_i\}_{i=1}^N$, with the downstream budget expressed directly in memory tokens and the task metric expressed in ROUGE-L (Bohdal et al., 24 Jan 2026). In this setting, learned memories are synthetic continuous vectors rather than standard vocabulary tokens, and the compression problem is explicitly constrained by small context windows, limited compute, tight latency constraints, and small parameter budgets in on-device LLMs (Bohdal et al., 24 Jan 2026).

In activation-centric settings, the objects are typically hidden-state or feature-map realizations. One generic formulation treats activation patterns as vectors $a_i \in \mathbb{R}^{d}$, or as aligned sequences that can be pooled into fixed-size vectors for clustering and then merged elementwise within clusters. A more specialized CNN formulation uses a convolutional feature tensor $$\mathcal{F} \in \mathbb{R}^{C_{\text{out}} \times H \times W}$$, then constructs region descriptors $\mathbf{r}_s \in \mathbb{R}^{C_{\text{out}}}$ and channel activation vectors $\mathbf{f}_c^{(k)} \in \mathbb{R}^{|\Omega_k|}$ so that both spatial locality and channel co-activation can drive compression (Hamlomo et al., 25 Apr 2026).

For sparse LLM feed-forward networks, activation patterns can be modeled as binary or continuous masks over neurons. The binary formulation uses an activation matrix $\mathbf{A} \in \{0,1\}^{N \times D}$, where $A_{ij}=1$ indicates that neuron $D_m$0 is active in pattern $D_m$1. Cluster representatives then define a corresponding centroid mask $D_m$2, and compression quality can be evaluated by how well centroids preserve active neurons rather than inactive ones (Dhar et al., 11 Jul 2025). This places activation pattern compression closer to mask coding and sparse execution than to ordinary feature quantization.

Across these settings, the shared compression objective is not merely dimensionality reduction. It is the preservation of downstream personalization quality, class discrimination, or language-model behavior under explicit operational constraints such as token budgets, parameter counts, cluster counts, FLOPs, latency, or inference-time prediction complexity (Bohdal et al., 24 Jan 2026, Hamlomo et al., 25 Apr 2026, Dhar et al., 11 Jul 2025).

2. Clustering criteria and partitioning strategies

The dominant clustering primitive is $D_m$3-means. In the memory-compression setting, memories are clustered into $D_m$4 groups before merging, with the number of clusters directly determining the final memory-token count $D_m$5. A generalized activation formulation uses the standard objective

$D_m$6

or, for sequence-valued representations, the same objective applied to pooled vectors $D_m$7 (Bohdal et al., 24 Jan 2026). In the CNN spatio-channel framework, $D_m$8-means appears twice: first on region descriptors $D_m$9 to produce spatial clusters, and then on channel activation vectors $D_e$0 within each spatial support $D_e$1, yielding a hierarchical partition indexed by spatial cluster $D_e$2 and channel cluster $D_e$3 (Hamlomo et al., 25 Apr 2026).

Not all activation-pattern clustering relies on symmetric Euclidean treatment of zeros and ones. For sparse FFN patterns in Mistral-7B, Activation-Aware Clustering (AWC), also called Activation-Aware Patterns Clustering (APC), restricts distance computation to the indices of active neurons in the data point, enforces balanced centroid capacity, and updates centroids using continuous activation sums followed by percentile sparsification. This differs from Binary Matrix Factorization and Binary-to-Real-and-Back K-means by explicitly matching active neurons instead of letting shared zeros dominate the geometry (Dhar et al., 11 Jul 2025).

A distinct alternative replaces geometric similarity with tree-induced discrete activation signatures. Random Forest Activation Pattern (RFAP) similarity maps a feature vector $D_e$4 to a path-code vector $D_e$5 through an unsupervised random forest, then defines similarity by averaged digitwise agreement across tree paths. The resulting similarity matrix $D_e$6 is used to supervise a clustering head, making clustering depend on shared activation routes through the forest rather than raw Euclidean or cosine proximity (Balasubramanian et al., 2021).

These variants illustrate that the “clustering” in the framework can target at least three different structures: Euclidean proximity of representations, co-activation structure under sparsity, and discrete activation signatures induced by another model. The choice matters because it determines whether the compression operator preserves semantic coherence, local functional structure, or sparse execution patterns.

3. Compression operators after clustering

Once clusters are formed, compression is carried out by a cluster-specific representative mechanism. The simplest operator is within-cluster averaging. In memory compression, each clustered memory is

$D_e$7

an elementwise average over aligned memory-token sequences. The key distinction from naive mean compression is that averaging is performed only within semantically coherent groups rather than across all memories, so the final sequence of cluster memories preserves multiple preference modes instead of collapsing them into a single blurred representation (Bohdal et al., 24 Jan 2026).

A more structured operator is cluster-wise low-rank factorization. In the hierarchical spatio-channel framework, filters corresponding to a channel cluster $D_e$8 are stacked into a matrix $D_e$9, decomposed by SVD, and truncated to a rank $\{M_i\}_{i=1}^N$0 chosen by an energy threshold $\{M_i\}_{i=1}^N$1 and a cap $\{M_i\}_{i=1}^N$2. At inference, each cluster is realized as a basis convolution followed by a $\{M_i\}_{i=1}^N$3 reconstruction convolution, so compression is local not only in representation space but also in computational structure (Hamlomo et al., 25 Apr 2026).

For sparse activation prediction in LLMs, the representative is a centroid activation mask or sparse prototype. Each cluster centroid is built from continuous activation sums within the cluster, then thresholded so that only the top $\{M_i\}_{i=1}^N$4 of features remain active. At inference, the intended mechanism is to predict a cluster index rather than individual neuron states and then use the corresponding centroid as an approximate activation pattern (Dhar et al., 11 Jul 2025). Here compression and prediction are coupled: the codebook of centroids compresses the activation-mask space, and cluster classification replaces per-neuron prediction.

Structured-codebook variants make the cluster representatives themselves compositional. In the acceleration framework for CNNs, the standard approximation $\{M_i\}_{i=1}^N$5 is replaced by

$\{M_i\}_{i=1}^N$6

where $\{M_i\}_{i=1}^N$7 is a small dictionary, $\{M_i\}_{i=1}^N$8 is sparse, and $\{M_i\}_{i=1}^N$9 is a one-hot assignment matrix. Representatives are therefore not arbitrary centroids but sparse linear combinations of shared atoms, enabling more representatives without losing acceleration (Pikoulis et al., 2021). In MetaCluster for KANs, per-edge coefficient vectors are first shaped onto a low-dimensional manifold by a meta-learner and are then replaced by shared $a_i \in \mathbb{R}^{d}$0-means centroids; after clustering, only the codebook and per-edge indices remain, and the meta-learner is discarded (Raffel et al., 21 Oct 2025).

The common pattern is cluster-conditioned parameter sharing. What differs is the representative class: arithmetic mean, low-rank factor pair, sparse centroid mask, structured centroid, or codebook vector. Each choice encodes a different inductive bias about what should be shared within a cluster.

4. Empirical regimes and observed trade-offs

The empirical literature shows that clustering-based compression is most effective when the baseline alternatives are either too coarse, such as global averaging, or too expensive, such as direct concatenation or uniform full-layer decomposition.

Setting	Clustered object	Reported outcome
On-device LLM personalization	Learned document memories	Clustering uses $a_i \in \mathbb{R}^{d}$1 memory tokens instead of $a_i \in \mathbb{R}^{d}$2 for concatenation and achieves average ROUGE-L $a_i \in \mathbb{R}^{d}$3 on news headlines, $a_i \in \mathbb{R}^{d}$4 on tweet paraphrasing, and $a_i \in \mathbb{R}^{d}$5 on movie tagging (Bohdal et al., 24 Jan 2026)
Sparse FFN execution in Mistral-7B	FFN activation patterns	APC achieves up to $a_i \in \mathbb{R}^{d}$6 clustering precision and PPL as low as $a_i \in \mathbb{R}^{d}$7; predicting cluster indices instead of neuron states gives a theoretical $a_i \in \mathbb{R}^{d}$8 reduction in prediction cost (Dhar et al., 11 Jul 2025)
CNN compression for MRI classification	Spatio-channel activation clusters	At $a_i \in \mathbb{R}^{d}$9 budget, FLOPs drop from $$\mathcal{F} \in \mathbb{R}^{C_{\text{out}} \times H \times W}$$0 to $$\mathcal{F} \in \mathbb{R}^{C_{\text{out}} \times H \times W}$$1, latency improves from $$\mathcal{F} \in \mathbb{R}^{C_{\text{out}} \times H \times W}$$2 ms to $$\mathcal{F} \in \mathbb{R}^{C_{\text{out}} \times H \times W}$$3 ms, and accuracy rises from $$\mathcal{F} \in \mathbb{R}^{C_{\text{out}} \times H \times W}$$4 to $$\mathcal{F} \in \mathbb{R}^{C_{\text{out}} \times H \times W}$$5 (Hamlomo et al., 25 Apr 2026)
KAN parameter/codebook compression	Per-edge coefficient vectors	MetaCluster reports up to $$\mathcal{F} \in \mathbb{R}^{C_{\text{out}} \times H \times W}$$6 reduction in parameter storage with no loss in accuracy (Raffel et al., 21 Oct 2025)

These results expose a recurring trade-off variable. In memory compression, the controlling knobs are the number of clusters $$\mathcal{F} \in \mathbb{R}^{C_{\text{out}} \times H \times W}$$7 and the per-memory length $$\mathcal{F} \in \mathbb{R}^{C_{\text{out}} \times H \times W}$$8; larger $$\mathcal{F} \in \mathbb{R}^{C_{\text{out}} \times H \times W}$$9 improves quality but consumes more tokens, while $\mathbf{r}_s \in \mathbb{R}^{C_{\text{out}}}$0 is equivalent to naive mean compression and performs significantly worse (Bohdal et al., 24 Jan 2026). In sparse FFN clustering, more clusters improve both clustering precision and perplexity, with the paper evaluating $\mathbf{r}_s \in \mathbb{R}^{C_{\text{out}}}$1 and showing severe degradation when codebooks are too small (Dhar et al., 11 Jul 2025). In spatio-channel low-rank compression, $\mathbf{r}_s \in \mathbb{R}^{C_{\text{out}}}$2, $\mathbf{r}_s \in \mathbb{R}^{C_{\text{out}}}$3, $\mathbf{r}_s \in \mathbb{R}^{C_{\text{out}}}$4, and $\mathbf{r}_s \in \mathbb{R}^{C_{\text{out}}}$5 trace out a Pareto frontier, with moderate clustering and adaptive rank selection yielding strong compression–accuracy trade-offs (Hamlomo et al., 25 Apr 2026).

A plausible implication is that cluster count is the primary budget knob in discrete or prototype-based schemes, while cluster count and per-cluster model complexity jointly define the trade-off in factorization-based schemes. This suggests that “clustering-based activation pattern compression framework” is less a single algorithm than a design family whose members differ in what the cluster representative stores.

5. Optimization principles and theoretical viewpoints

A theoretical basis for activation-informed compression is given by the bound on network loss change in terms of layerwise activation perturbations. For compressed linear layers $\mathbf{r}_s \in \mathbb{R}^{C_{\text{out}}}$6, the loss change satisfies

$\mathbf{r}_s \in \mathbb{R}^{C_{\text{out}}}$7

which directly links compression quality to distortion of actual layer outputs rather than to weight error alone. This leads to a bi-objective compression view in which parameter cost and activation-based error are optimized jointly, and under the paper’s assumptions a single uniform tolerance yields surrogate Pareto-optimal heterogeneous ranks across layers (Solgi et al., 7 Oct 2025). For clustering-based frameworks, this suggests using cluster-induced output distortion as the primary design metric and treating cluster counts or codebook sizes as layerwise analogs of heterogeneous ranks.

A second theoretical line comes from hard clustering with regularization. The objective

$\mathbf{r}_s \in \mathbb{R}^{C_{\text{out}}}$8

combines a $\mathbf{r}_s \in \mathbb{R}^{C_{\text{out}}}$9-means distortion term with an $\mathbf{f}_c^{(k)} \in \mathbb{R}^{|\Omega_k|}$0 sparsity penalty, yielding joint quantization and pruning through hard assignments to cluster centers and a zero cluster during hard tying (Yang et al., 2018). Although formulated for weights, the same block-coordinate view extends naturally to activation compression when activations rather than parameters are tied to a small codebook. This suggests a general principle: clustering can be imposed not only post hoc but as an optimization prior.

A third perspective comes from deterministic compression of high-dimensional discrete observations by scaled positional encoding. The mapping is injective for discrete inputs, yields approximate Gaussian compressed coordinates under mild regularity conditions, and preserves separation between cluster centroids in compressed space (D'Angelo et al., 9 Jun 2026). This suggests that when activation patterns are discretized or binarized, deterministic precompression can coexist with clustering and even support model-based clustering in the compressed domain.

Together, these viewpoints imply that the framework has three theoretical layers: compression as activation-distortion control, clustering as an optimization prior, and low-dimensional compressed coordinates as a statistically tractable domain for downstream clustering.

6. Limitations, failure modes, and design principles

The main failure mode is overcompression. In memory merging, $\mathbf{f}_c^{(k)} \in \mathbb{R}^{|\Omega_k|}$1 is equivalent to naive mean compression and performs significantly worse than $\mathbf{f}_c^{(k)} \in \mathbb{R}^{|\Omega_k|}$2, indicating that global averaging mixes heterogeneous content too aggressively (Bohdal et al., 24 Jan 2026). In sparse FFN clustering, small codebooks produce very high perplexity, and even the best clustered model remains worse than the $\mathbf{f}_c^{(k)} \in \mathbb{R}^{|\Omega_k|}$3 sparse unclustered baseline of PPL $\mathbf{f}_c^{(k)} \in \mathbb{R}^{|\Omega_k|}$4, because the reported evaluations assume $\mathbf{f}_c^{(k)} \in \mathbb{R}^{|\Omega_k|}$5 accurate cluster prediction and still incur information loss from centroid substitution (Dhar et al., 11 Jul 2025). In hierarchical CNN compression, overly fine spatial or channel partitioning creates very small clusters, making statistics less reliable and reducing the effectiveness of adaptive rank selection (Hamlomo et al., 25 Apr 2026).

Another limitation is the assumption that the representation space supports meaningful averaging, centroid substitution, or Euclidean partitioning. The memory-compression work explicitly notes that effectiveness depends on memory tokens residing in a space where Euclidean averaging makes semantic sense, and it does not introduce recency weighting or importance weighting within clusters (Bohdal et al., 24 Jan 2026). The medical-imaging framework depends on representative activations collected from a subset of data, so distribution shift can make the learned clusters suboptimal (Hamlomo et al., 25 Apr 2026). APC likewise relies on offline clustering of about $\mathbf{f}_c^{(k)} \in \mathbb{R}^{|\Omega_k|}$6 million activation patterns and leaves the actual cluster predictor for future work (Dhar et al., 11 Jul 2025).

The literature nevertheless converges on several design principles. Cluster first, then compress within clusters rather than compressing globally. Use the number of clusters as a first-class budget parameter. Preserve local structure, whether semantic coherence in memory tokens, co-activation within spatial regions, or active-neuron overlap in sparse FFN masks. Compare against both the global-mean baseline and the full-detail baseline under matched budgets. When representatives are vectors rather than scalars, exploit codebooks, sparse combinations, or low-dimensional manifolds so that multiple coefficients or pattern positions are amortized by a single assignment (Bohdal et al., 24 Jan 2026, Hamlomo et al., 25 Apr 2026, Raffel et al., 21 Oct 2025).

Adjacent structured-compression work reinforces this picture. ADMM-based semi-structured pattern pruning for Transformers formulates block-sparsity constraints explicitly and reshapes dense feature maps into regionally sparsified ones, especially in attention maps, showing that pattern sets and projection operators can serve a role analogous to cluster sets and assignment operators even when the method is not expressed as clustering (Wang, 2024). A plausible implication is that future clustering-based activation pattern compression systems will combine cluster assignment, block-structured sparsity, activation-aware distortion bounds, and hardware-oriented codebooks in a single pipeline rather than treating these as separate compression families.