Clustering and Merge-Based Compression

Updated 24 April 2026

Clustering and merge-based compression are techniques that group similar elements to reduce redundancy and computational costs across various applications.
They employ distance metrics and optimization strategies, such as k-means and greedy merges, to build hierarchies and efficiently quantize model parameters.
These methods are applied in neural networks, memory systems, and statistical models to achieve high compression ratios with minimal loss in fidelity.

Clustering and merge-based compression refer to a broad class of algorithmic strategies that exploit similarity among objects—contexts, weights, memories, or data blocks—to reduce redundancy, size, and computational cost. These approaches span statistical modeling, neural network quantization, memory and matrix compression, and structural data summarization. They are unified by the central paradigm of grouping (clustering) representations and merging (sharing, tying, or averaging) elements within each group, thereby achieving substantial gains in storage, speed, and generalization with only marginal loss of information.

1. Theoretical Principles of Clustering and Merge-Based Compression

Clustering as a precursor to compression is grounded in the information-theoretic observation that high-dimensional or combinatorial data spaces often feature regions of local homogeneity, where the distinction between elements contributes negligible predictive or coding value. By forming clusters using similarity or distance metrics (e.g., Kullback-Leibler divergence for distributions, squared Euclidean for vectors, normalized compression distance for strings), one can replace many fine-grained entities with a smaller number of representative prototypes or joint models.

Merge-based (agglomerative) algorithms explicitly construct hierarchies or partitions by greedily selecting merges that minimally increase some cost function—typically entropy, coding rate, reconstruction error, or other fidelity metrics. The merge process generates a sequence of hierarchical states, each corresponding to a particular trade-off between representational economy and descriptive detail.

For parametric model compression, clustering implements "parameter tying," where weights, probabilities, or other coefficients are constrained to share values within each cluster. The theoretical efficiency is achieved by minimizing a loss function that incorporates both the original task objective and terms penalizing deviation from cluster centroids, such as in k-means objectives or regularization-based formulations (Yang et al., 2018, Cho et al., 2021).

2. Clustering and Binning in Sequential and Statistical Models

In the context of statistical sequence modeling, such as context-based (Markov) models for compression, the explosion of possible contexts with increasing order ( $|\mathcal{A}|^l$ for order- $l$ ) makes storing and updating all context-specific statistics infeasible for large alphabets. Context binning is a principled solution that partitions the vast space of contexts into $K \ll |\mathcal{A}|^l$ bins, assigning to each bin a shared conditional distribution (Duda, 2022):

Let $C = \mathcal{A}^l$ denote all possible contexts; assign $s = \text{bin}(c)$ for $c\in C$ .
All $c$ in the same bin $s$ share aggregate distribution $P_s(a) = (1/p_s)\sum_{c:\text{bin}(c)=s} p(c)P_c(a)$ .
The binned model's code length penalty $J$ is the KL divergence between original and binned distributions.
Optimal (greedy) binning merges the context pairs with the smallest increase in rate, building a merge tree for flexible model size control.

In empirical studies, context binning achieves drastic reductions in the number of states (e.g., $l$ 0 to $l$ 1 for quality scores at $l$ 2 bpv penalty), enabling efficient lookup-based implementations and near-optimal compression rates (Duda, 2022).

For block-wise or non-stationary sources, model clustering via k-means in the space of entire statistical models (viewed as high-dimensional probability vectors) selects a small number of prototype distributions ("centroids") used to encode each sequence or block with minimal per-symbol overhead (Duda, 2022).

3. Neural Network Compression via Clustering and Parameter Merging

Clustering-based neural network compression enforces weight sharing by partitioning model parameters into clusters tied to shared centroids. This achieves quantization and enables parameter pruning, often through additional regularization (Yang et al., 2018):

The compression objective combines the original task loss, a k-means quantization term, and an $l$ 3 penalty for sparsity:

$l$ 4

where $l$ 5 is the k-means loss and $l$ 6 enforces sparsity.

Alternating block coordinate descent updates parameter assignments (clustering) and centroids, followed by a "hard tying" phase where all weights in a cluster strictly share the same value.
After training, only the centroids and cluster indices need storage; theoretical and empirical results confirm high compression factors (up to $l$ 7) with minor impact on test accuracy (Yang et al., 2018).

Differentiable clustering, as in DKM, formulates k-means as a soft-attention process within the standard training loop, optimizing both weights and centroids under the original task loss. Gradients are backpropagated through the assignment layer, and final model storage is based on centroid tables and assignment indices, yielding large compression ratios with minimal post-processing (Cho et al., 2021).

4. Merge-Based Compression of Mixture-of-Expert and Structural Models

For large-scale models using structural modularity, such as Mixture-of-Experts (MoE) architectures, expert merging addresses memory bottlenecks by reducing the number of distinct experts without significantly affecting inference quality (Miao et al., 16 Oct 2025):

MergeMoE formulates merging as an optimization over cluster assignments $l$ 8 and merge weights $l$ 9, seeking to minimize the approximation error of expert outputs.
Hard clustering of experts is performed using frequency-based heuristics or parameter similarity.
Merged experts are parameterized via block concatenation and weighted linear maps, and least-squares procedures optimize the compression mappings.
Empirical evaluations confirm state-of-the-art compression/accuracy trade-offs for LLM-scale MoE models (Miao et al., 16 Oct 2025).

In topological and scientific data, tree-based objects (merge trees) can be compressed using auto-encoder architectures with clustering penalties in the latent space, ensuring that both metric structure and semantic clusters are preserved after dimensionality reduction and compression (Pont et al., 2023).

5. Compression-Aware Clustering in Memory Systems and Matrix Collections

Context-aware memory and retrieval tasks in LLMs increasingly leverage clustering-driven memory compression. By grouping memory representations by similarity (in high-dimensional embedding space) and merging within clusters, these methods deliver substantially reduced context size while preserving personalization and semantic diversity of memories (Bohdal et al., 24 Jan 2026):

Documents are represented as $K \ll |\mathcal{A}|^l$ 0, and the set is clustered (typically with k-means or similar), then merged within clusters.
Compression ratio is directly controlled by cluster count $K \ll |\mathcal{A}|^l$ 1; $K \ll |\mathcal{A}|^l$ 2 governs the token budget.
Empirical results in generation and retrieval show superiority over both naive concatenation and mean-pooling for a fixed context budget (Bohdal et al., 24 Jan 2026).

For collections of matrices, as occur in multi-view learning and blockwise model parameterization, the question of which matrices can be safely merged into shared low-rank representations is central. Error-constrained clustering via SVD compression establishes spectral upper bounds and provides greedy algorithms for cluster formation consistent with user-specified error budgets (Shamrai, 12 Jan 2026):

Two main classes of upper bounds are employed: monotonicity-based (Weyl), which is fast but conservative, and residual-based, which are tighter but more computation-intensive.
Incremental SVD estimators allow for scalable, online, and memory-efficient tracking of joint compression error, enabling merge decisions for extremely large collections.
Classical clustering methods are inadequate since they do not control approximation error after joint low-rank representation (Shamrai, 12 Jan 2026).

6. Compression-Based Clustering Using Algorithmic Information Theory

Compression-based clustering leverages the normalized compression distance (NCD) between objects (e.g., strings derived from EEG segments) as a universal, parameter-free similarity metric (Sarasa et al., 31 Jan 2025):

Objects are first formed by transforming and discretizing the data into compression-friendly strings.
NCD between pairs is calculated via off-the-shelf lossless compressors; agglomerative or quartet-based algorithms generate hierarchical cluster trees.
This strategy is robust to noise, requires no domain-specific feature engineering, and excels in applications where standard vector-based distances are suboptimal.
Computational cost scales quadratically with sample size, limiting application to cohorts up to a few hundred objects without approximation (Sarasa et al., 31 Jan 2025).

7. Trade-offs, Extensions, and Cross-Domain Generalization

All clustering and merge-based compression schemes embody a balance between compression ratio and fidelity, with multiple tunable parameters governing cluster count, merge thresholds, and error tolerances. Increasing the granularity of clusters diminishes redundancy but incurs increased complexity and risk of information loss. Empirical and theoretical analyses consistently show diminishing returns for very large cluster counts and rapid quality loss for excessive merging.

These methods generalize across modalities (sequences, matrices, trees, model parameters), semantic content, and even algorithmic regimes (neural, probabilistic, symbolic), provided a suitable similarity measure or objective function exists. They underpin memory efficiency and adaptability in large-scale learning and scientific computing domains, and, when paired with adaptive or online updating (e.g., EMA for nonstationarity, fine-tuning centroids), enable flexible handling of dynamic statistics and growing datasets (Duda, 2022, Bohdal et al., 24 Jan 2026).

Merge-based clustering also serves as a foundation for subsequent tasks such as knowledge distillation, continual learning, and cross-domain compression, often enabling efficient storage, fast retrieval, and enhanced generalization without substantial overhead or domain limits (Miao et al., 16 Oct 2025, Pont et al., 2023).

References: