---
title: Similarity-Based Pruning in Neural Networks
url: https://www.emergentmind.com/topics/similarity-based-pruning-sp
type: topic
---

# Similarity-Based Pruning in Neural Networks

Searching arXiv for recent and foundational work on similarity-based pruning across CNNs, LLMs, MLLMs, and related settings.
Similarity-Based Pruning (SP) denotes pruning strategies that remove, merge, or skip structures judged redundant by an explicit similarity criterion rather than by magnitude alone. Across the literature, the pruned object varies substantially—convolutional filters, channels, residual blocks, Transformer layers, visual tokens, training pairs, reasoning branches, or streaming candidates—but the shared premise is that highly similar objects contribute overlapping information and can often be removed with limited loss of function or accuracy [2210.17416][2604.19520][2506.10967][2603.17205].

## 1. Conceptual scope and usage

In CNN compression, SP is often defined as a passive, data-free pruning framework that relies only on filter parameters and inter-filter similarity. The canonical motivation is that highly similar filters produce redundant feature maps, so pruning one filter from each similar pair or group reduces computation while preserving representational diversity [2210.17416]. This framing explicitly contrasts with active pruning, which scores filters using feature maps and other data-dependent quantities.

The same label is used more broadly in later work. In multimodal large language models, similarity-based pruning removes redundant visual tokens by examining pairwise similarity between token embeddings; in this setting, redundancy means that tokens are near-duplicates in feature space and keeping all of them increases inference cost without adding useful visual information [2506.10967]. In depth pruning for Transformer-based language models, SP refers to identifying layers whose outputs are highly similar to their inputs, interpreting such layers as near-identity transformations and therefore prunable [2604.19520]. In dense retrieval adaptation, SP denotes a static data-pruning regime that keeps only the highest-similarity positive query-document pairs before finetuning [2603.17205].

This variety of usage shows that SP is not tied to a single architecture or granularity. The invariant component is the ranking principle: pruning is driven by explicit similarity or distance among candidate objects, or by similarity between a pruned candidate and a reference system.

## 2. Core mathematical patterns

Despite domain differences, SP methods tend to instantiate one of a small number of recurring constructions: pairwise similarity matrices, similarity to an anchor or centroid, similarity between adjacent representations, or similarity between original and candidate-pruned models.

| Setting | Pruned object | Similarity signal |
|---|---|---|
| Passive CNN filter pruning | Convolutional filters | Cosine similarity of unit-norm rank-1 SVD representatives |
| Cluster/reconstruction CNN pruning | Filters or channels | Euclidean or cosine similarity to centroids or similar retained channels |
| Layer/depth pruning | Residual blocks or Transformer layers | CKA between original and candidate-pruned representations, or cosine similarity between adjacent hidden states |
| MLLM and LLM token pruning | Visual tokens or decoding-time attention steps | Cosine similarity between token embeddings, semantic embeddings, or joint key-value similarity |
| Retrieval/data pruning | Positive training pairs or candidate pairs | Cosine similarity of normalized embeddings or distance-threshold tests |

For passive CNN filter pruning, a convolutional layer with $F$ filters is represented by reshaping each filter to $F^l \in \mathbb{R}^{d \times c}$ with $d = wh$, computing a rank-1 SVD
$$
F^l \approx \sigma_1 u_1 v_1^\top,
$$
and taking the unit-norm left singular vector $r_l = u_1$ as the filter representative. Stacking representatives gives $R = [r_1,\dots,r_F] \in \mathbb{R}^{d \times F}$, from which the pairwise cosine similarity matrix is
$$
S = R^\top R,
$$
and the cosine distance matrix is $Z = 1 - S$ [2210.17416].

For depth pruning in LLMs, similarity-only methods compute the cosine dissimilarity between adjacent hidden states,
$$
L_{\text{sim}}^{(i)} = 1 - \frac{1}{B \cdot S} \sum_{j=1}^{B \cdot S} \cos(H_j^{(i)}, H_j^{(i+1)}),
$$
so that a small value indicates a layer that acts close to identity [2604.19520]. A different layer-pruning formulation compares the penultimate representations of the dense model and a candidate model with one layer removed using CKA; the layer whose removal yields the highest preservation of representation is treated as least important [2405.17081].

For static retrieval pruning, similarity is simply the cosine between normalized query and document embeddings,
$$
s(q,d) = e_q^\top e_d,
$$
and pruning keeps the top-scoring positive pairs globally [2603.17205].

## 3. CNN filter and channel pruning

The most developed SP literature concerns CNNs. A representative passive method computes a layerwise pairwise similarity matrix, identifies for each filter its closest neighbor
$$
q(l) = \arg\min_{j \neq l} z_{lj},
$$
sorts the resulting pairs by ascending distance, and then constructs an important list and a redundant list by retaining one filter from each closest pair and marking the counterpart as redundant. Removing a filter also removes the corresponding output feature map and the associated input channel in the next convolutional layer, after which the pruned network is fine-tuned [2210.17416].

A central practical issue is the cost of computing the full pairwise similarity matrix. In the passive CNN formulation above, exhaustive construction of $S$ requires $O(F^2 d)$ operations and $O(F^2)$ memory, which becomes expensive for layers with many filters. The Nyström-based approximation replaces the full kernel with
$$
\hat{S} = C W_k^{+} C^\top,
$$
using only $m \ll F$ sampled columns, and thereby reduces the cost to $O(m^3 + Fmk)$; empirically, this yields approximately $3\times$ faster pruning while preserving pruning decisions and downstream accuracy at fixed pruning ratios [2210.17416].

Other CNN variants reinterpret similarity more structurally. ASCP clusters filters with k-means, reconstructs the most redundant filters toward their assigned centroids during training, and finally merges asymptotically identical filters through channel addition in the next layer; the paper reports, for example, 48.5% FLOPs reduction on ImageNet ResNet-50 with 0.43% Top-1 drop and 0.15% Top-5 drop [2206.08186]. FSCL moves from within-layer similarity to cross-layer reuse: it scores a filter in layer $l$ by how strongly the corresponding input channels of layer $l+1$ reuse that feature, using a convolution-based similarity between the filter and the next-layer channel slices; this explicitly exploits the one-to-one coupling between an output channel of layer $l$ and an input channel of layer $l+1$ [2304.13397]. SSM-based pruning instead computes a self-similarity matrix over vectorized 2D filters, then ranks filters either by their nearest-neighbor distance or by a global row-area criterion; pruning is integrated into training, and no post-training fine-tuning is required [2211.01814].

Data-free SP can also be combined with reconstruction. AutoDFP uses cosine distance among channels together with BatchNorm-derived bias terms, formulates pruning and reconstruction as an optimization problem, and learns per-layer preserved ratios and similarity-bias trade-offs through Soft Actor-Critic. Its reconstruction rule
$$
\tilde{W}_r^{(\ell+1)} = W_r^{(\ell+1)} + s_{pr} W_p^{(\ell+1)}
$$
treats a pruned channel as partially compensable by a similar retained channel, with the scaling factor computed from weights and BatchNorm statistics rather than data [2403.08204].

A recurring theme across these CNN methods is that SP seeks redundancy directly. Norm-based pruning can remove low-norm but unique filters, whereas SP explicitly targets filters that are close to others in weight or representation space.

## 4. Layer and depth pruning

SP at the layer level appears in two notably different forms. One form evaluates a candidate layer removal by comparing the representation of the pruned model with that of the original model. In the CKA-based method, the penultimate representation of the original network on an unlabeled input set is compared with the penultimate representation of a temporary model with one block removed; the layer importance is
$$
I(\ell) = 1 - \mathrm{CKA}(R, R_\ell).
$$
The layer with minimal $I(\ell)$ is pruned, then the resulting model is fine-tuned, and the process repeats. Because the criterion measures preservation of learned representation rather than weight scale, it directly targets functional redundancy [2405.17081].

A second form, common in LLM depth pruning, measures similarity only between the input and output of the same layer. In this formulation, high cosine similarity between adjacent hidden states is interpreted as evidence that the layer performs a near-identity transformation and can be removed. However, this heuristic can be brittle. SimDiff reports that similarity-only pruning can exhibit abrupt performance collapse across architectures: on Mistral-7B-v0.3, pruning 12 layers with the similarity-only ShortGPT baseline raises WikiText2 perplexity from 42.2 to 1,833.0, and similar instability appears on LLaMA3.1-8B [2604.19520].

The critique is structural. Residual architectures can preserve directional alignment while still applying important corrective magnitude changes. SimDiff therefore augments similarity with two difference metrics—MSSD, which is outlier-sensitive, and MASD, which measures average contribution more robustly—and combines them into a joint importance score. This does not negate SP, but it narrows its safe operating regime: mild pruning can be effective with similarity alone, whereas moderate to aggressive pruning may require additional signals [2604.19520].

## 5. Token pruning and online pruning in large models

In multimodal large language models, SP commonly refers to instruction-agnostic pruning of visual tokens. The standard workflow is to compute pairwise cosine similarity
$$
L_{ij} = \frac{v_i \cdot v_j}{\|v_i\| \, \|v_j\|},
$$
then retain a subset through thresholding, clustering, or diversity-first heuristics. This works when redundancy is largely prompt-independent, such as in high-resolution images or long videos, but it ignores instruction relevance and may preserve visually diverse yet irrelevant tokens while discarding instruction-critical ones [2506.10967].

This limitation motivated methods that explicitly go beyond similarity. CDPruner defines instruction-conditioned relevance, constructs a conditional kernel $\tilde{L} = \mathrm{diag}(\tilde{r}) L \mathrm{diag}(\tilde{r})$, and uses a DPP objective to maximize conditional diversity. The paper is framed explicitly as moving “beyond attention or similarity,” which makes it a useful boundary marker: similarity-based token pruning is a strong baseline, but not necessarily sufficient when the prompt determines what redundancy means [2506.10967].

Online SP has also emerged in reasoning and autoregressive inference. SSDP inserts semantic clustering into parallel Tree-of-Thought search: each generated reasoning step is embedded, sibling candidates are clustered by cosine similarity, and only the highest-reward representative of each cluster is kept. With a default threshold of 0.75, this sibling-local merging reduces generated nodes by 85–90% on average and yields up to about 2.3× speedup over DPTS while maintaining competitive accuracy, typically within 5% of the strongest baseline [2511.08595].

Token Filtering applies online SP during LLM decoding rather than search. For each head and layer, it compares the current token’s key and value with exponential-moving-average anchors of prior keys and values, fuses the resulting similarities with inverse-variance weighting, averages across heads to obtain $S_{KV}^{(\ell)}(t)$, and skips attention in tail layers when this score exceeds a learned threshold. On LLaMA-2-13B at batch 128 and 50% pruning, the method reports 46.6% latency reduction and 33.6% memory reduction while preserving stronger accuracy than prior structured pruning baselines on commonsense reasoning and MMLU [2512.07090].

## 6. Data pruning and broader uses beyond model compression

SP is also used for pruning data rather than model structure. In OPERA, static pruning computes cosine similarity for every positive query-document training pair under the pretrained retriever, retains only the highest-similarity pairs, and fine-tunes on that subset. This produces a clear quality-coverage tradeoff: average NDCG@10 rises from 0.535 to 0.537, but average Recall@20 drops from 0.530 to 0.512 because queries with few surviving positives become underrepresented [2603.17205].

The same paper makes the denoising interpretation explicit. High-similarity positives are more likely to provide clean ranking supervision, so SP can improve ranking quality and greatly accelerate convergence. On FEVER, the pruned setup reaches competitive NDCG in fewer than 500 iterations, whereas standard finetuning and InfoBatch require about 16,000 steps to peak [2603.17205]. At the same time, the paper argues that hard pruning is often too static, motivating its two-stage dynamic pruning alternative.

An older non-neural example appears in DRSP for time-series data streams. There, similarity matching is followed by a pruning stage that filters pseudo data-object pairs, and the framework combines Multi-level Segment Means dimension reduction with a center-radius pruning lemma. The paper states the MSM cost as $O(l \cdot n_i)$ and the pruning cost as $O(\mathrm{DRF} \cdot wsize \cdot d)$, where DRF is the Dimension Reduction Factor [1312.2669]. Although this setting predates contemporary network compression, it shows that “similarity-based pruning” also has a broader algorithmic meaning: candidate elimination driven by similarity or distance structure.

## 7. Advantages, limitations, and recurrent points of contention

The main advantage of SP is that it targets redundancy directly. In passive CNN pruning, this is the explicit argument for why SP can outperform norm-based baselines at the same computation and parameter budget: norm criteria do not distinguish unique low-norm filters from redundant high-norm ones, whereas pairwise similarity models inter-filter redundancy directly [2210.17416]. The same logic recurs elsewhere. In retrieval adaptation, SP acts as a denoiser by concentrating training on high-quality pairs [2603.17205]. In online decoding and reasoning, similarity gives an input-adaptive proxy for redundancy that does not require offline calibration [2511.08595][2512.07090].

Its limitations are equally persistent. First, similarity computation itself can be expensive: full pairwise filter matrices have quadratic time and memory cost, which is precisely why Nyström approximation was introduced for passive CNN pruning [2210.17416]. Second, metric choice matters. In SSM-based CNN pruning, L2 and Cityblock distances produce small accuracy drops at modest compression, whereas cosine and KL variants can degrade accuracy severely on CIFAR-10 VGG and ResNet-18 [2211.01814]. Third, similarity alone may be semantically incomplete. In MLLM token pruning, instruction-agnostic similarity ignores what the question asks; in depth pruning, cosine similarity between adjacent hidden states can miss decisive residual corrections and produce catastrophic collapse under aggressive pruning [2506.10967][2604.19520].

A common misconception is that SP is necessarily data-free. This is true for passive CNN variants built only from filters or BatchNorm statistics, but false for many later uses. Layer pruning via CKA depends on a calibration set of inputs, similarity-only depth pruning in LLMs uses hidden states computed on calibration data, retrieval SP scores training pairs under a pretrained model, and MLLM token pruning uses the current image and prompt [2210.17416][2405.17081][2603.17205][2506.10967].

Another misconception is that SP is a single fixed algorithm. The literature instead presents a design space. Similarity may be measured in weight space, activation space, hidden-state space, token-embedding space, key-value space, or representation-preservation space; selection may be pairwise, cluster-based, kernel-based, or online threshold-based; and pruning may remove parameters, layers, tokens, data pairs, or search branches. This suggests that SP is best understood as a general pruning paradigm centered on explicit redundancy modeling rather than as one standardized procedure.

Source: https://www.emergentmind.com/topics/similarity-based-pruning-sp