Similarity-Based Pruning in Neural Networks

• Similarity-Based Pruning (SP) is a strategy that identifies and removes redundant components in neural networks by evaluating their similarity in weight, activation, or token space.
• The approach spans various architectures—CNNs, LLMs, MLLMs, and data-pruning—by employing metrics like cosine similarity, CKA, and cluster-based measures to guide pruning decisions.
• Practical implementations of SP have demonstrated notable reductions in FLOPs and enhanced training efficiency, while highlighting trade-offs in computational cost and semantic relevance.

Searching arXiv for recent and foundational work on similarity-based pruning across CNNs, LLMs, MLLMs, and related settings. Similarity-Based Pruning (SP) denotes pruning strategies that remove, merge, or skip structures judged redundant by an explicit similarity criterion rather than by magnitude alone. Across the literature, the pruned object varies substantially—convolutional filters, channels, residual blocks, Transformer layers, visual tokens, training pairs, reasoning branches, or streaming candidates—but the shared premise is that highly similar objects contribute overlapping information and can often be removed with limited loss of function or accuracy (Singh et al., 2022, Chen et al., 21 Apr 2026, Zhang et al., 12 Jun 2025, Fang et al., 17 Mar 2026).

1. Conceptual scope and usage

In CNN compression, SP is often defined as a passive, data-free pruning framework that relies only on filter parameters and inter-filter similarity. The canonical motivation is that highly similar filters produce redundant feature maps, so pruning one filter from each similar pair or group reduces computation while preserving representational diversity (Singh et al., 2022). This framing explicitly contrasts with active pruning, which scores filters using feature maps and other data-dependent quantities.

The same label is used more broadly in later work. In multimodal LLMs, similarity-based pruning removes redundant visual tokens by examining pairwise similarity between token embeddings; in this setting, redundancy means that tokens are near-duplicates in feature space and keeping all of them increases inference cost without adding useful visual information (Zhang et al., 12 Jun 2025). In depth pruning for Transformer-based LLMs, SP refers to identifying layers whose outputs are highly similar to their inputs, interpreting such layers as near-identity transformations and therefore prunable (Chen et al., 21 Apr 2026). In dense retrieval adaptation, SP denotes a static data-pruning regime that keeps only the highest-similarity positive query-document pairs before finetuning (Fang et al., 17 Mar 2026).

This variety of usage shows that SP is not tied to a single architecture or granularity. The invariant component is the ranking principle: pruning is driven by explicit similarity or distance among candidate objects, or by similarity between a pruned candidate and a reference system.

2. Core mathematical patterns

Despite domain differences, SP methods tend to instantiate one of a small number of recurring constructions: pairwise similarity matrices, similarity to an anchor or centroid, similarity between adjacent representations, or similarity between original and candidate-pruned models.

Setting	Pruned object	Similarity signal
Passive CNN filter pruning	Convolutional filters	Cosine similarity of unit-norm rank-1 SVD representatives
Cluster/reconstruction CNN pruning	Filters or channels	Euclidean or cosine similarity to centroids or similar retained channels
Layer/depth pruning	Residual blocks or Transformer layers	CKA between original and candidate-pruned representations, or cosine similarity between adjacent hidden states
MLLM and LLM token pruning	Visual tokens or decoding-time attention steps	Cosine similarity between token embeddings, semantic embeddings, or joint key-value similarity
Retrieval/data pruning	Positive training pairs or candidate pairs	Cosine similarity of normalized embeddings or distance-threshold tests

For passive CNN filter pruning, a convolutional layer with $F$ filters is represented by reshaping each filter to $F^l \in \mathbb{R}^{d \times c}$ with $d = wh$, computing a rank-1 SVD

$F^l \approx \sigma_1 u_1 v_1^\top,$

and taking the unit-norm left singular vector $r_l = u_1$ as the filter representative. Stacking representatives gives $R = [r_1,\dots,r_F] \in \mathbb{R}^{d \times F}$, from which the pairwise cosine similarity matrix is

$S = R^\top R,$

and the cosine distance matrix is $Z = 1 - S$ (Singh et al., 2022).

For depth pruning in LLMs, similarity-only methods compute the cosine dissimilarity between adjacent hidden states,

$$L_{\text{sim}}^{(i)} = 1 - \frac{1}{B \cdot S} \sum_{j=1}^{B \cdot S} \cos(H_j^{(i)}, H_j^{(i+1)}),$$

so that a small value indicates a layer that acts close to identity (Chen et al., 21 Apr 2026). A different layer-pruning formulation compares the penultimate representations of the dense model and a candidate model with one layer removed using CKA; the layer whose removal yields the highest preservation of representation is treated as least important (Pons et al., 2024).

For static retrieval pruning, similarity is simply the cosine between normalized query and document embeddings,

$s(q,d) = e_q^\top e_d,$

and pruning keeps the top-scoring positive pairs globally (Fang et al., 17 Mar 2026).

3. CNN filter and channel pruning

The most developed SP literature concerns CNNs. A representative passive method computes a layerwise pairwise similarity matrix, identifies for each filter its closest neighbor

$F^l \in \mathbb{R}^{d \times c}$0

sorts the resulting pairs by ascending distance, and then constructs an important list and a redundant list by retaining one filter from each closest pair and marking the counterpart as redundant. Removing a filter also removes the corresponding output feature map and the associated input channel in the next convolutional layer, after which the pruned network is fine-tuned (Singh et al., 2022).

A central practical issue is the cost of computing the full pairwise similarity matrix. In the passive CNN formulation above, exhaustive construction of $F^l \in \mathbb{R}^{d \times c}$1 requires $F^l \in \mathbb{R}^{d \times c}$2 operations and $F^l \in \mathbb{R}^{d \times c}$3 memory, which becomes expensive for layers with many filters. The Nyström-based approximation replaces the full kernel with

$F^l \in \mathbb{R}^{d \times c}$4

using only $F^l \in \mathbb{R}^{d \times c}$5 sampled columns, and thereby reduces the cost to $F^l \in \mathbb{R}^{d \times c}$6; empirically, this yields approximately $F^l \in \mathbb{R}^{d \times c}$7 faster pruning while preserving pruning decisions and downstream accuracy at fixed pruning ratios (Singh et al., 2022).

Other CNN variants reinterpret similarity more structurally. ASCP clusters filters with k-means, reconstructs the most redundant filters toward their assigned centroids during training, and finally merges asymptotically identical filters through channel addition in the next layer; the paper reports, for example, 48.5% FLOPs reduction on ImageNet ResNet-50 with 0.43% Top-1 drop and 0.15% Top-5 drop (Niu et al., 2022). FSCL moves from within-layer similarity to cross-layer reuse: it scores a filter in layer $F^l \in \mathbb{R}^{d \times c}$8 by how strongly the corresponding input channels of layer $F^l \in \mathbb{R}^{d \times c}$9 reuse that feature, using a convolution-based similarity between the filter and the next-layer channel slices; this explicitly exploits the one-to-one coupling between an output channel of layer $d = wh$0 and an input channel of layer $d = wh$1 (Wang et al., 2023). SSM-based pruning instead computes a self-similarity matrix over vectorized 2D filters, then ranks filters either by their nearest-neighbor distance or by a global row-area criterion; pruning is integrated into training, and no post-training fine-tuning is required (Rakshith et al., 2022).

Data-free SP can also be combined with reconstruction. AutoDFP uses cosine distance among channels together with BatchNorm-derived bias terms, formulates pruning and reconstruction as an optimization problem, and learns per-layer preserved ratios and similarity-bias trade-offs through Soft Actor-Critic. Its reconstruction rule

$d = wh$2

treats a pruned channel as partially compensable by a similar retained channel, with the scaling factor computed from weights and BatchNorm statistics rather than data (Li et al., 2024).

A recurring theme across these CNN methods is that SP seeks redundancy directly. Norm-based pruning can remove low-norm but unique filters, whereas SP explicitly targets filters that are close to others in weight or representation space.

4. Layer and depth pruning

SP at the layer level appears in two notably different forms. One form evaluates a candidate layer removal by comparing the representation of the pruned model with that of the original model. In the CKA-based method, the penultimate representation of the original network on an unlabeled input set is compared with the penultimate representation of a temporary model with one block removed; the layer importance is

$d = wh$3

The layer with minimal $d = wh$4 is pruned, then the resulting model is fine-tuned, and the process repeats. Because the criterion measures preservation of learned representation rather than weight scale, it directly targets functional redundancy (Pons et al., 2024).

A second form, common in LLM depth pruning, measures similarity only between the input and output of the same layer. In this formulation, high cosine similarity between adjacent hidden states is interpreted as evidence that the layer performs a near-identity transformation and can be removed. However, this heuristic can be brittle. SimDiff reports that similarity-only pruning can exhibit abrupt performance collapse across architectures: on Mistral-7B-v0.3, pruning 12 layers with the similarity-only ShortGPT baseline raises WikiText2 perplexity from 42.2 to 1,833.0, and similar instability appears on LLaMA3.1-8B (Chen et al., 21 Apr 2026).

The critique is structural. Residual architectures can preserve directional alignment while still applying important corrective magnitude changes. SimDiff therefore augments similarity with two difference metrics—MSSD, which is outlier-sensitive, and MASD, which measures average contribution more robustly—and combines them into a joint importance score. This does not negate SP, but it narrows its safe operating regime: mild pruning can be effective with similarity alone, whereas moderate to aggressive pruning may require additional signals (Chen et al., 21 Apr 2026).

5. Token pruning and online pruning in large models

In multimodal LLMs, SP commonly refers to instruction-agnostic pruning of visual tokens. The standard workflow is to compute pairwise cosine similarity

$d = wh$5

then retain a subset through thresholding, clustering, or diversity-first heuristics. This works when redundancy is largely prompt-independent, such as in high-resolution images or long videos, but it ignores instruction relevance and may preserve visually diverse yet irrelevant tokens while discarding instruction-critical ones (Zhang et al., 12 Jun 2025).

This limitation motivated methods that explicitly go beyond similarity. CDPruner defines instruction-conditioned relevance, constructs a conditional kernel $d = wh$6, and uses a DPP objective to maximize conditional diversity. The paper is framed explicitly as moving “beyond attention or similarity,” which makes it a useful boundary marker: similarity-based token pruning is a strong baseline, but not necessarily sufficient when the prompt determines what redundancy means (Zhang et al., 12 Jun 2025).

Online SP has also emerged in reasoning and autoregressive inference. SSDP inserts semantic clustering into parallel Tree-of-Thought search: each generated reasoning step is embedded, sibling candidates are clustered by cosine similarity, and only the highest-reward representative of each cluster is kept. With a default threshold of 0.75, this sibling-local merging reduces generated nodes by 85–90% on average and yields up to about 2.3× speedup over DPTS while maintaining competitive accuracy, typically within 5% of the strongest baseline (Kim et al., 30 Oct 2025).

Token Filtering applies online SP during LLM decoding rather than search. For each head and layer, it compares the current token’s key and value with exponential-moving-average anchors of prior keys and values, fuses the resulting similarities with inverse-variance weighting, averages across heads to obtain $d = wh$7, and skips attention in tail layers when this score exceeds a learned threshold. On LLaMA-2-13B at batch 128 and 50% pruning, the method reports 46.6% latency reduction and 33.6% memory reduction while preserving stronger accuracy than prior structured pruning baselines on commonsense reasoning and MMLU (Lee et al., 8 Dec 2025).

6. Data pruning and broader uses beyond model compression

SP is also used for pruning data rather than model structure. In OPERA, static pruning computes cosine similarity for every positive query-document training pair under the pretrained retriever, retains only the highest-similarity pairs, and fine-tunes on that subset. This produces a clear quality-coverage tradeoff: average NDCG@10 rises from 0.535 to 0.537, but average Recall@20 drops from 0.530 to 0.512 because queries with few surviving positives become underrepresented (Fang et al., 17 Mar 2026).

The same paper makes the denoising interpretation explicit. High-similarity positives are more likely to provide clean ranking supervision, so SP can improve ranking quality and greatly accelerate convergence. On FEVER, the pruned setup reaches competitive NDCG in fewer than 500 iterations, whereas standard finetuning and InfoBatch require about 16,000 steps to peak (Fang et al., 17 Mar 2026). At the same time, the paper argues that hard pruning is often too static, motivating its two-stage dynamic pruning alternative.

An older non-neural example appears in DRSP for time-series data streams. There, similarity matching is followed by a pruning stage that filters pseudo data-object pairs, and the framework combines Multi-level Segment Means dimension reduction with a center-radius pruning lemma. The paper states the MSM cost as $d = wh$8 and the pruning cost as $d = wh$9, where DRF is the Dimension Reduction Factor (Vishwanath et al., 2013). Although this setting predates contemporary network compression, it shows that “similarity-based pruning” also has a broader algorithmic meaning: candidate elimination driven by similarity or distance structure.

7. Advantages, limitations, and recurrent points of contention

The main advantage of SP is that it targets redundancy directly. In passive CNN pruning, this is the explicit argument for why SP can outperform norm-based baselines at the same computation and parameter budget: norm criteria do not distinguish unique low-norm filters from redundant high-norm ones, whereas pairwise similarity models inter-filter redundancy directly (Singh et al., 2022). The same logic recurs elsewhere. In retrieval adaptation, SP acts as a denoiser by concentrating training on high-quality pairs (Fang et al., 17 Mar 2026). In online decoding and reasoning, similarity gives an input-adaptive proxy for redundancy that does not require offline calibration (Kim et al., 30 Oct 2025, Lee et al., 8 Dec 2025).

Its limitations are equally persistent. First, similarity computation itself can be expensive: full pairwise filter matrices have quadratic time and memory cost, which is precisely why Nyström approximation was introduced for passive CNN pruning (Singh et al., 2022). Second, metric choice matters. In SSM-based CNN pruning, L2 and Cityblock distances produce small accuracy drops at modest compression, whereas cosine and KL variants can degrade accuracy severely on CIFAR-10 VGG and ResNet-18 (Rakshith et al., 2022). Third, similarity alone may be semantically incomplete. In MLLM token pruning, instruction-agnostic similarity ignores what the question asks; in depth pruning, cosine similarity between adjacent hidden states can miss decisive residual corrections and produce catastrophic collapse under aggressive pruning (Zhang et al., 12 Jun 2025, Chen et al., 21 Apr 2026).

A common misconception is that SP is necessarily data-free. This is true for passive CNN variants built only from filters or BatchNorm statistics, but false for many later uses. Layer pruning via CKA depends on a calibration set of inputs, similarity-only depth pruning in LLMs uses hidden states computed on calibration data, retrieval SP scores training pairs under a pretrained model, and MLLM token pruning uses the current image and prompt (Singh et al., 2022, Pons et al., 2024, Fang et al., 17 Mar 2026, Zhang et al., 12 Jun 2025).

Another misconception is that SP is a single fixed algorithm. The literature instead presents a design space. Similarity may be measured in weight space, activation space, hidden-state space, token-embedding space, key-value space, or representation-preservation space; selection may be pairwise, cluster-based, kernel-based, or online threshold-based; and pruning may remove parameters, layers, tokens, data pairs, or search branches. This suggests that SP is best understood as a general pruning paradigm centered on explicit redundancy modeling rather than as one standardized procedure.