---
title: Similarity-Based Filter Pruning
url: https://www.emergentmind.com/topics/similarity-based-filter-pruning
type: topic
---

# Similarity-Based Filter Pruning

Similarity-based filter pruning is a structured compression paradigm for convolutional neural networks in which whole filters are removed because they are redundant with other filters, with feature maps, or with synthesized functional patterns, rather than because they merely have small magnitude. Across the literature, redundancy is defined in several spaces: weight space, feature-map space, “functionality space” derived from Activation Maximization, cross-layer consumption in consecutive layers, and second-order covariance structure. These formulations lead to different pruning rules—greedy pairing, clustering, representative election, sketch-based synthesis, and online cluster regularization—but share the goal of preserving expressive diversity while reducing FLOPs, parameters, or MACs [1811.02639][2105.06052][2001.08514][2304.13397].

## 1. Conceptual basis

Similarity-based pruning begins from the hypothesis that similar filters produce similar responses and therefore encode redundant information. In passive variants, pruning decisions are made using only learned parameters, without training data, feature maps, or gradients; in active variants, data-dependent signals such as feature-map similarity, entropy, sparsity, or gradients are also used. This distinction is explicit in acoustic-scene and passive CNN pruning work, where passive methods are described as dataset-agnostic and post hoc, while active methods require forward or backward passes over data [2210.17416][2203.15751].

The central contrast is with norm-based pruning. Magnitude criteria such as $\ell_1$ norm evaluate each filter independently and can retain multiple redundant filters if they all have large norm. In “Demystifying Neural Network Filter Pruning,” this problem is reframed as a failure to model filter functionality: high-magnitude filters may be repetitive, while low-magnitude filters may encode distinct patterns that are important for feature diversity [1811.02639]. Related work makes the same point in different terms: REPrune argues that norm-based methods may keep “adjacent large filters” in weight space, while pruning filters that better represent the whole filter set [2007.06932]. This suggests that “importance” and “redundancy” are not equivalent quantities.

Similarity-based pruning also differs from methods that only detect weak signals. QSFM uses pairwise similarity of output feature maps and then uses matrix rank only as an auxiliary tie-breaker; the feature-statistics method of diversity-aware selection and similarity-aware selection treats low diversity and high similarity as complementary, not interchangeable, conditions [2105.06052][2005.12193]. Second-order approaches such as FilterSketch go further by preserving covariance structure rather than ranking original filters directly, implying that redundancy can be interpreted as a property of the layer’s overall representation, not just of filter pairs [2001.08514].

## 2. Similarity spaces and redundancy measures

The literature uses several distinct similarity spaces, each with its own operational definition of redundancy.

| Similarity space | Representative formulation | Example papers |
|---|---|---|
| Weight space | Cosine, Euclidean, Cityblock, KL, or Ward-clustering distances on vectorized filters | [2203.15751], [2211.01814], [2007.06932] |
| Feature-map space | SSIM, negative Euclidean distance, Pearson correlation, absolute cosine similarity | [2105.06052], [2112.05493], [2005.12193] |
| Functionality / cross-layer / second-order | AM-image distance, consecutive-layer weight convolution, covariance preservation | [1811.02639], [2304.13397], [2001.08514] |

In weight-space methods, a convolutional filter is typically reshaped into a vector or matrix and compared directly. Passive pruning for acoustic scene classification reshapes each filter to a $(wh \times c)$ matrix, computes a rank-1 SVD approximation, normalizes it, and then measures cosine distance
$$
d_{ij} = 1 - \frac{w_i \cdot w_j}{\|w_i\|_2 \, \|w_j\|_2}.
$$
The efficient Nyström-based method uses the same general representation principle, forming a representative matrix $R \in \mathbb{R}^{d \times n}$ and a cosine-similarity Gram matrix
$$
S = R^\top R, \qquad Z = 1 - S,
$$
then approximates $S$ by Nyström sampling to reduce the cost of exact pairwise similarity computation [2203.15751][2210.17416].

Feature-map methods move the redundancy test from weights to outputs. QSFM defines a layer output tensor $O_i^k \in \mathbb{R}^{N_i \times X_i \times Y_i}$ and measures similarity between channel slices either with SSIM or with negative Euclidean distance,
$$
S_{m,n}^{(i,k)} = -\|F_{(i,m)}^k - F_{(i,n)}^k\|_2,
$$
followed by averaging over $M$ images. Central Filter instead computes Pearson correlation between flattened feature maps and builds a similarity graph from the average correlations; the feature-statistics method computes per-feature mean cosine similarity and mean standard deviation to combine similarity-aware and diversity-aware selection [2105.06052][2112.05493][2005.12193].

Functionality-oriented pruning defines similarity even more semantically. In “Demystifying Neural Network Filter Pruning,” each filter $F_i^l$ is assigned an Activation Maximization image
$$
V(F_i^l) = \arg\max_X A_i^l(X), \qquad X \leftarrow X + \eta \cdot \frac{\partial A_i^l(X)}{\partial X},
$$
and functionality redundancy is measured by Euclidean distance in this visualization space,
$$
d_{ij}^l = \|V(F_i^l) - V(F_j^l)\|_2.
$$
This is explicitly presented as different from weight or activation similarity, because the target is the filter’s preferred feature-extraction pattern rather than its raw parameters [1811.02639].

A further alternative is cross-layer similarity. FSCL argues that the usefulness of a filter in layer $l$ depends on how strongly its output channel is used by layer $l+1$. It therefore computes the importance of a filter $w^c_{j_0^c}$ by convolving it with the replicated input-channel slices of the next layer and averaging the resulting $\ell_1$ norms:
$$
\mathcal{L}\!\left(w^{c}_{j_0^c}\right) = \frac{1}{N^n} \sum_{j^n=1}^{N^n} \left\|\, w^{c}_{j_0^c} \;\otimes\; \hat{w}^{n}_{j^n, j_0^c} \,\right\|_1.
$$
This formulation treats redundancy as a property of inter-layer consumption rather than purely intra-layer similarity [2304.13397].

## 3. Algorithmic families

A large class of methods starts from pairwise similarity and prunes greedily. Passive cosine-distance pruning builds a pairwise distance matrix, finds each filter’s nearest neighbor, sorts pairs by ascending distance, and greedily marks one filter as important and the other as redundant. The Nyström-based variant preserves the same logic while approximating the similarity matrix as
$$
\hat{S} = C W_k^{+} C^\top,
$$
reducing the main cost from $O(n^2 d)$ to $O(m^3 + nmk)$ when $m,k \ll n$ [2203.15751][2210.17416].

A second family uses clustering and representative election. REPrune applies agglomerative hierarchical clustering with Ward’s linkage,
$$
d(\mathcal{C}_A, \mathcal{C}_B) =
\frac{| \mathcal{C}_A |\,| \mathcal{C}_B |}{| \mathcal{C}_A | + | \mathcal{C}_B |}\,
\big\| \mathbf{m}_{\mathcal{C}_A} - \mathbf{m}_{\mathcal{C}_B} \big\|_2^2,
$$
searches over the number of clusters using the mean Silhouette Coefficient, and then selects one representative per valid cluster as the filter closest to the centroid. The feature-statistics method similarly retains one representative feature and deletes others whose similarity to that reference exceeds a threshold $\nu$, after first discarding low-diversity maps by a global threshold $\beta$ on mean standard deviation [2007.06932][2005.12193].

Functionality-aware clustering replaces weight-space geometry with visualization-space geometry. “Demystifying Neural Network Filter Pruning” clusters Activation Maximization images by K-means, interprets large clusters as repetitive functionality, and then prunes within clusters according to a gradient-based contribution index
$$
\gamma_i = \frac{1}{N} \sum_{n=1}^N \left\| \frac{\partial Z}{\partial A_i(x_n)} \right\|.
$$
The stated rationale is that larger clusters reflect more repetitive functionality and can be pruned more aggressively, whereas filters with “extremely minimal similarity” are preserved to protect unique functions [1811.02639].

Several methods incorporate similarity during training rather than only after training. “Online Filter Clustering and Pruning for Efficient Convnets” fixes size-1 and size-2 clusters in advance and adds a cluster loss
$$
f(K) = E(K)+\delta R(K)+\lambda \sum_{i=1}^{L}\sum_{t=1}^{T}\|k_i^t-c_i^t\|^2,
$$
with $c_i^t$ the mean filter in cluster $S_i^t$, so that filters within each cluster become nearly equal during optimization. “Self Similarity Matrix based CNN Filter Pruning” recomputes a self-similarity matrix from vectorized filters after each training epoch, scores each filter either by its minimum row similarity or by an area-under-row criterion, and prunes a fixed fraction such as $\rho=10\%$ per epoch without a separate finetuning stage [1905.11787][2211.01814].

A related but distinct formulation is Asymptotic Soft Cluster Pruning. ASCP clusters filters by Euclidean distance, reconstructs the most redundant filters by setting them to cluster centroids, and allows them to continue updating. The pruning rate increases asymptotically, so redundancy is introduced gradually rather than by hard deletion. Once filters in a cluster become identical, they can be removed by channel addition in the next layer with almost no accuracy drop [2206.08186].

FilterSketch occupies a separate algorithmic category. Instead of selecting representatives from the original set, it computes a Frequent Directions sketch $\Omega^i$ such that $\Omega^i(\Omega^i)^\top$ approximates $W^i(W^i)^\top$, thereby preserving second-order structure. In this sense it is similarity-based through covariance preservation rather than through explicit pairwise pruning [2001.08514].

## 4. Reconstruction, fine-tuning, and structural consistency

A recurrent issue in this literature is whether retraining is compensatory or merely restorative. “Demystifying Neural Network Filter Pruning” argues that after $\ell_1$ pruning, retraining substantially changes the functionality of retained filters—for example, an Activation Maximization image changing from a “cat” to a “deer”—which is interpreted as network-wide functionality reconstruction after critical filters were wrongly removed. By contrast, when pruning is guided by functionality redundancy, retained filters preserve their visualized patterns and retraining becomes “relatively unnecessary” or limited to short fine-tuning [1811.02639].

Other methods also try to reduce the capacity gap introduced by hard pruning. Online clustering makes filters within a cluster nearly equal during training and then uses channel merging in the next layer; ASCP reconstructs redundant filters to centroids before removing them; Central Filter explicitly adjusts next-layer weights so that a central filter absorbs the contribution of pruned neighbors. In the linearized form used by Central Filter, if $Y_j^{(l)} \approx \gamma_j Y_c^{(l)}$, then the next-layer weights are updated by
$$
W^{(l+1)}_{p,c} \leftarrow W^{(l+1)}_{p,c} + \sum_{j \in S_c} \gamma_j W^{(l+1)}_{p,j},
$$
so that the downstream pre-activation remains close to its original value [1905.11787][2206.08186][2112.05493].

Structured pruning also requires exact architectural surgery. Across the surveyed methods, pruning a filter implies removing the corresponding output channel, deleting the matching input channel of the next convolution, pruning associated biases, and pruning BatchNorm parameters $(\gamma,\beta,\text{running mean/variance})$ where present. Residual and multi-branch networks require additional constraints: REPrune avoids pruning identity shortcuts and the last point-wise convolution in ResNet-50 bottlenecks; QSFM prunes only the first convolution in each residual block of ResNet-56 to preserve tensor shapes; FSCL defines separate handling for Inception branches and for residual paths with convolutional shortcuts [2007.06932][2105.06052][2304.13397].

Fine-tuning schedules vary substantially. Passive acoustic-scene pruning uses Adam with learning rate $0.001$ for $30$ epochs, approximately seven times fewer than the $200$ epochs used to train the unpruned baseline [2203.15751]. Efficient passive similarity pruning fine-tunes with the same optimizer and loss as training for fewer epochs, using $100$ epochs instead of $200$ originally [2210.17416]. FilterSketch uses SGD with Nesterov momentum $0.9$, with separate schedules for CIFAR-10 and ImageNet, and attributes its effectiveness to a “second-order warm start” from the sketch initialization [2001.08514]. This variation indicates that similarity-based pruning does not imply a single retraining doctrine; rather, the need for fine-tuning depends on how redundancy is modeled and how aggressively structure is altered.

## 5. Empirical performance and application regimes

Reported results span image classification, acoustic scene classification, and edge deployment. The evidence is heterogeneous: some papers emphasize qualitative analyses of functional stability, while others report accuracy, FLOPs, parameter, MAC, latency, or pruning-time reductions.

| Method | Setting | Reported outcome |
|---|---|---|
| FilterSketch [2001.08514] | ResNet-110, CIFAR-10 | FLOPs reduced by $\approx 63.3\%$; parameters reduced by $\approx 59.9\%$; Top-1 $93.50\% \rightarrow 93.44\%$ |
| REPrune [2007.06932] | ResNet-110, CIFAR-10 | More than $49\%$ FLOPs reduction with a $0.53\%$ accuracy gain |
| QSFM [2105.06052] | ResNet-56, CIFAR-10, without fine-tuning | $48.7\%$ FLOPs and $57.9\%$ parameters reduced with only a loss of $0.54\%$ in the Top-1 accuracy |
| Efficient passive similarity pruning [2210.17416] | VGGish_Net, acoustic scene classification | Accuracy $64.69\% \rightarrow 62.00\%$; MACs $903$M $\rightarrow 595$M; params $55.361$M $\rightarrow 42.89$M; pruning time $34.80$s $\rightarrow 11.70$s |
| Central Filter [2112.05493] | ResNet-50, ImageNet | Approximately $47.9\%$ FLOPs reduction and $36.9\%$ parameter reduction with only a small loss of $1.07\%$ in Top-1 accuracy |
| FSCL [2304.13397] | ResNet-50, ImageNet | Top-1 $75.84\%$ with drop $0.31\%$; Top-5 $92.79\%$ with drop $0.08\%$; FLOPs $1.80$B; params $11.78$M |

The application scope is notably broad. Similarity-based passive pruning has been evaluated on DCASE 2021 Task 1A baseline and VGGish networks for acoustic scene classification, where the Nyström approximation reproduces the exact similarity-based decisions while being about three times faster on the larger model [2210.17416]. QSFM extends feature-map similarity pruning to MobileNet-V2 and reports practical acceleration on edge devices, including inference speedup from $22.97$ ms to $15.03$ ms on NVIDIA Jetson TX2 GPU and from $63.23$ ms to $43.35$ ms on NVIDIA Jetson TX2 CPU for ImageNet [2105.06052].

Some results are primarily qualitative rather than numerical. The functionality-oriented study centers on VGG-16 on CIFAR-10 and emphasizes slower accuracy drop under layer-wise and model-wise pruning, stable Activation Maximization patterns after pruning, and faster stabilization during recovery, but it does not report exact FLOP reductions, parameter counts, or accuracy numbers in the text [1811.02639]. This is important for interpretation: similarity-based pruning has both quantitative and qualitative evidence, and not all papers optimize the same evaluation axis.

## 6. Limitations, controversies, and research directions

The major limitation is that redundancy is representation-dependent. Weight similarity, feature-map similarity, functional similarity, cross-layer consumption, and second-order covariance do not induce the same pruning decisions. “Demystifying Neural Network Filter Pruning” explicitly argues that magnitude and even weight redundancy can miss repetitive functionality [1811.02639], whereas passive methods argue that weight-only similarity is attractive because it is data-free and simple to deploy [2210.17416][2203.15751]. This suggests that there is no universally accepted answer to the question of what a “redundant filter” is.

Computational overhead is a second recurring issue. Exact pairwise similarity matrices scale quadratically in the number of filters; feature-map methods require passes over calibration data or even full training sets; Activation Maximization must be run per filter and pairwise distances then computed in visualization space; clustering plus silhouette search can become expensive in wide layers [2210.17416][2105.06052][1811.02639][2007.06932]. Approximation strategies such as Nyström sampling and sketching via Frequent Directions are direct responses to this bottleneck [2210.17416][2001.08514].

Sensitivity to hyperparameters and architecture is also repeatedly noted. REPrune depends on the minimum cluster rate $\lambda$ and on silhouette-driven cluster search; QSFM depends on the choice between SSIM and negative Euclidean distance and on dataset size $M$ for stable similarity estimates; the feature-statistics method depends on thresholds $\beta$ and $\nu$; ASCP depends on the number of clusters and on the pruning-rate schedule; functionality-oriented pruning depends on the number of K-means clusters and on the quality of Activation Maximization images [2007.06932][2105.06052][2005.12193][2206.08186][1811.02639]. Several papers also caution that early layers, very narrow layers, or layers with weak similarity structure may be poor pruning targets.

Structural constraints remain a practical barrier. Residual paths, inception branches, bottlenecks, BatchNorm bookkeeping, and depthwise or grouped convolutions can all complicate direct filter removal. Some methods explicitly avoid certain layers; others require special handling or only demonstrate results on standard spatial convolutions [2007.06932][2304.13397][2105.06052]. Generalization beyond the demonstrated settings is also uneven: the functionality-oriented method is mainly shown on VGG-16/CIFAR-10, while passive and feature-map methods cover a wider architecture set [1811.02639][2210.17416][2105.06052].

A plausible implication is that future progress will continue to combine three themes already visible in the literature: better redundancy representations, cheaper similarity estimation, and tighter integration with other compression tools. The efficient passive pruning work explicitly notes that similarity-based pruning can be combined with quantization and low-rank factorization [2210.17416], while second-order and cross-layer formulations indicate that pruning criteria may increasingly move from isolated filter scores toward layer-level or inter-layer structure [2001.08514][2304.13397]. Within the current literature, the unifying conclusion is narrower but clear: when redundancy is modeled directly, rather than inferred from magnitude alone, filter pruning can preserve feature diversity more effectively and often recover accuracy with less reconstruction overhead.

Source: https://www.emergentmind.com/topics/similarity-based-filter-pruning