Similarity-Based Filter Pruning

• Similarity-based filter pruning is a structured compression technique that removes redundant CNN filters based on similarities in weight, feature-map, or functionality spaces.
• It employs methods such as greedy pairing, clustering, and sketch-based synthesis to preserve feature diversity while efficiently reducing computational costs.
• Techniques like Nyström sampling and frequent directions help manage computational overhead, yielding significant reductions in FLOPs and parameters with minimal accuracy loss.

Similarity-based filter pruning is a structured compression paradigm for convolutional neural networks in which whole filters are removed because they are redundant with other filters, with feature maps, or with synthesized functional patterns, rather than because they merely have small magnitude. Across the literature, redundancy is defined in several spaces: weight space, feature-map space, “functionality space” derived from Activation Maximization, cross-layer consumption in consecutive layers, and second-order covariance structure. These formulations lead to different pruning rules—greedy pairing, clustering, representative election, sketch-based synthesis, and online cluster regularization—but share the goal of preserving expressive diversity while reducing FLOPs, parameters, or MACs (Qin et al., 2018, Wang et al., 2021, Lin et al., 2020, Wang et al., 2023).

1. Conceptual basis

Similarity-based pruning begins from the hypothesis that similar filters produce similar responses and therefore encode redundant information. In passive variants, pruning decisions are made using only learned parameters, without training data, feature maps, or gradients; in active variants, data-dependent signals such as feature-map similarity, entropy, sparsity, or gradients are also used. This distinction is explicit in acoustic-scene and passive CNN pruning work, where passive methods are described as dataset-agnostic and post hoc, while active methods require forward or backward passes over data (Singh et al., 2022, Singh et al., 2022).

The central contrast is with norm-based pruning. Magnitude criteria such as $\ell_1$ norm evaluate each filter independently and can retain multiple redundant filters if they all have large norm. In “Demystifying Neural Network Filter Pruning,” this problem is reframed as a failure to model filter functionality: high-magnitude filters may be repetitive, while low-magnitude filters may encode distinct patterns that are important for feature diversity (Qin et al., 2018). Related work makes the same point in different terms: REPrune argues that norm-based methods may keep “adjacent large filters” in weight space, while pruning filters that better represent the whole filter set (Park et al., 2020). This suggests that “importance” and “redundancy” are not equivalent quantities.

Similarity-based pruning also differs from methods that only detect weak signals. QSFM uses pairwise similarity of output feature maps and then uses matrix rank only as an auxiliary tie-breaker; the feature-statistics method of diversity-aware selection and similarity-aware selection treats low diversity and high similarity as complementary, not interchangeable, conditions (Wang et al., 2021, Li et al., 2020). Second-order approaches such as FilterSketch go further by preserving covariance structure rather than ranking original filters directly, implying that redundancy can be interpreted as a property of the layer’s overall representation, not just of filter pairs (Lin et al., 2020).

2. Similarity spaces and redundancy measures

The literature uses several distinct similarity spaces, each with its own operational definition of redundancy.

Similarity space	Representative formulation	Example papers
Weight space	Cosine, Euclidean, Cityblock, KL, or Ward-clustering distances on vectorized filters	(Singh et al., 2022, Rakshith et al., 2022, Park et al., 2020)
Feature-map space	SSIM, negative Euclidean distance, Pearson correlation, absolute cosine similarity	(Wang et al., 2021, Duan et al., 2021, Li et al., 2020)
Functionality / cross-layer / second-order	AM-image distance, consecutive-layer weight convolution, covariance preservation	(Qin et al., 2018, Wang et al., 2023, Lin et al., 2020)

In weight-space methods, a convolutional filter is typically reshaped into a vector or matrix and compared directly. Passive pruning for acoustic scene classification reshapes each filter to a $(wh \times c)$ matrix, computes a rank-1 SVD approximation, normalizes it, and then measures cosine distance

$$d_{ij} = 1 - \frac{w_i \cdot w_j}{\|w_i\|_2 \, \|w_j\|_2}.$$

The efficient Nyström-based method uses the same general representation principle, forming a representative matrix $R \in \mathbb{R}^{d \times n}$ and a cosine-similarity Gram matrix

$S = R^\top R, \qquad Z = 1 - S,$

then approximates $S$ by Nyström sampling to reduce the cost of exact pairwise similarity computation (Singh et al., 2022, Singh et al., 2022).

Feature-map methods move the redundancy test from weights to outputs. QSFM defines a layer output tensor $O_i^k \in \mathbb{R}^{N_i \times X_i \times Y_i}$ and measures similarity between channel slices either with SSIM or with negative Euclidean distance,

$$S_{m,n}^{(i,k)} = -\|F_{(i,m)}^k - F_{(i,n)}^k\|_2,$$

followed by averaging over $M$ images. Central Filter instead computes Pearson correlation between flattened feature maps and builds a similarity graph from the average correlations; the feature-statistics method computes per-feature mean cosine similarity and mean standard deviation to combine similarity-aware and diversity-aware selection (Wang et al., 2021, Duan et al., 2021, Li et al., 2020).

Functionality-oriented pruning defines similarity even more semantically. In “Demystifying Neural Network Filter Pruning,” each filter $F_i^l$ is assigned an Activation Maximization image

$(wh \times c)$0

and functionality redundancy is measured by Euclidean distance in this visualization space,

$(wh \times c)$1

This is explicitly presented as different from weight or activation similarity, because the target is the filter’s preferred feature-extraction pattern rather than its raw parameters (Qin et al., 2018).

A further alternative is cross-layer similarity. FSCL argues that the usefulness of a filter in layer $(wh \times c)$2 depends on how strongly its output channel is used by layer $(wh \times c)$3. It therefore computes the importance of a filter $(wh \times c)$4 by convolving it with the replicated input-channel slices of the next layer and averaging the resulting $(wh \times c)$5 norms:

$(wh \times c)$6

This formulation treats redundancy as a property of inter-layer consumption rather than purely intra-layer similarity (Wang et al., 2023).

3. Algorithmic families

A large class of methods starts from pairwise similarity and prunes greedily. Passive cosine-distance pruning builds a pairwise distance matrix, finds each filter’s nearest neighbor, sorts pairs by ascending distance, and greedily marks one filter as important and the other as redundant. The Nyström-based variant preserves the same logic while approximating the similarity matrix as

$(wh \times c)$7

reducing the main cost from $(wh \times c)$8 to $(wh \times c)$9 when $$d_{ij} = 1 - \frac{w_i \cdot w_j}{\|w_i\|_2 \, \|w_j\|_2}.$$0 (Singh et al., 2022, Singh et al., 2022).

A second family uses clustering and representative election. REPrune applies agglomerative hierarchical clustering with Ward’s linkage,

$$d_{ij} = 1 - \frac{w_i \cdot w_j}{\|w_i\|_2 \, \|w_j\|_2}.$$1

searches over the number of clusters using the mean Silhouette Coefficient, and then selects one representative per valid cluster as the filter closest to the centroid. The feature-statistics method similarly retains one representative feature and deletes others whose similarity to that reference exceeds a threshold $$d_{ij} = 1 - \frac{w_i \cdot w_j}{\|w_i\|_2 \, \|w_j\|_2}.$$2, after first discarding low-diversity maps by a global threshold $$d_{ij} = 1 - \frac{w_i \cdot w_j}{\|w_i\|_2 \, \|w_j\|_2}.$$3 on mean standard deviation (Park et al., 2020, Li et al., 2020).

Functionality-aware clustering replaces weight-space geometry with visualization-space geometry. “Demystifying Neural Network Filter Pruning” clusters Activation Maximization images by K-means, interprets large clusters as repetitive functionality, and then prunes within clusters according to a gradient-based contribution index

$$d_{ij} = 1 - \frac{w_i \cdot w_j}{\|w_i\|_2 \, \|w_j\|_2}.$$4

The stated rationale is that larger clusters reflect more repetitive functionality and can be pruned more aggressively, whereas filters with “extremely minimal similarity” are preserved to protect unique functions (Qin et al., 2018).

Several methods incorporate similarity during training rather than only after training. “Online Filter Clustering and Pruning for Efficient Convnets” fixes size-1 and size-2 clusters in advance and adds a cluster loss

$$d_{ij} = 1 - \frac{w_i \cdot w_j}{\|w_i\|_2 \, \|w_j\|_2}.$$5

with $$d_{ij} = 1 - \frac{w_i \cdot w_j}{\|w_i\|_2 \, \|w_j\|_2}.$$6 the mean filter in cluster $$d_{ij} = 1 - \frac{w_i \cdot w_j}{\|w_i\|_2 \, \|w_j\|_2}.$$7, so that filters within each cluster become nearly equal during optimization. “Self Similarity Matrix based CNN Filter Pruning” recomputes a self-similarity matrix from vectorized filters after each training epoch, scores each filter either by its minimum row similarity or by an area-under-row criterion, and prunes a fixed fraction such as $$d_{ij} = 1 - \frac{w_i \cdot w_j}{\|w_i\|_2 \, \|w_j\|_2}.$$8 per epoch without a separate finetuning stage (Zhou et al., 2019, Rakshith et al., 2022).

A related but distinct formulation is Asymptotic Soft Cluster Pruning. ASCP clusters filters by Euclidean distance, reconstructs the most redundant filters by setting them to cluster centroids, and allows them to continue updating. The pruning rate increases asymptotically, so redundancy is introduced gradually rather than by hard deletion. Once filters in a cluster become identical, they can be removed by channel addition in the next layer with almost no accuracy drop (Niu et al., 2022).

FilterSketch occupies a separate algorithmic category. Instead of selecting representatives from the original set, it computes a Frequent Directions sketch $$d_{ij} = 1 - \frac{w_i \cdot w_j}{\|w_i\|_2 \, \|w_j\|_2}.$$9 such that $R \in \mathbb{R}^{d \times n}$0 approximates $R \in \mathbb{R}^{d \times n}$1, thereby preserving second-order structure. In this sense it is similarity-based through covariance preservation rather than through explicit pairwise pruning (Lin et al., 2020).

4. Reconstruction, fine-tuning, and structural consistency

A recurrent issue in this literature is whether retraining is compensatory or merely restorative. “Demystifying Neural Network Filter Pruning” argues that after $R \in \mathbb{R}^{d \times n}$2 pruning, retraining substantially changes the functionality of retained filters—for example, an Activation Maximization image changing from a “cat” to a “deer”—which is interpreted as network-wide functionality reconstruction after critical filters were wrongly removed. By contrast, when pruning is guided by functionality redundancy, retained filters preserve their visualized patterns and retraining becomes “relatively unnecessary” or limited to short fine-tuning (Qin et al., 2018).

Other methods also try to reduce the capacity gap introduced by hard pruning. Online clustering makes filters within a cluster nearly equal during training and then uses channel merging in the next layer; ASCP reconstructs redundant filters to centroids before removing them; Central Filter explicitly adjusts next-layer weights so that a central filter absorbs the contribution of pruned neighbors. In the linearized form used by Central Filter, if $R \in \mathbb{R}^{d \times n}$3, then the next-layer weights are updated by

$R \in \mathbb{R}^{d \times n}$4

so that the downstream pre-activation remains close to its original value (Zhou et al., 2019, Niu et al., 2022, Duan et al., 2021).

Structured pruning also requires exact architectural surgery. Across the surveyed methods, pruning a filter implies removing the corresponding output channel, deleting the matching input channel of the next convolution, pruning associated biases, and pruning BatchNorm parameters $R \in \mathbb{R}^{d \times n}$5 where present. Residual and multi-branch networks require additional constraints: REPrune avoids pruning identity shortcuts and the last point-wise convolution in ResNet-50 bottlenecks; QSFM prunes only the first convolution in each residual block of ResNet-56 to preserve tensor shapes; FSCL defines separate handling for Inception branches and for residual paths with convolutional shortcuts (Park et al., 2020, Wang et al., 2021, Wang et al., 2023).

Fine-tuning schedules vary substantially. Passive acoustic-scene pruning uses Adam with learning rate $R \in \mathbb{R}^{d \times n}$6 for $R \in \mathbb{R}^{d \times n}$7 epochs, approximately seven times fewer than the $R \in \mathbb{R}^{d \times n}$8 epochs used to train the unpruned baseline (Singh et al., 2022). Efficient passive similarity pruning fine-tunes with the same optimizer and loss as training for fewer epochs, using $R \in \mathbb{R}^{d \times n}$9 epochs instead of $S = R^\top R, \qquad Z = 1 - S,$0 originally (Singh et al., 2022). FilterSketch uses SGD with Nesterov momentum $S = R^\top R, \qquad Z = 1 - S,$1, with separate schedules for CIFAR-10 and ImageNet, and attributes its effectiveness to a “second-order warm start” from the sketch initialization (Lin et al., 2020). This variation indicates that similarity-based pruning does not imply a single retraining doctrine; rather, the need for fine-tuning depends on how redundancy is modeled and how aggressively structure is altered.

5. Empirical performance and application regimes

Reported results span image classification, acoustic scene classification, and edge deployment. The evidence is heterogeneous: some papers emphasize qualitative analyses of functional stability, while others report accuracy, FLOPs, parameter, MAC, latency, or pruning-time reductions.

Method	Setting	Reported outcome
FilterSketch (Lin et al., 2020)	ResNet-110, CIFAR-10	FLOPs reduced by $S = R^\top R, \qquad Z = 1 - S,$2; parameters reduced by $S = R^\top R, \qquad Z = 1 - S,$3; Top-1 $S = R^\top R, \qquad Z = 1 - S,$4
REPrune (Park et al., 2020)	ResNet-110, CIFAR-10	More than $S = R^\top R, \qquad Z = 1 - S,$5 FLOPs reduction with a $S = R^\top R, \qquad Z = 1 - S,$6 accuracy gain
QSFM (Wang et al., 2021)	ResNet-56, CIFAR-10, without fine-tuning	$S = R^\top R, \qquad Z = 1 - S,$7 FLOPs and $S = R^\top R, \qquad Z = 1 - S,$8 parameters reduced with only a loss of $S = R^\top R, \qquad Z = 1 - S,$9 in the Top-1 accuracy
Efficient passive similarity pruning (Singh et al., 2022)	VGGish_Net, acoustic scene classification	Accuracy $S$0; MACs $S$1M $S$2M; params $S$3M $S$4M; pruning time $S$5s $S$6s
Central Filter (Duan et al., 2021)	ResNet-50, ImageNet	Approximately $S$7 FLOPs reduction and $S$8 parameter reduction with only a small loss of $S$9 in Top-1 accuracy
FSCL (Wang et al., 2023)	ResNet-50, ImageNet	Top-1 $O_i^k \in \mathbb{R}^{N_i \times X_i \times Y_i}$0 with drop $O_i^k \in \mathbb{R}^{N_i \times X_i \times Y_i}$1; Top-5 $O_i^k \in \mathbb{R}^{N_i \times X_i \times Y_i}$2 with drop $O_i^k \in \mathbb{R}^{N_i \times X_i \times Y_i}$3; FLOPs $O_i^k \in \mathbb{R}^{N_i \times X_i \times Y_i}$4B; params $O_i^k \in \mathbb{R}^{N_i \times X_i \times Y_i}$5M

The application scope is notably broad. Similarity-based passive pruning has been evaluated on DCASE 2021 Task 1A baseline and VGGish networks for acoustic scene classification, where the Nyström approximation reproduces the exact similarity-based decisions while being about three times faster on the larger model (Singh et al., 2022). QSFM extends feature-map similarity pruning to MobileNet-V2 and reports practical acceleration on edge devices, including inference speedup from $O_i^k \in \mathbb{R}^{N_i \times X_i \times Y_i}$6 ms to $O_i^k \in \mathbb{R}^{N_i \times X_i \times Y_i}$7 ms on NVIDIA Jetson TX2 GPU and from $O_i^k \in \mathbb{R}^{N_i \times X_i \times Y_i}$8 ms to $O_i^k \in \mathbb{R}^{N_i \times X_i \times Y_i}$9 ms on NVIDIA Jetson TX2 CPU for ImageNet (Wang et al., 2021).

Some results are primarily qualitative rather than numerical. The functionality-oriented study centers on VGG-16 on CIFAR-10 and emphasizes slower accuracy drop under layer-wise and model-wise pruning, stable Activation Maximization patterns after pruning, and faster stabilization during recovery, but it does not report exact FLOP reductions, parameter counts, or accuracy numbers in the text (Qin et al., 2018). This is important for interpretation: similarity-based pruning has both quantitative and qualitative evidence, and not all papers optimize the same evaluation axis.

6. Limitations, controversies, and research directions

The major limitation is that redundancy is representation-dependent. Weight similarity, feature-map similarity, functional similarity, cross-layer consumption, and second-order covariance do not induce the same pruning decisions. “Demystifying Neural Network Filter Pruning” explicitly argues that magnitude and even weight redundancy can miss repetitive functionality (Qin et al., 2018), whereas passive methods argue that weight-only similarity is attractive because it is data-free and simple to deploy (Singh et al., 2022, Singh et al., 2022). This suggests that there is no universally accepted answer to the question of what a “redundant filter” is.

Computational overhead is a second recurring issue. Exact pairwise similarity matrices scale quadratically in the number of filters; feature-map methods require passes over calibration data or even full training sets; Activation Maximization must be run per filter and pairwise distances then computed in visualization space; clustering plus silhouette search can become expensive in wide layers (Singh et al., 2022, Wang et al., 2021, Qin et al., 2018, Park et al., 2020). Approximation strategies such as Nyström sampling and sketching via Frequent Directions are direct responses to this bottleneck (Singh et al., 2022, Lin et al., 2020).

Sensitivity to hyperparameters and architecture is also repeatedly noted. REPrune depends on the minimum cluster rate $$S_{m,n}^{(i,k)} = -\|F_{(i,m)}^k - F_{(i,n)}^k\|_2,$$0 and on silhouette-driven cluster search; QSFM depends on the choice between SSIM and negative Euclidean distance and on dataset size $$S_{m,n}^{(i,k)} = -\|F_{(i,m)}^k - F_{(i,n)}^k\|_2,$$1 for stable similarity estimates; the feature-statistics method depends on thresholds $$S_{m,n}^{(i,k)} = -\|F_{(i,m)}^k - F_{(i,n)}^k\|_2,$$2 and $$S_{m,n}^{(i,k)} = -\|F_{(i,m)}^k - F_{(i,n)}^k\|_2,$$3; ASCP depends on the number of clusters and on the pruning-rate schedule; functionality-oriented pruning depends on the number of K-means clusters and on the quality of Activation Maximization images (Park et al., 2020, Wang et al., 2021, Li et al., 2020, Niu et al., 2022, Qin et al., 2018). Several papers also caution that early layers, very narrow layers, or layers with weak similarity structure may be poor pruning targets.

Structural constraints remain a practical barrier. Residual paths, inception branches, bottlenecks, BatchNorm bookkeeping, and depthwise or grouped convolutions can all complicate direct filter removal. Some methods explicitly avoid certain layers; others require special handling or only demonstrate results on standard spatial convolutions (Park et al., 2020, Wang et al., 2023, Wang et al., 2021). Generalization beyond the demonstrated settings is also uneven: the functionality-oriented method is mainly shown on VGG-16/CIFAR-10, while passive and feature-map methods cover a wider architecture set (Qin et al., 2018, Singh et al., 2022, Wang et al., 2021).

A plausible implication is that future progress will continue to combine three themes already visible in the literature: better redundancy representations, cheaper similarity estimation, and tighter integration with other compression tools. The efficient passive pruning work explicitly notes that similarity-based pruning can be combined with quantization and low-rank factorization (Singh et al., 2022), while second-order and cross-layer formulations indicate that pruning criteria may increasingly move from isolated filter scores toward layer-level or inter-layer structure (Lin et al., 2020, Wang et al., 2023). Within the current literature, the unifying conclusion is narrower but clear: when redundancy is modeled directly, rather than inferred from magnitude alone, filter pruning can preserve feature diversity more effectively and often recover accuracy with less reconstruction overhead.