Similarity Group Pruning in Deep Models

• Similarity Group Pruning (SGP) is a structured pruning method that groups similar neural components to identify and remove redundancy while preserving key representations.
• In explicit SGP formulations like FedSODA, angular distance metrics are used to select non-overlapping layer groups, achieving significant compression in large language models.
• Broader SGP approaches extend to channel, filter, and token pruning in CNNs and multimodal models, though they may require complementary metrics to balance efficiency and accuracy.

Searching arXiv for the cited SGP-related papers to ground the article in current arXiv records. Similarity Group Pruning (SGP) denotes a family of structured pruning methods that use similarity relations to identify redundant model components and retain a smaller representative subset. In current arXiv usage, the term appears explicitly in FedSODA, where an SGP module prunes redundant transformer layers from an LLM emulator by selecting non-overlapping layer groups with minimal representational angular distance on a public dataset (Zhu et al., 18 Aug 2025). Closely related work applies the same redundancy intuition to residual blocks, channels, filters, feature maps, and visual tokens, but often without explicit group formation; accordingly, SGP spans both strict group-based formulations and a broader family of similarity-driven pruning methods (Zhang et al., 2019).

1. Terminology and scope

One recurrent source of confusion is terminological. In the supplied literature, “Similarity Group Pruning” is explicit in FedSODA. Other papers are conceptually adjacent but not terminologically identical: some use similarity without explicit grouping, some use grouping without an explicit learned similarity metric, and some approximate grouping through graph neighborhoods rather than clusters. This suggests a useful distinction between a strict reading of SGP, in which pruning acts on explicitly selected similarity groups, and a broader reading, in which similarity merely supplies a pruning score.

Formulation	Representative paper	Salient property
Explicit SGP	FedSODA (Zhu et al., 18 Aug 2025)	Non-overlapping layer groups via angular distance
Similarity without groups	(Pons et al., 2024)	Each layer scored individually against the dense model
Hierarchical similarity clustering	(Zhang et al., 2019)	One representative channel kept per cluster
Implicit graph neighborhoods	SAINT (Jeddi et al., 14 Mar 2025)	Thresholded token-similarity graph

A second source of confusion is false attribution. In the supplied record for “SGLP: A Similarity Guided Fast Layer Partition Pruning for Compressing Large Deep Models,” the document is described as an Elsevier LaTeX template/sample manuscript and therefore cannot be used as evidence about SGLP, SGP, CKA-based segmentation, GradNorm, or pruning experiments (Li et al., 2024). In encyclopedic treatment, that record is better regarded as non-evidentiary for the technical history of SGP.

2. Explicit layer-group pruning in LLMs

The clearest explicit SGP formulation in the supplied literature appears in FedSODA. There the full model is decomposed as $\mathcal{M} = [\mathcal{E}, \mathcal{A}]$, where $\mathcal{E}$ is an emulator and $\mathcal{A}$ is an adapter. SGP compresses the emulator into a sub-emulator $\mathcal{E}^*$ by scanning contiguous groups of size $n+1$, computing the angular distance between the hidden states at the first and last layer of each group, selecting the top $p$ non-overlapping groups with the smallest summed angular distances, and then preserving the first layer while removing the next $n$ layers in each selected group. The similarity measure is

$$d(x^{(\ell)}, x^{(\ell+n)}) \equiv \frac{1}{\pi} \arccos{\frac{x_S^{(\ell)} \cdot x_S^{(\ell+n)}}{\|x_S^{(\ell)}\| \|x_S^{(\ell+n)}\|}}.$$

After pruning, the sub-emulator depth becomes $L_{\mathcal{E}^*} = L_{\mathcal{E} - n \times p}$, and an index mapping $\mathcal{I}_i=(t_i,s_i)$ records the correspondence between the starting layers in the full emulator and the sub-emulator (Zhu et al., 18 Aug 2025).

The rationale is tied to residual transformers. FedSODA states that residual connections allow the final output to be expressed as a sum of all previous layer outputs and the input embedding, so groups showing minimal change across adjacent layers are natural pruning candidates. In this strict SGP setting, similarity is not merely a ranking heuristic; it defines the groups themselves.

Within the broader FedSODA pipeline, SGP is a server-side preprocessing stage. The server first prunes the emulator, then forms the sub-LLM $\mathcal{E}$0, then applies QLoRA and orchestrated distillation alignment. The full framework reports an average 70.6% reduction in communication overhead, a 75.6% decrease in storage usage, and a 3.1% improvement in task accuracy across three open-source LLMs and multiple downstream tasks (Zhu et al., 18 Aug 2025).

3. Similarity-driven pruning without explicit group formation

A large part of the SGP-adjacent literature uses similarity as a pruning signal but does not form groups explicitly. “Effective Layer Pruning Through Similarity Metric Perspective” is a canonical example. Its criterion compares the penultimate representation of the dense network, $\mathcal{E}$1, with the penultimate representation of a one-layer-removed candidate, $\mathcal{E}$2, and defines

$\mathcal{E}$3

The layer with minimum score is pruned at each greedy iteration, followed by fine-tuning. This is highly similarity-driven, but the paper explicitly does not compute pairwise layer-layer similarity matrices, does not form clusters of similar layers, and does not prune sets of layers jointly because they are mutually redundant (Pons et al., 2024).

That distinction becomes important in later critiques. SimDiff argues that similarity-only depth pruning can be unstable because representational similarity is not identical to functional redundancy. It augments cosine-based hidden-state similarity with transformation-difference metrics, MSSD and MASD, and scores layers by

$\mathcal{E}$4

Its empirical evidence is specifically directed against similarity-only pruning: on Mistral-7B-v0.3 with 12 layers pruned, ShortGPT reaches WikiText2 perplexity 1833.0, whereas MASD reaches 47.7; on Qwen2.5-0.5B with 12-layer pruning, ShortGPT reaches 10916.14, while MSSD and MASD remain at 208.23 and 399.39, respectively (Chen et al., 21 Apr 2026). This suggests that similarity is a central but incomplete ingredient of SGP-like systems.

4. Channel-, filter-, and feature-map formulations

In CNN pruning, the most direct analogues of SGP appear at channel and filter granularity. “Exploiting Channel Similarity for Accelerating Deep Convolutional Neural Networks” explicitly groups channels within each layer by hierarchical clustering. The practical similarity metric is estimated from BatchNorm parameters,

$\mathcal{E}$5

after which channels are clustered with a global threshold and one representative per cluster is retained by $\mathcal{E}$6. This is close to a classical SGP formulation: similarity defines groups, and pruning removes all but a representative. On ImageNet, its pruned ResNet-50 achieves 45.90% FLOPs reduction with a 0.80% Top-1 drop and a 0.34% Top-5 drop (Zhang et al., 2019).

A second line of work shifts similarity from parameter space to output space. QSFM computes pairwise similarity between output feature maps within the same convolutional layer, averages that similarity across samples, and then greedily deletes one member of the most similar remaining pair using average feature-map rank as the representative-selection criterion. The feature-map similarity matrix is

$\mathcal{E}$7

with $\mathcal{E}$8 instantiated by SSIM or by negative Euclidean distance. On ResNet-56 for CIFAR-10, QSFM reports 48.7% FLOPs reduction and 57.9% parameter reduction with only a 0.54% Top-1 accuracy loss and no fine-tuning (Wang et al., 2021). Although the method does not run explicit clustering, it behaves like greedy representative selection inside latent similarity groups.

Within-layer self-similarity matrices offer an even simpler formulation. “Self Similarity Matrix based CNN Filter Pruning” defines $\mathcal{E}$9 on flattened filters and ranks filters either by nearest-neighbor redundancy or by the trapezoidal area of each matrix row. The area-based rule is closer to group-style redundancy because it prefers filters globally similar to many others rather than merely one close neighbor (Rakshith et al., 2022). “Efficient Similarity-based Passive Filter Pruning for Compressing CNNs” keeps the same pairwise-similarity logic but approximates the similarity matrix by a Nyström method, reducing pruning time by about 3x while preserving the same accuracy and computation reduction as the full-matrix similarity method (Singh et al., 2022).

5. Token- and multimodal variants

In transformer inference, SGP-like behavior often appears as token pruning. SAINT is a training-free framework for ViTs and VLMs that computes cosine similarity on head-averaged keys, forms a thresholded bipartite similarity graph, and scores redundancy by neighborhood degree and average valid-edge similarity. Its key quantities are the degree

$\mathcal{A}$0

the mean valid-edge similarity

$\mathcal{A}$1

and an adaptive prune rate based on how many tokens satisfy $\mathcal{A}$2. The method does not explicitly cluster tokens, but it approximates grouping through local thresholded neighborhoods. On ViT-H/14 at 224px, it roughly doubles throughput with only about 0.6% accuracy loss on ImageNet-1K, and on LLaVA-13B it drops 75% of visual tokens with latency comparable to LLaVA-7B and less than 1% performance loss across benchmarks (Jeddi et al., 14 Mar 2025).

Multimodal token pruning also reveals a limitation of pure similarity grouping. CDPruner argues that similarity-based pruning can overlook instruction relevance and “always prune the same tokens,” regardless of the question. It therefore replaces instruction-agnostic similarity grouping with conditional diversity maximization via a DPP kernel

$\mathcal{A}$3

so that token-token similarity is reweighted by instruction relevance. The paper does not present itself as SGP, but it can be read as a critique of similarity grouping alone: on LLaVA, CDPruner reduces FLOPs by 95% and CUDA latency by 78% while maintaining 94% of the original accuracy (Zhang et al., 12 Jun 2025).

6. Empirical behavior, limitations, and recurrent misconceptions

Across architectures, the strongest empirical advantage of SGP-style pruning is that it targets structurally meaningful redundancy rather than scalar sparsity. Layer removal in residual networks and LLMs yields direct depth reduction; channel and filter removal yields hardware-friendly shrinkage; token pruning reduces sequence length and therefore attention cost. Several papers also report robustness-related side effects. The CKA-based layer-pruning method states that its pruned models exhibit robustness to adversarial and out-of-distribution samples, including CIFAR-C, ImageNet-C, CIFAR-10.2, and FGSM, even though no defense mechanism is included during pruning (Pons et al., 2024).

The principal limitation is that many methods called “similarity-based” are not group pruning in a strict sense. Some score one component at a time against the dense model; some approximate groups by graph neighborhoods; some define groups manually from architecture rather than from a learned metric. “Group channel pruning and spatial attention distilling for object detection,” for example, divides YOLOv4 into five groups according to feature-layer scale and similarity of module structure, then applies different pruning thresholds to each group. Its grouping is genuinely group-based, but its “similarity” is structural rather than numerically estimated (Chu et al., 2023).

A second limitation is that similarity alone may not be sufficient. SimDiff shows collapse under aggressive similarity-only depth pruning on several LLMs, and CDPruner shows that similarity-only token pruning can ignore instruction relevance. A plausible implication is that mature SGP systems will often require a second criterion—difference, variance, task conditioning, or distillation—to separate redundancy from genuine function.

A third limitation is architectural and systems dependence. FedSODA requires a public dataset for server-side similarity estimation; QSFM requires repeated feature-map extraction over data; thresholded graph methods require hyperparameters such as $\mathcal{A}$4 and $\mathcal{A}$5; filter-similarity methods incur quadratic pairwise-matrix costs unless approximated. These are not incidental details: they determine whether a similarity formulation is merely descriptive or practically deployable.

Taken together, the literature supports a precise but non-unitary definition. In a strict sense, SGP refers to pruning methods that explicitly identify similarity groups and remove redundant members while preserving representatives, as in grouped layer pruning for federated LLM adaptation or hierarchical channel clustering. In a broader and now common sense, it denotes similarity-guided structured pruning more generally, including methods that never materialize the groups but still act on the same redundancy hypothesis: components that preserve representations, duplicate feature content, or inhabit dense similarity neighborhoods can be removed with limited loss of model function.