Information-Based Feature-Pruning Strategy

• Information-based feature pruning is a strategy that selects key network representations based on task relevance, mutual information, and redundancy reduction.
• It employs various proxies—such as loss gradients, matrix rank, and stochastic gating—to quantify the informativeness of features.
• Empirical results demonstrate that these methods effectively reduce model size and computation while maintaining performance across domains like classification, segmentation, and NLP.

Searching arXiv for the cited and closely related pruning papers to ground the article in the referenced literature. Searching arXiv for "Channel Pruning guided by classification Loss and feature Importance". An information-based feature-pruning strategy is a pruning approach in which the pruning decision is driven by information carried by features, feature maps, channels, hidden dimensions, tokens, or other intermediate representations, rather than solely by weight magnitude or a purely local saliency heuristic. In the literature, the term “information” is used in several distinct senses: discriminative or classification information from loss gradients, mutual information or conditional mutual information, feature-map discriminant power, activation-based redundancy, structural richness such as matrix rank, stochastic signal-passing probability, and deterministic feature usefulness defined by actual query support. Several papers are explicit that this usage is often not information-theoretic in the strict Shannon sense, but instead operationalizes information through task relevance, reconstructability, or redundancy of learned representations (Guo et al., 2020, Hou et al., 2020, Lin et al., 2020, Trelin et al., 2019, Fan et al., 2021, Vu-Van et al., 2024).

1. Conceptual scope and objects of pruning

Information-based feature pruning is not tied to a single structural unit. In convolutional networks, the most common units are output channels or filters. CPLI prunes channels of a convolutional layer’s input feature map through a binary channel-selection vector $\bm{\beta}^{(l)}$, and the corresponding adjacent convolutional weights are physically removed as pruning proceeds from shallow to deep layers (Guo et al., 2020). HRank scores filters indirectly through the average rank of the feature maps they generate and prunes the filters with the lowest average feature-map rank (Lin et al., 2020). DI-based pruning also targets CNN channels, but ranks them by how much their removal reduces a multi-class discriminant criterion computed from feature maps and labels (Hou et al., 2020).

The same general idea extends beyond standard CNN channels. In the mutual-information-based layer-wise method for Transformers and related architectures, the pruned object is the hidden representation dimension $m^l \in \{0,1\}^{D \times 1}$, and associated matrix rows and columns are then removed to produce smaller dense matrices rather than irregular sparsity (Fan et al., 2021). BSF places stochastic gates either on raw input features, hidden activations, or convolutional feature maps, so the same mechanism can realize supervised feature selection, neuron pruning, or kernel pruning (Trelin et al., 2019). Later work applies information-based pruning to full-resolution segmentation feature maps through redundancy minimization (Wu et al., 2024), to feature tokens in multi-frame pose estimation (Wang et al., 7 Mar 2025), and even to hash-grid entries in Gaussian-splatting compression, where a feature is pruned if it is never queried by any Gaussian coordinate during interpolation (Ma et al., 28 Dec 2025).

This breadth suggests that “feature-pruning strategy” denotes a family of representation-selection procedures rather than a single canonical algorithm. What is held fixed is the emphasis on feature usefulness, informativeness, or nonredundancy at the representation level; what varies is the formal proxy used to quantify it.

2. Meanings of “information” in pruning criteria

One major family uses task-discriminative information. CPLI augments layerwise reconstruction with the gradient of the final classification loss, $\frac{\partial \mathcal{C}}{\partial y_{i,m}}$, so that reconstruction errors are weighted by downstream classification sensitivity. Its per-layer objective is

$$\arg\min_{\bm{\beta},\mathbf{W}} \sum_{i=1}^{c_{out}}\sum_{m=1}^{M} \left[ \frac{\partial \mathcal{C}}{\partial y_{i,m}} \cdot \left(y_{i,m}^{0}-\gamma y_{i,m}^{*}\cdot y_{i,m}\right) \right]^2, \quad \text{s.t. } \|\bm{\beta}\|_0 \leq B.$$

Here the gradient term is treated explicitly as an importance weight, while $y_{i,m}^*$ suppresses features likely to be removed at the next pruning stage (Guo et al., 2020). FGP follows a related discriminative logic, but in a class-aware CAM-style formulation: feature maps $\mathcal{F}_j^m$ are weighted by class-conditioned gradients $G_j^m$, aggregated into class-specific heatmaps, and then summed across classes to rank channels by broad class support rather than by a single-source criterion (Lv et al., 2024).

A second family uses discriminant information in a more formal statistical sense. DI defines

$DI = \trace((\bar{\mathbf{K}}+\rho\mathbf{I})^{-1}\mathbf{K}_B),$

where $\mathbf{K}_B$ is a feature signal matrix and $\bar{\mathbf{K}}$ is a feature noise matrix. The same quantity also appears as the term that determines the minimum ridge least-square error for predicting labels from the layer’s features, since

$m^l \in \{0,1\}^{D \times 1}$0

The paper proves the non-decreasing property $m^l \in \{0,1\}^{D \times 1}$1 whenever $m^l \in \{0,1\}^{D \times 1}$2, and then prunes channels with minimum influence on DI (Hou et al., 2020).

A third family uses mutual information or conditional mutual information directly. The layer-wise model-pruning method defines a top-down objective

$m^l \in \{0,1\}^{D \times 1}$3

with the top hidden layer anchored to the softmax/output set, and interprets pruning as selecting hidden dimensions that are most informative for already preserved higher-level dimensions (Fan et al., 2021). The CMI-based CNN filter-pruning method instead asks how much information a candidate feature map still provides about labels once the already retained features are known. Its conceptual decomposition,

$m^l \in \{0,1\}^{D \times 1}$4

directly motivates pruning the non-selected set when $m^l \in \{0,1\}^{D \times 1}$5 becomes small. To compute CMI, it uses a matrix-based Rényi $m^l \in \{0,1\}^{D \times 1}$6-order entropy estimator (Vu-Van et al., 2024).

A fourth family uses structural or operational surrogates for information. HRank defines information richness by the matrix rank of each 2D feature map, under the hypothesis that low-rank feature maps contain less information and are easier for the remaining network to reproduce (Lin et al., 2020). BSF treats “the amount of information involved in the training process” as the amount of signal allowed to pass through stochastic Bernoulli gates,

$m^l \in \{0,1\}^{D \times 1}$7

with an $m^l \in \{0,1\}^{D \times 1}$8 penalty on gate parameters pushing the expected transmitted signal downward (Trelin et al., 2019). Hash-grid feature pruning takes an even more operational stance: a hash-grid feature is informative only if at least one Gaussian query ever reaches it, so unused entries are pruned deterministically (Ma et al., 28 Dec 2025).

3. Redundancy, diversity, and set-dependent feature selection

A recurring theme is that information-based pruning is often set-dependent, not a matter of assigning independent intrinsic scores to features. SIRFP makes this explicit by abandoning per-channel importance scoring in favor of retained-set redundancy minimization. For segmentation feature maps $m^l \in \{0,1\}^{D \times 1}$9 and $\frac{\partial \mathcal{C}}{\partial y_{i,m}}$0, it defines redundancy through a Jensen–Shannon-based similarity

$\frac{\partial \mathcal{C}}{\partial y_{i,m}}$1

computed on full unpooled feature maps so that spatial organization is preserved. Channel selection is then cast as a maximum edge weight clique problem in which large edge weight means low redundancy, and a greedy deletion rule removes the remaining node with the smallest total incident edge weight (Wu et al., 2024).

Feature-statistics-guided filter pruning separates low diversity within a feature map from high similarity across feature maps. DFS uses mean standard deviation,

$\frac{\partial \mathcal{C}}{\partial y_{i,m}}$2

to remove low-diversity maps globally, while SFS uses cosine similarity to remove highly redundant maps within each layer (Li et al., 2020). QSFM likewise defines redundancy through similarity between feature maps in the same output tensor and prunes one feature from the most similar remaining pair, using average feature-map rank only as a tie-breaker for which one to keep (Wang et al., 2021).

Even ostensibly local methods often encode future or contextual dependence. CPLI’s $\frac{\partial \mathcal{C}}{\partial y_{i,m}}$3 term is a heuristic estimate of whether reconstructing a feature is worthwhile because that feature may be removed at the next layerwise stage (Guo et al., 2020). DI pruning computes per-layer channel influence scores, but the actual network-level budget is achieved by a greedy search that evaluates candidate one-layer pruning moves on a validation set (Hou et al., 2020). CMI-based pruning is explicitly sequential: a filter is valuable only relative to what the selected set already explains (Vu-Van et al., 2024). This recurring structure contradicts the common assumption that channels can always be ranked independently.

4. Optimization procedures and pruning pipelines

The optimization mechanisms used by information-based pruning strategies are heterogeneous. CPLI frames channel selection under an $\frac{\partial \mathcal{C}}{\partial y_{i,m}}$4 budget as NP-hard, relaxes it with an $\frac{\partial \mathcal{C}}{\partial y_{i,m}}$5 regularizer, and solves a LASSO problem for $\frac{\partial \mathcal{C}}{\partial y_{i,m}}$6 while gradually increasing $\frac{\partial \mathcal{C}}{\partial y_{i,m}}$7 until the budget $\frac{\partial \mathcal{C}}{\partial y_{i,m}}$8 is met. Once channels are selected, it drops the information weights and updates weights by least squares,

$\frac{\partial \mathcal{C}}{\partial y_{i,m}}$9

then moves to the next layer and performs only one final fine-tuning stage after all layers are pruned (Guo et al., 2020).

DI pruning uses an analytical first-order approximation of leave-one-channel-out DI loss and then a greedy global resource-allocation procedure. At each step, it constructs $$\arg\min_{\bm{\beta},\mathbf{W}} \sum_{i=1}^{c_{out}}\sum_{m=1}^{M} \left[ \frac{\partial \mathcal{C}}{\partial y_{i,m}} \cdot \left(y_{i,m}^{0}-\gamma y_{i,m}^{*}\cdot y_{i,m}\right) \right]^2, \quad \text{s.t. } \|\bm{\beta}\|_0 \leq B.$$0 candidate subnetworks, each obtained by pruning one additional layer-specific subset of bottom-ranked channels, evaluates them on a validation set, and chooses the candidate with the highest validation accuracy. Structure Distillation then transfers stage-wise pruning ratios from shallower to deeper networks (Hou et al., 2020).

The layer-wise mutual-information method faces an NP-hard subset-selection problem and therefore uses greedy forward selection with an approximate scoring rule derived from Brown et al. It then builds masks, removes the corresponding rows and columns from weight matrices, and physically squeezes the remaining parameters into smaller dense matrices. The method is used in both one-shot and iterative prune–retrain form, with reported gains saturating after two iterations in one ablation (Fan et al., 2021). BSF differs fundamentally: pruning is embedded inside ordinary training through stochastic gates, with straight-through gradients

$$\arg\min_{\bm{\beta},\mathbf{W}} \sum_{i=1}^{c_{out}}\sum_{m=1}^{M} \left[ \frac{\partial \mathcal{C}}{\partial y_{i,m}} \cdot \left(y_{i,m}^{0}-\gamma y_{i,m}^{*}\cdot y_{i,m}\right) \right]^2, \quad \text{s.t. } \|\bm{\beta}\|_0 \leq B.$$1

and the final subset is obtained by thresholding the learned gate weights after end-to-end optimization (Trelin et al., 2019).

Other procedures are even more specialized. FSM ranks channels by estimated downstream feature shift,

$$\arg\min_{\bm{\beta},\mathbf{W}} \sum_{i=1}^{c_{out}}\sum_{m=1}^{M} \left[ \frac{\partial \mathcal{C}}{\partial y_{i,m}} \cdot \left(y_{i,m}^{0}-\gamma y_{i,m}^{*}\cdot y_{i,m}\right) \right]^2, \quad \text{s.t. } \|\bm{\beta}\|_0 \leq B.$$2

prunes layer by layer, and then applies a distribution-optimization correction to compensate mean and variance shifts before short fine-tuning (Duan et al., 2022). FTP-Pose prunes feature tokens by density peaks clustering, with scores

$$\arg\min_{\bm{\beta},\mathbf{W}} \sum_{i=1}^{c_{out}}\sum_{m=1}^{M} \left[ \frac{\partial \mathcal{C}}{\partial y_{i,m}} \cdot \left(y_{i,m}^{0}-\gamma y_{i,m}^{*}\cdot y_{i,m}\right) \right]^2, \quad \text{s.t. } \|\bm{\beta}\|_0 \leq B.$$3

and retains the top $$\arg\min_{\bm{\beta},\mathbf{W}} \sum_{i=1}^{c_{out}}\sum_{m=1}^{M} \left[ \frac{\partial \mathcal{C}}{\partial y_{i,m}} \cdot \left(y_{i,m}^{0}-\gamma y_{i,m}^{*}\cdot y_{i,m}\right) \right]^2, \quad \text{s.t. } \|\bm{\beta}\|_0 \leq B.$$4 tokens in both high-resolution and low-resolution branches (Wang et al., 7 Mar 2025). Hash-grid feature pruning is post-training and deterministic: valid entries are identified from Gaussian coordinates, only those entries are entropy coded, and the decoder reconstructs the same valid set from decoded positions (Ma et al., 28 Dec 2025).

5. Empirical domains and reported performance

The empirical record shows that information-based feature pruning is not confined to image classification. CPLI reports gains on CIFAR-10, ImageNet, and UCF-101. On ImageNet, ResNet-50 at $$\arg\min_{\bm{\beta},\mathbf{W}} \sum_{i=1}^{c_{out}}\sum_{m=1}^{M} \left[ \frac{\partial \mathcal{C}}{\partial y_{i,m}} \cdot \left(y_{i,m}^{0}-\gamma y_{i,m}^{*}\cdot y_{i,m}\right) \right]^2, \quad \text{s.t. } \|\bm{\beta}\|_0 \leq B.$$5 compression yields a $$\arg\min_{\bm{\beta},\mathbf{W}} \sum_{i=1}^{c_{out}}\sum_{m=1}^{M} \left[ \frac{\partial \mathcal{C}}{\partial y_{i,m}} \cdot \left(y_{i,m}^{0}-\gamma y_{i,m}^{*}\cdot y_{i,m}\right) \right]^2, \quad \text{s.t. } \|\bm{\beta}\|_0 \leq B.$$6 Top-5 accuracy drop, better than DCP, ThiNet, CP, WM, and GAL in the reported comparison, while UCF-101 with C3D at $$\arg\min_{\bm{\beta},\mathbf{W}} \sum_{i=1}^{c_{out}}\sum_{m=1}^{M} \left[ \frac{\partial \mathcal{C}}{\partial y_{i,m}} \cdot \left(y_{i,m}^{0}-\gamma y_{i,m}^{*}\cdot y_{i,m}\right) \right]^2, \quad \text{s.t. } \|\bm{\beta}\|_0 \leq B.$$7 compression gives a $$\arg\min_{\bm{\beta},\mathbf{W}} \sum_{i=1}^{c_{out}}\sum_{m=1}^{M} \left[ \frac{\partial \mathcal{C}}{\partial y_{i,m}} \cdot \left(y_{i,m}^{0}-\gamma y_{i,m}^{*}\cdot y_{i,m}\right) \right]^2, \quad \text{s.t. } \|\bm{\beta}\|_0 \leq B.$$8 clip-level accuracy drop (Guo et al., 2020). DI-based pruning reports that a pruned ResNet-50 on ImageNet achieves $$\arg\min_{\bm{\beta},\mathbf{W}} \sum_{i=1}^{c_{out}}\sum_{m=1}^{M} \left[ \frac{\partial \mathcal{C}}{\partial y_{i,m}} \cdot \left(y_{i,m}^{0}-\gamma y_{i,m}^{*}\cdot y_{i,m}\right) \right]^2, \quad \text{s.t. } \|\bm{\beta}\|_0 \leq B.$$9 FLOPs reduction without any Top-1 accuracy loss, and its DI-SD ResNet-50 preserves $y_{i,m}^*$0 Top-1 while reducing FLOPs to $y_{i,m}^*$1 and parameters to $y_{i,m}^*$2 (Hou et al., 2020). HRank reports $y_{i,m}^*$3 FLOPs reduction with $y_{i,m}^*$4 parameter removal and only $y_{i,m}^*$5 top-1 loss on ResNet-110/CIFAR-10, and $y_{i,m}^*$6 FLOPs reduction with $y_{i,m}^*$7 parameter removal and $y_{i,m}^*$8 top-1 loss on ResNet-50/ImageNet (Lin et al., 2020).

The same perspective has been extended to segmentation and detection, where spatial structure is more critical. On Cityscapes with DeepLabv3-ResNet50, SIRFP at $y_{i,m}^*$9 pruning reports $\mathcal{F}_j^m$0 mIoU with $\mathcal{F}_j^m$1G FLOPs and $\mathcal{F}_j^m$2M parameters, compared with an unpruned baseline of $\mathcal{F}_j^m$3 mIoU and $\mathcal{F}_j^m$4G FLOPs. At $\mathcal{F}_j^m$5 and $\mathcal{F}_j^m$6 pruning, it reports $\mathcal{F}_j^m$7 and $\mathcal{F}_j^m$8 mIoU, respectively (Wu et al., 2024). On SSD-ResNet50 for COCO2017 detection, SIRFP reports $\mathcal{F}_j^m$9 AP at $G_j^m$0 FLOPs reduction (Wu et al., 2024). FTP-Pose integrates token pruning into a multi-grained feature encoder and reports PoseTrack2017 performance increasing from $G_j^m$1 mAP and $G_j^m$2 FPS in the baseline to $G_j^m$3 mAP and $G_j^m$4 FPS in the full model, a $G_j^m$5 speed improvement compared to the baseline (Wang et al., 7 Mar 2025).

Information-based pruning also appears in NLP and compression. The mutual-information-based layer-wise pruning method reports, on WMT14 En$G_j^m$6Fr, that pruning an Extra-Large model to Large size yields $G_j^m$7 BLEU with $G_j^m$8 speedup, compared with $G_j^m$9 BLEU and $DI = \trace((\bar{\mathbf{K}}+\rho\mathbf{I})^{-1}\mathbf{K}_B),$0 speedup for magnitude pruning (Fan et al., 2021). The CMI-based CNN method reports on VGG16/CIFAR-10 that bi-directional pruning with cross-layer compact CMI prunes $DI = \trace((\bar{\mathbf{K}}+\rho\mathbf{I})^{-1}\mathbf{K}_B),$1 of filters, reduces parameters from $DI = \trace((\bar{\mathbf{K}}+\rho\mathbf{I})^{-1}\mathbf{K}_B),$2M to $DI = \trace((\bar{\mathbf{K}}+\rho\mathbf{I})^{-1}\mathbf{K}_B),$3M, and after retraining reaches $DI = \trace((\bar{\mathbf{K}}+\rho\mathbf{I})^{-1}\mathbf{K}_B),$4 test accuracy versus $DI = \trace((\bar{\mathbf{K}}+\rho\mathbf{I})^{-1}\mathbf{K}_B),$5 for the original model, a $DI = \trace((\bar{\mathbf{K}}+\rho\mathbf{I})^{-1}\mathbf{K}_B),$6 drop (Vu-Van et al., 2024). Hash-grid feature pruning reports an average bitrate reduction of $DI = \trace((\bar{\mathbf{K}}+\rho\mathbf{I})^{-1}\mathbf{K}_B),$7 under the AVS-VRU Common Test Conditions, with PSNR unchanged because unused entries never contribute to inference (Ma et al., 28 Dec 2025).

6. Limitations, misconceptions, and recurring trade-offs

A central misconception is that “information-based” necessarily means Shannon-style information optimization. Several of the best-known methods explicitly use the term in looser senses. HRank states that rank measures linear structural diversity rather than entropy or mutual information (Lin et al., 2020). BSF says that “information” means the amount of signal allowed to pass through stochastic gates, not a mutual-information objective (Trelin et al., 2019). CPLI and FSM use task loss gradients or expected downstream shift rather than formal information measures (Guo et al., 2020, Duan et al., 2022). This suggests that the label covers both formal information-theoretic methods and feature-centric operational proxies.

A second misconception is that better pruning always comes from a better intrinsic importance score. Many methods show that the decisive issue is context. SIRFP argues that channel value depends on which other channels are retained, hence the MEWCP formulation (Wu et al., 2024). CMI pruning measures the incremental label information left in the remaining set after conditioning on already retained features (Vu-Van et al., 2024). DI proves monotonicity for the full-set criterion, but still relies on validation-guided greedy architecture search to allocate pruning budgets across layers (Hou et al., 2020). These methods converge on the idea that redundancy, complementarity, and downstream dependence are inseparable from feature importance.

The principal trade-off is between fidelity of the information proxy and cost of estimating it. DI avoids backward passes but still requires feature extraction and matrix computations (Hou et al., 2020). Mutual-information and CMI methods require kernel matrices, entropy estimation, and repeated greedy evaluations (Fan et al., 2021, Vu-Van et al., 2024). SIRFP computes pairwise redundancy on full unpooled feature maps and acknowledges extra overhead relative to weight-based criteria (Wu et al., 2024). CPLI and FSM reduce cost through approximations—respectively a heuristic future-pruning guide $DI = \trace((\bar{\mathbf{K}}+\rho\mathbf{I})^{-1}\mathbf{K}_B),$8 and BN-based expectation estimates—but each paper is explicit that these are approximations rather than exact estimators (Guo et al., 2020, Duan et al., 2022).

Finally, information-based feature pruning spans both input-feature selection and hidden-feature pruning, and the distinction matters. BSF and induced feature selection by structured pruning can remove raw input features directly (Trelin et al., 2019, Hubens et al., 2023). MI-based layer pruning, CPLI, DI, CMI pruning, and most CNN channel methods prune learned hidden representations first and then remove associated weights (Guo et al., 2020, Hou et al., 2020, Fan et al., 2021, Vu-Van et al., 2024). PruneGrad is input-specific and prunes hidden activations to improve attribution, not to compress a deployed model (Khakzar et al., 2019). DAPNet prunes channels only during training and removes the pruning module at inference (Zhu et al., 2019). Consequently, “feature-pruning strategy” in the literature designates a broad methodological field whose common denominator is representation-level selection by informativeness, but whose objectives, guarantees, and deployment implications differ substantially across tasks and architectures.