Entropy-Based Block Pruning

• Entropy-Based Block Pruning is a model compression strategy that leverages Shannon entropy to quantify and remove low-information, redundant neural blocks and channels.
• It applies information-theoretic techniques like global average pooling and conditional entropy computation across CNNs and Transformers for structured sparsity.
• Empirical results demonstrate significant speedups and compression ratios with minimal accuracy drop, supported by fine-tuning and hybrid metric strategies.

Entropy-Based block pruning is a class of model compression and acceleration strategies that utilize information-theoretic metrics—specifically, entropy—to quantify the import, redundancy, or discriminative capability of neural blocks, filters, channels, or other architectural subunits. These methods are prominent in both convolutional architectures and large-scale Transformer models, offering rigorous, interpretable criteria for structured model sparsification, with practical gains in computational efficiency and minimal accuracy degradation.

1. Information-Theoretic Foundations

The core quantitative basis for entropy-based block pruning is Shannon entropy, defined for a discrete probability distribution $p_i$ as $H = -\sum_i p_i \log p_i$. In convolutional networks, entropy is typically computed over bin-counts of activation outputs following global average pooling or over the eigen-spectrum of layer weight matrices. Low-entropy filters are interpreted as having uniformly predictable activations (e.g., always zero, always constant, or low diversity), indicating minimal representational contribution.

In the context of transformer models and LLMs, entropy is employed to measure the uncertainty or information richness of the hidden state features in each computation block (attention or MLP). Block-level entropy, calculated via bucket-based histogram estimators or KNN-based entropy, provides a direct measure of the distributional diversity entering and exiting each block (Yang et al., 4 Apr 2025).

Conditional entropy extends the foundational metric by quantifying the uncertainty of filter activations given the observed loss, demonstrating superior linkage between information reduction and predictive performance (Min et al., 2018).

2. Entropy-Based Pruning Methodologies

Classical entropy-based block pruning in CNNs involves:

• Summarizing per-filter activations via global average pooling across a dataset.
• Bin-counting these pooled responses to estimate activation distributions per filter.
• Computing entropy for each channel as $H_j = -\sum_{i=1}^m p_i \log p_i$, where $p_i$ are bin probabilities (Luo et al., 2017).
• Ranking and pruning filters in ascending order of entropy, preserving only those with the highest information richness.

More advanced frameworks use conditional entropy, where filter activations are jointly binned with their associated loss values, directly linking filter informativeness to network accuracy (Min et al., 2018).

For transformer architectures, entropy-based metrics quantify the change in information content between sequential blocks ($\Delta H^l = H(Z^l) - H(Z^{l-1})$), with blocks exhibiting minimal entropy increase considered redundant and candidates for removal (Yang et al., 4 Apr 2025).

When applied to N:M sparsity in LLMs, entropy is fused with other metrics (e.g., parameter amplitude) to build composite importance scores: $$S_{cj} = |w_e^{(j)}| \cdot (\mathrm{IR}_c + \alpha \cdot \mathrm{AM}_c)$$, prioritizing the retention of highly informative channels (Li et al., 2023).

3. Learning Schedules and Fine-Tuning Strategies

Pruning—by eliminating low-entropy blocks—introduces deleterious effects on generalization. Empirical results show that gradual fine-tuning following block or filter removal (one or two epochs per layer, then extended fine-tuning after the final pruning stage) recovers most performance loss (Luo et al., 2017). Maximum-entropy filter freezing, which locks high-entropy neuron weights during post-pruning adaptation, is shown to further mitigate overfitting (Min et al., 2018).

In reinforcement learning-based pruning, the information-theoretic reward function is defined via spatial entropy minimization of convolutional activations, encouraging an agent to sequentially prune blocks while monitoring entropy reduction rather than accuracy directly (Musat et al., 2023).

4. Empirical Results and Benchmark Performance

Entropy-based block pruning methods yield state-of-the-art speedup and compression ratios with minimal accuracy loss. For example:

• On VGG-16, entropy-based pruning achieved a 3.3× training/inference speedup and 16.64× compression, incurring only a ~1% drop in top-5 accuracy (Luo et al., 2017).
• Conditional entropy-based pruning (2PFPCE) delivered 10× FLOPs reduction and 46% improvement in inference time, with merely a 2% accuracy drop (Min et al., 2018).
• In ResNet-50, the entropy criterion provided around 1.54× acceleration and 1.47× compression, again with a minor accuracy decrease (Luo et al., 2017).
• In LLMs (e.g., Llama3.1-8B, Mistral-7B), the entropy-based approach retained over 95% performance after pruning up to 12 blocks, and offered near-linear speedup in inference (Yang et al., 4 Apr 2025).
• For channel pruning, fusing entropy and rank into a single information concentration metric enables data-driven layerwise pruning ratios, achieving compressions of ≥40% FLOPs and parameters with negligible accuracy loss or even slight improvements (Chen et al., 14 Aug 2024).

5. Comparative Analysis with Alternative Criteria

Entropy offers clear advantages over classical pruning metrics:

• APoZ (Average Percentage of Zeros) may miss nearly-uniform but nonzero activations, whereas entropy robustly identifies low-information outputs (Luo et al., 2017).
• Cosine similarity-based pruning focuses on geometric alignment, failing to measure actual information content; entropy directly quantifies uncertainty, providing a more reliable redundancy measure (Yang et al., 4 Apr 2025).
• Gradient-based Head Importance Scores (HIS) alone may misidentify which attention heads are functionally specialized; hybrid approaches like HIES blend attention entropy with HIS, yielding up to 15.2% improvements in model quality and 2.04× better stability (Choi et al., 10 Oct 2025).
• In block-level pruning of ViTs, pure loss-driven block benefit measures (P3B) are complementary, often intended to catch late-converging blocks that become critical later in training (Glandorf et al., 30 Jun 2025), while entropy-based methods capture statistical redundancy ab initio.

6. Impact on Generalization, Robustness, and Deployment

The generalization benefits of entropy-based block pruning are evidenced by improved performance when transferring pruned models to domains with differing distributions; redundant, low-entropy layers tend to encourage overfitting and can be safely removed (Luo et al., 2017). In LLMs, data pruning based on sample-level entropy has resulted in reduced overfitting and improved downstream metrics, with models trained on pruned datasets attaining improved perplexity and accuracy versus those trained on full datasets (Kim et al., 20 Jun 2024).

Structured sparsity produced by entropy-based techniques is compatible with off-the-shelf deep learning libraries and hardware accelerators, thereby enabling practical deployment in edge computing and real-time scenarios (Luo et al., 2017, Wu et al., 2023). Reduction in depth and intermediate activation size translates directly to memory footprint savings and reduced critical path latency in modern hardware (Liao et al., 24 Apr 2024).

7. Future Directions and Broader Implications

The applicability of entropy-based block pruning continues to expand:

• Extensions to hybrid metrics (combining rank and entropy, or entropy with Shapley values) offer more interpretable and flexible resource allocation (Chen et al., 14 Aug 2024).
• Differentiable proxies for entropy could facilitate direct integration into learning objectives and regularization schemes (Liao et al., 24 Apr 2024).
• Adaptive strategies informed by task sensitivity are proposed for domain adaptation in transformers, highlighting the importance of late block reactivation and global budget scheduling (Glandorf et al., 30 Jun 2025).
• In the blockchain domain, entropy minimization has been shown to yield lower orphan rates and more efficient consensus finality (Kreder et al., 2023).

Research in entropy-based block pruning is increasingly significant for scalable, efficient, and generalizable deep models, with proven impact on computational savings, interpretability, and robustness across diverse architectures and tasks.