Gradient-Based Pruning Methods

• Gradient-based pruning is a method that uses gradients to rank the importance of model parameters, offering a more dynamic alternative to magnitude-based criteria.
• It employs various gradient-derived metrics such as first-order sensitivity, Hessian interactions, and cumulative gradient statistics to decide which weights, filters, or layers to remove.
• This approach enhances computational efficiency and can improve task performance, addressing challenges like backdoor unlearning and enabling effective model compression.

Gradient-based pruning denotes a class of pruning methods that rank model parameters or structured units using gradients or gradient-derived quantities, rather than relying only on static weight magnitudes. In the literature, these quantities include first-order loss sensitivity, Hessian–gradient interactions, cumulative early-training gradients, Fisher-style estimates, gradient alignment, and gradient-flow criteria. The resulting pruning decisions can target individual weights, convolutional filters or channels, attention heads, intra-attention substructures, entire layers, or even synaptic connectivity in spiking networks. Across CNNs, transformers, diffusion models, RBF networks, and LLMs, gradient-based pruning is used to reduce computational cost, maintain task performance under compression, and, in some cases, remove undesirable functionality such as backdoors (Lubana et al., 2020, Redman et al., 2021, Yang et al., 2022, Das et al., 2023, Lv et al., 2024).

1. Conceptual foundations and theoretical framing

A recurring theme in the pruning literature is that gradient information is meant to approximate how much the loss, optimization dynamics, or downstream adaptation would change if a parameter or structured group were removed. This is distinct from pure magnitude pruning, which scores parameters by size alone. Theoretical work has clarified that these families are related rather than disjoint. A gradient-flow analysis shows that magnitude-based pruning, loss-preservation pruning, and gradient-norm-based pruning can be interpreted through successive time derivatives of the parameter norm during infinitesimal-step gradient descent. In that view, magnitude corresponds to the parameter norm itself, loss-preservation to a first-order time derivative, and GraSP-style criteria to second-order dynamics (Lubana et al., 2020).

An operator-theoretic account reaches a related conclusion through Koopman mode decomposition. There, magnitude-based pruning is linked to a principal mode predicting the training fixed point, while gradient-based pruning is linked to decaying modes that track training dynamics. This establishes a formal relationship between magnitude- and gradient-based pruning and helps explain why magnitude pruning can remain competitive even before convergence. The same analysis reports that, early in training, Koopman magnitude pruning and ordinary magnitude pruning can have more than 95% mask overlap, and that before useful structure emerges, “intelligent” pruning may not outperform random layer-wise selection (Redman et al., 2021).

More recent theory places pruning in a statistical learning framework. For sparse high-dimensional single-index and multi-index models, pruning neural networks in proportion to the sparsity level of the relevant directions improves sample complexity relative to unpruned networks. In that setting, a pruned network trained with gradient descent can match the sample complexity implied by correlational statistical query lower bounds, whereas basis-independent methods trained by standard gradient descent with rotationally invariant random initialization are provably suboptimal in the sparse regime. This suggests that pruning is not merely a deployment heuristic; under sparsity assumptions it can alter the statistical efficiency of learning itself (Vural et al., 2024).

2. Importance scores and gradient-derived criteria

The most common gradient-based criterion is first-order Taylor sensitivity. GRAIN, for example, assigns each pruning unit $w$ the score

$$\mathrm{IS}(w) = \mathbb{E}_{x\sim X}\left|\frac{\partial \mathcal{L}(x)}{\partial w} \, w\right|,$$

and applies this to intra-attention units rather than whole heads. It also introduces a regularized score,

$$\mathrm{IS}^r(w) = \mathrm{IS}(w)\cdot \tanh\!\left(\frac{D(\mathcal{M},\mathcal{W})}{\alpha}\right),$$

to promote block sparsity and practical speedup (Yang et al., 2022).

A closely related formulation is loss-preservation pruning, where a parameter group $\theta_p$ is scored by

$$I(\theta_p(t)) = \theta_p^\top(t)\, \mathbf{g}(\theta(t)),$$

using the first-order approximation of the loss change induced by removal. GraSP-style methods instead use the Hessian–gradient interaction

$$I(\theta_p(t)) = \theta_p^\top(t)\, \mathbf{H}(\theta(t))\, \mathbf{g}(\theta(t)),$$

which corresponds to the effect of pruning on the gradient norm. The gradient-flow analysis warns that increasing gradient norm via pruning can damage second-order optimization dynamics and produce poorly performing models; preserving, rather than increasing, gradient norm is the more robust principle (Lubana et al., 2020).

Several later methods replace first-order Taylor scores with accumulated or aggregated gradient statistics. For LLM layer pruning, GradPruner computes an Initial Gradient Information Accumulation Matrix by accumulating squared simulated gradients over the first $t$ fine-tuning steps,

$$F_W = \sum_{i=1}^{t} \left(\nabla W_i^{\mathrm{sim}}\right)^2,$$

then sums these over linear sublayers to obtain a layer importance score (Huang et al., 27 Jan 2026). GradMAP uses a global layer score based on average per-sample gradient magnitudes,

$$s_i = \frac{1}{N} \sum_{j\in\mathcal{D}} \sum_{\theta \in \mathcal{W}_i} \left\|\frac{\partial \mathcal{L}_{CE}(\mathbf{x}^{(j)},\mathbf{y}^{(j)})}{\partial \theta}\right\|_2,$$

arguing that this gives an efficient Fisher-like estimate of layer importance (Liu et al., 16 Feb 2026).

Other work explicitly combines gradient information with complementary signals. GAPrune defines a parameter-wise gradient-alignment score

$$s_g^j = \frac{\langle g_j^{gen}, g_j^{dom}\rangle}{\|g_j^{gen}\|\,\|g_j^{dom}\|+\varepsilon},$$

and a Domain Alignment Importance score

$$\mathrm{IS}(w) = \mathbb{E}_{x\sim X}\left|\frac{\partial \mathcal{L}(x)}{\partial w} \, w\right|,$$0

so that pruning reflects both domain importance and compatibility with general-language behavior (Tang et al., 13 Sep 2025). In CNN channel pruning, FGP similarly argues that gradient-only criteria can miss global feature structure and therefore combines gradients and activations through CAM-derived heatmaps before aggregating channel importance across classes (Lv et al., 2024).

Dynamic rescoring is another important axis. Greedy-Gnorm assigns each attention head the product of Q/K/V gradient-block norms,

$$\mathrm{IS}(w) = \mathbb{E}_{x\sim X}\left|\frac{\partial \mathcal{L}(x)}{\partial w} \, w\right|,$$1

and recomputes all scores after every pruning step. The stated motivation is that static rankings become stale as pruning changes the roles of the remaining heads (Guo et al., 4 Feb 2026).

3. Procedural patterns in gradient-based pruning

Despite variation in scoring, most methods follow a common procedural template: collect gradient-related statistics, rank parameters or structured groups, remove the lowest-importance units, and optionally fine-tune or compensate for the induced distribution shift. The main differences concern when gradients are measured, what unit is pruned, and whether pruning is discrete, soft, or intertwined with optimization.

One family is training-free or nearly training-free. GBLM-Pruner computes gradients from a small calibration set on pretrained LLMs and scores each weight by

$$\mathrm{IS}(w) = \mathbb{E}_{x\sim X}\left|\frac{\partial \mathcal{L}(x)}{\partial w} \, w\right|,$$2

optionally augmenting this with an activation term. It requires no retraining or weight updates after pruning (Das et al., 2023). DRIVE is not training-free, but uses only a short dense pretraining phase before iterative pruning with the dual metric

$$\mathrm{IS}(w) = \mathbb{E}_{x\sim X}\left|\frac{\partial \mathcal{L}(x)}{\partial w} \, w\right|,$$3

thereby combining magnitude, connection sensitivity, and convergence sensitivity (Saikumar et al., 2024).

A second family prunes during training. Progressive Gradient Pruning computes filter-wise gradient norms during training, progressively reduces capacity with an exponential schedule, mixes hard and soft pruning, and explicitly updates momentum buffers so that zeroed or removed filters do not retain stale velocity terms (Nguyen-Meidine et al., 2019). In diffusion models, gradient-flow pruning uses progressive soft masks $$\mathrm{IS}(w) = \mathbb{E}_{x\sim X}\left|\frac{\partial \mathcal{L}(x)}{\partial w} \, w\right|,$$4, gradually decays $$\mathrm{IS}(w) = \mathbb{E}_{x\sim X}\left|\frac{\partial \mathcal{L}(x)}{\partial w} \, w\right|,$$5 from 1 to 0 while increasing sparsity $$\mathrm{IS}(w) = \mathbb{E}_{x\sim X}\left|\frac{\partial \mathcal{L}(x)}{\partial w} \, w\right|,$$6, and prunes weights whose removal least harms—or can even improve—the gradient-flow objective (Wan et al., 16 Jan 2025).

A third family alters optimization itself so that pruning is induced by the update rule. PSG scales each gradient element according to distance from a target value,

$$\mathrm{IS}(w) = \mathbb{E}_{x\sim X}\left|\frac{\partial \mathcal{L}(x)}{\partial w} \, w\right|,$$7

with $$\mathrm{IS}(w) = \mathbb{E}_{x\sim X}\left|\frac{\partial \mathcal{L}(x)}{\partial w} \, w\right|,$$8 for pruning. The method is interpreted as gradient descent in a warped weight space and is presented as a way to obtain pruning-ready models without explicit pruning schedules or fine-tuning (Kim et al., 2020). In deep SNNs, Grad R introduces a continuous synaptic parameter $$\mathrm{IS}(w) = \mathbb{E}_{x\sim X}\left|\frac{\partial \mathcal{L}(x)}{\partial w} \, w\right|,$$9 with $$\mathrm{IS}^r(w) = \mathrm{IS}(w)\cdot \tanh\!\left(\frac{D(\mathcal{M},\mathcal{W})}{\alpha}\right),$$0, redefines the gradient so that pruned synapses can regrow, and thus treats connectivity and weight as jointly optimized variables (Chen et al., 2021). A related differentiable-topology approach for feedforward networks uses Gumbel-Softmax gates $$\mathrm{IS}^r(w) = \mathrm{IS}(w)\cdot \tanh\!\left(\frac{D(\mathcal{M},\mathcal{W})}{\alpha}\right),$$1 and backpropagates through both weights and gate probabilities to learn a sparse architecture end-to-end (Zhang et al., 2023).

4. Architectures, pruning units, and domain-specific realizations

Gradient-based pruning has diversified mainly by changing the unit of removal and the meaning of “importance.” In CNNs, units are typically filters, channels, or layers. In transformers, they may be weights, heads, intra-attention rows, or full layers. In SNNs, connectivity itself becomes the pruned object. Domain-specific work further adjusts the loss being differentiated, as in backdoor unlearning or domain-aware embedding compression.

Method	Pruning unit	Core gradient signal
GRAIN (Yang et al., 2022)	Intra-attention query/value units	First-order Taylor score with gradient separation
FGP (Lv et al., 2024)	Convolutional channels	Multi-class feature × gradient CAM aggregation
GBLM-Pruner (Das et al., 2023)	Pretrained LLM weights	$$\mathrm{IS}^r(w) = \mathrm{IS}(w)\cdot \tanh\!\left(\frac{D(\mathcal{M},\mathcal{W})}{\alpha}\right),$$2 from calibration gradients
Greedy-Gnorm (Guo et al., 4 Feb 2026)	Attention heads	Product of Q/K/V gradient-block norms
GradPruner (Huang et al., 27 Jan 2026)	LLM layers	Early cumulative squared gradients (IGIA-Matrix)
GradMAP (Liu et al., 16 Feb 2026)	LLM layers	Average per-layer gradient magnitudes

CNN work illustrates how gradient information can be specialized to task structure. FGP uses Seg-Grad-CAM and inspiration from Score-CAM to produce per-class, per-channel heatmaps,

$$\mathrm{IS}^r(w) = \mathrm{IS}(w)\cdot \tanh\!\left(\frac{D(\mathcal{M},\mathcal{W})}{\alpha}\right),$$3

then retains the top $$\mathrm{IS}^r(w) = \mathrm{IS}(w)\cdot \tanh\!\left(\frac{D(\mathcal{M},\mathcal{W})}{\alpha}\right),$$4 channels, with $$\mathrm{IS}^r(w) = \mathrm{IS}(w)\cdot \tanh\!\left(\frac{D(\mathcal{M},\mathcal{W})}{\alpha}\right),$$5–$$\mathrm{IS}^r(w) = \mathrm{IS}(w)\cdot \tanh\!\left(\frac{D(\mathcal{M},\mathcal{W})}{\alpha}\right),$$6 reported as the best speed–accuracy trade-off (Lv et al., 2024). A distinct CNN use case reframes backdoor defense as unlearning: parameters are ranked by the mean absolute gradient magnitude of an unlearning loss,

$$\mathrm{IS}^r(w) = \mathrm{IS}(w)\cdot \tanh\!\left(\frac{D(\mathcal{M},\mathcal{W})}{\alpha}\right),$$7

and the highest-scoring filters are pruned iteratively until validation accuracy or unlearning-loss patience criteria are triggered (Dunnett et al., 2024).

Transformer work is even more varied. GRAIN prunes rows inside attention modules rather than whole heads and separates pruning gradients from hidden-state distillation gradients (Yang et al., 2022). Greedy-Gnorm prunes one head at a time with dynamic re-estimation (Guo et al., 4 Feb 2026). GBLM-Pruner operates at the individual-weight level in pretrained LLMs without retraining (Das et al., 2023). GradPruner and GradMAP both perform layer pruning, but the former merges sparsified pruned layers into retained layers using same-sign parameters, whereas the latter compensates the largest post-pruning mean shift with a projection matrix fused into an FFN down-projection (Huang et al., 27 Jan 2026, Liu et al., 16 Feb 2026). GAPrune extends the framework to domain-adapted embedding models by combining Fisher information with general–domain gradient alignment (Tang et al., 13 Sep 2025).

Outside standard discriminative networks, gradient-based pruning has also been adapted to generative, scientific, and neuromorphic settings. Diffusion-model pruning uses gradient flow and progressive soft masks rather than one-shot deletion (Wan et al., 16 Jan 2025). In materials physics, RBF networks are pruned by minimizing the expected squared difference between a large model and a smaller one over an assumed data distribution, using closed-form expectations and gradient-based optimization of centroids, weights, bias, and kernel width (Määttä et al., 2020). In deep SNNs, gradient rewiring makes regrowth a first-class component of pruning dynamics rather than a post hoc repair mechanism (Chen et al., 2021).

5. Empirical performance and reported trade-offs

Across the surveyed papers, empirical gains are usually reported as a trade-off among accuracy retention, compression ratio, latency, and the additional cost of the pruning procedure itself. FGP reports, on CIFAR-10 with VGG-16, a reduction of FLOPs from 314.03M to 83.15M and parameters from 17.7M to 1.73M, with accuracy changing from 93.92% to 93.49%. The same summary states that FGP yields higher accuracy at the same or lower computational cost than feature-only or gradient-only baselines, and that for segmentation with a ResNet-50 backbone on Cityscapes, mIoU is retained or improved relative to both generic and segmentation-specific pruning methods (Lv et al., 2024).

PSG emphasizes high-sparsity behavior after training rather than structured speedup. On ResNet-32 for CIFAR-100, pruning by magnitude after PSGD training gives 69.25% accuracy at 50% sparsity, 68.62% at 70%, 67.27% at 80%, and 64.33% at 90%, outperforming SGD, $$\mathrm{IS}^r(w) = \mathrm{IS}(w)\cdot \tanh\!\left(\frac{D(\mathcal{M},\mathcal{W})}{\alpha}\right),$$8 regularization, and SNIP at high sparsity in the reported table. The method is presented as requiring no explicit $$\mathrm{IS}^r(w) = \mathrm{IS}(w)\cdot \tanh\!\left(\frac{D(\mathcal{M},\mathcal{W})}{\alpha}\right),$$9 penalty, pruning mask, schedule, or post-pruning fine-tuning (Kim et al., 2020).

For pretrained LLMs, GRAIN reports that at 5% model density it retains 93%–99% of original BERT$\theta_p$0 performance across GLUE, SQuAD, and CoNLL 2003, achieves $\theta_p$1 speedups, and remains competitive even when only 3% of transformer weights remain (Yang et al., 2022). Training-free pruning in LLMs also shows competitive results: on LLaMA-2 7B at 50% unstructured sparsity, GBLM-Pruner reports perplexity 6.86, compared with 6.92 for Wanda, 7.00 for SparseGPT, and 16.03 for magnitude pruning, while also improving mean zero-shot accuracy over Wanda and SparseGPT in the summarized results (Das et al., 2023).

Early-stage and layer-level methods focus strongly on pruning efficiency. DRIVE is reported to be $\theta_p$2 to $\theta_p$3 faster than IMP for pruning, while consistently outperforming other training-agnostic early pruning methods in accuracy on VGG and ResNet models over CIFAR-10/100, Tiny ImageNet, and ImageNet (Saikumar et al., 2024). For LLM fine-tuning, GradPruner reports a 40% parameter reduction with only a 0.99% decrease in accuracy across eight downstream datasets, along with about 36% reductions in training time and GPU memory and about 39% reductions in inference time and memory relative to the dense model (Huang et al., 27 Jan 2026). GradMAP reports an average $\theta_p$4 pruning speedup over earlier layer-pruning methods and, at 25% layer pruning on LLaMA2-7B, 58.59 average zero-shot accuracy, 20.56 perplexity, and 169 seconds pruning time with compensation, compared with 55.96, 21.76, and 801 seconds for SLEB (Liu et al., 16 Feb 2026).

Dynamic head pruning and SNN rewiring show similar patterns. Greedy-Gnorm reports that pruning BERT down to roughly 20% of heads preserves 90.08% accuracy versus 96.82% before pruning, and reports model-size reductions of 22.5% for BERT, 18.1% for RoBERTa, 7.43% for XLM-RoBERTa, and 5.05% for ALBERT (Guo et al., 4 Feb 2026). Grad R reports 92.03% accuracy on CIFAR-10 at 5% connectivity and 89.32% accuracy at 0.73% connectivity, corresponding to an approximately 3.5% loss, while requiring no retraining after the structural optimization (Chen et al., 2021).

Gradient-based pruning has also been used for objectives other than generic compression. The backdoor-unlearning method reports, for BadNets on CIFAR-10 with PreActResNet-18 and 10 samples per class, ACC 90.02, ASR 8.72, and RA 83.71 after pruning, compared with a baseline ACC 91.82 and ASR 94.76. The same summary emphasizes comparatively low variance and good behavior in limited-data regimes (Dunnett et al., 2024).

6. Debates, limitations, and adjacent meanings

A central debate is whether gradient-based pruning is actually superior to simpler baselines. The operator-theoretic analysis explicitly notes that naive magnitude pruning has often been found “as robust as more complex, state-of-the-art algorithms,” and uses Koopman theory to argue that part of this robustness is theoretically explainable rather than accidental (Redman et al., 2021). The gradient-flow analysis likewise shows that not all gradient criteria are equally sound: first-order loss-preservation is appropriate for minimally trained models, whereas pruning rules that deliberately increase gradient norm can disproportionately damage early layers and degrade accuracy (Lubana et al., 2020).

Another limitation concerns the stability of importance estimates. Static scores can become stale under iterative pruning, since removing one structured unit changes the role of the rest; Greedy-Gnorm treats this as the main weakness of attention-entropy ranking and re-estimates all scores at every greedy step (Guo et al., 4 Feb 2026). Conversely, relying on gradients alone may ignore global representational structure. FGP is motivated precisely by the claim that feature-only methods retain redundant channels while gradient-only methods can overlook global feature structure, so it integrates both types of information (Lv et al., 2024).

There is also a persistent gap between structured and fine-grained sparsity. GradMAP reports that, at very high pruning rates, structured layer pruning still trails unstructured pruning in accuracy, even when importance estimation and compensation are improved (Liu et al., 16 Feb 2026). This suggests that the practical value of gradient-based pruning depends not only on how importance is estimated, but also on the granularity at which hardware, software, and optimization can exploit the resulting sparsity.

Finally, the phrase “gradient pruning” can denote a neighboring but distinct idea: pruning communicated gradients rather than model parameters. Dual Gradient Pruning in collaborative learning removes both the top-$\theta_p$5 largest and bottom-$\theta_p$6 smallest gradient entries to improve privacy and communication efficiency, with theoretical bounds on reconstruction error and convergence. This is adjacent to, but not the same as, model pruning based on gradients (Xue et al., 2024).

Taken together, the literature portrays gradient-based pruning less as a single method than as a design space. The common premise is that gradient information encodes functional relevance more directly than parameter size alone. The main points of variation are the object being pruned, the statistic extracted from gradients, the timing of measurement, and the recovery mechanism after deletion. The strongest recent results tend to appear where these choices are matched to architectural structure and task objective, rather than where gradients are used in isolation.