Differential Sensitivity Fusion Pruning

• The paper proposes DSFP, a method that fuses gradient-based, Taylor, and KL divergence signals to achieve effective single-shot filter pruning in DCNNs.
• It demonstrates significant computational savings and minimal accuracy loss, reporting up to 80% FLOPs reduction on architectures like VGG-16 and AlexNet.
• The approach is extensible, with variants using diverse sensitivity signals for channel, transformer layer, and token pruning across different neural network models.

Searching arXiv for the core paper and closely related pruning/fusion papers to ground the article in the literature. Differential Sensitivity Fusion Pruning (DSFP) denotes pruning schemes that combine multiple sensitivity signals to decide which model components to remove. In current arXiv usage, the term is applied most explicitly to single-shot filter pruning in DCNNs, where gradient-based sensitivity, a first-order Taylor score, and KL divergence of activation distributions are fused into a differential score; closely related formulations fuse image- and instance-level channel deviations for adaptive detection, representational similarity and transformation difference for LLM depth pruning, or interval sensitivity and Sobolev differential training for surrogate-model compression (Chowdhury et al., 5 Sep 2025, Wang et al., 3 Jun 2025, Chen et al., 21 Apr 2026, Kichler et al., 2023). This suggests that DSFP is best understood as a family of sensitivity-fusion pruning methods rather than a single standardized algorithm.

1. Core formulation in convolutional filter pruning

The clearest canonical DSFP formulation appears in work on DCNN filter pruning. Let $F=\{F_i\}$ be the set of convolutional filters in a pretrained network, let $L$ denote the loss on a held-out batch, and let $F_i=[F_{i,1},\dots,F_{i,n}]^\top$ denote the weights of filter $i$. Three scalar sensitivity scores are defined per filter: $$S_{\mathrm{grad}}(i)=\frac{1}{n}\sum_{j=1}^n \left|\frac{\partial L}{\partial F_{i,j}}\right|,$$

$$S_{\mathrm{taylor}}(i)=\sum_{j=1}^n \left|\left(\frac{\partial L}{\partial F_{i,j}}\right)\cdot F_{i,j}\right|,$$

$S_{\mathrm{KL}}(i)=D_{KL}(p_i\|q_i),$

where $p_i$ is the empirical distribution of output logits or next-layer activations when $F_i$ is kept and $q_i$ is the corresponding distribution when $L$0 is dropped by zeroing it (Chowdhury et al., 5 Sep 2025).

The fused differential sensitivity score is then

$L$1

This fusion emphasizes disagreement among the constituent signals. The paper describes such disagreement as indicative of filters with inconsistent importance across metrics, and applies exponential scaling to magnify large pairwise discrepancies. In the pruning stage, filters are ranked within each layer by $L$2, and the bottom $L$3 filters are removed for each layer $L$4, where $L$5 is the layerwise pruning ratio and $L$6 is the number of filters in that layer.

Conceptually, this formulation departs from single-criterion heuristics by treating inconsistency across saliency metrics as information rather than noise. A plausible implication is that DSFP is designed not merely to estimate “importance,” but to detect stability or instability in the importance estimate itself.

2. Single-shot workflow and optimization protocol

The one-shot DSFP pipeline proceeds in six steps. First, a forward pass on a calibration batch collects baseline activations and logits. Second, each filter is zeroed out temporarily, a forward pass is executed, and the corresponding $L$7 is measured. Third, a backward pass computes $L$8 for all filters. Fourth, $L$9, $F_i=[F_{i,1},\dots,F_{i,n}]^\top$0, and $F_i=[F_{i,1},\dots,F_{i,n}]^\top$1 are computed. Fifth, the lowest-ranked filters per layer are pruned. Sixth, the resulting student network is fine-tuned with knowledge distillation from the original pretrained teacher using

$F_i=[F_{i,1},\dots,F_{i,n}]^\top$2

with $F_i=[F_{i,1},\dots,F_{i,n}]^\top$3 decaying from $F_i=[F_{i,1},\dots,F_{i,n}]^\top$4 and temperature $F_i=[F_{i,1},\dots,F_{i,n}]^\top$5 (Chowdhury et al., 5 Sep 2025).

The same work characterizes DSFP as efficient, deterministic, and single shot. Its detailed complexity discussion is more specific: it requires one forward pass to obtain the baseline $F_i=[F_{i,1},\dots,F_{i,n}]^\top$6, $F_i=[F_{i,1},\dots,F_{i,n}]^\top$7 additional forward passes to estimate $F_i=[F_{i,1},\dots,F_{i,n}]^\top$8 for KL divergence, and one backward pass to obtain gradients, for a total of approximately $F_i=[F_{i,1},\dots,F_{i,n}]^\top$9 network evaluations. The method is therefore “single shot” in the sense of avoiding iterative prune–retrain cycles for scoring, rather than in the sense of a literal single network evaluation.

This distinction matters for implementation. The scoring stage is substantially cheaper than multi-round RL-based pruning or repeated retraining, but the per-filter KL estimation can still dominate cost when $i$0 is large. The paper explicitly notes that KL may be approximated for groups or computed on sampled subsets of filters to reduce overhead.

3. Reported performance and operating regime

The reported experiments for DSFP use CIFAR-10, AlexNet with $i$1 convolutional layers and $i$2K filters, and VGG-16 with $i$3 convolutional layers and $i$4K filters. Pruning rates of $i$5, $i$6, and $i$7 of filters per layer are considered. Fine-tuning uses knowledge distillation for $i$8 epochs on VGG-16 and up to $i$9 epochs on AlexNet, with AdamW at learning rate $$S_{\mathrm{grad}}(i)=\frac{1}{n}\sum_{j=1}^n \left|\frac{\partial L}{\partial F_{i,j}}\right|,$$0 and Cosine Annealing restarts (Chowdhury et al., 5 Sep 2025).

For AlexNet, the baseline fine-tuned accuracy is $$S_{\mathrm{grad}}(i)=\frac{1}{n}\sum_{j=1}^n \left|\frac{\partial L}{\partial F_{i,j}}\right|,$$1. After DSFP, the fine-tuned accuracies are $$S_{\mathrm{grad}}(i)=\frac{1}{n}\sum_{j=1}^n \left|\frac{\partial L}{\partial F_{i,j}}\right|,$$2 at $$S_{\mathrm{grad}}(i)=\frac{1}{n}\sum_{j=1}^n \left|\frac{\partial L}{\partial F_{i,j}}\right|,$$3 pruning, $$S_{\mathrm{grad}}(i)=\frac{1}{n}\sum_{j=1}^n \left|\frac{\partial L}{\partial F_{i,j}}\right|,$$4 at $$S_{\mathrm{grad}}(i)=\frac{1}{n}\sum_{j=1}^n \left|\frac{\partial L}{\partial F_{i,j}}\right|,$$5, and $$S_{\mathrm{grad}}(i)=\frac{1}{n}\sum_{j=1}^n \left|\frac{\partial L}{\partial F_{i,j}}\right|,$$6 at $$S_{\mathrm{grad}}(i)=\frac{1}{n}\sum_{j=1}^n \left|\frac{\partial L}{\partial F_{i,j}}\right|,$$7, corresponding to retention rates of $$S_{\mathrm{grad}}(i)=\frac{1}{n}\sum_{j=1}^n \left|\frac{\partial L}{\partial F_{i,j}}\right|,$$8, $$S_{\mathrm{grad}}(i)=\frac{1}{n}\sum_{j=1}^n \left|\frac{\partial L}{\partial F_{i,j}}\right|,$$9, and $$S_{\mathrm{taylor}}(i)=\sum_{j=1}^n \left|\left(\frac{\partial L}{\partial F_{i,j}}\right)\cdot F_{i,j}\right|,$$0. The number of filters drops from $$S_{\mathrm{taylor}}(i)=\sum_{j=1}^n \left|\left(\frac{\partial L}{\partial F_{i,j}}\right)\cdot F_{i,j}\right|,$$1 to $$S_{\mathrm{taylor}}(i)=\sum_{j=1}^n \left|\left(\frac{\partial L}{\partial F_{i,j}}\right)\cdot F_{i,j}\right|,$$2, $$S_{\mathrm{taylor}}(i)=\sum_{j=1}^n \left|\left(\frac{\partial L}{\partial F_{i,j}}\right)\cdot F_{i,j}\right|,$$3, and $$S_{\mathrm{taylor}}(i)=\sum_{j=1}^n \left|\left(\frac{\partial L}{\partial F_{i,j}}\right)\cdot F_{i,j}\right|,$$4, while parameters drop from $$S_{\mathrm{taylor}}(i)=\sum_{j=1}^n \left|\left(\frac{\partial L}{\partial F_{i,j}}\right)\cdot F_{i,j}\right|,$$5M to $$S_{\mathrm{taylor}}(i)=\sum_{j=1}^n \left|\left(\frac{\partial L}{\partial F_{i,j}}\right)\cdot F_{i,j}\right|,$$6M, $$S_{\mathrm{taylor}}(i)=\sum_{j=1}^n \left|\left(\frac{\partial L}{\partial F_{i,j}}\right)\cdot F_{i,j}\right|,$$7M, and $$S_{\mathrm{taylor}}(i)=\sum_{j=1}^n \left|\left(\frac{\partial L}{\partial F_{i,j}}\right)\cdot F_{i,j}\right|,$$8M.

For VGG-16, the baseline fine-tuned accuracy is $$S_{\mathrm{taylor}}(i)=\sum_{j=1}^n \left|\left(\frac{\partial L}{\partial F_{i,j}}\right)\cdot F_{i,j}\right|,$$9. After DSFP, the fine-tuned accuracies are $S_{\mathrm{KL}}(i)=D_{KL}(p_i\|q_i),$0 at $S_{\mathrm{KL}}(i)=D_{KL}(p_i\|q_i),$1 pruning, $S_{\mathrm{KL}}(i)=D_{KL}(p_i\|q_i),$2 at $S_{\mathrm{KL}}(i)=D_{KL}(p_i\|q_i),$3, and $S_{\mathrm{KL}}(i)=D_{KL}(p_i\|q_i),$4 at $S_{\mathrm{KL}}(i)=D_{KL}(p_i\|q_i),$5, corresponding to retention rates of $S_{\mathrm{KL}}(i)=D_{KL}(p_i\|q_i),$6, $S_{\mathrm{KL}}(i)=D_{KL}(p_i\|q_i),$7, and $S_{\mathrm{KL}}(i)=D_{KL}(p_i\|q_i),$8. The number of filters decreases from $S_{\mathrm{KL}}(i)=D_{KL}(p_i\|q_i),$9 to $p_i$0, $p_i$1, and $p_i$2, and parameters decrease from $p_i$3M to $p_i$4M, $p_i$5M, and $p_i$6M.

The paper further states that FLOPs reduction exceeds $p_i$7 in many cases, for example for VGG-16 at $p_i$8 pruning. It also reports that DSFP outperforms magnitude-based and single-metric heuristics, and achieves compression comparable to RL-based methods with substantially less overhead. These results place DSFP in the class of post-training compression methods that seek high retention under aggressive structured pruning.

4. Other sensitivity-fusion pruning formulations

The broader literature uses closely related sensitivity-fusion ideas at different pruning granularities and with different signals.

Paper	Pruned unit	Fused signals
(Wang et al., 3 Jun 2025)	BN channels	image-level and instance-level domain sensitivity
(Chen et al., 21 Apr 2026)	Transformer layers	representational similarity and transformation difference
(Rahman, 5 Jan 2025)	Transformer layers	12 activation, MI, gradient, weight, and attention signals

In continual test-time adaptive object detection, channel pruning is guided by a fused channel score $p_i$9, where image-level sensitivity is the average $F_i$0 deviation from a source-domain mean feature and instance-level sensitivity is computed analogously over foreground RoIs after RoI-Align. This fused score weights a channel-structured sparsity regularizer on BN scaling parameters $F_i$1, channels with $F_i$2 are removed from the computation graph, and a stochastic channel reactivation mechanism restores pruned channels with Bernoulli probability $F_i$3. On Cityscapes$F_i$4Cityscapes-C, UAVDT$F_i$5UAVDT-C, and Cityscapes$F_i$6ACDC, the method improves ten-round average mAP by $F_i$7, $F_i$8, and $F_i$9 over WHW while reducing computational overhead by about $q_i$0–$q_i$1 in FLOPs relative to the SOTA alignment method (Wang et al., 3 Jun 2025).

In LLM depth pruning, SimDiff fuses two orthogonal layer-importance signals: cosine-dissimilarity between layer inputs and outputs, and transformation difference measured either by MSSD or MASD. After normalization, the final importance is

$q_i$2

with $q_i$3 selected by ternary search on validation perplexity. The method retains over $q_i$4 of LLaMA2-7B performance at a $q_i$5 pruning ratio and reaches up to $q_i$6 inference speedup when pruning $q_i$7 layers on LLaMA3.1-8B (Chen et al., 21 Apr 2026).

In transformer layer pruning for NLP, “strategic fusion” treats layer pruning as a supervised regression problem over $q_i$8 signals derived from activations, mutual information, gradients, weights, and attention. Linear regression and random forest are used to predict the expected accuracy drop from pruning each layer, and the lowest-predicted-drop layer is removed at each step. Random forest fusion is reported as best in $q_i$9 datasets and near-optimal in the remaining $L$00, while distilled random forest surpasses original accuracy in $L$01 datasets and mitigates losses in the other $L$02 (Rahman, 5 Jan 2025).

Taken together, these works show that “fusion” in pruning can mean summation of normalized sensitivities, discrepancy amplification across metrics, or learned aggregation of heterogeneous signals.

5. Relation to neighboring compression paradigms

Not all sensitivity-aware pruning uses fusion. HQP is an instructive counterexample: it uses a single dynamic weight sensitivity metric derived from a diagonal Fisher Information Matrix approximation,

$L$03

and prunes iteratively subject to a hard accuracy-drop constraint $L$04. Only after this constrained structural pruning does the method proceed to INT8 post-training quantization. On MobileNetV3 Small and ResNet-18 deployed on Jetson Xavier NX, the reported gains include $L$05 and $L$06 inference speedup respectively, with the accuracy drop kept below the $L$07 constraint (Gopalan et al., 2 Feb 2026). Relative to DSFP, HQP illustrates a sensitivity-aware but non-fusion design.

The term DSFP is also used in a distinct surrogate-modelling context. In “Towards Sobolev Pruning,” DSFP is a three-step procedure: train a large network, prune it via Interval Adjoint Significance Analysis (IASA), and then fine-tune the pruned model with Sobolev training so that first- and second-order differential information is preserved. The IASA significance score for neuron $L$08 is

$L$09

and the Sobolev objective augments value matching with gradient matching. On the Gaussian basket-option example, pruning yields a network that is approximately $L$10 smaller by parameter count; value $L$11 remains above $L$12, while Sobolev fine-tuning restores Delta and Gamma fidelity to above $L$13 $L$14 (Kichler et al., 2023).

A broader neighboring idea appears in token compression for Vision Transformers. “Token Fusion” combines token pruning and token merging by selecting between them according to model sensitivity: pruning is advantageous when the model is sensitive to input interpolations, while merging is effective when responses are close to linear. The method further replaces average merging with MLERP merging to preserve feature-norm distribution (Kim et al., 2023). Although this is not labeled DSFP in the paper, it belongs to the same wider class of sensitivity-dependent fusion mechanisms for compression.

6. Limitations, ambiguities, and open directions

Several limitations recur across DSFP-style methods. In filter-level DSFP, per-filter KL estimation is explicitly identified as costly when the number of filters is large, and the method relies on a post-training knowledge-distillation phase that requires keeping a teacher model. The same source also contains a noteworthy interpretive tension: the abstract describes inconsistent filters as candidates identified by the fused differential score, whereas the detailed pruning routine removes the lowest-$L$15 filters after ascending sort. The literature therefore supports the algorithmic ranking rule unambiguously, but the semantic interpretation of “inconsistency” versus “prunability” requires careful reading (Chowdhury et al., 5 Sep 2025).

Channel-level fusion in adaptive detection highlights other practical constraints. Because pruning is implemented through BN scaling parameters, architectures without BN require an alternative gating mechanism. The fusion itself is a simple linear sum of image- and instance-level sensitivities, and the paper explicitly suggests that learned weighting, attention-based fusion, or per-layer thresholding could improve robustness. It also identifies extension to one-stage or anchor-free detectors and to multimodal adaptation as future work (Wang et al., 3 Jun 2025).

LLM depth pruning introduces a different cautionary lesson: methods that rely solely on cosine similarity can exhibit unpredictable performance and even catastrophic collapse across architectures. The SimDiff results suggest that fusion across orthogonal criteria is valuable precisely because one-dimensional heuristics may confuse directional redundancy with true functional redundancy. This is an argument for multi-view sensitivity estimation rather than for any one particular fusion operator (Chen et al., 21 Apr 2026).

Finally, sensitivity-aware pruning more generally must negotiate trade-offs between theoretical fidelity and operational simplicity. HQP notes that diagonal-FIM pruning ignores parameter cross-correlations, avoids mixed-precision quantization, and makes monotonic pruning decisions without regrowth (Gopalan et al., 2 Feb 2026). A plausible implication across the literature is that future DSFP variants will continue to explore richer sensitivity estimators, more adaptive fusion rules, and tighter coupling between pruning and downstream recovery procedures such as distillation, quantization, or differential fine-tuning.