PruneFuse: Pruning-Based Data Selection & Fusion

• PruneFuse is a pruning-based data selection and network fusion method that uses a structurally pruned surrogate to efficiently guide active learning.
• It replaces costly dense-model scoring with a pruned proxy and aligns its learned weights into the dense target through weight-aligned fusion and optional knowledge distillation.
• Empirical results on datasets like CIFAR and ImageNet show that PruneFuse maintains or improves accuracy while significantly reducing computational FLOPs and model size.

PruneFuse is a pruning-based data selection and network fusion method for efficient deep learning. It was introduced to reduce the computational overhead of active learning and related subset-selection pipelines by replacing repeated large-model scoring with a structurally pruned surrogate of the target network, then transferring the surrogate’s learned weights back into the dense model through weight-aligned fusion and final training with optional knowledge distillation (Kousar et al., 2 Jan 2025). A later formulation presents the same core idea as a two-stage pipeline, adds an optional synchronization mechanism between selector and target, and reports results beyond the initial CIFAR and Tiny-ImageNet setting, including ImageNet-1K, vision backbones beyond ResNet, and NLP models (Kousar et al., 27 Mar 2026).

1. Problem setting and motivation

PruneFuse is situated in active learning (AL), where a labeled set $L$ is iteratively expanded from a large unlabeled pool $U$ by querying the most informative samples. The motivation is that traditional AL pipelines repeatedly train large, dense models to score and acquire samples in each round, which incurs substantial computational overhead as data and model sizes grow (Kousar et al., 2 Jan 2025).

The method addresses two limitations identified in prior AL practice. First, dense-model acquisition is expensive because training and scoring are repeated across rounds. Second, proxy-model approaches reduce cost but often require costly pretraining and suffer structural mismatches between proxy and target models, which can degrade selection quality and transferability (Kousar et al., 2 Jan 2025). PruneFuse responds by using a pruned surrogate of the target model for acquisition and then fusing the trained surrogate back into the dense model.

The central goal is to reduce data selection cost and total training FLOPs while matching or exceeding baseline accuracy at a given label budget $b$, preserve selection quality by ensuring the surrogate shares structure with the target, and accelerate final training via fusion and knowledge distillation (Kousar et al., 2 Jan 2025). In the later formulation, this is expressed as efficient data selection from a labeled pool $D$ of fixed budget $b$, where the selected subset should train a dense model that approximates full-data performance while minimizing compute (Kousar et al., 27 Mar 2026).

A representative objective given for the method is

$$\arg\min_{s_p} \left|\, \mathbb{E}_{(x,y)\in s_p}\big[l(x,y;\theta,\theta_p)\big] - \mathbb{E}_{(x,y)\in D}\big[l(x,y;\theta)\big]\,\right|,$$

where $s_p$ is the subset selected using the pruned surrogate $\theta_p$, and $\theta$ is the dense target model (Kousar et al., 2 Jan 2025).

2. Pipeline and algorithmic structure

PruneFuse proceeds in five stages: structured pruning at initialization, surrogate training for selection, iterative selection, network fusion, and final training with distillation (Kousar et al., 2 Jan 2025). The later version presents the same process as a two-stage pipeline—pruned selector for acquisition, followed by fusion into the dense target and fine-tuning—and adds optional synchronization every $T_{\text{sync}}$ rounds (Kousar et al., 27 Mar 2026).

Stage	Operation	Function
1	Structured pruning at initialization	Create a smaller surrogate $U$0 from dense $U$1
2	Surrogate training for selection	Train $U$2 on the current labeled set
3	Iterative selection	Score $U$3, acquire top-$U$4, update $U$5
4	Network fusion	Copy trained surrogate weights into aligned dense channels
5	Final training with distillation	Train fused model $U$6 on $U$7

The iterative selection phase begins from a small labeled seed $U$8. In each round, the pruned surrogate is trained from scratch on $U$9, used to compute scores on $b$0, and then used to acquire the top-$b$1 samples; the process stops when $b$2 reaches the budget $b$3 or a desired validation performance threshold is reached (Kousar et al., 2 Jan 2025). A reported acquisition schedule starts with $b$4 seed, adds $b$5 in the first round, then $b$6 per subsequent round until budget $b$7 (Kousar et al., 2 Jan 2025).

The acquisition rules are standard. Uncertainty-based rules include Least Confidence and Entropy. Diversity-based selection uses Greedy $b$8-centers in feature space $b$9, typically penultimate-layer embeddings (Kousar et al., 2 Jan 2025). In the later version, the same acquisition layer can wrap LC, entropy, greedy $D$0-centers, BALD, ALSE, and coreset-style criteria (Kousar et al., 27 Mar 2026).

The optional synchronization mechanism periodically fuses the current selector into the dense model, fine-tunes the fused dense model, prunes it again to regenerate a selector, and resumes acquisition. This is intended to improve proxy–target alignment during long AL runs (Kousar et al., 27 Mar 2026).

3. Pruning, fusion, and distillation mechanics

The pruning operation is structured channel pruning at initialization with a magnitude-based $D$1 criterion. For layer $D$2 with channels $D$3 and channel weights $D$4, the score is

$D$5

A binary mask $D$6 selects the top $D$7 fraction of channels globally in one-shot pruning, and the network sparsity is

$D$8

The method retains initialization without reinitializing after pruning (Kousar et al., 2 Jan 2025).

The selector scores are computed from the trained pruned surrogate $D$9. The Least Confidence and Entropy scores are

$b$0

$b$1

For diversity selection, Greedy $b$2-centers chooses

$b$3

with Euclidean distance in the learned feature space (Kousar et al., 2 Jan 2025).

The fusion operation is a one-way, structured weight copy from the trained pruned model into the untrained dense model. For each layer $b$4, if $b$5 is the index set of surviving channels, then the fused weights satisfy

$b$6

Equivalently,

$b$7

This preserves trained substructures while leaving non-surviving weights at initialization, which the paper describes as retaining exploration capacity (Kousar et al., 2 Jan 2025).

The final dense model $b$8 is trained on $b$9 with cross-entropy and a distillation loss from the trained surrogate teacher. With temperature $$\arg\min_{s_p} \left|\, \mathbb{E}_{(x,y)\in s_p}\big[l(x,y;\theta,\theta_p)\big] - \mathbb{E}_{(x,y)\in D}\big[l(x,y;\theta)\big]\,\right|,$$0 and balance $$\arg\min_{s_p} \left|\, \mathbb{E}_{(x,y)\in s_p}\big[l(x,y;\theta,\theta_p)\big] - \mathbb{E}_{(x,y)\in D}\big[l(x,y;\theta)\big]\,\right|,$$1,

$$\arg\min_{s_p} \left|\, \mathbb{E}_{(x,y)\in s_p}\big[l(x,y;\theta,\theta_p)\big] - \mathbb{E}_{(x,y)\in D}\big[l(x,y;\theta)\big]\,\right|,$$2

$$\arg\min_{s_p} \left|\, \mathbb{E}_{(x,y)\in s_p}\big[l(x,y;\theta,\theta_p)\big] - \mathbb{E}_{(x,y)\in D}\big[l(x,y;\theta)\big]\,\right|,$$3

$$\arg\min_{s_p} \left|\, \mathbb{E}_{(x,y)\in s_p}\big[l(x,y;\theta,\theta_p)\big] - \mathbb{E}_{(x,y)\in D}\big[l(x,y;\theta)\big]\,\right|,$$4

A later version emphasizes the same loss and gives $$\arg\min_{s_p} \left|\, \mathbb{E}_{(x,y)\in s_p}\big[l(x,y;\theta,\theta_p)\big] - \mathbb{E}_{(x,y)\in D}\big[l(x,y;\theta)\big]\,\right|,$$5 as a default practical setting (Kousar et al., 27 Mar 2026).

The computational rationale is straightforward. Pruning cost is negligible compared to training; selector training and scoring scale with the pruned architecture rather than the dense one; fusion is $$\arg\min_{s_p} \left|\, \mathbb{E}_{(x,y)\in s_p}\big[l(x,y;\theta,\theta_p)\big] - \mathbb{E}_{(x,y)\in D}\big[l(x,y;\theta)\big]\,\right|,$$6 weight copying; final training is identical in nominal architecture to dense training but converges faster due to fused initialization and KD (Kousar et al., 2 Jan 2025). A later analysis states that selector compute scales by $$\arg\min_{s_p} \left|\, \mathbb{E}_{(x,y)\in s_p}\big[l(x,y;\theta,\theta_p)\big] - \mathbb{E}_{(x,y)\in D}\big[l(x,y;\theta)\big]\,\right|,$$7, where $$\arg\min_{s_p} \left|\, \mathbb{E}_{(x,y)\in s_p}\big[l(x,y;\theta,\theta_p)\big] - \mathbb{E}_{(x,y)\in D}\big[l(x,y;\theta)\big]\,\right|,$$8 and $$\arg\min_{s_p} \left|\, \mathbb{E}_{(x,y)\in s_p}\big[l(x,y;\theta,\theta_p)\big] - \mathbb{E}_{(x,y)\in D}\big[l(x,y;\theta)\big]\,\right|,$$9 are the parameter counts of the pruned and dense models, respectively, and reports 70–96% reductions in selector FLOPs across datasets (Kousar et al., 27 Mar 2026).

4. Empirical performance

The original evaluation covers CIFAR-10, CIFAR-100, and Tiny-ImageNet-200 for image classification, using ResNet-56, ResNet-164, and ResNet-50, with test accuracy averaged over 3 independent runs (Kousar et al., 2 Jan 2025). The later version expands to ImageNet-1K, Wide-ResNet, MobileNetV2, ViT, VDCNN, Amazon Review Polarity, Amazon Review Full, and CIFAR-10-C (Kousar et al., 27 Mar 2026).

Setting	Baseline AL	PruneFuse
CIFAR-10, ResNet-56, $s_p$0, LC	93.00	93.65
CIFAR-100, ResNet-56, $s_p$1, LC	66.72	67.87
Tiny-ImageNet-200, ResNet-50, $s_p$2, LC	54.65	55.89
ImageNet-1K, ResNet-50, $s_p$3	73.56	73.64

On CIFAR-10 with ResNet-56 and LC selection, baseline AL achieves 80.53, 87.74, 90.85, 92.24, and 93.00 at budgets $s_p$4, whereas PruneFuse with $s_p$5 achieves 80.92, 88.35, 91.44, 92.77, and 93.65; the reported gains are up to $s_p$6 at $s_p$7 (Kousar et al., 2 Jan 2025). On CIFAR-100 with the same architecture and acquisition rule, the corresponding baseline values are 35.99, 52.99, 59.29, 63.68, and 66.72, while PruneFuse with $s_p$8 obtains 40.26, 53.90, 60.80, 64.98, and 67.87, with gains up to $s_p$9 at $\theta_p$0 (Kousar et al., 2 Jan 2025). On Tiny-ImageNet-200 with ResNet-50 and LC selection, baseline AL reports 14.86, 33.62, 43.96, 49.86, and 54.65, while PruneFuse with $\theta_p$1 reports 18.71, 39.70, 47.41, 51.84, and 55.89 (Kousar et al., 2 Jan 2025).

The later version emphasizes compute reductions as well as accuracy. On CIFAR-10 with ResNet-56 at $\theta_p$2, baseline AL uses selector FLOPs $\theta_p$3, whereas PruneFuse with $\theta_p$4 reaches 93.40 with selector FLOPs $\theta_p$5, described as an approximately $\theta_p$6 reduction (Kousar et al., 27 Mar 2026). On ImageNet-1K with ResNet-50 at $\theta_p$7 label budget and $\theta_p$8, PruneFuse attains 73.64 versus 73.56 for AL while reducing selector FLOPs by approximately $\theta_p$9 (Kousar et al., 27 Mar 2026).

Selector size is substantially reduced. For ResNet-56, parameter count falls from 0.85M in the baseline model to 0.21M at $\theta$0, 0.13M at $\theta$1, 0.07M at $\theta$2, and 0.03M at $\theta$3. For ResNet-50, the corresponding counts are 25.56M, 6.10M, 3.92M, 2.23M, and 1.02M (Kousar et al., 2 Jan 2025).

Comparisons with Selection via Proxy (SVP) are central to the method’s positioning. On CIFAR-10 with ResNet-56 as target and LC selection at $\theta$4, SVP with a ResNet-20 selector of 0.26M parameters gives target accuracy 92.95, whereas PruneFuse with a pruned ResNet-56 selector at $\theta$5 and 0.21M parameters gives target accuracy 93.69 (Kousar et al., 2 Jan 2025). In a small-model comparison with ResNet-20 as target on CIFAR-10 at $\theta$6, SVP with a ResNet-8 selector of 0.074M reaches 91.88, whereas PruneFuse with a pruned ResNet-20 selector at $\theta$7 and 0.066M reaches 92.29 (Kousar et al., 2 Jan 2025).

Ablations report that fusion alone accelerates convergence and improves final accuracy, while KD is beneficial but optional. On CIFAR-100 with ResNet-56 and $\theta$8, LC selection yields 66.72 for baseline, 67.49 for PruneFuse without KD, and 67.87 with KD (Kousar et al., 2 Jan 2025). The paper also states that fusion yields faster convergence and higher peak accuracy than non-fused training on the same selected subsets, especially early in training (Kousar et al., 2 Jan 2025).

5. Position within the broader pruning–fusion literature

PruneFuse is not a general name for any method that combines pruning and fusion. In the original usage, it specifically denotes pruning-based data selection plus weight-aligned network fusion for efficient deep learning (Kousar et al., 2 Jan 2025). This is distinct from several neighboring lines of work.

Within active learning, the most immediate comparison is to uncertainty-based, Bayesian, core-set, meta-learning, gradient matching, and influence-based baselines, all of which typically retrain large models at each round or require heavy pretraining (Kousar et al., 2 Jan 2025). PruneFuse differs by replacing large-model scoring with a pruned surrogate and then reusing the surrogate’s learned weights via fusion and KD. It also differs from SVP, which uses small proxies to select subsets but discards the proxy and may suffer structural mismatch (Kousar et al., 2 Jan 2025).

Within pruning research, the method adopts structured channel pruning at initialization for speed and hardware-friendly compression, but its objective is not only model compression. The pruned model is used as a selector, and the dense model remains the final training target (Kousar et al., 2 Jan 2025). This distinguishes it from pruning methods oriented purely toward inference efficiency or static model compression.

Related work in the wider pruning–fusion space uses the terms “fusion” and “pruning” differently. FuPruner combines aggressive operator fusion with dynamic filter pruning so that non-parametric operators can be pruned alongside convolutions for edge-device inference (Li et al., 2020). ResConv-based layer pruning inserts a fusible residual convolutional block into each convolution, learns sparse layer scaling factors, and fuses the residual back into an ordinary convolution after pruning (Xu et al., 2020). FuseGPT recycles pruned transformer blocks by fusing their linear-layer parameters into neighboring blocks and then applying lightweight group-level fine-tuning (Pei et al., 2024). TinyFusion learns depth pruning for diffusion transformers with differentiable mask sampling and co-optimized recovery parameters (Fang et al., 2024). “Strategic fusion” for transformer compression uses linear regression or random forest models to combine multiple pruning signals and schedule layer removal (Rahman, 5 Jan 2025).

These works share a family resemblance—pruning followed by some form of functional reuse or fusion—but they are conceptually distinct from PruneFuse’s channel-aligned selector–target transfer. The original PruneFuse paper further states that its fusion is conceptually distinct from SWA and model soups because it is one-way, structured weight copying from a trained subnetwork into an untrained dense network, rather than weight averaging across trained models (Kousar et al., 2 Jan 2025).

6. Limitations, practical guidance, and prospective extensions

The reported limitations are specific. Excessive pruning ratios may degrade selection quality and final accuracy; empirical results show modest degradation at $\theta$9, and the paper notes that $T_{\text{sync}}$0 may be harmful (Kousar et al., 2 Jan 2025). Channel pruning at random initialization using $T_{\text{sync}}$1 scores is simple and fast but can be brittle for certain architectures. Fusion relies on architectural coherence, so complex skip topologies require careful masking and alignment. Surrogate uncertainty and diversity estimates may be less calibrated than those from a large model, and KD helps but does not fully mitigate this. Tasks beyond image classification and non-CNN architectures require adaptation of structured pruning and fusion rules, because channel-based mapping is less direct for attention modules (Kousar et al., 2 Jan 2025).

The practical guidance in the paper is correspondingly narrow. A pruning ratio $T_{\text{sync}}$2 is recommended as a strong accuracy–efficiency trade-off; $T_{\text{sync}}$3 consistently yields the best accuracy among pruned variants while saving about $T_{\text{sync}}$4 of parameters (Kousar et al., 2 Jan 2025). Least Confidence is reported as effective across datasets, while Entropy and $T_{\text{sync}}$5-centers also perform well. Retraining the surrogate from scratch each round is recommended to avoid overfitting to previously selected samples. The paper specifies SGD, $T_{\text{sync}}$6 epochs and batch size $T_{\text{sync}}$7 for CIFAR-10/100, and $T_{\text{sync}}$8 epochs and batch size $T_{\text{sync}}$9 for Tiny-ImageNet-200 (Kousar et al., 2 Jan 2025). The later formulation recommends $U$00, with $U$01 preferred if compute allows, and identifies large unlabeled pools, constrained compute for AL, and CNN targets amenable to structured pruning as the primary use case (Kousar et al., 27 Mar 2026).

A plausible implication is that PruneFuse is best understood as an efficiency mechanism for sample acquisition and optimization warm-starting rather than as a compression method in the narrow deployment sense. The method’s empirical pattern—reduced selector FLOPs, competitive or improved accuracy at fixed budgets, and faster convergence after fusion—supports that interpretation (Kousar et al., 2 Jan 2025). The later paper points to future directions including streaming or online AL, multi-task and multi-modal settings, transformers with token or channel pruning, learned dispersal maps, and tighter theory linking proxy–target alignment to expected loss reduction (Kousar et al., 27 Mar 2026).