Multi-Loss-Aware SCPM for Classification

• The paper presents a multi-loss-aware channel pruning method that integrates feature-map reconstruction, semantic correlation, and classification loss to preserve model accuracy.
• It employs layer-wise supervision and a greedy, sensitivity-based algorithm to select and refine channels, ensuring both local feature fidelity and global task performance.
• Empirical evaluations demonstrate that the pruned models achieve 2–3× speedup while matching or even exceeding the accuracy of denser networks on benchmark datasets.

Multi-loss-aware Structured Channel Pruning Method (SCPM) is a channel pruning framework targeting the compression and acceleration of deep neural networks for classification tasks. Unlike standard channel pruning approaches that focus exclusively on layer-wise reconstruction errors, multi-loss-aware SCPM explicitly incorporates a joint objective combining feature-map reconstruction, feature and semantic correlation, and classification loss. This enables more effective preservation of both local feature distributions and global task performance during the pruning procedure, promoting efficient models that retain—or sometimes exceed—the accuracy of their dense counterparts (Hu et al., 2019).

1. Multi-Loss Objective Formulation

At the core of multi-loss-aware SCPM is an explicitly defined multi-term objective for selecting informative channels in convolutional layers. Consider a convolutional layer receiving an input tensor $X\in\mathbb{R}^{C_{\rm in}\times H\times W}$ and parameterized by filters $$W\in\mathbb{R}^{M\times C_{\rm in}\times k\times k}$$. The method selects a retained channel set $\mathcal{P}$, forming the pruned filter sub-tensor $W_{\mathcal{P}}$. The framework defines three loss components:

• Feature-Map Reconstruction Loss ($\mathcal{L}_r$):

$$\mathcal{L}_r(W_{\mathcal{P}}) = \frac{1}{2T} \| X\otimes W - X\otimes W_{\mathcal{P}} \|_2^2,\quad T = MHW,$$

enforcing similarity between the baseline and pruned feature maps.

• Feature & Semantic Correlation Loss ($\mathcal{L}_s$):

$$\mathcal{L}_s(W_{\mathcal{P}}) = \frac{1}{4N^2M^2}\|G^f - G^f_{\mathcal{P}}\|_2^2 + \frac{1}{4N^2M^2}\|G^s - G^s_{\mathcal{P}}\|_2^2,\,N=HW,$$

where $G^f$ and $G^s$ are Gram matrices in feature and spatial domains, aligned between baseline and pruned feature maps, preserving channel-channel and spatial-spatial statistics.

• Classification Loss ($$W\in\mathbb{R}^{M\times C_{\rm in}\times k\times k}$$0):

$$W\in\mathbb{R}^{M\times C_{\rm in}\times k\times k}$$1

being the cross-entropy computed over a mini-batch, directly supervising the channel importance with respect to the end-task.

The joint pruning objective is:

$$W\in\mathbb{R}^{M\times C_{\rm in}\times k\times k}$$2

where $$W\in\mathbb{R}^{M\times C_{\rm in}\times k\times k}$$3 is the channel budget and $$W\in\mathbb{R}^{M\times C_{\rm in}\times k\times k}$$4 are balancing coefficients.

2. Layer-Wise Supervision Strategy

The supervision occurs at every layer, leveraging intermediate outputs of the baseline (unpruned) and the current pruned model:

• For each layer $$W\in\mathbb{R}^{M\times C_{\rm in}\times k\times k}$$5, a batch is passed through both networks to extract feature maps $$W\in\mathbb{R}^{M\times C_{\rm in}\times k\times k}$$6 (baseline) and $$W\in\mathbb{R}^{M\times C_{\rm in}\times k\times k}$$7 (pruned).
• Gram matrices for both sets are computed to evaluate $$W\in\mathbb{R}^{M\times C_{\rm in}\times k\times k}$$8; $$W\in\mathbb{R}^{M\times C_{\rm in}\times k\times k}$$9 is calculated from the raw feature activations.
• The pruned network’s output is then forwarded through remaining layers to compute the downstream $\mathcal{P}$0.
• This process ties local channel selection to both immediate feature reconstruction and ultimate classification accuracy, balancing local and global supervision at every layer.

Joint optimization over all losses efficiently guides the identification of channels that are structurally and functionally critical for the network’s classification capability.

3. Greedy Channel-Selection and Optimization Algorithm

As the exact subset selection in the multi-loss-aware objective is combinatorial, the method employs a greedy, sensitivity-based approach at each layer:

1. For each layer $\mathcal{P}$1, perform:
   • Mini-batch forward passes through both baseline and pruned models.
   • Compute total loss $\mathcal{P}$2 for the layer.
   • Calculate channel sensitivities:

   $\mathcal{P}$3

   capturing the influence of each channel’s pruning on the total objective.

2. Retain the top $\mathcal{P}$4 channels ranked by sensitivity $\mathcal{P}$5.
3. Freeze the pruned mask, then perform local SGD to refine the remaining channel weights.
4. After progressing through all layers (using this procedure), globally fine-tune the pruned network for classification.

This local-greedy algorithm decomposes the high-dimensional subset selection problem into tractable, performance-driven steps, combining sensitivity assessment and targeted SGD refinement.

4. Hyperparameter Selection and Training Regimens

Empirical guidelines for hyperparameter selection and training schedules, as validated on benchmark datasets, are as follows:

• Loss balancing: $\mathcal{P}$6, $\mathcal{P}$7 yield robust performance; grid search in $\mathcal{P}$8 and $\mathcal{P}$9 is suggested for tuning.
• Per-layer keep ratio: Uniform ratios (e.g., retaining 70% of channels per layer) are effective, though ratios can be adapted per layer based on baseline sensitivity for finer control.
• Local SGD for retained channels: Use learning rates $W_{\mathcal{P}}$0, momentum $W_{\mathcal{P}}$1, weight decay $W_{\mathcal{P}}$2, over 10–50 epochs for local convergence.
• Global fine-tuning: Undertake 100–200 epochs with initial learning rate $W_{\mathcal{P}}$3 (decayed at prescribed milestones), standard data augmentation (random crop/flip), and batch size 128 for dataset-wide adaptation.

These recipes are suitable for instantiating the method in modern frameworks (such as PyTorch) and achieving reproducible pruning-performance tradeoffs (Hu et al., 2019).

5. Empirical Evaluation and Ablation Analysis

Experimental validation spans classification networks on CIFAR-10 and CIFAR-100:

Model & Dataset	Baseline Error	Pruned Error (ratio × params / FLOPs)	Speedup
VGG-16, CIFAR-10	6.01%	5.88% (2.26× params, 2.23× FLOPs)	2.23×
ResNet-56, CIFAR-10 (30%)	6.20%	5.96% (1.97× params, 1.99× FLOPs)	1.99×
ResNet-18, CIFAR-100 (30%, 50%, 70%)	21.89%	24.39%, 24.91%, 25.15%	—
ResNet-34, CIFAR-100 (30%, 50%, 70%)	21.16%	21.87%, 22.41%, 22.89%	—

Ablation studies confirm the efficacy of multi-loss integration. On ResNet-56 with 30% pruning and no fine-tuning, using only $W_{\mathcal{P}}$4 yields 9.74% error, while employing the full fusion of $W_{\mathcal{P}}$5 reduces error to 8.00%. This suggests that combining feature, semantic, and task-level losses prevents deleterious effects of naive layer-wise pruning.

The pruned models produced by this SCPM deliver compact representations with 2–3× speedup, and in many cases, accuracy that matches or exceeds the uncompressed model’s performance.

6. Reproducibility and Implementation Considerations

All protocols can be implemented in standard deep learning libraries with mini-batch forward/backward passes, Gram matrix computations, and SGD routines, as prescribed in Algorithm 1. The explicit loss definitions (Eqs. (1)–(4)) and training schedules are sufficient for direct reproduction on canonical image classification datasets, and the procedure scales across architectures and pruning ratios. This structure enables systematic and deployable compression protocols without reliance on black-box heuristics (Hu et al., 2019).