CHIP: CHannel Independence-based Pruning for Compact Neural Networks (2110.13981v3)

Abstract: Filter pruning has been widely used for neural network compression because of its enabled practical acceleration. To date, most of the existing filter pruning works explore the importance of filters via using intra-channel information. In this paper, starting from an inter-channel perspective, we propose to perform efficient filter pruning using Channel Independence, a metric that measures the correlations among different feature maps. The less independent feature map is interpreted as containing less useful information$/$knowledge, and hence its corresponding filter can be pruned without affecting model capacity. We systematically investigate the quantification metric, measuring scheme and sensitiveness$/$reliability of channel independence in the context of filter pruning. Our evaluation results for different models on various datasets show the superior performance of our approach. Notably, on CIFAR-10 dataset our solution can bring $0.90\%$ and $0.94\%$ accuracy increase over baseline ResNet-56 and ResNet-110 models, respectively, and meanwhile the model size and FLOPs are reduced by $42.8\%$ and $47.4\%$ (for ResNet-56) and $48.3\%$ and $52.1\%$ (for ResNet-110), respectively. On ImageNet dataset, our approach can achieve $40.8\%$ and $44.8\%$ storage and computation reductions, respectively, with $0.15\%$ accuracy increase over the baseline ResNet-50 model. The code is available at https://github.com/Eclipsess/CHIP_NeurIPS2021.

Overview of "CHIP: CHannel Independence-based Pruning for Compact Neural Networks"

The paper "CHIP: CHannel Independence-based Pruning for Compact Neural Networks" presents a novel approach to filter pruning in neural networks by introducing the concept of channel independence. This approach addresses the challenges related to the computational intensity and storage demands of convolutional neural networks (CNNs), which hinder their efficient deployment on resource-limited platforms.

Key Contributions

The authors propose that traditional filter pruning techniques, which largely rely on intra-channel information to determine filter importance, can benefit from an inter-channel perspective. By leveraging channel independence—a metric assessing correlations among feature maps—filters that generate less independent and therefore less informative feature maps can be successfully identified and pruned. The significant contributions from this paper include:

1. Channel Independence Metric: A robust metric is introduced to measure the independence of feature maps across channels, enabling a globally informed pruning decision based on inter-channel correlations.
2. Efficient and Reliable Pruning Scheme: The paper details a computationally efficient scheme for calculating channel independence across multiple feature maps, with systematic evaluation of its robustness and reliability.
3. Empirical Evaluation: Extensive experiments demonstrate the superior performance of this approach, achieving significant reductions in model size and computational costs while preserving or even improving model accuracy.

Empirical Results

The authors provide empirical evidence supporting the efficacy of channel independence-based pruning on various datasets and models. Notable results include:

• On the CIFAR-10 dataset, the CHIP approach resulted in a 0.90% and 0.94% accuracy increase over baseline ResNet-56 and ResNet-110 models, respectively, with model size and FLOPs reductions of up to 48.3%.
• For the ImageNet dataset, the proposed method achieved storage and computation reductions of 40.8% and 44.8%. Moreover, an accuracy improvement of 0.15% was observed for the ResNet-50 model.

Implications and Future Directions

CHIP's inter-channel approach to filter importance opens up avenues for more informed and potentially more effective model pruning strategies. Practically, this could lead to more efficient deployment of CNNs on edge devices, improving the trade-off between model performance and resources consumed. Theoretically, it contributes to the understanding of network pruning dynamics and the role of feature map correlation in determining filter importance.

Future developments could involve exploring the integration of channel independence with other model compression techniques, such as quantization and knowledge distillation, further optimizing the trade-offs of accuracy, size, and inference speed. Additionally, extending this line of research to novel architectures and training regimes will be valuable in adapting to the evolving demands of AI systems in real-world applications.

In summary, the paper makes a significant contribution to the field of neural network pruning by leveraging inter-channel dependencies, demonstrating that important reductions in computational requirements can be achieved with minimal impact on model performance.