HRank: Filter Pruning using High-Rank Feature Map (2002.10179v2)

Abstract: Neural network pruning offers a promising prospect to facilitate deploying deep neural networks on resource-limited devices. However, existing methods are still challenged by the training inefficiency and labor cost in pruning designs, due to missing theoretical guidance of non-salient network components. In this paper, we propose a novel filter pruning method by exploring the High Rank of feature maps (HRank). Our HRank is inspired by the discovery that the average rank of multiple feature maps generated by a single filter is always the same, regardless of the number of image batches CNNs receive. Based on HRank, we develop a method that is mathematically formulated to prune filters with low-rank feature maps. The principle behind our pruning is that low-rank feature maps contain less information, and thus pruned results can be easily reproduced. Besides, we experimentally show that weights with high-rank feature maps contain more important information, such that even when a portion is not updated, very little damage would be done to the model performance. Without introducing any additional constraints, HRank leads to significant improvements over the state-of-the-arts in terms of FLOPs and parameters reduction, with similar accuracies. For example, with ResNet-110, we achieve a 58.2%-FLOPs reduction by removing 59.2% of the parameters, with only a small loss of 0.14% in top-1 accuracy on CIFAR-10. With Res-50, we achieve a 43.8%-FLOPs reduction by removing 36.7% of the parameters, with only a loss of 1.17% in the top-1 accuracy on ImageNet. The codes can be available at https://github.com/lmbxmu/HRank.

Overview of HRank: Filter Pruning using High-Rank Feature Map

The presented paper introduces HRank, a novel approach to filter pruning in Convolutional Neural Networks (CNNs) by exploring the high rank of feature maps. Pruning is a method to reduce the computational demands of deep neural networks, making them viable for deployment on resource-constrained devices.

Methodology

HRank leverages an empirical observation that feature map ranks, generated by individual filters, remain consistent across varying input datasets. This finding allows for an efficient estimation of filter importance. The proposed method capitalizes on the rank of feature maps, mathematically proving that lower-rank feature maps contribute less to a network's predictive capacity. Therefore, filters associated with lower-rank feature maps can be pruned safely with minimal impact on performance.

Experimental Results

Experiments demonstrate HRank's efficacy across diverse architectures including VGG, ResNet, GoogLeNet, and DenseNet. Notably, in ResNet-110, a 58.2% reduction in FLOPs and a 59.2% reduction in parameters were achieved with only a 0.14% drop in accuracy. Similarly, ResNet-50 on ImageNet experienced a 43.8% reduction in FLOPs and a 36.7% reduction in parameters, evidencing HRank's ability to maintain high accuracy despite significant architecture compression.

Key Findings

1. Consistency of Average Rank: The paper highlights a consistent average rank of feature maps irrespective of input data variability, paving the way for reliable filter importance analysis.
2. Mathematical Validation: The paper provides a theoretical basis, proving the correlation between feature map rank and information content, distinguishing high-rank feature maps as vital for network performance.
3. Integration into CNN Workflows: HRank seamlessly integrates into existing CNN frameworks without additional constraints, simplifying the pruning process while outperforming state-of-the-art methods in both parameters and FLOPs reduction.

Implications and Future Work

HRank offers practical implications for optimizing CNN deployment on edge devices, addressing critical issues of computational efficiency and memory constraints. The methodological elegance lies in its ability to prune effectively without compromising performance, highlighting the potential for broader applications in model compression tasks.

The authors suggest further exploration into the theoretical grounding of why the average rank consistency holds, which could provide deeper insights into network optimization strategies.

Conclusion

HRank presents a compelling alternative to traditional pruning methods, offering a mathematically sound and empirically validated approach to filter pruning. Its effectiveness across multiple architectures marks a significant contribution to the field of neural network optimization, providing a framework that balances complexity reduction with performance retention.