2000 character limit reached
Accelerating Convolutional Neural Network Pruning via Spatial Aura Entropy
Published 8 Dec 2023 in cs.CV and cs.LG | (2312.04926v1)
Abstract: In recent years, pruning has emerged as a popular technique to reduce the computational complexity and memory footprint of Convolutional Neural Network (CNN) models. Mutual Information (MI) has been widely used as a criterion for identifying unimportant filters to prune. However, existing methods for MI computation suffer from high computational cost and sensitivity to noise, leading to suboptimal pruning performance. We propose a novel method to improve MI computation for CNN pruning, using the spatial aura entropy. The spatial aura entropy is useful for evaluating the heterogeneity in the distribution of the neural activations over a neighborhood, providing information about local features. Our method effectively improves the MI computation for CNN pruning, leading to more robust and efficient pruning. Experimental results on the CIFAR-10 benchmark dataset demonstrate the superiority of our approach in terms of pruning performance and computational efficiency.
- A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in Advances in neural information processing systems, 2012, pp. 1097–1105.
- K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778.
- J. Long, E. Shelhamer, and T. Darrell, “Fully convolutional networks for semantic segmentation,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2015, pp. 3431–3440.
- C. Sun, L. Wang, X. Liu, and J. Shi, “Patient knowledge distillation for bert model compression,” in Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 2019, pp. 3873–3879.
- Y. LeCun, J. S. Denker, and S. A. Solla, “Optimal brain damage,” in Advances in neural information processing systems, 1990, pp. 598–605.
- X. Wu, R. He, Z. Sun, and T. Tan, “A survey of compressing deep neural networks,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 43, no. 3, pp. 705–723, 2021.
- S. Han, J. Pool, J. Tran, and W. J. Dally, “Learning both weights and connections for efficient neural networks,” in Advances in neural information processing systems, 2015, pp. 1135–1143.
- H. Li, A. Kadav, I. Durdanovic, H. Samet, and H. P. Graf, “Pruning filters for efficient convnets,” in Proceedings of the International Conference on Learning Representations, 2017.
- D. Molchanov, A. Ashukha, and D. Vetrov, “Importance-driven neural network pruning,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019, pp. 11 281–11 289.
- S. Han, X. Liu, H. Mao, J. Pu, A. Pedram, M. A. Horowitz, and W. J. Dally, “Eie: efficient inference engine on compressed deep neural network,” in Proceedings of the 43rd ACM/IEEE Annual International Symposium on Computer Architecture. IEEE, 2016, pp. 243–254.
- Z. Liu, M. Sun, T. Zhou, G. Huang, and T. Darrell, “Rethinking the value of network pruning,” in Proceedings of the International Conference on Computer Vision, 2019, pp. 1070–1078.
- J. Frankle and M. Carbin, “The lottery ticket hypothesis: Finding sparse, trainable neural networks,” in International Conference on Learning Representations, 2019. [Online]. Available: https://openreview.net/forum?id=rJl-b3RcF7
- C. Sarvani, M. Ghorai, S. R. Dubey, and S. S. Basha, “Hrel: Filter pruning based on high relevance between activation maps and class labels,” Neural Networks, vol. 147, pp. 186–197, 2022. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0893608021004962
- N. Tishby, F. C. Pereira, and W. Bialek, “The information bottleneck method,” arXiv preprint physics/0004057, 1999.
- B. M. sat and R. azvan Andonie, “Information bottleneck in deep learning - a semiotic approach,” International Journal of Computers Communications & Control, vol. 17, no. 1, 2022. [Online]. Available: http://univagora.ro/jour/index.php/ijccc/article/view/4650
- Z. Li, S. Liu, X. Yu, K. Bhavya, J. Cao, D. J. Daniel, P.-T. Bremer, and V. Pascucci, “”understanding robustness lottery”: A comparative visual analysis of neural network pruning approaches,” arXiv preprint arXiv:2206.07918, 2022.
- B. Kovalerchuk, R. Andonie, N. Datia, K. Nazemi, and E. Banissi, “Visual knowledge discovery with artificial intelligence: Challenges and future directions,” in Integrating Artificial Intelligence and Visualization for Visual Knowledge Discovery, B. Kovalerchuk, K. Nazemi, R. Andonie, N. Datia, and E. Banissi, Eds. Cham: Springer International Publishing, 2022, pp. 1–27.
- Y. He, X. Zhang, and S. Sun, “Channel pruning for accelerating very deep neural networks,” International Conference on Computer Vision, pp. 1398–1406, 2017.
- H. Li, A. Kadav, I. Durdanovic, H. Samet, and H. P. Graf, “Pruning filters for efficient convnets,” arXiv preprint arXiv:1608.08710, 2016.
- B. Mu¸sat and R. a. Andonie, “Semiotic aggregation in deep learning,” Entropy, vol. 22, no. 12, 2020. [Online]. Available: https://www.mdpi.com/1099-4300/22/12/1365
- K. Wickstrøm, S. Løkse, M. Kampffmeyer, S. Yu, J. Principe, and R. Jenssen, “Information plane analysis of deep neural networks via matrix-based renyi’s entropy and tensor kernels,” arXiv preprint arXiv:1909.11396, 2019.
- A. Gordon, E. Eban, O. Nachum, B. Chen, H. Wu, T. Yang, and E. Choi, “Morphnet: Fast & simple resource-constrained structure learning of deep networks,” in Advances in neural information processing systems, 2018, pp. 10 205–10 215.
- S. Han, H. Zhu, J. Liu, and W. J. Dally, “Once-for-all: Train one network and specialize it for efficient deployment,” Transactions on Pattern Analysis and Machine Intelligence, 2021.
- Z. Liu, M. Sun, T. Zhou, G. Huang, and T. Darrell, “Autocompress: An automated neural network compression framework,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 9251–9260.
- Y. Cao, J. Xu, and W. Lin, “Learning a compression-agnostic representation with sparsity regularization,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 10 930–10 938.
- A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam, “Mobilenets: Efficient convolutional neural networks for mobile vision applications,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 1382–1391.
- M. Tan and Q. V. Le, “Efficientnet: Rethinking model scaling for convolutional neural networks,” in Proceedings of the IEEE international conference on computer vision, 2019, pp. 6105–6114.
- E. Volden, G. Giraudon, and M. Berthod, “Modelling image redundancy,” in 1995 International Geoscience and Remote Sensing Symposium, IGARSS ’95. Quantitative Remote Sensing for Science and Applications, vol. 3, 1995, pp. 2148–2150.
- A. G. Journel and C. V. Deutsch, “Entropy and spatial disorder,” Mathematical Geology, vol. 25, no. 3, pp. 329–355, 1993.
- N. I. Tapia and P. A. Estévez, “On the information plane of autoencoders,” CoRR, vol. abs/2005.07783, 2020. [Online]. Available: https://arxiv.org/abs/2005.07783
- K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” in Proceedings of the International Conference on Learning Representations (ICLR), 2015.
- A. Krizhevsky, V. Nair, and G. Hinton, “Cifar-10 (canadian institute for advanced research).” [Online]. Available: http://www.cs.toronto.edu/ kriz/cifar.html
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.