Towards Explaining Deep Neural Network Compression Through a Probabilistic Latent Space (2403.00155v3)
Abstract: Despite the impressive performance of deep neural networks (DNNs), their computational complexity and storage space consumption have led to the concept of network compression. While DNN compression techniques such as pruning and low-rank decomposition have been extensively studied, there has been insufficient attention paid to their theoretical explanation. In this paper, we propose a novel theoretical framework that leverages a probabilistic latent space of DNN weights and explains the optimal network sparsity by using the information-theoretic divergence measures. We introduce new analogous projected patterns (AP2) and analogous-in-probability projected patterns (AP3) notions for DNNs and prove that there exists a relationship between AP3/AP2 property of layers in the network and its performance. Further, we provide a theoretical analysis that explains the training process of the compressed network. The theoretical results are empirically validated through experiments conducted on standard pre-trained benchmarks, including AlexNet, ResNet50, and VGG16, using CIFAR10 and CIFAR100 datasets. Through our experiments, we highlight the relationship of AP3 and AP2 properties with fine-tuning pruned DNNs and sparsity levels.
- “Pruning by explaining: A novel criterion for deep neural network pruning,” Pattern Recognition, vol. 115, pp. 107899, 2021.
- “What is the state of neural network pruning?,” in Proceedings of Machine Learning and Systems, I. Dhillon, D. Papailiopoulos, and V. Sze, Eds., 2020, vol. 2, pp. 129–146.
- “Centripetal sgd for pruning very deep convolutional networks with complicated structure,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019, pp. 4943–4953.
- “A signal propagation perspective for pruning neural networks at initialization,” arXiv preprint arXiv:1906.06307, 2019.
- “Lookahead: A far-sighted alternative of magnitude-based pruning,” arXiv preprint arXiv:2002.04809, 2020.
- “Group fisher pruning for practical network compression,” in International Conference on Machine Learning. PMLR, 2021, pp. 7021–7032.
- “Gate decorator: Global filter pruning method for accelerating deep convolutional neural networks,” Advances in neural information processing systems, vol. 32, 2019.
- T. Cover and J. Thomas, Elements of information theory, 1st edn. John Wiley & Sons, Chichester, 1991.
- Information theory: coding theorems for discrete memoryless systems, Cambridge University Press, 2011.
- Tomohiro Nishiyama, “Lower bounds for the total variation distance given means and variances of distributions,” arXiv preprint arXiv:2212.05820, 2022.
- “Structured pruning of deep convolutional neural networks,” ACM Journal on Emerging Technologies in Computing Systems (JETC), vol. 13, no. 3, pp. 1–18, 2017.
- “Learning structured sparsity in deep neural networks,” Advances in neural information processing systems, vol. 29, 2016.
- “Pruning filters for efficient convnets,” arXiv preprint arXiv:1608.08710, 2016.
- “Channel pruning for accelerating very deep neural networks,” in Proceedings of the IEEE international conference on computer vision, 2017, pp. 1389–1397.
- “Thinet: A filter level pruning method for deep neural network compression,” in Proceedings of the IEEE international conference on computer vision, 2017, pp. 5058–5066.
- “Learning both weights and connections for efficient neural network,” Advances in neural information processing systems, vol. 28, 2015.
- “Movement pruning: Adaptive sparsity by fine-tuning,” Advances in Neural Information Processing Systems, vol. 33, pp. 20378–20389, 2020.
- “Pruning convolutional neural networks for resource efficient inference,” arXiv preprint arXiv:1611.06440, 2016.
- “To prune, or not to prune: exploring the efficacy of pruning for model compression,” arXiv preprint arXiv:1710.01878, 2017.
- “Rethinking the smaller-norm-less-informative assumption in channel pruning of convolution layers,” arXiv preprint arXiv:1802.00124, 2018.
- “Importance estimation for neural network pruning,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019, pp. 11264–11272.
- “Learning multiple layers of features from tiny images,” 2009.
- “Deep residual learning for image recognition,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
- “Very deep convolutional networks for large-scale image recognition,” arXiv preprint arXiv:1409.1556, 2014.
- “Imagenet classification with deep convolutional neural networks,” in Advances in Neural Information Processing Systems, F. Pereira, C.J. Burges, L. Bottou, and K.Q. Weinberger, Eds. 2012, vol. 25, Curran Associates, Inc.
- “Slimming neural networks using adaptive connectivity scores,” IEEE Transactions on Neural Networks and Learning Systems, 2022.
- “Mint: Deep network compression via mutual information-based neuron trimming,” in 2020 25th International Conference on Pattern Recognition (ICPR). IEEE, 2021, pp. 8251–8258.
- “Theoretical understanding of the information flow on continual learning performance,” in European Conference on Computer Vision. Springer, 2022, pp. 86–101.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.
