Filter Pruning For CNN With Enhanced Linear Representation Redundancy (2310.06344v1)
Abstract: Structured network pruning excels non-structured methods because they can take advantage of the thriving developed parallel computing techniques. In this paper, we propose a new structured pruning method. Firstly, to create more structured redundancy, we present a data-driven loss function term calculated from the correlation coefficient matrix of different feature maps in the same layer, named CCM-loss. This loss term can encourage the neural network to learn stronger linear representation relations between feature maps during the training from the scratch so that more homogenous parts can be removed later in pruning. CCM-loss provides us with another universal transcendental mathematical tool besides L*-norm regularization, which concentrates on generating zeros, to generate more redundancy but for the different genres. Furthermore, we design a matching channel selection strategy based on principal components analysis to exploit the maximum potential ability of CCM-loss. In our new strategy, we mainly focus on the consistency and integrality of the information flow in the network. Instead of empirically hard-code the retain ratio for each layer, our channel selection strategy can dynamically adjust each layer's retain ratio according to the specific circumstance of a per-trained model to push the prune ratio to the limit. Notably, on the Cifar-10 dataset, our method brings 93.64% accuracy for pruned VGG-16 with only 1.40M parameters and 49.60M FLOPs, the pruned ratios for parameters and FLOPs are 90.6% and 84.2%, respectively. For ResNet-50 trained on the ImageNet dataset, our approach achieves 42.8% and 47.3% storage and computation reductions, respectively, with an accuracy of 76.23%. Our code is available at https://github.com/Bojue-Wang/CCM-LRR.
- Centripetal sgd for pruning very deep convolutional networks with complicated structure. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 4943–4953.
- Depgraph: Towards any structural pruning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 16091–16101.
- A survey of quantization methods for efficient neural network inference. In Low-Power Computer Vision, 291–326. Chapman and Hall/CRC.
- Ghostnet: More features from cheap operations. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 1580–1589.
- Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, 770–778.
- Soft filter pruning for accelerating deep convolutional neural networks. In Proceedings of the 27th International Joint Conference on Artificial Intelligence, 2234–2240.
- Filter pruning via geometric median for deep convolutional neural networks acceleration. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 4340–4349.
- Channel pruning for accelerating very deep neural networks. In Proceedings of the IEEE international conference on computer vision(ICCV), 1389–1397.
- Teacher-Student Architecture for Knowledge Learning: A Survey. arXiv preprint arXiv:2210.17332.
- Network trimming: A data-driven neuron pruning approach towards efficient deep architectures. arXiv preprint arXiv:1607.03250.
- Data-driven sparse structure selection for deep neural networks. In Proceedings of the European conference on computer vision (ECCV), 304–320.
- Learning multiple layers of features from tiny images.
- Pruning filters with L1-norm and capped L1-norm for CNN compression. Applied Intelligence, 51: 1152–1160.
- Pruning filters for efficient convnets. arXiv preprint arXiv:1608.08710.
- Provable filter pruning for efficient neural networks. arXiv preprint arXiv:1911.07412.
- Hrank: Filter pruning using high-rank feature map. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition(CVPR), 1529–1538.
- Towards optimal structured cnn pruning via generative adversarial learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition(CVPR), 2790–2799.
- Autopruner: An end-to-end trainable filter pruning method for efficient deep model inference. Pattern Recognition, 107: 107461.
- Thinet: A filter level pruning method for deep neural network compression. In Proceedings of the IEEE international conference on computer vision, 5058–5066.
- Designing efficient convolutional neural network structure: A survey. Neurocomputing, 489: 139–156.
- Importance estimation for neural network pruning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 11264–11272.
- Automatic differentiation in pytorch.
- Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556.
- Chip: Channel independence-based pruning for compact neural networks. Advances in Neural Information Processing Systems(NeurIPS), 34: 24604–24616.
- Scientific control for reliable neural network pruning. Neural Information Processing Systems (NeurIPS), 1(2): 7.
- Randomized algorithms for low-rank matrix approximation: Design, analysis, and applications. arXiv preprint arXiv:2306.12418.
- Learning structured sparsity in deep neural networks. Advances in neural information processing systems(NeurIPS), 29.
- Nisp: Pruning networks using neuron importance score propagation. In Proceedings of the IEEE conference on computer vision and pattern recognition(CVPR), 9194–9203.
- Model selection and estimation in regression with grouped variables. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 68(1): 49–67.
- Accelerate cnn via recursive bayesian pruning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 3306–3315.
- Bojue Wang (1 paper)
- Chunmei Ma (3 papers)
- Bin Liu (441 papers)
- Nianbo Liu (1 paper)
- Jinqi Zhu (1 paper)