Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
102 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Reduced storage direct tensor ring decomposition for convolutional neural networks compression (2405.10802v2)

Published 17 May 2024 in cs.CV and cs.LG

Abstract: Convolutional neural networks (CNNs) are among the most widely used machine learning models for computer vision tasks, such as image classification. To improve the efficiency of CNNs, many CNNs compressing approaches have been developed. Low-rank methods approximate the original convolutional kernel with a sequence of smaller convolutional kernels, which leads to reduced storage and time complexities. In this study, we propose a novel low-rank CNNs compression method that is based on reduced storage direct tensor ring decomposition (RSDTR). The proposed method offers a higher circular mode permutation flexibility, and it is characterized by large parameter and FLOPS compression rates, while preserving a good classification accuracy of the compressed network. The experiments, performed on the CIFAR-10 and ImageNet datasets, clearly demonstrate the efficiency of RSDTR in comparison to other state-of-the-art CNNs compression approaches.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (97)
  1. Cp-decomposition with tensor power method for convolutional neural networks compression, in: BigComp, pp. 115–118.
  2. Softer pruning, incremental regularization, in: 2020 25th international conference on pattern recognition (ICPR), IEEE. pp. 224–230.
  3. Whc: Weighted hybrid criterion for filter pruning on convolutional neural networks, in: ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE. pp. 1–5.
  4. Compressing neural networks with the hashing trick, in: International conference on machine learning, PMLR. pp. 2285–2294.
  5. Ccprune: Collaborative channel pruning for learning compact convolutional networks. Neurocomputing 451, 35–45.
  6. Dynamical channel pruning by conditional accuracy change for deep neural networks. IEEE transactions on neural networks and learning systems 32, 799–813.
  7. A novel rank selection scheme in tensor ring decomposition based on reinforcement learning for deep neural networks, in: ICASSP, IEEE. pp. 3292–3296.
  8. Dynamic structured pruning with novel filter importance and leaky masking based on convolution and batch normalization parameters. IEEE Access 9, 165005–165013.
  9. Neural network relief: a pruning algorithm based on neural activity. Machine Learning , 1–22.
  10. On the channel pruning using graph convolution network for convolutional neural network acceleration, in: Proc. Int. Joint Conf. Artif. Intell, pp. 3107–3113.
  11. More is less: A more complicated network with less inference complexity, in: CVPR, pp. 5840–5848.
  12. Fire together wire together: A dynamic pruning approach with self-supervised mask prediction, in: CVPR, pp. 12454–12463.
  13. An effective low-rank compression with a joint rank selection followed by a compression-friendly training. Neural Networks 161, 165–177.
  14. Convolutional neural network compression via tensor-train decomposition on permuted weight tensor with automatic rank determination, in: International Conference on Computational Science, Springer. pp. 654–667.
  15. Compressing convolutional neural networks with hierarchical tucker-2 decomposition. Applied Soft Computing 132, 109856.
  16. Discrete model compression with resource constraint for deep neural networks, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1899–1908.
  17. Efficient neural network compression inspired by compressive sensing. IEEE Transactions on Neural Networks and Learning Systems .
  18. Ultimate tensorization: compressing convolutional and fc layers alike. [Online] arXiv preprint arXiv:1611.03214 .
  19. Dais: Automatic channel pruning via differentiable annealing indicator search. IEEE Transactions on Neural Networks and Learning Systems .
  20. Automated multi-stage compression of neural networks, in: CVPR Workshops, pp. 0–0.
  21. Deep residual learning for image recognition, in: CVPR, pp. 770–778.
  22. Asymptotic soft filter pruning for deep convolutional neural networks. IEEE transactions on cybernetics 50, 3594–3604.
  23. Soft filter pruning for accelerating deep convolutional neural networks, in: IJCAI, pp. 2234–2240.
  24. Filter pruning via geometric median for deep convolutional neural networks acceleration, in: CVPR, pp. 4340–4349.
  25. Filter pruning by switching to neighboring cnns with good attributes. IEEE Transactions on Neural Networks and Learning Systems .
  26. Low-rank compression of neural nets: Learning the rank of each layer, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8049–8059.
  27. Learning to quantize deep networks by optimizing quantization intervals with task loss, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4350–4359.
  28. Compression of deep convolutional neural networks for fast and low power mobile applications, in: ICLR.
  29. Tensor decompositions and applications. SIAM Review 51, 455–500.
  30. Factorized higher-order cnns with an application to spatio-temporal emotion estimation, in: CVPR, pp. 6060–6069.
  31. Imagenet classification with deep convolutional neural networks. NIPS 25, 1097–1105.
  32. Speeding-up convolutional neural networks using fine-tuned cp-decomposition, in: ICLR.
  33. Optimal brain damage. NIPS 2.
  34. Alternating local enumeration (tnale): Solving tensor network structure search with fewer evaluations, in: International Conference on Machine Learning, PMLR. pp. 20384–20411.
  35. Permutation search of tensor network structures via local sampling, in: International Conference on Machine Learning, PMLR. pp. 13106–13124.
  36. Pruning filters for efficient convnets, in: ICLR.
  37. Dmpp: Differentiable multi-pruner and predictor for neural network pruning. Neural Networks 147, 103–112.
  38. Heuristic rank selection with progressively searching tensor ring network. Complex & Intelligent Systems , 1–15.
  39. Compressing convolutional neural networks via factorized convolutional filters, in: CVPR, pp. 3977–3986.
  40. Provable filter pruning for efficient neural networks, in: ICLR.
  41. Compressing neural networks: Towards determining the optimal layer-wise decomposition, in: NeurIPS.
  42. Hrank: Filter pruning using high-rank feature map, in: CVPR, pp. 1529–1538.
  43. Channel pruning via automatic structure search. arXiv preprint arXiv:2001.08565 .
  44. Toward compact convnets via structure-sparsity regularized filter pruning. IEEE transactions on neural networks and learning systems 31, 574–588.
  45. Towards optimal structured cnn pruning via generative adversarial learning, in: CVPR, pp. 2790–2799.
  46. Soks: Automatic searching of the optimal kernel shapes for stripe-wise network pruning. IEEE Transactions on Neural Networks and Learning Systems .
  47. Attention-based adaptive structured continuous sparse network pruning. Neurocomputing , 127698.
  48. Discrimination-aware network pruning for deep model compression. IEEE Transactions on Pattern Analysis and Machine Intelligence .
  49. Channel pruning guided by spatial and channel attention for dnns in intelligent edge computing. Applied Soft Computing 110, 107636.
  50. Eacp: An effective automatic channel pruning for neural networks. Neurocomputing 526, 131–142.
  51. Learning efficient convolutional networks through network slimming, in: Proceedings of the IEEE international conference on computer vision, pp. 2736–2744.
  52. Fully convolutional networks for semantic segmentation, in: CVPR, pp. 3431–3440.
  53. On algorithms for and computing with the tensor ring decomposition. Numerical Linear Algebra with Applications 27, e2289.
  54. Importance estimation for neural network pruning, in: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11264–11272.
  55. Stacked hourglass networks for human pose estimation, in: ECCV, Springer. pp. 483–499.
  56. Tednet: A pytorch toolkit for tensor decomposition networks. Neurocomputing 469, 234–238.
  57. Stable low-rank tensor decomposition for compression of convolutional neural network, in: ECCV, Springer. pp. 522–539.
  58. Hierarchical threshold pruning based on uniform response criterion. IEEE Transactions on Neural Networks and Learning Systems .
  59. U-net: Convolutional networks for biomedical image segmentation, in: MICCAI, pp. 234–241.
  60. Edp: An efficient decomposition and pruning scheme for convolutional neural network compression. IEEE Transactions on Neural Networks and Learning Systems .
  61. Hrel: Filter pruning based on high relevance between activation maps and class labels. Neural Networks 147, 186–197.
  62. Neural network pruning by cooperative coevolution. arXiv preprint arXiv:2204.05639 .
  63. Vngep: Filter pruning based on von neumann graph entropy. Neurocomputing 528, 113–124.
  64. Very deep convolutional networks for large-scale image recognition, in: ICLR.
  65. Chip: Channel independence-based pruning for compact neural networks. NIPS 34.
  66. Scop: Scientific control for reliable neural network pruning. Advances in Neural Information Processing Systems 33, 10936–10947.
  67. Eigendamage: Structured pruning in the kronecker-factored eigenbasis, in: International conference on machine learning, PMLR. pp. 6566–6575.
  68. Wide compression: Tensor ring nets, in: CVPR, pp. 9329–9338.
  69. Cop: customized correlation-based filter level pruning method for deep cnn compression. Neurocomputing 464, 533–545.
  70. Network pruning using sparse learning and genetic algorithm. Neurocomputing 404, 247–256.
  71. Automatic group-based structured pruning for deep convolutional networks. IEEE Access 10, 128824–128834.
  72. Efficient layer compression without pruning. IEEE Transactions on Image Processing .
  73. Cprnc: Channels pruning via reverse neuron crowding for model compression. Computer Vision and Image Understanding 240, 103942.
  74. Pruning filter via gaussian distribution feature for deep neural networks acceleration, in: 2022 International Joint Conference on Neural Networks (IJCNN), IEEE. pp. 1–8.
  75. Efficient structured pruning based on deep feature stabilization. Neural Computing and Applications 33, 7409–7420.
  76. Trained rank pruning for efficient deep neural networks, in: EMC2-NIPS, IEEE. pp. 14–17.
  77. Trp: Trained rank pruning for efficient deep neural networks, in: Bessiere, C. (Ed.), IJCAI, pp. 977–983.
  78. Channel pruning via multi-criteria based on weight dependency, in: 2021 International Joint Conference on Neural Networks (IJCNN), IEEE. pp. 1–8.
  79. Soft independence guided filter pruning. Pattern Recognition , 110488.
  80. Block-term tensor neural networks. Neural Networks 130, 11–21.
  81. Tutor-instructing global pruning for accelerating convolutional neural networks, in: ECAI 2020. IOS Press, pp. 2792–2799.
  82. Gate trimming: One-shot channel pruning for efficient convolutional neural networks, in: ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE. pp. 1365–1369.
  83. Nisp: Pruning networks using neuron importance score propagation, in: CVPR, pp. 9194–9203.
  84. Arpruning: An automatic channel pruning based on attention map ranking. Neural Networks , 106220.
  85. Nested compression of convolutional neural networks with tucker-2 decomposition, in: 2022 International Joint Conference on Neural Networks (IJCNN), IEEE. pp. 1–8.
  86. Group-based network pruning via nonlinear relationship between convolution filters. Applied Intelligence 52, 9274–9288.
  87. Akecp: Adaptive knowledge extraction from feature maps for fast and efficient channel pruning, in: ACM Multimedia, pp. 648–657.
  88. Fchp: Exploring the discriminative feature and feature correlation of feature maps for hierarchical dnn pruning and compression. IEEE Transactions on Circuits and Systems for Video Technology 32, 6807–6820.
  89. Multi-domain clustering pruning: Exploring space and frequency similarity based on gan. Neurocomputing 542, 126279.
  90. Rufp: Reinitializing unimportant filters for soft pruning. Neurocomputing 483, 311–321.
  91. Fpfs: Filter-level pruning via distance weight measuring filter similarity. Neurocomputing 512, 40–51.
  92. Automatic attention pruning: Improving and automating model pruning using attentions, in: International Conference on Artificial Intelligence and Statistics, PMLR. pp. 10470–10486.
  93. Tensor ring decomposition. [Online] arXiv preprint arXiv:1606.05535 .
  94. Rasp: Regularization-based amplitude saliency pruning. Neural Networks 168, 1–13.
  95. Svdinstn: An integrated method for tensor network representation with efficient structure search. arXiv preprint arXiv:2305.14912 .
  96. Towards effective low-bitwidth convolutional neural networks, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7920–7928.
  97. Discrimination-aware channel pruning for deep neural networks, in: NeurIPS, pp. 881–892.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (2)
  1. Mateusz Gabor (6 papers)
  2. Rafał Zdunek (2 papers)
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets