Papers
Topics
Authors
Recent
Search
2000 character limit reached

Feather: An Elegant Solution to Effective DNN Sparsification

Published 3 Oct 2023 in cs.LG | (2310.02448v1)

Abstract: Neural Network pruning is an increasingly popular way for producing compact and efficient models, suitable for resource-limited environments, while preserving high performance. While the pruning can be performed using a multi-cycle training and fine-tuning process, the recent trend is to encompass the sparsification process during the standard course of training. To this end, we introduce Feather, an efficient sparse training module utilizing the powerful Straight-Through Estimator as its core, coupled with a new thresholding operator and a gradient scaling technique, enabling robust, out-of-the-box sparsification performance. Feather's effectiveness and adaptability is demonstrated using various architectures on the CIFAR dataset, while on ImageNet it achieves state-of-the-art Top-1 validation accuracy using the ResNet-50 architecture, surpassing existing methods, including more complex and computationally heavy ones, by a considerable margin. Code is publicly available at https://github.com/athglentis/feather .

Definition Search Book Streamline Icon: https://streamlinehq.com
References (41)
  1. Deep learning. Nature, 521(7553):436–444, 2015.
  2. Imagenet classification with deep convolutional neural networks. Advances in Neural Information Processing Systems, 25:1097–1105, 2012.
  3. Deepface: Closing the gap to human-level performance in face verification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1701–1708, 2014.
  4. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770–778, 2016.
  5. Efficient processing of deep neural networks: A tutorial and survey. Proceedings of the IEEE, 105(12):2295–2329, 2017.
  6. Model compression and acceleration for deep neural networks: The principles, progress, and challenges. IEEE Signal Processing Magazine, 35(1):126–136, 2018.
  7. Model compression and hardware acceleration for neural networks: A comprehensive survey. Proceedings of the IEEE, 108(4):485–532, 2020.
  8. Pruning and quantization for deep neural network acceleration: A survey. Neurocomputing, 461:370–403, 2021.
  9. Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks. The Journal of Machine Learning Research, 22(1):10882–11005, 2021.
  10. Pruning filters for efficient convnets. arXiv preprint arXiv:1608.08710, 2016.
  11. Pruning convolutional neural networks for resource efficient inference. arXiv preprint arXiv:1611.06440, 2016.
  12. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149, 2015.
  13. Learning both weights and connections for efficient neural network. Advances in Neural Information Processing Systems, 28:1135–1143, 2015.
  14. To prune, or not to prune: exploring the efficacy of pruning for model compression. arXiv preprint arXiv:1710.01878, 2017.
  15. The state of sparsity in deep neural networks. arXiv preprint arXiv:1902.09574, 2019.
  16. Optimal brain surgeon and general network pruning. In Proceedings of the IEEE International Conference on Neural Networks, pages 293–299, 1993.
  17. Learning to prune deep neural networks via layer-wise optimal brain surgeon. Advances in Neural Information Processing Systems, 30:4860–4874, 2017.
  18. Fast convex pruning of deep neural networks. SIAM Journal on Mathematics of Data Science, 2(1):158–188, 2020.
  19. Soft threshold weight reparameterization for learnable sparsity. In Proceedings of the International Conference on Machine Learning, pages 5544–5555, 2020.
  20. Comparing rewinding and fine-tuning in neural network pruning. arXiv preprint arXiv:2003.02389, 2020.
  21. Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science. Nature Communications, 9(1):2383, 2018.
  22. Sparse networks from scratch: Faster training without losing performance. arXiv preprint arXiv:1907.04840, 2019.
  23. Sparse training theory for scalable and efficient agents. In Proceedings of the International Conference on Autonomous Agents and Multiagent Systems, 2021.
  24. Online weight pruning via adaptive sparsity loss. In Proceedings of the IEEE International Conference on Image Processing, pages 3517–3521, 2021.
  25. Top-kast: Top-k always sparse training. Advances in Neural Information Processing Systems, 33:20744–20754, 2020.
  26. Spartan: Differentiable sparsity via regularized transportation. Advances in Neural Information Processing Systems, 35:4189–4202, 2022.
  27. Antoine Vanderschueren and Christophe De Vleeschouwer. Are straight-through gradients and soft-thresholding all you need for sparse training? In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 3808–3817, 2023.
  28. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432, 2013.
  29. David L Donoho. De-noising by soft-thresholding. IEEE Transactions on Information Theory, 41(3):613–627, 1995.
  30. Alex Krizhevsky. Learning multiple layers of features from tiny images. Master’s thesis, Department of Computer Science, University of Toronto, 2009.
  31. Imagenet: A large-scale hierarchical image database. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 248–255, 2009.
  32. Haoyang Liu and Rina Foygel Barber. Between hard and soft thresholding: optimal iterative thresholding algorithms. Information and Inference: A Journal of the IMA, 9(4):899–933, 2020.
  33. Katsuyuki Hagiwara. Bridging between soft and hard thresholding by scaling. IEICE Transactions on Information and Systems, 105(9):1529–1536, 2022.
  34. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861, 2017.
  35. Densely connected convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4700–4708, 2017.
  36. Effective sparsification of neural networks with global sparsity constraint. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3599–3608, 2021.
  37. Picking winning tickets before training by preserving gradient flow. In Proceedings of the International Conference on Learning Representations, 2020.
  38. Dropout: a simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15(1):1929–1958, 2014.
  39. Discovering neural wirings. Advances in Neural Information Processing Systems, 32:2684–2694, 2019.
  40. Optimizing gradient-driven criteria in network sparsity: Gradient is all you need. arXiv preprint arXiv:2201.12826, 2022.
  41. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 115:211–252, 2015.

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now