Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
169 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Stochastic Subnetwork Annealing: A Regularization Technique for Fine Tuning Pruned Subnetworks (2401.08830v1)

Published 16 Jan 2024 in cs.LG

Abstract: Pruning methods have recently grown in popularity as an effective way to reduce the size and computational complexity of deep neural networks. Large numbers of parameters can be removed from trained models with little discernible loss in accuracy after a small number of continued training epochs. However, pruning too many parameters at once often causes an initial steep drop in accuracy which can undermine convergence quality. Iterative pruning approaches mitigate this by gradually removing a small number of parameters over multiple epochs. However, this can still lead to subnetworks that overfit local regions of the loss landscape. We introduce a novel and effective approach to tuning subnetworks through a regularization technique we call Stochastic Subnetwork Annealing. Instead of removing parameters in a discrete manner, we instead represent subnetworks with stochastic masks where each parameter has a probabilistic chance of being included or excluded on any given forward pass. We anneal these probabilities over time such that subnetwork structure slowly evolves as mask values become more deterministic, allowing for a smoother and more robust optimization of subnetworks at high levels of sparsity.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (45)
  1. What is the state of neural network pruning?, 2020.
  2. Autoaugment: Learning augmentation policies from data, 2019.
  3. An image is worth 16x16 words: Transformers for image recognition at scale, 2021.
  4. Fire together wire together: A dynamic pruning approach with self-supervised mask prediction, 2022.
  5. The lottery ticket hypothesis: Finding sparse, trainable neural networks, 2019.
  6. The state of sparsity in deep neural networks, 2019.
  7. Loss surfaces, mode connectivity, and fast ensembling of dnns, 2018.
  8. A loss curvature perspective on training instability in deep learning, 2021.
  9. A closer look at deep learning heuristics: Learning rate restarts, warmup and distillation, 2018.
  10. Learning both weights and connections for efficient neural networks, 2015.
  11. L.K. Hansen and P. Salamon. Neural network ensembles. IEEE Transactions on Pattern Analysis and Machine Intelligence, 12(10):993–1001, 1990.
  12. Training independent subnetworks for robust prediction, 2021.
  13. Deep residual learning for image recognition. In IEEE Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2016.
  14. Channel gating neural networks, 2019.
  15. Snapshot ensembles: Train 1, get m for free, 2017.
  16. Adam: A method for stochastic optimization, 2017.
  17. Variational dropout and the local reparameterization trick, 2015.
  18. Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical Report, University of Toronto, May 2012.
  19. Neural network ensembles, cross validation, and active learning. In G. Tesauro, D. Touretzky, and T. Leen, editors, Advances in Neural Information Processing Systems, volume 7. MIT Press, 1994.
  20. Ya Le and Xuan S. Yang. Tiny imagenet visual recognition challenge. 2015.
  21. Layer-adaptive sparsity for the magnitude-based pruning, 2021.
  22. Why m heads are better than one: Training a diverse ensemble of deep networks. arXiv preprint:1511.06314, 2015.
  23. Pruning filters for efficient convnets, 2017.
  24. Runtime neural pruning. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
  25. Deep ensembling with no overhead for either training or testing: The all-round blessings of dynamic sparsity, 2022.
  26. The unreasonable effectiveness of random pruning: Return of the most naive baseline for sparse training, 2022.
  27. Rethinking the value of network pruning, 2019.
  28. On the adequacy of untuned warmup for adaptive optimization, 2019.
  29. On the adequacy of untuned warmup for adaptive optimization, 2021.
  30. Yashwant Malaiya. Antirandom testing: getting the most out of black-box testing. In Proceedings of Sixth International Symposium on Software Reliability Engineering. ISSRE’95, pages 86 – 95, 1995.
  31. Variational dropout sparsifies deep neural networks, 2017.
  32. Uncertainty Baselines: Benchmarks for uncertainty & robustness in deep learning. arXiv preprint arXiv:2106.04015, 2021.
  33. Omiita. Vit-cifar. https://github.com/omihub777/ViT-CIFAR, 2022.
  34. Comparing rewinding and fine-tuning in neural network pruning, 2020.
  35. Super-convergence: Very fast training of neural networks using large learning rates, 2018.
  36. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(56):1929–1958, 2014.
  37. Neural networks with late-phase weights, 2022.
  38. Mothernets: Rapid deep ensemble learning. arXiv preprint:1809.04270, 2018.
  39. Learning structured sparsity in deep neural networks, 2016.
  40. Batchensemble: An alternative approach to efficient ensemble and lifelong learning, 2020.
  41. Prune and tune ensembles: Low-cost ensemble learning with sparse independent subnetworks, 2022.
  42. Antirandom testing: A distance-based approach. Journal of VLSI Design, 2008.
  43. How does learning rate decay help modern neural networks?, 2019.
  44. Wide residual networks, 2017.
  45. Scaling vision transformers, 2022.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets