Papers
Topics
Authors
Recent
Search
2000 character limit reached

Stochastic Gradient Langevin Dynamics Based on Quantization with Increasing Resolution

Published 30 May 2023 in cs.LG and cs.AI | (2305.18864v2)

Abstract: Stochastic learning dynamics based on Langevin or Levy stochastic differential equations (SDEs) in deep neural networks control the variance of noise by varying the size of the mini-batch or directly those of injecting noise. Since the noise variance affects the approximation performance, the design of the additive noise is significant in SDE-based learning and practical implementation. In this paper, we propose an alternative stochastic descent learning equation based on quantized optimization for non-convex objective functions, adopting a stochastic analysis perspective. The proposed method employs a quantized optimization approach that utilizes Langevin SDE dynamics, allowing for controllable noise with an identical distribution without the need for additive noise or adjusting the mini-batch size. Numerical experiments demonstrate the effectiveness of the proposed algorithm on vanilla convolution neural network(CNN) models and the ResNet-50 architecture across various data sets. Furthermore, we provide a simple PyTorch implementation of the proposed algorithm.

Authors (2)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (61)
  1. Privacy of noisy stochastic gradient descent: More iterations without more privacy loss. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 3788–3800. Curran Associates, Inc., 2022. URL https://proceedings.neurips.cc/paper_files/paper/2022/file/18561617ca0b4ffa293166b3186e04b0-Paper-Conference.pdf.
  2. Sigma-delta quantization and finite frames. In 2004 IEEE International Conference on Acoustics, Speech, and Signal Processing, volume 3, pages iii–937, 2004. doi: 10.1109/ICASSP.2004.1326700.
  3. A Note on the Generation of Random Normal Deviates. The Annals of Mathematical Statistics, 29(2):610 – 611, 1958. doi: 10.1214/aoms/1177706645. URL https://doi.org/10.1214/aoms/1177706645.
  4. The promises and pitfalls of stochastic gradient langevin dynamics. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018. URL https://proceedings.neurips.cc/paper_files/paper/2018/file/335cd1b90bfa4ee70b39d08a4ae0cf2d-Paper.pdf.
  5. Stochastic gradient descent performs variational inference, converges to limit cycles for deep networks. In 2018 Information Theory and Applications Workshop, ITA 2018, SanDiego, CA, USA, February 11-16, 2018, pages 1–10. IEEE, 2018. doi: 10.1109/ITA.2018.8503224. URL https://doi.org/10.1109/ITA.2018.8503224.
  6. Stochastic gradient and langevin processes. In Proceedings of the 37th International Conference on Machine Learning, ICML’20. JMLR.org, 2020a.
  7. Stochastic gradient and Langevin processes. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 1810–1819. PMLR, 13–18 Jul 2020b. URL https://proceedings.mlr.press/v119/cheng20e.html.
  8. User-friendly guarantees for the langevin monte carlo with inaccurate gradient. Stochastic Processes and their Applications, 129(12):5278–5311, 2019. ISSN 0304-4149. doi: https://doi.org/10.1016/j.spa.2019.02.016. URL https://www.sciencedirect.com/science/article/pii/S0304414918304824.
  9. Taming the wild: A unified analysis of hogwild-style algorithms. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems 28, pages 2674–2682. NIPS, 2015.
  10. Timothy Dozat. Incorporating Nesterov Momentum into Adam. In Proceedings of the 4th International Conference on Learning Representations: Workshop Track, pages 1–4, 2016.
  11. Statistical learning and inverse problems: A stochastic gradient approach. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 9591–9602. Curran Associates, Inc., 2022. URL https://proceedings.neurips.cc/paper_files/paper/2022/file/3e8b1835833ef809059efa74b9df6805-Paper-Conference.pdf.
  12. Diffusions for global optimization. SIAM Journal on Control and Optimization, 24(5):1031–1043, 1986.
  13. Accurate, large minibatch sgd: Training imagenet in 1 hour, 2018. URL https://arxiv.org/abs/1706.02677.
  14. Learning rates as a function of batch size: A random matrix theory approach to neural network training. Journal of Machine Learning Research, 23(173):1–65, 2022. URL http://jmlr.org/papers/v23/20-1258.html.
  15. Quantization. IEEE Transactions on Information Theory, 44(6):2325–2383, 2006.
  16. Learning both weights and connections for efficient neural network. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems 28, pages 1135–1143. Curran Associates, Inc., 2015.
  17. Train longer, generalize better: Closing the generalization gap in large batch training of neural networks. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS’17, page 1729–1739, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 9781510860964.
  18. White noise hypothesis for uniform quantization errors. SIAM J. Math. Analysis, 38(6):2042–2056, 2007.
  19. Learning to quantize deep networks by optimizing quantization intervals with task loss. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 4350–4359, 2019.
  20. Approaching quartic convergence rates for quasi-stochastic approximation with application to gradient-free optimization. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 15743–15756. Curran Associates, Inc., 2022. URL https://proceedings.neurips.cc/paper_files/paper/2022/file/6530274c68e81047e1f4a2ceb0b8c0ef-Paper-Conference.pdf.
  21. On the insufficiency of existing momentum schemes for stochastic optimization. In ITA, pages 1–9. IEEE, 2018. ISBN 978-1-7281-0124-8. URL http://dblp.uni-trier.de/db/conf/ita/ita2018.html#KidambiN0K18.
  22. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http://arxiv.org/abs/1412.6980.
  23. Fima C. Klebaner. Introduction To Stochastic Calculus With Applications. Imperial College Press, 3rd edition, 2012. ISBN 1848168322.
  24. Alex Krizhevsky. One weird trick for parallelizing convolutional neural networks, 2014. URL https://arxiv.org/abs/1404.5997.
  25. Harold J. Kushner. On the Weak Convergence of Interpolated Markov Chains to a Diffusion. The Annals of Probability, 2(1):40 – 50, 1974. doi: 10.1214/aop/1176996750. URL https://doi.org/10.1214/aop/1176996750.
  26. Stochastic modified equations and dynamics of stochastic gradient algorithms i: Mathematical foundations. Journal of Machine Learning Research, 20(40):1–47, 2019. URL http://jmlr.org/papers/v20/17-526.html.
  27. Generalization error analysis of quantized compressive learning. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 15124–15134. NIPS, 2019.
  28. On the validity of modeling sgd with stochastic differential equations (sdes). In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 12712–12725. Curran Associates, Inc., 2021. URL https://proceedings.neurips.cc/paper_files/paper/2021/file/69f62956429865909921fa916d61c1f8-Paper.pdf.
  29. Fast mixing of stochastic gradient descent with normalization and weight decay. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 9233–9248. Curran Associates, Inc., 2022. URL https://proceedings.neurips.cc/paper_files/paper/2022/file/3c215225324f9988858602dc92219615-Paper-Conference.pdf.
  30. Don’t use large mini-batches, use local sgd. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=B1eyO1BFPr.
  31. Accelerating sgd with momentum for over-parameterized learning. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=r1gixp4FPH.
  32. On the variance of the adaptive learning rate and beyond. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=rkgz2aEKDr.
  33. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=Bkg6RiCqY7.
  34. The power of interpolation: Understanding the effectiveness of SGD in modern over-parametrized learning. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 3325–3334. PMLR, 10–15 Jul 2018. URL https://proceedings.mlr.press/v80/ma18a.html.
  35. On the sdes and scaling rules for adaptive gradient algorithms. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 7697–7711. Curran Associates, Inc., 2022. URL https://proceedings.neurips.cc/paper_files/paper/2022/file/32ac710102f0620d0f28d5d05a44fe08-Paper-Conference.pdf.
  36. Stochastic gradient descent as approximate bayesian inference. Journal of Machine Learning Research, 18(134):1–35, 2017. URL http://jmlr.org/papers/v18/17-214.html.
  37. D. Marco and D.L. Neuhoff. The validity of the additive noise model for uniform scalar quantizers. IEEE Transactions on Information Theory, 51(5):1739–1755, 2005. doi: 10.1109/TIT.2005.846397.
  38. G. Marsaglia. Generating a variable from the tail of the normal distribution. In Mathematical Note No. 322, pages 1–3. Boeing Scientific Research Labs., 1963.
  39. The ziggurat method for generating random variables. Journal of Statistical Software, 5(8):1–7, 2000. doi: 10.18637/jss.v005.i08. URL https://www.jstatsoft.org/index.php/jss/article/view/v005i08.
  40. Stochastic second-order methods improve best-known sample complexity of sgd for gradient-dominated functions. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 10862–10875. Curran Associates, Inc., 2022. URL https://proceedings.neurips.cc/paper_files/paper/2022/file/46323351ebc2afa42b30a6122815cb95-Paper-Conference.pdf.
  41. Generalization bounds of sgld for non-convex learning: Two theoretical viewpoints. In Sébastien Bubeck, Vianney Perchet, and Philippe Rigollet, editors, Proceedings of the 31st Conference On Learning Theory, volume 75 of Proceedings of Machine Learning Research, pages 605–638. PMLR, 06–09 Jul 2018. URL https://proceedings.mlr.press/v75/mou18a.html.
  42. First exit time analysis of stochastic gradient descent under heavy-tailed gradient noise. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https://proceedings.neurips.cc/paper_files/paper/2019/file/a97da629b098b75c294dffdc3e463904-Paper.pdf.
  43. Non-convex learning via stochastic gradient langevin dynamics: a nonasymptotic analysis. In Satyen Kale and Ohad Shamir, editors, Proceedings of the 2017 Conference on Learning Theory, volume 65 of Proceedings of Machine Learning Research, pages 1674–1703. PMLR, 07–10 Jul 2017. URL https://proceedings.mlr.press/v65/raginsky17a.html.
  44. 1-bit stochastic gradient descent and application to data-parallel distributed training of speech dnns. In Interspeech 2014, September 2014.
  45. Measuring the effects of data parallelism on neural network training. Journal of Machine Learning Research (JMLR), 2018. URL https://arxiv.org/pdf/1811.03600.pdf.
  46. Stochastic gradient descent for non-smooth optimization: Convergence results and optimal averaging schemes. In Sanjoy Dasgupta and David McAllester, editors, Proceedings of the 30th International Conference on Machine Learning, volume 28 of Proceedings of Machine Learning Research, pages 71–79, Atlanta, Georgia, USA, 17–19 Jun 2013. PMLR. URL https://proceedings.mlr.press/v28/shamir13.html.
  47. A tail-index analysis of stochastic gradient noise in deep neural networks. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 5827–5837. PMLR, 09–15 Jun 2019. URL https://proceedings.mlr.press/v97/simsekli19a.html.
  48. Momentum enables large batch training. In ICML workshop on Theoretical Physics in Deep Learning, 2019. URL https://proceedings.neurips.cc/paper_files/paper/2019/file/e0eacd983971634327ae1819ea8b6214-Paper.pdf.
  49. On the generalization benefit of noise in stochastic gradient descent. In Proceedings of the 37th International Conference on Machine Learning, ICML’20. JMLR.org, 2020.
  50. Gaussian random number generators. ACM Comput. Surv., 39(4):11–es, nov 2007. ISSN 0360-0300. doi: 10.1145/1287620.1287622. URL https://doi.org/10.1145/1287620.1287622.
  51. Analyzing the generalization capability of sgld using properties of gaussian channels. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 24222–24234. Curran Associates, Inc., 2021. URL https://proceedings.neurips.cc/paper_files/paper/2021/file/cb77649f5d53798edfa0ff40dae46322-Paper.pdf.
  52. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th International Conference on International Conference on Machine Learning, ICML’11, page 681–688, Madison, WI, USA, 2011. Omnipress. ISBN 9781450306195.
  53. Learning structured sparsity in deep neural networks. In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems 29, pages 2074–2082. Curran Associates, Inc., 2016.
  54. On the noisy gradient descent that generalizes as sgd. In Proceedings of the 37th International Conference on Machine Learning, ICML’20. JMLR.org, 2020.
  55. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms, 2017.
  56. A diffusion theory for deep learning dynamics: Stochastic gradient descent exponentially favors flat minima. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=wXgk_iCiYGo.
  57. Global convergence of langevin dynamics based algorithms for nonconvex optimization. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018. URL https://proceedings.neurips.cc/paper_files/paper/2018/file/9c19a2aa1d84e04b0bd4bc888792bd1e-Paper.pdf.
  58. R. Zamir and M. Feder. On lattice quantization noise. IEEE Transactions on Information Theory, 42(4):1152–1159, 1996. doi: 10.1109/18.508838.
  59. Which algorithmic choices matter at which batch sizes? insights from a noisy quadratic model. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https://proceedings.neurips.cc/paper_files/paper/2019/file/e0eacd983971634327ae1819ea8b6214-Paper.pdf.
  60. Low-precision stochastic gradient langevin dynamics. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 26624–26644. PMLR, 2022. URL https://proceedings.mlr.press/v162/zhang22ag.html.
  61. Bernt Øksendal. Stochastic Differential Equations:An Introduction with Applications. Springer-Verlag, 6th edition, 2003. ISBN 978-3-540-04758-2.

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up