Papers
Topics
Authors
Recent
Search
2000 character limit reached

Batch size invariant Adam

Published 29 Feb 2024 in cs.LG | (2402.18824v1)

Abstract: We propose a batch size invariant version of Adam, for use in large-scale, distributed settings, in which the mini-batch is divided into micro-batches which are distributed among worker nodes. For the v term, standard Adam first computes the average over micro-batch gradients, then squares, while in the batch size invariant Adam proposed here, we first square the micro-batch gradients, then average. Previous work (e.g. Malladi et al. 2022) used an alternative approach that involved a square-root scaling of the learning rate, but this approach requires strong assumptions to work; in particular that the gradient variance dominates the square of the expected gradient. In contrast, the approach proposed here gives batch size invariance without this assumption. We confirm that in practice our scheme gives batch size invariance in a much larger range of scenarios than the previous approach.

Authors (2)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (27)
  1. Autoaugment: Learning augmentation policies from data. arXiv preprint arXiv:1805.09501, 2018.
  2. BackPACK: Packing more into backprop. In International Conference on Learning Representations, 2020.
  3. Scaling vision transformers to 22 billion parameters. In International Conference on Machine Learning, pp.  7480–7512. PMLR, 2023.
  4. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2020.
  5. On the computational inefficiency of large batch sizes for stochastic gradient descent. arXiv preprint arXiv:1811.12941, 2018.
  6. Accurate, large minibatch sgd: Training imagenet in 1 hour. arXiv preprint arXiv:1706.02677, 2017.
  7. Learning rates as a function of batch size: A random matrix theory approach to neural network training. The Journal of Machine Learning Research, 23(1):7795–7859, 2022.
  8. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.  770–778, 2016.
  9. Batch size-invariance for policy optimization. Advances in Neural Information Processing Systems, 35:17086–17098, 2022.
  10. On large-batch training for deep learning: Generalization gap and sharp minima. arXiv preprint arXiv:1609.04836, 2016.
  11. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
  12. Krizhevsky, A. One weird trick for parallelizing convolutional neural networks. arXiv preprint arXiv:1404.5997, 2014.
  13. Learning multiple layers of features from tiny images. 2009.
  14. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017.
  15. The power of interpolation: Understanding the effectiveness of sgd in modern over-parametrized learning. In International Conference on Machine Learning, pp.  3325–3334. PMLR, 2018.
  16. On the sdes and scaling rules for adaptive gradient algorithms. Advances in Neural Information Processing Systems, 35:7697–7711, 2022.
  17. An empirical model of large-batch training. arXiv preprint arXiv:1812.06162, 2018.
  18. Momo: Momentum models for adaptive learning rates. arXiv preprint arXiv:2305.07583, 2023.
  19. Measuring the effects of data parallelism on neural network training. Journal of Machine Learning Research, 20(112):1–49, 2019.
  20. A bayesian perspective on generalization and stochastic gradient descent. arXiv preprint arXiv:1710.06451, 2017.
  21. Don’t decay the learning rate, increase the batch size. arXiv preprint arXiv:1711.00489, 2017.
  22. Stochastic gradient descent as approximate bayesian inference. Journal of Machine Learning Research, 18(134):1–35, 2017.
  23. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023.
  24. Small-scale proxies for large-scale transformer training instabilities. arXiv preprint arXiv:2309.14322, 2023.
  25. Which algorithmic choices matter at which batch sizes? insights from a noisy quadratic model. Advances in neural information processing systems, 32, 2019.
  26. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022.
  27. Adam accumulation to reduce memory footprints of both activations and gradients for large-scale dnn training. arXiv preprint arXiv:2305.19982, 2023.
Citations (1)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up

Tweets

Sign up for free to view the 2 tweets with 12 likes about this paper.

Sign Up for Free