Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
166 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

On the Trajectories of SGD Without Replacement (2312.16143v2)

Published 26 Dec 2023 in cs.LG, math.OC, and stat.ML

Abstract: This article examines the implicit regularization effect of Stochastic Gradient Descent (SGD). We consider the case of SGD without replacement, the variant typically used to optimize large-scale neural networks. We analyze this algorithm in a more realistic regime than typically considered in theoretical works on SGD, as, e.g., we allow the product of the learning rate and Hessian to be $O(1)$ and we do not specify any model architecture, learning task, or loss (objective) function. Our core theoretical result is that optimizing with SGD without replacement is locally equivalent to making an additional step on a novel regularizer. This implies that the expected trajectories of SGD without replacement can be decoupled in (i) following SGD with replacement (in which batches are sampled i.i.d.) along the directions of high curvature, and (ii) regularizing the trace of the noise covariance along the flat ones. As a consequence, SGD without replacement travels flat areas and may escape saddles significantly faster than SGD with replacement. On several vision tasks, the novel regularizer penalizes a weighted trace of the Fisher Matrix, thus encouraging sparsity in the spectrum of the Hessian of the loss in line with empirical observations from prior work. We also propose an explanation for why SGD does not train at the edge of stability (as opposed to GD).

Definition Search Book Streamline Icon: https://streamlinehq.com
References (99)
  1. Efficient approaches for escaping higher order saddle points in non-convex optimization. arXiv:1602.05908 [cs, stat].
  2. On the Optimization of Deep Networks: Implicit Acceleration by Overparameterization. arXiv:1802.06509 [cs].
  3. Implicit regularization in deep matrix factorization. In Advances in Neural Information Processing Systems, pages 7411–7422.
  4. Neural networks and principal component analysis: Learning from examples without local minima. Neural Networks, 2(1):53–58.
  5. Implicit Gradient Regularization. arXiv:2009.11162 [cs, stat]. arXiv: 2009.11162.
  6. Nearly-tight VC-dimension and Pseudodimension Bounds for Piecewise Linear Neural Networks. Journal of Machine Learning Research, 20(63):1–17.
  7. Benign Overfitting in Linear Regression. Proceedings of the National Academy of Sciences, 117(48):30063–30070. arXiv:1906.11300 [cs, math, stat].
  8. Reconciling modern machine learning practice and the bias-variance trade-off. Proceedings of the National Academy of Sciences, 116(32):15849–15854. arXiv:1812.11118 [cs, stat].
  9. Bengio, Y. (2012). Practical recommendations for gradient-based training of deep architectures. arXiv:1206.5533 [cs].
  10. Implicit regularization for deep neural networks driven by an Ornstein-Uhlenbeck like process. arXiv preprint arXiv:1904.09080, page 31.
  11. Bottou, L. (2009). Curiously Fast Convergence of some Stochastic Gradient Descent Algorithms. page 4.
  12. Bottou, L. (2012). Stochastic Gradient Descent Tricks. In Montavon, G., Orr, G. B., and Müller, K.-R., editors, Neural Networks: Tricks of the Trade: Second Edition, Lecture Notes in Computer Science, pages 421–436. Springer, Berlin, Heidelberg.
  13. On the Implicit Bias of Adam. arXiv:2309.00079 [cs, math, stat].
  14. Stochastic gradient descent performs variational inference, converges to limit cycles for deep networks. In 2018 Information Theory and Applications Workshop (ITA), pages 1–10. IEEE.
  15. Stochastic Collapse: How Gradient Noise Attracts SGD Dynamics Towards Simpler Subnetworks. arXiv:2306.04251 [cs, stat].
  16. Adaptive Gradient Methods at the Edge of Stability. arXiv:2207.14484 [cs].
  17. Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability. arXiv:2103.00065 [cs, stat]. arXiv: 2103.00065.
  18. Label Noise SGD Provably Prefers Flat Global Minimizers. arXiv:2106.06530 [cs, math, stat]. arXiv: 2106.06530.
  19. Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stability. arXiv:2209.15594 [cs, math, stat].
  20. Escaping Saddles with Stochastic Gradients. arXiv:1803.05999 [cs, math, stat].
  21. Identifying and attacking the saddle point problem in high-dimensional non-convex optimization. arXiv:1406.2572 [cs, math, stat].
  22. ImageNet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248–255. ISSN: 1063-6919.
  23. Essentially No Barriers in Neural Network Energy Landscape. arXiv:1803.00885 [cs, stat]. arXiv: 1803.00885.
  24. Gradient Descent Can Take Exponential Time to Escape Saddle Points. arXiv:1705.10412 [cs, math, stat].
  25. Sharp Analysis for Nonconvex SGD Escaping from Saddle Points. arXiv:1902.00247 [cs, math].
  26. Topology and Geometry of Half-Rectified Network Optimization. arXiv:1611.01540 [cs, stat]. arXiv: 1611.01540.
  27. Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs. arXiv:1802.10026 [cs, stat]. arXiv: 1802.10026.
  28. Escaping from saddle points—online stochastic gradient for tensor decomposition. In Conference on Learning Theory, pages 797–842.
  29. Stochastic Training is Not Necessary for Generalization. arXiv:2109.14119 [cs, math]. arXiv: 2109.14119.
  30. Implicit regularization in Heavy-ball momentum accelerated stochastic gradient descent. arXiv:2302.00849 [cs, math].
  31. Deep learning. MIT press.
  32. Qualitatively characterizing neural network optimization problems. arXiv:1412.6544 [cs, stat].
  33. Accurate, large minibatch sgd: Training imagenet in 1 hour. arXiv preprint arXiv:1706.02677.
  34. Characterizing implicit bias in terms of optimization geometry. arXiv preprint arXiv:1802.08246.
  35. Implicit bias of gradient descent on linear convolutional networks. In Advances in Neural Information Processing Systems, pages 9461–9471.
  36. Implicit regularization in matrix factorization. In Advances in Neural Information Processing Systems, pages 6151–6159.
  37. Why Random Reshuffling Beats Stochastic Gradient Descent. Mathematical Programming, 186(1-2):49–84. arXiv:1510.08560 [math].
  38. Geometric numerical integration: structure-preserving algorithms for ordinary differential equations. Number 31 in Springer series in computational mathematics. Springer, Berlin ; New York, 2nd ed edition. OCLC: ocm69223213.
  39. Random Shuffling Beats SGD after Finite Epochs. arXiv:1806.10077 [math, stat].
  40. Shape Matters: Understanding the Implicit Bias of the Noise Covariance. arXiv:2006.08680 [cs, stat]. arXiv: 2006.08680.
  41. Control Batch Size and Learning Rate to Generalize Well: Theoretical and Empirical Evidence. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.
  42. Deep Residual Learning for Image Recognition. arXiv:1512.03385 [cs]. arXiv: 1512.03385.
  43. Densely Connected Convolutional Networks. arXiv:1608.06993 [cs] version: 5.
  44. Catastrophic Fisher Explosion: Early Phase Fisher Matrix Impacts Generalization. arXiv:2012.14193 [cs, stat]. arXiv: 2012.14193.
  45. Three Factors Influencing Minima in SGD. arXiv:1711.04623 [cs, stat]. arXiv: 1711.04623.
  46. On the relation between the sharpest direc- tions of DNN loss and the SGD step length.
  47. The Break-Even Point on Optimization Trajectories of Deep Neural Networks. arXiv:2002.09572 [cs, stat]. arXiv: 2002.09572.
  48. Directional convergence and alignment in deep learning. arXiv:2006.06657 [cs, math, stat].
  49. Fantastic Generalization Measures and Where to Find Them. arXiv:1912.02178 [cs, stat]. arXiv: 1912.02178.
  50. How to Escape Saddle Points Efficiently. arXiv:1703.00887 [cs, math, stat].
  51. On Nonconvex Optimization for Machine Learning: Gradients, Stochasticity, and Saddle Points. Journal of the ACM, 68(2):1–29.
  52. Kawaguchi, K. (2016). Deep Learning without Poor Local Minima. arXiv:1605.07110 [cs, math, stat].
  53. On large-batch training for deep learning: Generalization gap and sharp minima. arXiv preprint arXiv:1609.04836.
  54. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
  55. An Alternative View: When Does SGD Escape Local Minima? arXiv:1802.06175 [cs].
  56. Big Transfer (BiT): General Visual Representation Learning. arXiv:1912.11370 [cs] version: 3.
  57. Explaining Landscape Connectivity of Low-cost Solutions for Multilayer Nets. arXiv:1906.06247 [cs, stat]. arXiv: 1906.06247.
  58. Efficient backprop. In Neural networks: Tricks of the trade, pages 9–48. Springer.
  59. Gradient Descent Only Converges to Minimizers.
  60. The large learning rate phase of deep learning: the catapult mechanism. arXiv:2003.02218 [cs, stat]. arXiv: 2003.02218.
  61. Algorithmic regularization in over-parameterized matrix sensing and neural networks with quadratic activations. arXiv preprint arXiv:1712.09203.
  62. Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networks. arXiv:1907.04595 [cs, stat]. arXiv: 1907.04595.
  63. On the Validity of Modeling SGD with Stochastic Differential Equations (SDEs). arXiv:2102.12470 [cs, stat]. arXiv: 2102.12470.
  64. What Happens after SGD Reaches Zero Loss? –A Mathematical Framework. arXiv:2110.06914 [cs, stat]. arXiv: 2110.06914.
  65. Understanding the Loss Surface of Neural Networks for Binary Classification. arXiv:1803.00909 [cs, stat]. arXiv: 1803.00909.
  66. Toward a theory of optimization for over-parameterized systems of non-linear equations: the lessons of deep learning. arXiv:2003.00307 [cs, math, stat]. arXiv: 2003.00307.
  67. Benign Oscillation of Stochastic Gradient Descent with Large Learning Rates. arXiv:2310.17074 [cs, math, stat].
  68. Gradient descent maximizes the margin of homogeneous neural networks. arXiv preprint arXiv:1906.05890.
  69. Martens, J. (2020). New Insights and Perspectives on the Natural Gradient Method.
  70. Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learning. arXiv:1810.01075 [cs, stat].
  71. Random Reshuffling: Simple Analysis with Vast Improvements. In Advances in Neural Information Processing Systems, volume 33, pages 17309–17320. Curran Associates, Inc.
  72. Exploring Generalization in Deep Learning. In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R., editors, Advances in Neural Information Processing Systems 30, pages 5947–5956. Curran Associates, Inc.
  73. Nguyen, Q. (2019). On Connected Sublevel Sets in Deep Learning. arXiv:1901.07417 [cs, stat]. arXiv: 1901.07417.
  74. ON THE LOSS LANDSCAPE OF A CLASS OF DEEP NEURAL NETWORKS WITH NO BAD LOCAL VALLEYS. page 20.
  75. Explicit Regularization in Overparametrized Models via Noise Injection. arXiv:2206.04613 [cs, stat].
  76. Papyan, V. (2019). The Full Spectrum of Deepnet Hessians at Scale: Dynamics with SGD Training and Sample Size. arXiv:1811.07062 [cs, stat].
  77. Papyan, V. (2020). Traces of Class/Cross-Class Structure Pervade Deep Learning Spectra. arXiv:2008.11865 [cs, stat].
  78. The Effect of Network Width on Stochastic Gradient Descent and Generalization: an Empirical Study. In Proceedings of the 36th International Conference on Machine Learning, pages 5042–5051. PMLR. ISSN: 2640-3498.
  79. Implicit Bias of SGD for Diagonal Linear Networks: a Provable Benefit of Stochasticity. arXiv:2106.09524 [cs]. arXiv: 2106.09524.
  80. Non-attracting Regions of Local Minima in Deep and Wide Neural Networks. arXiv:1812.06486 [cs, stat].
  81. Parallel stochastic gradient algorithms for large-scale matrix completion. Math. Prog. Comp., 5(2):201–226.
  82. Roberts, D. A. (2021). SGD Implicitly Regularizes Generalization Error. arXiv:2104.04874 [cs, stat]. arXiv: 2104.04874.
  83. Spurious Local Minima are Common in Two-Layer ReLU Neural Networks. arXiv:1712.08968 [cs, stat].
  84. The Effects of Mild Over-parameterization on the Optimization Landscape of Shallow ReLU Neural Networks. arXiv:2006.01005 [cs, stat].
  85. Eigenvalues of the Hessian in Deep Learning: Singularity and Beyond.
  86. Measuring the effects of data parallelism on neural network training. arXiv preprint arXiv:1811.03600.
  87. On the Origin of Implicit Regularization in Stochastic Gradient Descent. arXiv:2101.12176 [cs, stat]. arXiv: 2101.12176.
  88. The Implicit Bias of Gradient Descent on Separable Data. page 57.
  89. Sun, R.-Y. (2020). Optimization for Deep Learning: An Overview. Journal of the Operations Research Society of China, 8(2):249–294.
  90. On the interplay between noise and curvature and its effect on optimization and generalization. In Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, pages 3503–3513. PMLR. ISSN: 2640-3498.
  91. Implicit Regularization in ReLU Networks with the Square Loss. In Proceedings of Thirty Fourth Conference on Learning Theory, pages 4224–4258. PMLR. ISSN: 2640-3498.
  92. Spurious Valleys in Two-layer Neural Network Optimization Landscapes. arXiv:1802.06384 [cs, math, stat]. arXiv: 1802.06384.
  93. Interplay between optimization and generalization of stochastic gradient descent with covariance noise. arXiv preprint arXiv:1902.08234.
  94. Wilkinson, J. H. (1960). Error analysis of floating-point computation. Numerische Mathematik, 2(1):319–340.
  95. Kernel and Rich Regimes in Overparametrized Models. In Proceedings of Thirty Third Conference on Learning Theory, pages 3635–3673. PMLR. ISSN: 2640-3498.
  96. Yaida, S. (2018). Fluctuation-dissipation relations for stochastic gradient descent. arXiv preprint arXiv:1810.00004.
  97. Understanding deep learning requires rethinking generalization. arXiv:1611.03530 [cs].
  98. Understanding Deep Learning (Still) Requires Rethinking Generalization | March 2021 | Communications of the ACM.
  99. Embedding Principle of Loss Landscape of Deep Neural Networks. arXiv:2105.14573 [cs, stat].
Citations (3)
View on Semantic Scholar

Summary

  • The paper reveals that SGD without replacement converges faster and exhibits lower variance than SGD with replacement and noised GD.
  • It demonstrates that an implicit regularizer penalizes gradient covariance, influencing the Hessian's spectrum towards sparsity.
  • The study shows SGD without replacement escapes saddle points rapidly, favoring flatter minima that likely generalize well.

Core Theoretical Result

This paper presents a theoretical paper of Stochastic Gradient Descent (SGD) without replacement, also known as random reshuffling, which is a commonly used variant in the optimization of large-scale neural networks. The central theoretical result posits that SGD without replacement diverges significantly from SGD with replacement and from gradient descent (GD) with noise injection. The divergence is evident in both the direction and the pace at which they traverse flat regions of the loss landscape. While SGD with replacement and noised GD move towards the lowest norm solution, SGD without replacement converges faster and with lower variance.

The paper establishes that, in expectation, SGD without replacement can escape saddle points substantially quicker than its counterparts. It introduces an implicit regularizer that penalizes the covariance of the gradients, which shows that SGD without replacement traverses through regions with lower gradient noise variance.

Implicit Regularization

The work explores the phenomenon known as implicit regularization, demonstrating that SGD without replacement adds a regularizer that penalizes the weighted trace of the noise covariance, which is related to the Fisher Matrix and the Hessian. This regularization encourages sparsity in the Hessian's spectrum and aligns with empirical observations from previous studies, suggesting that while SGD converges to almost-global minima, it also shapes the Hessian's spectrum during training.

Moreover, it is suggested that SGD without replacement regularizes along the eigenspaces corresponding to small and negative Hessian eigenvalues, which are associated with controlling model overfitting and effective complexity. This leads to the model fitting better and finding flatter minima that likely generalize well.

Additional Insights

The paper explores why SGD without replacement does not train at the edge of stability contrary to what is often noticed with GD. The authors provide evidence indicating the presence of a phase transition when the largest eigenvalue of the Hessian exceeds a threshold, causing the expected trajectory of SGD without replacement to deviate from that of GD.

In comparing SGD with and without replacement, the paper reveals qualitative differences in how they navigate the loss landscape. SGD without replacement moves through regions where the loss is nearly constant markedly faster and exhibits smaller oscillations due to its lower variance and the noise penalization effect inherent to the trajectory.

Concluding Remarks

The findings indicate that SGD without replacement offers a bias toward areas with lower variance, differing fundamentally from SGD with replacement whose behavior may be attributed to noise diffusion. These insights contribute to a better understanding of the nature of SGD without replacement, its regularization effects, and its practical implications for training neural networks.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown