Don't be so Monotone: Relaxing Stochastic Line Search in Over-Parameterized Models (2306.12747v2)

Abstract: Recent works have shown that line search methods can speed up Stochastic Gradient Descent (SGD) and Adam in modern over-parameterized settings. However, existing line searches may take steps that are smaller than necessary since they require a monotone decrease of the (mini-)batch objective function. We explore nonmonotone line search methods to relax this condition and possibly accept larger step sizes. Despite the lack of a monotonic decrease, we prove the same fast rates of convergence as in the monotone case. Our experiments show that nonmonotone methods improve the speed of convergence and generalization properties of SGD/Adam even beyond the previous monotone line searches. We propose a POlyak NOnmonotone Stochastic (PoNoS) method, obtained by combining a nonmonotone line search with a Polyak initial step size. Furthermore, we develop a new resetting technique that in the majority of the iterations reduces the amount of backtracks to zero while still maintaining a large initial step size. To the best of our knowledge, a first runtime comparison shows that the epoch-wise advantage of line-search-based methods gets reflected in the overall computational time.

• The paper introduces the POlyak NOnmonotone Stochastic (PoNoS) method, combining a Polyak step size with nonmonotone line search to allow larger steps while retaining fast convergence.
• It proves theoretical convergence rates for strongly convex, convex, and PL-condition functions, showing linear and O(1/k) performance guarantees.
• Empirical experiments on benchmarks like MNIST and CIFAR demonstrate that the relaxed line search improves both convergence speed and generalization performance.

Overview of "Don't be so Monotone: Relaxing Stochastic Line Search in Over-Parameterized Models"

The paper presents advancements in optimizing the training of over-parameterized models by introducing a nonmonotone line search method in stochastic optimization. The authors examine this approach's applications to Stochastic Gradient Descent (SGD) and Adam, two prevalent optimization methods in deep learning. A key motivation behind this work is that traditional monotone line searches, which require a consistent decrease in the objective function, might limit the step sizes and thus the efficiency in complex landscapes typical in modern deep learning models.

Theoretical Contributions

The paper advances the theory of nonmonotone line searches, a concept traditionally associated with deterministic optimization, and adapts it for stochastic settings. A major theoretical strength of the work is the introduction of the POlyak NOnmonotone Stochastic (PoNoS) method. This method combines a Polyak-based initial step size with a nonmonotone line search, thereby permitting larger step sizes and potentially quicker convergence while still proving fast convergence rates similar to those in monotone line search scenarios.

Three main theoretical results are established:

1. For Strongly Convex Functions: The authors establish a linear convergence rate, showing that under specific conditions, nonmonotone line searches do not compromise convergence speed.
2. For Convex Functions: A $O(1/k)$ convergence rate is demonstrated, validating the method's applicability and efficiency for convex functions.
3. For Functions under the Polyak-Lojasiewicz (PL) Condition: The paper also demonstrates that the convergence results extend to this condition, which is particularly relevant for deep learning models represented by non-convex functions.

Each of these results underscores the potential of the proposed method to achieve accelerated convergence in various system states.

Practical Implications and Experimentation

The practical section of the paper provides empirical evidence supporting the theoretical claims through extensive experiments involving deep learning benchmarks (e.g., MNIST, CIFAR). The results show that the nonmonotone line search methods generally outperform their monotone counterparts in both convergence speed and generalization performance.

Significant observations include:

• Larger Step Sizes: The ability to use larger step sizes typically leads to faster convergence, particularly in the non-convex landscapes of deep neural networks.
• Reduced Computational Load: The nonmonotone techniques, when implemented effectively, reduce the backtrack overhead typically associated with line search methods, as shown in their developed resetting techniques.
• Generalization Improvements: Improved generalization performance was consistently observed, aligning with recent insights that nonmonotone behavior can be beneficial in over-parameterized models, possibly due to phenomena like the edge of stability.

Future Directions

This research opens multiple avenues for future exploration. The authors suggest potential links between nonmonotone behavior and stability phenomena in neural networks. Moreover, enhancing these methods through more sophisticated initial step size strategies, such as those linked to specific model architectures (e.g., transformers), offers fertile ground for further paper. The proposed methodology and theoretical framework also provide a blueprint for integrating nonmonotone dynamics with other optimization techniques, promising further efficiency gains in AI model training.

The paper marks a significant step towards rethinking the constraints of traditional line search methods, especially in rapidly evolving deep learning environments, suggesting a shift towards more flexible strategies that align better with the characteristics of modern AI models.