Lipschitz minimization and the Goldstein modulus (2405.12655v1)
Abstract: Goldstein's 1977 idealized iteration for minimizing a Lipschitz objective fixes a distance - the step size - and relies on a certain approximate subgradient. That "Goldstein subgradient" is the shortest convex combination of objective gradients at points within that distance of the current iterate. A recent implementable Goldstein-style algorithm allows a remarkable complexity analysis (Zhang et al. 2020), and a more sophisticated variant (Davis and Jiang, 2022) leverages typical objective geometry to force near-linear convergence. To explore such methods, we introduce a new modulus, based on Goldstein subgradients, that robustly measures the slope of a Lipschitz function. We relate near-linear convergence of Goldstein-style methods to linear growth of this modulus at minimizers. We illustrate the idea computationally with a simple heuristic for Lipschitz minimization.
- A unified analysis of descent sequences in weakly convex optimization, including convergence rates for bundle methods. SIAM Journal on Optimization, 33:89–115, 2023.
- N. Boumal. An Introduction to Optimization on Smooth Manifolds. Cambridge University Press, Cambridge, 2023.
- F.H. Clarke. Optimization and Nonsmooth Analysis. Wiley Interscience, New York, 1983.
- A gradient sampling method with complexity guarantees for lipschitz functions in high and low dimensions. In NeurIPS Proceedings, 2022.
- D. Davis and Liwei Jiang. A nearly linearly convergent first-order method for nonsmooth functions with quadratic growth. Found. Comput. Math., to appear, 2024. arXiv:2205.00064v3.
- D. Drusvyatskiy and A.S. Lewis. Optimality, identifiability, and sensitivity. Math. Program., 147:467–498, 2014.
- D. Drusvyatskiy and A.S. Lewis. Error bounds, quadratic growth, and linear convergence of proximal methods. Preprint arXiv:1602.06661, 2016.
- A.A. Goldstein. Optimization of Lipschitz continuous functions. Math. Programming, 13:14–22, 1977.
- E. Hazan and S. Kale. Beyond the regret minimization barrier: optimal algorithms for stochastic strongly-convex optimization. J. Mach. Learn. Res., 15:2489–2512, 2014.
- A.D Ioffe. Variational Analysis of Regular Mappings. Springer US, 2017.
- Deterministic nonsmooth nonconvex optimization. In Proceedings of Machine Learning Research, volume 195, pages 1–28, 2023.
- On the complexity of deterministic nonsmooth and nonconvex optimization. arXiv:2209.12463, 2022.
- The cost of nonconvexity in deterministic nonsmooth optimization. Mathematics of Operations Research, doi.org/10.1287/moor.2022.0289, 2023.
- G. Kornowski and O. Shamir. On the complexity of finding small subgradients in nonsmooth optimization. arXiv:2209.10346, 2022.
- Identifiability, the KL property in metric spaces, and subgradient curves. Fourndations of Computational Mathematics, 2024. To appear.
- A.S. Lewis and S. Zhang. Partial smoothness, tilt stability, and generalized Hessians. SIAM J. Optim., 23(1):74–94, 2013.
- J. Nocedal and S.J. Wright. Numerical Optimization. Springer Series in Operations Research and Financial Engineering. Springer, New York, second edition, 2006.
- S.M. Robinson. Linear convergence of epsilon-subgradient descent methods for a class of convex functions. Math. Program., 86:41–50, 1999.
- Lai Tian and Anthony Man-Cho So. Computing Goldstein (ϵ,δ)italic-ϵ𝛿(\epsilon,\delta)( italic_ϵ , italic_δ )-stationary points of Lipschitz functions in O~(ϵ−3δ−1)~𝑂superscriptitalic-ϵ3superscript𝛿1\widetilde{O}(\epsilon^{-3}\delta^{-1})over~ start_ARG italic_O end_ARG ( italic_ϵ start_POSTSUPERSCRIPT - 3 end_POSTSUPERSCRIPT italic_δ start_POSTSUPERSCRIPT - 1 end_POSTSUPERSCRIPT ) iterations via random conic perturbation. arxiv.org/abs/2112.09002, 2021.
- Complexity of finding stationary points of nonconvex nonsmooth functions. In ICML Proceedings, 2020.
Collections
Sign up for free to add this paper to one or more collections.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.