Lower Bounds for Anytime Acceleration of Gradient Descent

Abstract: Recent work suggests that the convergence rate of gradient descent (GD) in smooth convex optimization can be significantly improved by employing large stepsizes that may violate the descent property. In particular, if the total number of iterations $n$ is given, an $O(n^{-1.271})$ convergence rate can be achieved for both function value and squared gradient norm minimization. On the other hand, in the setting of anytime convergence, where $n$ is not known in advance, the best known rates of GD are much slower: $O(n^{-1.119})$ for function value minimization and $O(n^{-1})$ for squared gradient norm minimization. It remains open whether any of these upper bounds can be improved, as they are far from the classical $Ω(n^{-2})$ lower bound for any first-order method. In this work, we establish two lower bounds on the anytime convergence of GD. We show that no positive stepsize schedule can achieve an $o(n^{-1.334})$ anytime rate for function value minimization, nor an $o(n^{-1})$ anytime rate for squared gradient norm minimization. The key ingredients of our analysis are novel upper bounds on the number and the magnitude of large stepsizes, derived by analyzing GD on quadratic functions and variants of Huber functions. Our work provides the first lower bounds for the COLT 2024 open problem posed by Kornowski and Shamir regarding the optimal anytime convergence rates of GD.

• The paper demonstrates that no positive stepsize schedule can achieve a function value convergence rate better than o(n^(-4/3)), marking a key lower bound for anytime GD.
• It employs novel analysis on quadratic objectives and Chebyshev systems to limit the frequency of large stepsizes, thus constraining accelerated convergence.
• The results highlight a fundamental gap between anytime and non-anytime settings, emphasizing the intrinsic limits of non-adaptive stepsize strategies in convex optimization.

Lower Bounds for Anytime Acceleration of Gradient Descent: An Expert Synthesis

Problem Context and Motivation

The paper "Lower Bounds for Anytime Acceleration of Gradient Descent" (2607.02053) addresses a central question in first-order optimization: the potential for accelerated convergence rates of Gradient Descent (GD) under arbitrary positive, non-adaptive stepsize schedules in the anytime regime. Recent works have demonstrated that GD, contrary to longstanding belief, can achieve accelerated (i.e., $o(n^{-1})$) rates via carefully chosen large stepsizes when the total number of iterations $n$ is known. However, the optimal rates attainable without advance knowledge of $n$—the "anytime" setting—have remained elusive, especially given the gap between known upper and lower bounds for function value and gradient norm minimization. This paper resolves core aspects of this issue by establishing fundamental lower bounds for the anytime setting.

Preliminaries and Definitions

Consider minimizing a smooth convex function $f$ via GD with an initial point $x_1 \in \mathbb{R}^d$:

$$x_{k+1} = x_k - \eta_k \nabla f(x_k),\quad \forall k\in \mathbb{N}$$

where the stepsize sequence $(\eta_k)$ is positive, non-adaptive, and not explicitly bounded. Convergence rates are quantified in two regimes:

• $R_n$ (Function Value): Worst-case ratio of terminal function value gap to squared initial distance.
• $G_n$ (Squared Gradient Norm): Worst-case ratio of terminal squared gradient norm to initial function value gap.

The setting distinguishes between non-anytime (stepsizes may depend on a known horizon $n$) and anytime (stepsizes fixed in advance, must simultaneously guarantee the rate for all $n$0). The latter is strictly more restrictive.

Summary of Main Results

The main theoretical contributions are two lower bounds for anytime GD:

• Function Value: No positive stepsize schedule achieves $n$1.
• Squared Gradient Norm: No positive stepsize schedule achieves $n$2.

These results rule out the possibility of closing the gap to the $n$3 rates of optimal first-order methods (e.g., Nesterov's accelerated method), providing the first unconditional anytime lower bounds for GD. They directly address an open problem posed by Kornowski and Shamir.

Technical Contributions

Novel Upper Bounds on Large Stepsizes

A central technical advance is the upper bound on the number and cumulative impact of large stepsizes in any convergent schedule. Analyzing GD on parameterized quadratic objectives, the authors establish that only $n$4 stepsizes can exceed a given threshold, tightly constraining how often GD can take large, potentially non-monotonic steps without violating worst-case convergence. This follows via a reduction to extremal polynomial properties, leveraging the structure of Chebyshev systems and Weierstrass-type inequalities.

Additionally, for schedules where a single stepsize becomes anomalously large, the authors introduce asymmetric Huber functions to demonstrate that such excursions cause unavoidable overshoot and slow convergence, unless compensated by sufficiently many small steps. The analysis yields explicit tradeoffs between maximum stepsize, partial sums, and achievable rates.

Tight Lower Bounds via Constructive Analysis

• For $n$5, the combination of large sum requirements (from quadratics) and the magnitude restriction (from Huber functions) leads to the $n$6 barrier.
• For $n$7, the argument is sharper: The lower bound precisely matches the best possible rate achievable even with large, non-monotonic steps, and no schedule can break the $n$8 barrier.

Both bounds hold for univariate functions, underscoring their fundamental character.

Position Relative to Known Upper Bounds

The established lower bounds place tight restrictions on what is achievable in the anytime regime. For $n$9, the result puts the optimal rate strictly between $n$0 (new lower bound) and the $n$1 upper bound of Zhang et al. (2025). For $n$2, the upper and lower bounds match at $n$3.

This delineates a separation between anytime and non-anytime settings: in the latter, $n$4 rates for both $n$5 and $n$6 are attainable, coinciding with strategies that exploit foreknowledge of the stopping time, often by concentrating large steps at late iterations. The impossibility in the anytime regime quantifies the "price of ignorance."

Implications and Future Directions

Theoretical Implications

1. Suboptimality of GD (Anytime): GD with positive stepsizes is strictly outperformed by optimal first-order methods (e.g., Nesterov acceleration) in the anytime regime for function value minimization; an $n$7 rate is unachievable.
2. Fundamental Effect of Step Scheduling: Only a limited fraction of large steps can be used before violating lower bounds, even if non-monotonicity or loss increases are permitted.
3. Separation Between $n$8 and $n$9 Rates: Unlike the non-anytime regime (where both can be accelerated), squared gradient norm minimization is uniquely resistant to anytime acceleration.

Practical Relevance

For practitioners and theorists designing optimization algorithms for convex objectives (especially in large-scale ML), these findings imply that attempts to accelerate GD by heuristically increasing stepsize while remaining anytime-safe face categorical performance limits. Methods must either exploit knowledge of $f$0, use negative or adaptive steps (outside this analysis), or shift to higher-order/auxiliary-method approaches.

Open Problems

1. Closing the Gap for $f$1 (Anytime): There remains a log-polynomial gap between lower and upper bounds. Constructing explicit schedules or instance-optimal methods that achieve the true minimax rate is open.
2. Non-anytime Lower Bounds: Extending the technique to show tighter lower bounds in the non-anytime regime (i.e., approaching $f$2) remains elusive, as even carefully permuted Chebyshev-based schedules are not ruled out by current analysis.
3. Beyond Positive, Non-adaptive Stepsizes: Lower bounds for average iterates, negative/adaptive stepsizes, or convex-concave problems are natural extensions, but require new tools.

Conclusion

This work establishes rigorous lower bounds for the anytime convergence of Gradient Descent in smooth convex optimization under positive, non-adaptive stepsize schedules. The results demarcate the intrinsic suboptimality of GD in this setting, explain the empirical limits of accelerated scheduling, and provide a baseline for future advances in both analytical techniques and algorithmic design within convex optimization theory.

---

Reference

"Lower Bounds for Anytime Acceleration of Gradient Descent" (2607.02053)