Gradient Descent's Last Iterate is Often (slightly) Suboptimal

Abstract: We consider the well-studied setting of minimizing a convex Lipschitz function using either gradient descent (GD) or its stochastic variant (SGD), and examine the last iterate convergence. By now, it is known that standard stepsize choices lead to a last iterate convergence rate of $\log T/\sqrt{T}$ after $T$ steps. A breakthrough result of Jain et al. [2019] recovered the optimal $1/\sqrt{T}$ rate by constructing a non-standard stepsize sequence. However, this sequence requires choosing $T$ in advance, as opposed to common stepsize schedules which apply for any time horizon. Moreover, Jain et al. conjectured that without prior knowledge of $T$, no stepsize sequence can ensure the optimal error for SGD's last iterate, a claim which so far remained unproven. We prove this conjecture, and in fact show that even in the noiseless case of GD, it is impossible to avoid an excess poly-log factor in $T$ when considering an anytime last iterate guarantee. Our proof further suggests that such (slightly) suboptimal stopping times are unavoidably common.

• The paper demonstrates that any anytime stepsize schedule must suffer a poly-log overhead, establishing inherent last-iterate suboptimality in GD and SGD.
• It employs novel lower-bound techniques on both deterministic and stochastic settings to show that optimal convergence without horizon awareness is impossible.
• Implications underscore the necessity for iterate averaging or horizon-aware scheduling to achieve minimax-optimal rates in online and streaming applications.

Gradient Descent’s Last Iterate is Often (slightly) Suboptimal

Introduction and Problem Context

The paper "Gradient Descent's Last Iterate is Often (slightly) Suboptimal" (2604.13870) investigates the convergence properties of the last iterate in (sub)gradient descent (GD) and stochastic gradient descent (SGD) in convex, Lipschitz-continuous optimization. While classical theory ensures that the average of iterates can achieve the minimax-optimal rate of $1/\sqrt{T}$ after $T$ iterations using standard stepsizes, common practice and software implementations typically return the last iterate $x_T$ rather than this average. Prior theoretical results showed a corresponding last-iterate rate of only $\log T/\sqrt{T}$ under typical anytime stepsize schedules, motivating deeper investigation into whether this "poly-log" ($\log T$) overhead is intrinsic and unavoidable in settings where $T$ is not fixed in advance.

Prior Work and Motivation

Jain et al. (2019) constructed a non-standard stepsize schedule depending explicitly on the known time horizon $T$ that achieves the optimal $1/\sqrt{T}$ rate for the last iterate. However, this non-anytime approach is impractical for settings with unknown or continually increasing $T$. Jain et al. also conjectured that, in the absence of prior knowledge of $T$, no stepsize schedule can achieve the optimal $T$0 last-iterate rate in SGD—leaving open whether the log-factor gap is a methodological artifact or a fundamental limitation.

Theoretical literature surrounding last-iterate performance has offered refinements, yet most either impose strong convexity, require averaging, or do not provide anytime guarantees. Lower bounds for standard smooth and non-smooth SGD with anytime stepsizes have typically matched or fallen short—but have not directly proved—the inevitability of last-iterate suboptimality with anytime guarantees.

Main Results and Technical Contributions

The principal result proves that for GD and, as a consequence, for SGD, no anytime stepsize schedule can guarantee $T$1 last-iterate convergence for all $T$2. Formally, for all stepsize schedules independent of $T$3 (i.e., anytime), the worst-case guarantee for the last-iterate error remains at least an $T$4 rate multiplied by a poly-log factor (specifically, at least $T$5 up to constants).

This lower bound is established constructively: even in the noiseless (deterministic) case of GD—where stochasticity is absent and thus optimism for improvement might be higher—one cannot eliminate this overhead factor without designing the stepsizes with full knowledge of $T$6. This both proves and strengthens the conjecture of Jain et al., showing the limitation holds for both stochastic and deterministic variants.

Further, the analysis demonstrates that achieving the optimal rate at a sequence of predetermined stopping times (e.g., through "doubling trick" schedules) is possible, but that the vast majority (in natural density) of stopping times will be suboptimal, suffering a poly-log overhead. In other words, optimality can only be achieved on a sparse (measure zero) subsequence of iterates.

Proof Techniques

The paper combines advanced lower bound techniques from optimization theory. The key tools are:

• 1D constructions demonstrating that the stepsize at step $T$7 is an inherent lower bound for the possible last-iterate error at $T$8.
• Quadratic and worst-case convex function constructions ensuring that the sum of stepsizes cannot be too small, else excess error arises.
• A high-dimensional lower bound drawing on and extending techniques from the analysis of "max-of-linear" functions where the behavior in different coordinate axes amplifies the inability to synchronize optimality with arbitrary stopping times.

The result leverages averaging arguments to show that for most $T$9 large enough, a randomly chosen stopping time exhibits this unavoidable poly-log overhead, crystallizing the "commonality" of suboptimal stopping.

Implications and Theoretical Impact

This work crystalizes a structural property of first-order methods (GD/SGD) in convex, Lipschitz settings. It formalizes that the inability to synchronize stepsize adaptation with unpredictable horizons is a fundamental barrier to optimal last-iterate convergence, even absent noise. Practically, this validates the widespread use of iterate averaging (or, in specific cases, schedule designs tied to anticipated $x_T$0) to achieve optimality and highlights that last-iterate solution quality can be inherently fragile in anytime/streaming contexts.

Theoretically, the result raises new questions regarding:

1. Strongly Convex Extension: The lower bound in the strongly convex case, where anytime rates remain separated from optimal, is open. The refined analysis of log factors vs. polynomial factors is potentially tractable via extensions of the present techniques.
2. Smoothness: In smooth optimization, the best-known anytime bounds involve (slightly worse) polynomial factors. The dichotomy between smooth and non-smooth cases—here, provably narrowed to a logarithmic exponent—invites further structural study.
3. Density of Optimal Stopping Times: The demonstration that optimal stopping times form a zero-density set under any stepsize schedule suggests avenues for characterizing, in the sense of asymptotic density, the sparsity of optimal last-iterate events and exploring potential schedule designs that target denser optimality.
4. Algorithmic Directions: For practitioners, this indicates that in online or open-horizon learning, one should use iteration averaging or develop early stopping rules tied to computational budgets rather than relying on last-iterate output.

Numerical and Analytical Strengths

The lower bound is strong in that it holds for all convex, Lipschitz objectives without requiring smoothness, second-order curvature, or strong convexity. The gap between the lower bound ($x_T$1) and the best-known upper bound ($x_T$2) is narrow, suggesting the analysis is near-tight for worst-case instances.

Conclusion

This paper resolves a fundamental conjecture about anytime last-iterate convergence for GD and SGD, establishing that a poly-log overhead is inherent when the computation budget (or horizon $x_T$3) is not predetermined. This result further sharpens the theory/practice gap regarding output rules in first-order methods and draws clear boundaries for what is information-theoretically achievable in unconstrained, anytime optimization. The implications for the design of optimization routines further underline the necessity of averaging or horizon-aware scheduling in practical large-scale machine learning. Prospective work aims to close the remaining tightness gap and extend the analysis to broader classes, including strongly convex and smooth objectives.