Anytime Acceleration of Gradient Descent

• Anytime Acceleration of Gradient Descent is a framework for designing non-adaptive stepsize schedules that uniformly boost last-iterate convergence without prior stopping time knowledge in smooth convex settings.
• It leverages recursively concatenated silver blocks and strategically placed long steps to manage overshoots while ensuring improved convergence rates.
• The analysis establishes tight upper and lower bounds on performance and outlines open directions for relaxing positivity or adapting methods beyond standard gradient descent.

Anytime acceleration of gradient descent denotes the search for stepsize schedules that improve the classical last-iterate convergence behavior of vanilla gradient descent without requiring prior knowledge of the stopping time. In the smooth convex setting, the central requirement is that a single predetermined schedule must work uniformly for all iteration counts $n$. This distinguishes anytime acceleration from known-horizon acceleration, where the schedule may be tuned to a prescribed terminal time and can exploit aggressive long steps placed near that horizon. Recent work established that anytime stepsize-only acceleration is possible for smooth convex minimization, but also that it is fundamentally limited: for positive, non-adaptive stepsizes, the best known upper bounds are separated from new impossibility results, and the gap is now concentrated in a narrow range for function-value minimization, while the squared-gradient-norm case is essentially settled (Zhang et al., 2024, Tsai et al., 2 Jul 2026).

1. Formal problem and notion of “anytime”

In the core smooth convex model, one considers minimizing an $L$-smooth convex function $f:\mathbb{R}^d\to\mathbb{R}$ with nonempty minimizer set $X_f^\star$, under the gradient descent recursion

$x_{k+1}=x_k-\alpha_k \nabla f(x_k),$

with $\alpha_k>0$. The recent lower-bound analysis normalizes $L=1$ without loss of generality and studies non-adaptive schedules $\alpha=(\alpha_k)_{k\in\mathbb{N}}$ that are fixed in advance, independent of the iterates, gradients, and horizon $n$ (Tsai et al., 2 Jul 2026).

Two worst-case performance measures are standard in this literature. For function value minimization,

$$R_n(\alpha)=\sup_{d,f,x^\star,x_1}\frac{f(x_{n+1})-f(x^\star)}{\tfrac12\|x_1-x^\star\|^2}.$$

For squared gradient norm minimization,

$L$0

An anytime rate bound $L$1 means that a single schedule $L$2 satisfies the bound uniformly for all $L$3. In contrast, non-anytime or known-horizon guarantees allow $L$4 to depend on the terminal iteration count. In this terminology, “anytime acceleration” means achieving $L$5 behavior for $L$6 or $L$7 with one horizon-free schedule (Tsai et al., 2 Jul 2026).

This last-iterate emphasis is essential. The COLT 2024 open problem isolated the gap between schedules that accelerate at selected horizons and schedules that guarantee improved behavior at every stopping time. The distinction is not merely terminological: known accelerated schedules can exhibit large intermediate spikes and therefore fail to provide a decaying last-iterate bound uniformly over all $L$8 (Kornowski et al., 2024).

2. Known-horizon acceleration and its mechanisms

The modern theory of stepsize-only acceleration began with the observation that gradient descent can outperform its textbook rate through carefully orchestrated long steps. The “silver” stepsize program showed that, for smooth convex optimization, a non-monotone, fractal-like schedule based on the silver ratio $L$9 can accelerate gradient descent without momentum (Altschuler et al., 2023).

In the known-horizon regime, the best reported rate for function value and squared gradient norm is

$f:\mathbb{R}^d\to\mathbb{R}$0

attained by non-anytime “silver” schedules that use stepsizes exceeding $f:\mathbb{R}^d\to\mathbb{R}$1, thereby violating the per-step descent property, and that require knowledge of $f:\mathbb{R}^d\to\mathbb{R}$2 in advance (Tsai et al., 2 Jul 2026). Related long-step constructions also proved a first big-$f:\mathbb{R}^d\to\mathbb{R}$3 improvement beyond $f:\mathbb{R}^d\to\mathbb{R}$4, obtaining

$f:\mathbb{R}^d\to\mathbb{R}$5

for smooth convex minimization through a nonconstant, nonperiodic schedule with occasional long steps (Grimmer et al., 2023).

The key mechanism is multi-step rather than per-step descent. Large steps can transiently increase the function value or gradient norm, yet still improve the final-iterate guarantee when they are positioned so that subsequent steps compensate for the overshoot. In the silver construction, the schedule is recursive, non-monotone, and approximately periodic, and its finite-horizon form is assembled through recursively split short and long steps (Altschuler et al., 2023).

This acceleration is structurally different from momentum methods. It preserves the basic update rule of gradient descent and changes only the stepsizes. However, it is also more fragile with respect to horizon uncertainty. The same feature that creates finite-horizon gains—placing aggressive steps near the stopping time—becomes an obstacle when the stopping time is unknown (Kornowski et al., 2024).

3. The anytime regime: upper bounds and explicit schedules

The central positive result in the anytime literature is that schedule-only acceleration is possible even when the stopping time is unknown. A constructive answer was given by a predetermined infinite schedule built from recursively concatenated primitive “silver” blocks with carefully chosen join steps. For smooth convex minimization, this yields the anytime last-iterate rate

$f:\mathbb{R}^d\to\mathbb{R}$6

more precisely with exponent

$f:\mathbb{R}^d\to\mathbb{R}$7

for all stopping times $f:\mathbb{R}^d\to\mathbb{R}$8 (Zhang et al., 2024).

The construction is based on primitive sequences satisfying a potential inequality of the form

$f:\mathbb{R}^d\to\mathbb{R}$9

and on a concatenation rule in which a join step

$X_f^\star$0

is inserted between two primitive blocks. Repeating silver blocks of increasing order sufficiently many times and controlling the join points produces an infinite horizon-free schedule with $X_f^\star$1 growth large enough to beat $X_f^\star$2, while separate intra-block estimates prevent uncontrolled oscillations between join points (Zhang et al., 2024).

In the strongly convex setting, a periodic variant of the same schedule gives

$X_f^\star$3

for $X_f^\star$4, improving on the classical gradient descent dependence $X_f^\star$5 while remaining weaker than momentum-based $X_f^\star$6 acceleration (Zhang et al., 2024).

The current upper-bound picture under positive, non-adaptive stepsizes is summarized below.

Setting	Best known anytime upper bound	Status
Smooth convex, function value	$X_f^\star$7	gap remains
Smooth convex, squared gradient norm	$X_f^\star$8	later shown tight under positive stepsizes
Smooth strongly convex	$X_f^\star$9	improved over classical GD

The open-problem formulation that preceded these constructions emphasized that best-iterate guarantees are insufficient: the challenge is uniform last-iterate control at every $x_{k+1}=x_k-\alpha_k \nabla f(x_k),$0. The affirmative result $x_{k+1}=x_k-\alpha_k \nabla f(x_k),$1 therefore resolved the specific question of whether $x_{k+1}=x_k-\alpha_k \nabla f(x_k),$2 anytime acceleration is possible for vanilla gradient descent via stepsizes alone (Kornowski et al., 2024, Zhang et al., 2024).

4. Obstacles, overshoot, and lower bounds

A complementary line of work established that anytime acceleration cannot simply reuse known-horizon long-step ideas. One result showed that any schedule achieving a uniform $x_{k+1}=x_k-\alpha_k \nabla f(x_k),$3 improvement in the smooth convex class must satisfy

$x_{k+1}=x_k-\alpha_k \nabla f(x_k),$4

so unbounded steps are necessary for anytime acceleration beyond the classical $x_{k+1}=x_k-\alpha_k \nabla f(x_k),$5 scale (Kornowski et al., 2024). The same paper also proved an overshoot lower bound: whenever a step $x_{k+1}=x_k-\alpha_k \nabla f(x_k),$6 is sufficiently large relative to the accumulated earlier steps, there exists an $x_{k+1}=x_k-\alpha_k \nabla f(x_k),$7-smooth convex function such that

$x_{k+1}=x_k-\alpha_k \nabla f(x_k),$8

This demonstrates that long-step acceleration is inherently linked to large spikes in last-iterate error and explains why the standard silver schedule cannot have any decaying anytime last-iterate bound (Kornowski et al., 2024).

The 2026 lower-bound work sharpened this picture substantially. For all positive, non-adaptive schedules $x_{k+1}=x_k-\alpha_k \nabla f(x_k),$9, it proves:

• Function-value barrier:

$\alpha_k>0$0

• Gradient-norm barrier:

$\alpha_k>0$1

These are impossibility statements for anytime acceleration under positive stepsizes. They do not assert $\alpha_k>0$2 or $\alpha_k>0$3 pointwise for every $\alpha_k>0$4, but they exclude uniformly faster asymptotic decay across all $\alpha_k>0$5 (Tsai et al., 2 Jul 2026).

The function-value result narrows the admissible exponent to the interval between the current upper bound $\alpha_k>0$6 and the lower-bound barrier $\alpha_k>0$7. For squared gradient norm, the lower bound matches the known anytime upper bound $\alpha_k>0$8, establishing a genuine separation between the anytime and known-horizon regimes in that metric (Tsai et al., 2 Jul 2026).

5. Proof techniques and adversarial constructions

The lower-bound analysis combines two adversarial families. The first is quadratic. For the one-dimensional quadratic

$\alpha_k>0$9

the trajectory yields

$L=1$0

where

$L=1$1

A fundamental sum bound states

$L=1$2

In particular, if $L=1$3, then $L=1$4, so large steps are unavoidable for acceleration (Tsai et al., 2 Jul 2026).

The quadratic analysis also limits how often large steps can occur. For

$L=1$5

the bound

$L=1$6

shows that super-large steps cannot be frequent if the worst-case quadratic performance is to remain small (Tsai et al., 2 Jul 2026).

The second adversarial family is asymmetric Huber functions, piecewise linear-quadratic objectives designed so that a single step $L=1$7 causes an overshoot inside the quadratic region that later linear segments cannot repair. For suitable $L=1$8,

$L=1$9

This yields the bounds

$\alpha=(\alpha_k)_{k\in\mathbb{N}}$0

and

$\alpha=(\alpha_k)_{k\in\mathbb{N}}$1

Unlike the quadratic lower bounds, these inequalities are permutation-sensitive: they depend on where the large step occurs, not only on the multiset of stepsizes (Tsai et al., 2 Jul 2026).

The function-value lower bound is obtained by combining necessity of large steps from $\alpha=(\alpha_k)_{k\in\mathbb{N}}$2, upper bounds on the number and cumulative size of such steps, and the Huber per-step overshoot constraints. The gradient-norm lower bound follows by assuming $\alpha=(\alpha_k)_{k\in\mathbb{N}}$3, deriving $\alpha=(\alpha_k)_{k\in\mathbb{N}}$4 from the quadratic sum bound, and then contradicting this by summing the Huber-derived per-step constraints (Tsai et al., 2 Jul 2026).

6. Relation to broader acceleration paradigms

Anytime acceleration of gradient descent occupies a narrow space between classical constant-step gradient descent and richer first-order methods. Classical oracle complexity gives an $\alpha=(\alpha_k)_{k\in\mathbb{N}}$5 lower bound for general first-order methods in smooth convex optimization, and accelerated methods such as Nesterov’s attain $\alpha=(\alpha_k)_{k\in\mathbb{N}}$6 rates for function value and squared gradient norm. These results, however, are not directly informative for positive-stepsize gradient descent, whose best known anytime and non-anytime rates are substantially slower (Tsai et al., 2 Jul 2026).

Within stepsize-only methods, the silver program also extends beyond unconstrained smooth convex minimization. Proximal and projected gradient descent can be accelerated by silver stepsizes with the same asymptotic rate

$\alpha=(\alpha_k)_{k\in\mathbb{N}}$7

in the convex setting and

$\alpha=(\alpha_k)_{k\in\mathbb{N}}$8

under strong convexity, while retaining the horizon-free, parameter-free character of the schedule apart from normalization by $\alpha=(\alpha_k)_{k\in\mathbb{N}}$9 (Bok et al., 2024). This suggests that the recursive gluing and multi-step hedging ideas are not specific to plain gradient descent.

At the same time, the 2026 lower bounds apply specifically to positive stepsizes. This leaves open the possibility that negative stepsizes, averaging, momentum, or more advanced adaptive rules might evade the barriers proved for positive non-adaptive schedules (Tsai et al., 2 Jul 2026). The fixed-point literature and Anderson-acceleration literature provide related but distinct notions of “anytime” improvement, typically through residual-based safeguarding or restart mechanisms rather than horizon-free scalar schedules (Jung, 2017, Ouyang et al., 2022).

There are also special problem classes where much faster anytime behavior is possible. For separable logistic regression under a margin condition, a non-adaptive increasing stepsize schedule yields monotone, stable loss decrease with stretched-exponential decay

$n$0

and stochastic gradient descent admits an anytime block-adaptive variant with exponential-type hitting-time guarantees (Kale et al., 21 Feb 2026). This does not contradict the smooth-convex lower bounds, because it exploits structure absent from the general convex class.

7. Current status and open directions

The present landscape is sharply structured. For smooth convex optimization with positive, non-adaptive stepsizes, anytime acceleration of function values is known to be possible at rate $n$1, but impossible faster than $n$2. For squared gradient norms, the picture is tight: $n$3 is achievable and no $n$4 anytime rate is possible (Zhang et al., 2024, Tsai et al., 2 Jul 2026).

Three open directions dominate the field. The first is closing the function-value exponent gap between $n$5 and $n$6. The second is understanding whether relaxing positivity or non-adaptivity changes the achievable anytime rates. The third is extending lower-bound technology beyond the current metrics and beyond plain gradient descent, particularly toward proximal, projected, or stochastic variants (Tsai et al., 2 Jul 2026, Bok et al., 2024).

A broader conceptual conclusion has nevertheless emerged. Horizon-free schedule design is not a minor modification of known-horizon acceleration; it is a distinct optimization problem with its own combinatorial and geometric constraints. The interplay between large-step rarity, cumulative stepsize growth, and local overshoot adversaries now appears to be the defining structure of anytime acceleration for gradient descent (Kornowski et al., 2024, Tsai et al., 2 Jul 2026).