Polyak-Style Adaptive Steps

• Polyak-style adaptive step sizes use the current optimality gap and gradient norms to dynamically adjust updates, eliminating the need for explicit curvature parameters.
• They enable parameter-free, curvature-aware convergence across deterministic, stochastic, high-dimensional, constrained, and momentum-augmented optimization scenarios.
• Extensions such as Sparse, momentum, slack, and proximal variants address the limitations of classical methods, improving stability and performance in non-smooth and distributed contexts.

Polyak-style adaptive step sizes are a family of optimization rules that set the gradient (or subgradient) step size using local function values and gradient norms, with the goal of adapting rapidly and robustly to the local geometry of the objective function. Unlike fixed step size schedules, Polyak-style steps are determined at each iteration by the current optimality gap and the magnitude of the gradient, promoting parameter-free and curvature-aware convergence across a broad spectrum of deterministic and stochastic optimization settings, including high-dimensional M-estimation, stochastic first-order and Newton-type methods, constrained and nonsmooth problems, and algorithms with momentum.

1. Classical Polyak Step Size and Its Limitations

The original Polyak step size for unconstrained minimization of a differentiable (possibly convex, possibly nonsmooth) function $f\colon\mathbb{R}^d\to\mathbb{R}$ with known minimum $f^* = \min_x f(x)$ is defined as

$\eta_k = \frac{f(x_k) - f^*}{\|\nabla f(x_k)\|^2}$

The rationale is to maximize the one-step descent achievable, under $L$-smoothness, without explicit knowledge of curvature constants such as $L$ (the Lipschitz constant of $\nabla f$). This step size can be derived by maximizing the descent in the $L$-smooth inequality for $f(x_{k+1})$ as a function of $\eta_k$.

While this rule provides aggressive, data-dependent step sizes that outperform conservative fixed-step sizes, it is not universally robust:

• In the stochastic setting, the step size may require knowledge of per-instance function minima $f_i^* = \min_x f_i(x)$, which are typically unavailable in regularized or non-interpolated regimes.
• In high-dimensional settings, the global smoothness constant $f^* = \min_x f(x)$0 can scale badly with $f^* = \min_x f(x)$1, causing Polyak’s rule to “see” artificially high gradient magnitudes and resulting in unnecessarily conservative steps—even when the problem is well-conditioned in the subspace of interest (Qiao et al., 11 Sep 2025).
• For constrained, non-smooth, or distributed Newton methods, additional adaptations of the Polyak rule are necessary to maintain robustness and computational efficiency.

2. Polyak-Style Rules in High-Dimensional M-Estimation

In high-dimensional M-estimation, e.g., sparse regression or low-rank matrix recovery, the ambient dimension $f^* = \min_x f(x)$2 can greatly exceed the sample size $f^* = \min_x f(x)$3. The global Lipschitz constant of $f^* = \min_x f(x)$4 typically diverges with $f^* = \min_x f(x)$5, and using Polyak’s rule based on $f^* = \min_x f(x)$6 leads to vanishingly small step sizes as $f^* = \min_x f(x)$7.

The central innovation of "Sparse Polyak" is the use of restricted Lipschitz smoothness: $f^* = \min_x f(x)$8 The effective smoothness in $f^* = \min_x f(x)$9-sparse directions is then $\eta_k = \frac{f(x_k) - f^*}{\|\nabla f(x_k)\|^2}$0, which can remain bounded as $\eta_k = \frac{f(x_k) - f^*}{\|\nabla f(x_k)\|^2}$1 if $\eta_k = \frac{f(x_k) - f^*}{\|\nabla f(x_k)\|^2}$2 and $\eta_k = \frac{f(x_k) - f^*}{\|\nabla f(x_k)\|^2}$3.

The Sparse Polyak step modifies the Polyak numerator/denominator to use only the sparse coordinates: $\eta_k = \frac{f(x_k) - f^*}{\|\nabla f(x_k)\|^2}$4 where $\eta_k = \frac{f(x_k) - f^*}{\|\nabla f(x_k)\|^2}$5 denotes hard-thresholding to the top $\eta_k = \frac{f(x_k) - f^*}{\|\nabla f(x_k)\|^2}$6 entries. Empirically and theoretically, this yields convergence rates and statistical precision that do not degrade with the ambient dimension (Qiao et al., 11 Sep 2025).

Summary Table: Step Size Comparison

Rule	Denominator term	Effective Smoothness ("L")	High-$\eta_k = \frac{f(x_k) - f^*}{\|\nabla f(x_k)\|^2}$7 Scaling
Classical Polyak	$\eta_k = \frac{f(x_k) - f^*}{\|\nabla f(x_k)\|^2}$8	Global $\eta_k = \frac{f(x_k) - f^*}{\|\nabla f(x_k)\|^2}$9	$L$0
Sparse Polyak	$L$1	Restricted $L$2	Bounded in $L$3

3. Momentum and Polyak-Style Step Sizes

Polyak-adaptive step sizes have been extended and generalized to first-order methods with momentum, including Heavy-ball and Nesterov-type schemes:

• The generalized Polyak step size for momentum algorithms is given as

$L$4

where $L$5 is the search direction (gradient plus momentum correction), $L$6 a lower bound on $L$7, $L$8, and $L$9 is the momentum parameter (Wang et al., 2023).

• For stochastic settings and stochastic heavy-ball (SHB) methods, multiple Polyak-type variants are formulated:
  • MomSPS$L$0: polyak step with a clipped maximum, converges at $L$1 up to a neighborhood, with exact convergence when interpolation holds.
  • MomDecSPS and MomAdaSPS: decaying and adapted Polyak steps, achieving $L$2 convergence to the minimizer without tuning or interpolation assumptions (Oikonomou et al., 2024).

These Polyak-style momentum schemes are substantially less sensitive to hyperparameters than classical momentum, and linear or sublinear rates are established for convex and smooth problems.

4. Extensions: Stochastic, Constrained, and Proximal Polyak Rules

Stochastic Polyak Variants

In stochastic optimization, precise knowledge of $L$3 or $L$4 is rarely available. Multiple workarounds exist:

• SPS with lower bounds: Use any $L$5 (often $L$6) in the denominator, yielding convergence to a ball of radius $L$7 where $L$8 (Orvieto et al., 2022).
• DecSPS: A decaying variant with denominator scaling as $L$9 (e.g., $\nabla f$0), provably converging to the exact minimizer in $\nabla f$1 steps without boundedness constraints on the domain (Orvieto et al., 2022).

Slack and Safeguarded Variants

Non-interpolation and non-smooth settings present stability challenges due to small or vanishing subgradients. Remedies include:

• SPS with slack: Track a global slack $\nabla f$2, adjusting the interpolation equation to be $\nabla f$3, and update $\nabla f$4 with a relaxation term to cap step sizes and prevent explosion or collapse (Gower et al., 2022).
• Safeguarded SPS (SPS$\nabla f$5): Use $\nabla f$6 as denominator, with a user-provided floor $\nabla f$7, to prevent instability when subgradients vanish. This yields robust $\nabla f$8 convergence in nonsmooth convex and deep neural net settings (Oikonomou et al., 2 Dec 2025).

Proximal Polyak Steps

Regularized empirical risk minimization can be handled via ProxSPS, in which a truncated linear model of the loss with an exact regularizer yields an update: $\nabla f$9 where $L$0 is derived to minimally reach the truncated linearized lower bound, and only a lower bound on the loss is required (not the composite objective) (Schaipp et al., 2023).

Constrained and Distributed Polyak Updates

• Constrained optimization: The Polyak minorant method (PMM) generalizes subgradients to arbitrary convex minorants of objectives and constraints, performing a projection onto the intersection of sublevel sets to define the next iterate (Devanathan et al., 2023).
• Distributed inexact Newton: DINAS uses a Polyak-style step size for inexact Newton directions, with adaptive $L$1 ensuring sufficient reduction in $L$2 and permitting local superlinear/quadratic convergence, with minimal communication overhead (Jakovetic et al., 2023).

5. Convergence Rates and Universality

The convergence properties of Polyak-style adaptive step sizes are extensively characterized:

• Tightness: For deterministic PolyakGD, both the $L$3 rate for $L$4-smooth strongly convex and $L$5 for $L$6-smooth convex are unimprovable, as established via explicit construction (He et al., 6 Dec 2025).
• Universality: Polyak step sizes are adaptive to Hölder continuity and Hölder growth conditions, achieving linear or sublinear rates, e.g.,
  • For $L$7-Hölder smooth and $L$8-growth, $L$9 convergence for $f(x_{k+1})$0; $f(x_{k+1})$1 for $f(x_{k+1})$2.
• Stochastic contraction: Under interpolation, SPS and variants can achieve deterministic rates; more generally, $f(x_{k+1})$3 rates to a noise floor (Orvieto et al., 2022, Oikonomou et al., 2 Dec 2025, Jiang et al., 2023).
• Last-iterate and momentum variants: Adaptive Polyak with momentum attains the optimal $f(x_{k+1})$4 rate, matching lower bounds for last-iterate subgradient methods (Zamani et al., 2024).

6. Comparative Properties and Practical Considerations

Variant/Class	Key Feature	Convergence	Extra Inputs	Applicability
Classical Polyak	Instant curvature adaptation	$f(x_{k+1})$5/$f(x_{k+1})$6	$f(x_{k+1})$7	Deterministic, smooth, unconstrained
Sparse Polyak	Restricted smoothness estimation	$f(x_{k+1})$8 to optimal radius	$f(x_{k+1})$9, $\eta_k$0	High-dimensional sparse/structured estimation
SPS$\eta_k$1/SPS$\eta_k$2	Handles stochastic and non-interpolated settings	$\eta_k$3 to $\eta_k$4-ball	Lower bound $\eta_k$5	ERM with regularization
DecSPS	Decaying denominator, exact minimizer	$\eta_k$6	Learning rate $\eta_k$7	General convex, no domain bound required
SPS with slack	Global step control	$\eta_k$8	Slack variable	Non-interpolated, non-smooth
SPS$\eta_k$9	Runaway step-size prevention	$f_i^* = \min_x f_i(x)$0	Floor $f_i^* = \min_x f_i(x)$1	Nonsmooth, stochastic, deep nets
ProxSPS	Handles regularizers prox-friendly	$f_i^* = \min_x f_i(x)$2, $f_i^* = \min_x f_i(x)$3 (SC)	Lower bound on loss	Regularized, possibly composite objectives
GPSS (momentum)	Momentum robust to $f_i^* = \min_x f_i(x)$4 choice	$f_i^* = \min_x f_i(x)$5	$f_i^* = \min_x f_i(x)$6, or lower bound	Deterministic/stochastic, with momentum
MomAdaSPS	Adaptive $f_i^* = \min_x f_i(x)$7, parameter-free	$f_i^* = \min_x f_i(x)$8/$f_i^* = \min_x f_i(x)$9	None	Large-scale stochastic, with momentum
PMM	Constraint-aware, minorant-based	Linear/sublinear	$f^* = \min_x f(x)$00 (or lower bound)	General convex constrained
DINAS/Polyak Newton	Distributed, inexact Newton	Linear/superlinear	Forcing parameter $f^* = \min_x f(x)$01	Distributed strongly-convex optimization

Practical guidance emphasizes choosing appropriate lower bounds, safeguard thresholds, and, where necessary, proxy minorants or curvature-aware projections. For high-dimensional, structured or overparameterized settings (e.g., sparse regression, deep networks), replacing global smoothness with subspace-restricted analogues is critical for scaling performance (Qiao et al., 11 Sep 2025).

7. Empirical Evidence and Application Domains

Extensive experiments support Polyak-style rules:

• In high-dimensional regression, Sparse Polyak achieves iteration counts that are invariant to ambient dimension, outperforming both fixed and classical Polyak steps (Qiao et al., 11 Sep 2025).
• On deep learning tasks (CIFAR-10/100, ResNet/DenseNet), momentum-augmented Polyak step sizes (e.g., MomSPS$f^* = \min_x f(x)$02, ALR-SMAG) yield faster training dynamics and competitive or superior generalization relative to Adam, SGD+momentum, and other adaptive-momentum methods (Oikonomou et al., 2024, Wang et al., 2023).
• In non-interpolated or regularized regimes, ProxSPS and SPS-slack variants demonstrate improved stability and ease of tuning compared to vanilla SPS or decaying SGD rules (Schaipp et al., 2023, Gower et al., 2022).
• Constrained, distributed, and Newton-type problems benefit from Polyak-style rules that exploit local information to ensure major convergence rate improvements without global parameter tuning (Devanathan et al., 2023, Jakovetic et al., 2023).

Collectively, Polyak-style adaptive step sizes constitute a unifying and universality-enabling framework for optimization across deterministic, stochastic, high-dimensional, momentum-accelerated, and constraint-rich problems.