---
title: Polyak-Style Adaptive Steps
url: https://www.emergentmind.com/topics/polyak-style-adaptive-step-sizes
type: topic
---

# Polyak-Style Adaptive Steps

Polyak-style adaptive step sizes are a family of optimization rules that set the gradient (or subgradient) step size using local function values and gradient norms, with the goal of adapting rapidly and robustly to the local geometry of the objective function. Unlike fixed step size schedules, Polyak-style steps are determined at each iteration by the current optimality gap and the magnitude of the gradient, promoting parameter-free and curvature-aware convergence across a broad spectrum of deterministic and stochastic optimization settings, including high-dimensional M-estimation, stochastic first-order and Newton-type methods, constrained and nonsmooth problems, and algorithms with momentum.

## 1. Classical Polyak Step Size and Its Limitations

The original Polyak step size for unconstrained minimization of a differentiable (possibly convex, possibly nonsmooth) function $f\colon\mathbb{R}^d\to\mathbb{R}$ with known minimum $f^* = \min_x f(x)$ is defined as
\[
\eta_k = \frac{f(x_k) - f^*}{\|\nabla f(x_k)\|^2}
\]
The rationale is to maximize the one-step descent achievable, under $L$-smoothness, without explicit knowledge of curvature constants such as $L$ (the Lipschitz constant of $\nabla f$). This step size can be derived by maximizing the descent in the $L$-smooth inequality for $f(x_{k+1})$ as a function of $\eta_k$.

While this rule provides aggressive, data-dependent step sizes that outperform conservative fixed-step sizes, it is not universally robust:
- In the stochastic setting, the step size may require knowledge of per-instance function minima $f_i^* = \min_x f_i(x)$, which are typically unavailable in regularized or non-interpolated regimes.
- In high-dimensional settings, the global smoothness constant $L$ can scale badly with $d$, causing Polyak’s rule to “see” artificially high gradient magnitudes and resulting in unnecessarily conservative steps—even when the problem is well-conditioned in the subspace of interest [2509.09802].
- For constrained, non-smooth, or distributed Newton methods, additional adaptations of the Polyak rule are necessary to maintain robustness and computational efficiency.

## 2. Polyak-Style Rules in High-Dimensional M-Estimation

In high-dimensional M-estimation, e.g., sparse regression or low-rank matrix recovery, the ambient dimension $d$ can greatly exceed the sample size $n$. The global Lipschitz constant of $\nabla f$ typically diverges with $d$, and using Polyak’s rule based on $\|\nabla f(x)\|^2$ leads to vanishingly small step sizes as $d \to \infty$.

The central innovation of "Sparse Polyak" is the use of restricted Lipschitz smoothness:
\[
f(\theta_1) - f(\theta_2) - \langle \nabla f(\theta_2), \theta_1-\theta_2\rangle \leq \frac{L}{2}\|\theta_1-\theta_2\|^2 + \frac{\tau}{2}\|\theta_1-\theta_2\|_1^2
\]
The effective smoothness in $s$-sparse directions is then $\bar L = L + 3\tau s$, which can remain bounded as $d\to\infty$ if $s\ll d$ and $n \sim s \log d$.

The Sparse Polyak step modifies the Polyak numerator/denominator to use only the sparse coordinates:
\[
\gamma_t = \frac{f(\theta_t)-\widehat f}{5\,\|\text{HT}_s(\nabla f(\theta_t))\|^2}
\]
where $\text{HT}_s(\cdot)$ denotes hard-thresholding to the top $s$ entries. Empirically and theoretically, this yields convergence rates and statistical precision that do not degrade with the ambient dimension [2509.09802].

**Summary Table: Step Size Comparison**

| Rule              | Denominator term         | Effective Smoothness ("L")  | High-$d$ Scaling    |
|-------------------|-------------------------|-----------------------------|---------------------|
| Classical Polyak  | $\|\nabla f(x)\|^2$     | Global $L$                  | $\propto d$         |
| Sparse Polyak     | $\|\mathrm{HT}_s(\nabla f(x))\|^2$ | Restricted $\bar L$         | Bounded in $d$      |

## 3. Momentum and Polyak-Style Step Sizes

Polyak-adaptive step sizes have been extended and generalized to first-order methods with momentum, including Heavy-ball and Nesterov-type schemes:

- The generalized Polyak step size for momentum algorithms is given as
\[
\eta_k^G = \frac{M_k + \gamma C_k}{\|d_k\|^2}
\]
where $d_k$ is the search direction (gradient plus momentum correction), $M_k$ a lower bound on $\langle d_k,x_k-x^*\rangle$, $C_k=\langle d_k, x_k-x_{k-1}\rangle$, and $\gamma$ is the momentum parameter [2305.12939].

- For stochastic settings and stochastic heavy-ball (SHB) methods, multiple Polyak-type variants are formulated:
    - **MomSPS$_{\max}$**: polyak step with a clipped maximum, converges at $O(1/T)$ up to a neighborhood, with exact convergence when interpolation holds.
    - **MomDecSPS** and **MomAdaSPS**: decaying and adapted Polyak steps, achieving $O(1/\sqrt{T})$ convergence to the minimizer without tuning or interpolation assumptions [2406.04142].

These Polyak-style momentum schemes are substantially less sensitive to hyperparameters than classical momentum, and linear or sublinear rates are established for convex and smooth problems.

## 4. Extensions: Stochastic, Constrained, and Proximal Polyak Rules

### Stochastic Polyak Variants

In stochastic optimization, precise knowledge of $f^*$ or $f_i^*$ is rarely available. Multiple workarounds exist:
- **SPS with lower bounds**: Use any $\ell_i^* \leq f_i^*$ (often $\ell_i^*=0$) in the denominator, yielding convergence to a ball of radius $O(\sigma^2)$ where $\sigma^2 = \mathbb{E}[f_i(x^*)-\ell_i^*]$ [2205.04583].
- **DecSPS**: A decaying variant with denominator scaling as $c_k$ (e.g., $c_k=\sqrt{k+1}$), provably converging to the exact minimizer in $O(1/\sqrt{K})$ steps without boundedness constraints on the domain [2205.04583].

### Slack and Safeguarded Variants

Non-interpolation and non-smooth settings present stability challenges due to small or vanishing subgradients. Remedies include:
- **SPS with slack**: Track a global slack $s_k$, adjusting the interpolation equation to be $\ell_{i_k}(x_k) + g_k^\top (x-x_k) \leq s_k$, and update $s_{k+1}$ with a relaxation term to cap step sizes and prevent explosion or collapse [2202.12328].
- **Safeguarded SPS (SPS$_{\text{safe}}$)**: Use $1/\max\{\|g_k\|^2, M\}$ as denominator, with a user-provided floor $M>0$, to prevent instability when subgradients vanish. This yields robust $O(1/\sqrt{T})$ convergence in nonsmooth convex and deep neural net settings [2512.02342].

### Proximal Polyak Steps

Regularized empirical risk minimization can be handled via ProxSPS, in which a truncated linear model of the loss with an exact regularizer yields an update:
\[
x^{k+1} = \mathrm{prox}_{\alpha_k \phi}\left(x^k - \tau_k^+ \nabla f(x^k; S_k)\right)
\]
where $\tau_k^+$ is derived to minimally reach the truncated linearized lower bound, and only a lower bound on the loss is required (not the composite objective) [2301.04935].

### Constrained and Distributed Polyak Updates

- **Constrained optimization**: The Polyak minorant method (PMM) generalizes subgradients to arbitrary convex minorants of objectives and constraints, performing a projection onto the intersection of sublevel sets to define the next iterate [2310.07922].
- **Distributed inexact Newton**: DINAS uses a Polyak-style step size for inexact Newton directions, with adaptive $\alpha_k$ ensuring sufficient reduction in $g^k$ and permitting local superlinear/quadratic convergence, with minimal communication overhead [2305.13985].

## 5. Convergence Rates and Universality

The convergence properties of Polyak-style adaptive step sizes are extensively characterized:
- **Tightness**: For deterministic PolyakGD, both the $O((1-1/\kappa)^K)$ rate for $L$-smooth strongly convex and $O(1/K)$ for $L$-smooth convex are unimprovable, as established via explicit construction [2512.06231].
- **Universality**: Polyak step sizes are adaptive to Hölder continuity and Hölder growth conditions, achieving linear or sublinear rates, e.g.,
    - For $(\nu, L_\nu)$-Hölder smooth and $(r,\rho_r)$-growth, $O((1-c)^k)$ convergence for $r=\nu+1$; $O(k^{-r(\nu+1)/[2(r-\nu-1)]})$ for $r>\nu+1$.
- **Stochastic contraction**: Under interpolation, SPS and variants can achieve deterministic rates; more generally, $O(1/\sqrt{K})$ rates to a noise floor [2205.04583, 2512.02342, 2308.06058].
- **Last-iterate and momentum variants**: Adaptive Polyak with momentum attains the optimal $O(1/\sqrt{N})$ rate, matching lower bounds for last-iterate subgradient methods [2407.15195].

## 6. Comparative Properties and Practical Considerations

| Variant/Class      | Key Feature                         | Convergence             | Extra Inputs                | Applicability                                 |
|--------------------|-------------------------------------|-------------------------|-----------------------------|-----------------------------------------------|
| Classical Polyak   | Instant curvature adaptation        | $O((1-1/\kappa)^K)$/$O(1/K)$| $f^*$                  | Deterministic, smooth, unconstrained          |
| Sparse Polyak      | Restricted smoothness estimation    | $O(\log(1/\epsilon))$ to optimal radius | $s$, $\widehat f$ | High-dimensional sparse/structured estimation |
| SPS$_{\max}$/SPS$_\ell$| Handles stochastic and non-interpolated settings | $O(1/\sqrt K)$ to $\sigma$-ball | Lower bound $\ell_i^*$         | ERM with regularization                      |
| DecSPS             | Decaying denominator, exact minimizer| $O(1/\sqrt{K})$          | Learning rate $\gamma_b$     | General convex, no domain bound required      |
| SPS with slack     | Global step control                 | $O(1/\sqrt K)$           | Slack variable              | Non-interpolated, non-smooth                  |
| SPS$_\text{safe}$  | Runaway step-size prevention        | $O(1/\sqrt{T})$          | Floor $M$                   | Nonsmooth, stochastic, deep nets              |
| ProxSPS            | Handles regularizers prox-friendly  | $O(1/\sqrt{K})$, $O(\frac{\ln K}{K})$ (SC) | Lower bound on loss        | Regularized, possibly composite objectives    |
| GPSS (momentum)    | Momentum robust to $\beta$ choice   | $O(1/\kappa)$           | $f^*$, or lower bound        | Deterministic/stochastic, with momentum       |
| MomAdaSPS          | Adaptive $\beta$, parameter-free    | $O(1/\sqrt{T})$/$O(1/T)$| None                        | Large-scale stochastic, with momentum         |
| PMM                | Constraint-aware, minorant-based    | Linear/sublinear         | $f^*$ (or lower bound)      | General convex constrained                   |
| DINAS/Polyak Newton| Distributed, inexact Newton         | Linear/superlinear       | Forcing parameter $\gamma_0$| Distributed strongly-convex optimization      |

Practical guidance emphasizes choosing appropriate lower bounds, safeguard thresholds, and, where necessary, proxy minorants or curvature-aware projections. For high-dimensional, structured or overparameterized settings (e.g., sparse regression, deep networks), replacing global smoothness with subspace-restricted analogues is critical for scaling performance [2509.09802].

## 7. Empirical Evidence and Application Domains

Extensive experiments support Polyak-style rules:
- In high-dimensional regression, Sparse Polyak achieves iteration counts that are invariant to ambient dimension, outperforming both fixed and classical Polyak steps [2509.09802].
- On deep learning tasks (CIFAR-10/100, ResNet/DenseNet), momentum-augmented Polyak step sizes (e.g., MomSPS$_{\max}$, ALR-SMAG) yield faster training dynamics and competitive or superior generalization relative to Adam, SGD+momentum, and other adaptive-momentum methods [2406.04142, 2305.12939].
- In non-interpolated or regularized regimes, ProxSPS and SPS-slack variants demonstrate improved stability and ease of tuning compared to vanilla SPS or decaying SGD rules [2301.04935, 2202.12328].
- Constrained, distributed, and Newton-type problems benefit from Polyak-style rules that exploit local information to ensure major convergence rate improvements without global parameter tuning [2310.07922, 2305.13985].

Collectively, Polyak-style adaptive step sizes constitute a unifying and universality-enabling framework for optimization across deterministic, stochastic, high-dimensional, momentum-accelerated, and constraint-rich problems.

Source: https://www.emergentmind.com/topics/polyak-style-adaptive-step-sizes