--- title: Polyak Step-Size Method url: https://www.emergentmind.com/topics/polyak-step-size type: topic --- # Polyak Step-Size Method Polyak step-size is an adaptive learning-rate rule that selects the step from the current objective gap divided by a squared gradient norm. For a differentiable convex objective with known optimum value $f^\star$, the classical rule is \[ \alpha_k=\frac{f(x_k)-f^\star}{\|\nabla f(x_k)\|^2}, \qquad x_{k+1}=x_k-\alpha_k\nabla f(x_k). \] It was derived by minimizing a quadratic upper bound on the next squared distance to an optimizer. The method does not require an externally supplied smoothness constant, but its classical form requires the optimal objective value, which is generally unavailable. Subsequent work has extended the construction to stochastic optimization, momentum, proximal and mirror descent, constrained problems, preconditioned updates, high-dimensional sparse estimation, decentralized optimization, reinforcement learning, and objectives whose curvature degenerates near the optimum. ## 1. Classical rule and geometric derivation Consider the convex subgradient iteration \[ x^{k+1}=x^k-\gamma_k g^k, \qquad g^k\in\partial f(x^k). \] For an optimizer $x^\star$ and optimal value $f^\star=f(x^\star)$, convexity gives \[ \langle g^k,x^k-x^\star\rangle\geq f(x^k)-f^\star. \] Consequently, \[ \|x^{k+1}-x^\star\|^2 \leq \|x^k-x^\star\|^2 -2\gamma_k\bigl(f(x^k)-f^\star\bigr) +\gamma_k^2\|g^k\|^2. \] Minimizing the right-hand side as a quadratic function of $\gamma_k$ yields \[ \boxed{ \gamma_k^{\mathrm{Polyak}} = \frac{f(x^k)-f^\star}{\|g^k\|^2} }. \] For smooth objectives, $g^k=\nabla f(x^k)$. The numerator is the current objective gap and the denominator normalizes the step by the local gradient scale. Thus the rule tends to take larger steps far from the optimum and smaller steps near it. If $f^\star$ is known exactly, no estimate of a global Lipschitz or smoothness constant is required. The same construction can be interpreted geometrically: the affine subgradient minorant \[ f(x^k)+\langle g^k,x-x^k\rangle \] is intersected with the optimal level $f^\star$, and the update is the point on that hyperplane closest to $x^k$. This projection interpretation is the basis of the Polyak Minorant Method, which replaces affine subgradient models with arbitrary convex lower models and incorporates constraints [2310.07922]. A scaled form is frequently used: \[ x^{k+1}=x^k-\gamma \frac{f(x^k)-f^\star}{\|\nabla f(x^k)\|^2} \nabla f(x^k), \qquad \gamma\in(0,2]. \] The case $\gamma=1$ is the usual Polyak method. For $\gamma\in(0,2)$, the elementary distance recursion contains the positive descent coefficient $\gamma(2-\gamma)$. The requirement that $f^\star$ be known is both the principal strength and the principal limitation of the rule. The method is adaptive once the target value is available, but is not parameter-free in an absolute sense. In nonsmooth convex optimization, exact last-iterate behavior is substantially weaker than many best-iterate or averaged-iterate guarantees: with uniformly bounded subgradients, the standard Polyak method has an exact worst-case last-iterate rate of order \[ O\!\left(\frac{BR}{N^{1/4}}\right), \] where $B$ bounds subgradients and $R$ bounds the initial distance to a solution [2407.15195]. ## 2. Stochastic Polyak step-size For a finite-sum objective \[ f(x)=\frac1n\sum_{i=1}^n f_i(x), \] stochastic gradient descent samples $i_k$ and updates \[ x^{k+1}=x^k-\gamma_k\nabla f_{i_k}(x^k). \] Applying the Polyak construction to the sampled component gives the stochastic Polyak step-size (SPS): \[ \boxed{ \gamma_k^{\mathrm{SPS}} = \frac{f_{i_k}(x^k)-f_{i_k}^\ast} {c\|\nabla f_{i_k}(x^k)\|^2} }, \] where \[ f_i^\ast=\inf_x f_i(x), \qquad c>0. \] A bounded version is \[ \boxed{ \gamma_k^{\mathrm{SPS}_{\max}} = \min\left\{ \frac{f_{i_k}(x^k)-f_{i_k}^\ast} {c\|\nabla f_{i_k}(x^k)\|^2}, \gamma_{\max} \right\}. } \] The cap prevents very large steps when a sampled gradient is small. The loss and gradient used by SPS are already computed in an SGD iteration, so unbounded SPS requires no full-objective or full-gradient evaluation and has essentially no additional per-iteration computational cost [2002.10542]. The component optimum $f_i^\ast$ is weaker information than the global optimum $f^\star$. For many nonnegative unregularized losses, $f_i^\ast=0$ is available or is the relevant infimum. Examples include squared loss in realizable regression, logistic loss in separable classification, exponential loss, and squared-hinge loss in linearly separable classification. For unregularized logistic loss, the infimum can be zero without being attained by any finite parameter vector. With regularization, individual minima are generally nonzero, although they can sometimes be computed analytically or cheaply offline. The parameter $c$ controls the scale of the update. The cited SPS analysis recommends $c=1/2$ in strongly convex and interpolation analyses and $c=1$ for a convenient convex-objective bound. A central inequality is \[ \gamma_k^2\|\nabla f_{i_k}(x^k)\|^2 \leq \frac{\gamma_k}{c} \bigl(f_{i_k}(x^k)-f_{i_k}^\ast\bigr), \] which is an equality for unbounded SPS. This permits convergence analysis without bounded stochastic gradients or a standard bounded-variance assumption in several settings. Define the stochastic inconsistency quantity \[ \sigma^2 = \mathbb E_i\bigl[f_i(x^\star)-f_i^\ast\bigr]. \] Under interpolation, \[ f_i(x^\star)=f_i^\ast\quad\forall i, \qquad \sigma=0. \] For convex smooth components and a strongly convex aggregate objective, capped SPS converges linearly in expected squared distance to a neighborhood whose size depends on $\sigma^2$ and $\gamma_{\max}$. Under interpolation, ordinary SPS with $c=1/2$ converges linearly to the true solution. For convex non-strongly-convex objectives, the averaged iterate has an $O(1/K)$ objective-gap rate up to a residual neighborhood; under interpolation, the residual vanishes. For smooth PL objectives, SPS yields linear objective convergence under the stated parameter restrictions. For general smooth nonconvex objectives, weak growth yields an $O(1/K)$ stationarity rate up to a neighborhood governed by the growth residual. The same analysis gives constant-step SGD bounds. If the cap is sufficiently small, capped SPS is effectively constant-step SGD. In the strongly convex case, \[ \mathbb E\|x^k-x^\star\|^2 \leq (1-\mu\gamma)^k\|x^0-x^\star\|^2 +\frac{2\sigma^2}{\mu}. \] Under interpolation, the residual term disappears. The neighborhood is expressed in terms of the optimal objective difference $\sigma^2$, rather than solely in terms of the conventional gradient-noise quantity \[ z^2=\mathbb E\|\nabla f_i(x^\star)\|^2. \] ## 3. Interpolation, non-interpolation, and optimal-value estimation Interpolation means that a common point minimizes every component: \[ f_i(x^\star)=f_i^\ast \qquad\text{for every }i. \] It occurs in realizable least-squares problems, consistent linear systems, separable classification, sufficiently expressive kernel models, and over-parameterized neural networks. Under interpolation, stochastic gradients vanish at a common solution and constant-step stochastic methods need not settle in a persistent noise neighborhood. For consistent linear systems, the interpolation interpretation is especially direct. A stochastic quadratic reformulation can satisfy \[ f_S(x)-f_S^\ast=\frac12\|\nabla f_S(x)\|^2. \] With $c=1/2$, SPS therefore produces a unit step and recovers the theoretically optimal step for randomized sketch-and-project methods, including randomized Kaczmarz. The associated distance guarantee depends on the smallest nonzero eigenvalue of the expected system matrix [2002.10542]. Outside interpolation, replacing $f_i^\ast$ by zero can produce systematically excessive steps. The residual $\sigma^2$ then determines the limiting neighborhood in many SPS bounds. Several methods address this limitation by estimating individual or global target values. **MOTAPS**, or Moving Targetted Polyak Stepsize, maintains one auxiliary scalar for every data point. These variables track individual losses and permit a moving estimate of the global target. TAPS assumes that the global optimum value is known; MOTAPS also estimates that value and therefore does not require interpolation. Its auxiliary objective contains a dampening parameter $\lambda$. Fixed $\lambda>0$ produces convergence to a neighborhood, with an error term proportional to \[ \frac{\lambda f(w^\star)^2}{\mu(n+1)} \] or \[ \frac{\lambda f(w^\star)^2}{2(n+1)}. \] Eventually decreasing the step-size can remove the fixed-$\lambda$ error floor asymptotically. The convergence theory is conditional on star-convexity or strong star-convexity of an auxiliary objective, which does not automatically follow from convexity of the original component losses [2106.11851]. **Twin-Polyak** maintains two iterates and updates the higher-valued one using the lower-valued objective as a proxy for $f^\star$. If $f(x^k)>f(y^k)$, \[ x^{k+1} = x^k - 2\frac{f(x^k)-f(y^k)} {\|\nabla f(x^k)\|^2} \nabla f(x^k), \qquad y^{k+1}=y^k. \] The lower-valued sequence supplies an observed target rather than a known optimum. The method is invariant to positive objective scaling and objective translation. Under deterministic twin-gap assumptions, it has linear convergence for strongly convex smooth objectives and an $O(k^{-1/2})$ best-iterate rate for convex bounded-gradient objectives. Its stochastic variants are primarily empirical because the stepsize depends on the same random minibatch as the gradient [2509.14854]. **PSADLA**, or Polyak Stepsize with Accelerated Decision-Guided Level Adjustment, maintains a lower level $\bar f_k