New Results on the Polyak Stepsize: Tight Convergence Analysis and Universal Function Classes

Abstract: In this paper, we revisit a classical adaptive stepsize strategy for gradient descent: the Polyak stepsize (\texttt{PolyakGD}), originally proposed in \cite{polyak1969minimization}. We study the convergence behavior of \texttt{PolyakGD} from two perspectives: tight worst-case analysis and universality across function classes. As our first main result, we establish the tightness of the known convergence rates of \texttt{PolyakGD} by explicitly constructing worst-case functions. In particular, we show that the $\mathcal{O}((1-\frac{1}κ)^K)$ rate for smooth strongly convex functions and the $\mathcal{O}(1/K)$ rate for smooth convex functions are both tight. Moreover, we theoretically show that \texttt{PolyakGD} automatically exploits floating-point errors to escape the worst-case behavior. Our second main result provides new convergence guarantees for \texttt{PolyakGD} under both Hölder smoothness and Hölder growth conditions. These findings show that the Polyak stepsize is universal, automatically adapting to various function classes without requiring prior knowledge of problem parameters.