Lectures on optimization

Published 7 May 2026 in math.OC | (2605.07006v1)

Abstract: These lecture notes cover the theory of convex optimization, with a particular emphasis on first-order methods.

Abstract PDF Upgrade to Chat

Authors (1)

Sinho Chewi

Summary

The paper rigorously develops an oracle-based framework, demonstrating tight worst-case complexity bounds for first-order methods in convex optimization.
It introduces a unified treatment of gradient flows, discrete algorithms, and acceleration schemes like Nesterov’s methods to achieve optimal convergence rates.
It connects advanced optimization techniques to statistical learning, providing explicit rates that highlight the computational-statistical tradeoffs.

Authoritative Essay on "Lectures on Optimization" (2605.07006)

Overview and Structure

"Lectures on Optimization" (2605.07006) offers a rigorous, theorem-proof based treatment of core topics in continuous optimization, centered on convexity, first-order and second-order methods, duality, and modern variants such as mirror descent and stochastic optimization. The content, prepared as advanced lecture notes, challenges the reader with a demanding mathematical presentation, encompassing both foundational theory and nuanced technical details. The material is organized to build progressively from the oracle-based formulation of optimization models through algorithmic paradigms, tight lower bounds, and complex algorithmic structures such as cutting-plane and proximal methods.

Key topics include: formal oracle models and query complexity, convexity and its quantitative refinements (e.g., strong convexity, smoothness), gradient-based dynamics in both continuous and discrete time, optimality conditions, equivalence of smoothness/convexity between primal and dual (Fenchel) perspectives, and stochastic approximation. Substantial attention is given to modern perspectives on acceleration, composite minimization, structure-exploiting algorithms, interplay between geometry and method design, and rigorous oracle lower bounds.

Optimization Framework: Oracles, Convexity, and Complexity

The notes open with a decisive emphasis on the oracle model as the fundamental abstraction for quantitative optimization theory. The predominance of the first-order (gradient) oracle for continuity-based problems is established, alongside a taxonomy of oracles (zeroth-, first-, linear optimization, and proximal) and associated query models. The interaction between the algorithm and the function is captured via the permitted queries and their information content, leading to a formalization of query complexity as a function of class properties and target accuracy $\varepsilon$ .

Within this framework, the necessity of nontrivial structure—specifically, convexity—is driven home by an explicit lower bound: for $L$ -Lipschitz functions minimized via zeroth-order oracles, the iteration complexity is exponential in dimension, $\Omega((L/\varepsilon)^d)$ . Thus, convexity is shown to be the minimal property under which local oracle information can, via algorithmic strategies, generate global guarantees.

The transition from abstract convexity to quantitative properties—strong convexity ( $\alpha$ -convexity), smoothness ( $\beta$ -smoothness), and the Polyak–Łojasiewicz (PL) condition—is developed through precise equivalence theorems, interlacing first- and second-order (Hessian) characterizations with secant and tangent inequalities. Notably, the PL condition is cast as an interpolation between strong convexity and quadratic growth, with implications for both convergence rates and the sharpness of algorithms in overparameterized models.

Gradient Flows, Discrete-time Algorithms, and Acceleration

The continuous-time gradient flow ODE, $\dot x_t = -\nabla f(x_t)$ , is systematically analyzed, producing descent and contraction properties under various regimes. In the strongly convex case, exponential convergence in both distance and function values is obtained, whereas for general convexity, $O(1/t)$ rates are established (with sharp $O(1/t^2)$ Lyapunov bounds derived via sophisticated functionals).

Discrete algorithms are obtained via forward Euler discretization (gradient descent) and require careful analysis of step size relative to smoothness for stability and provable convergence. The condition number $\kappa = \beta/\alpha$ enters as the central parameter governing iteration complexity. For instance, with step size $1/\beta$ , $L$ 0 convergence is shown for strongly convex, smooth objectives.

A significant technical highlight is the treatment of lower bounds in the first-order oracle model. The notes give both "gradient span" lower bounds (demonstrated for both convex smooth and non-smooth cases) and reductions connecting the convex and strongly convex settings. The constructions utilize hard-to-optimize quadratic chains and decoupled maxima, establishing the sharpness of $L$ 1 and $L$ 2 rates for non-smooth and stochastic problems, respectively.

Acceleration is treated with both depth and technical precision. The conjugate gradient method is developed for quadratics, including explicit algebraic and polynomial approximation characterizations via Chebyshev polynomials and their recurrences. The general case is handled through both ODE-based Lyapunov methods and discrete Nesterov's schemes, yielding the $L$ 3 and $L$ 4 rates associated with the optimal "accelerated" regime.

Figure 1: Comparison of the gradient flow with the continuous-time accelerated gradient flow (AGF), visualizing non-monotonic objective dynamics and the need for specialized Lyapunov analyses for AGF.

Non-smooth Optimization, Proximal, Mirror, and Cutting-plane Methods

The lectures address the structural and algorithmic challenges posed by non-smooth objectives and constraints, utilizing subgradient calculus, lower semicontinuity, and projection operators (including Bregman projections via mirror maps). Projected subgradient, cutting-plane, and ellipsoid methods are analyzed, contrasting their iteration complexities and their applicability given the cost structure of projections versus linear-minimization oracles.

The optimality of cutting-plane and ellipsoid methods in the context of low-dimensional problems is established, matching dimension-dependent lower bounds and demonstrating that, under certain oracle models, further improvements are unattainable. These results are complemented by technical treatments of error guarantees for average iterates and feasibility-based reductions.

Proximal algorithms are introduced for composite minimization, formalizing the proximal operator and Moreau envelope. Complete analyses are given for both smooth and non-smooth composite objectives, including tight convergence rates that match those achieved for smooth problems under appropriate regularity.

Figure 2: The soft thresholding operator, central to proximal updates for the $L$ 5 regularization in LASSO (ISTA/FISTA algorithms).

Mirror descent and Bregman divergence geometry are integrated with duality and relative smoothness/convexity. The lectures demonstrate affine-invariant and geometry-adaptive variants, generalized to online and stochastic scenarios, and illuminate the role of the Legendre transformation and Fenchel duality in algorithm design and analysis. Multiplicative weights and entropic regularization receive a technically detailed and geometric presentation.

Stochastic Optimization and Statistical Implications

The latter sections explore stochastic approximation, offering precise iteration complexities for stochastic first-order and projection-type methods—accounting for both variance bounds and function class regularity. The convergence of stochastic mirror/proximal gradient descent is established, including optimal $L$ 6 and $L$ 7 rates for non-smooth and strongly convex settings. The likelihood of acceleration in the stochastic regime is refuted via matching lower bounds.

The link to statistical generalization is rigorously exposed. One-pass stochastic gradient descent is shown to yield excess risk bounds that match minimax rates for empirical risk minimization, realizing statistical optimality with computational economy. This extends to fast rates under strong convexity (curvature conditions) and demonstrates the duality between statistical complexity and algorithmic iteration requirements.

A particularly robust result is the central limit theorem for Polyak–Ruppert averaged iterates of SGD, handled for both quadratic and general smooth strongly convex objectives. The technical apparatus covers martingale difference sequences, time-inhomogeneous step-size schedules, and offers full characterization of the limiting covariance matrix in terms of the Hessian and asymptotic noise covariance.

Noteworthy Numerical Results and Claims

Sharp lower bounds: For first-order optimization, no deterministic algorithm admits better worst-case query complexity than $L$ 8 for smooth strongly convex, $L$ 9 for non-smooth strongly convex, and $\Omega((L/\varepsilon)^d)$ 0 for smooth or non-smooth convex stochastic settings.
Existence of PL regime: The Polyak–Łojasiewicz condition yields exponential rates without full strong convexity, particularly relevant for overparameterized models (e.g., deep learning).
Accelerated convergence: Accelerated gradient and conjugate gradient methods attain the lower bounds associated with optimal polynomial approximation— $\Omega((L/\varepsilon)^d)$ 1 iterations.
Mirror-primal/dual adaptability: Algorithms achieve affine-invariant and geometry-adapted rates in mirror and norm-based settings via suitable choice of Bregman divergences and Fenchel-dual settings.
Statistical optimality: Stochastic mirror descent, with appropriate step sizes and averaging, matches the statistical minimax rates for excess risk under convex (and strongly convex) settings, serving as an optimal computational strategy for massive data regimes.

Theoretical and Practical Implications

The multi-faceted treatment of optimization in these lectures has both deep theoretical consequences and far-reaching practical implications. The demonstration of tight lower and upper bounds in various oracle models provides a foundation for certifying algorithms as optimal or suboptimal within their regime. The reduction-based equivalence between convex and strongly convex settings elucidates the true role of curvature in both global and local optimization strategies.

The mirror descent and proximal frameworks, established for both online and stochastic regimes, provide the tools necessary to adapt algorithms to the intrinsic geometry of the problem—crucial for high-dimensional learning, signal reconstruction, energy-efficient algorithms, and methods under memory or norm constraints.

The probabilistic analyses connecting optimization (in both deterministic and stochastic settings) with generalization guarantees formalize the computational-statistical tradeoff at the heart of contemporary large-scale machine learning, deep learning, and nonparametric inference.

The unification of geometric, variational, and algebraic perspectives, notably in the treatment of Lyapunov methods, duality, and SGD CLT, signals the maturation of optimization as a central discipline for computational mathematics, statistics, and data-driven modeling.

Future Directions

Adaptive and non-uniform sampling schemes: Extensions to adaptive step size, sampling without replacement, and structure-exploiting variance reduction methods remain active areas, poised to further narrow the computational-statistical gap.
Non-Euclidean optimization and optimal transport: Building on mirror and Bregman methods, future work may push toward algorithms inherently adapted to statistical divergence geometries and non-linear data manifolds.
Compositional stochastic and distributed optimization: The integration of compositional and federated architectures with strong theoretical guarantees remains a crucial direction for modern applications.
High-dimensional phenomena and implicit regularization: Investigations into the role of optimization in high-dimensional statistical inference, generalization beyond explicit model constraints, and implicit bias of algorithms.

Conclusion

"Lectures on Optimization" (2605.07006) presents a mathematically rigorous, technically advanced, and comprehensive analysis of modern continuous optimization, weaving together oracle-based complexity, non-asymptotic convergence, geometry-adapted methods, and deep connections to statistical learning theory. The work establishes tight upper and lower bounds, formalizes advanced algorithmic paradigms (including acceleration and mirror methods), and frames stochastic and statistical optimization within a unified, rigorous analytical structure. This resource is essential for researchers in optimization, mathematical statistics, and algorithmic machine learning seeking a definitive and technically precise treatment of contemporary optimization theory.

Markdown Report Issue