Lecture Notes: Convex Optimization

Abstract: Lecture notes for a course on convex optimization taught by Andreas Habring at Graz University of Technology in 2026. The notes cover mathematical preliminaries, fundamental results about existence and of solutions for minimization problems, projected subgradient descent, proximal-gradient methods, heavy ball gradient descent, Nesterov accelerated gradient descent and FISTA, primal-dual methods, and a short intro to optimal transport.

• The paper presents rigorous proofs of convex optimization fundamentals, including vector space properties, topology, and continuity concepts.
• It details comprehensive derivations of first-order and proximal algorithms, subdifferential calculus, and primal-dual methods for efficient optimization.
• The notes connect advanced theoretical constructs to practical applications such as large-scale learning, empirical risk minimization, and optimal transport.

Authoritative Summary of "Lecture Notes: Convex Optimization" (2607.11664)

These lecture notes provide an extensive and rigorous treatment of the core principles and techniques underlying convex optimization, following a classic mathematical exposition suitable for advanced researchers and practitioners. The material systematically develops foundational vector space concepts, topological preliminaries, convex analysis, subdifferential calculus, and modern first-order optimization methods, with rigorous attention both to theoretical details and algorithmic implications.

---

Preliminaries: Vector Spaces, Topology, and Continuity

The notes begin by establishing the formal underpinning of convex optimization: finite-dimensional vector spaces over $\mathbb{R}$, equipped with norms and inner products. The equivalence of all norms in $\mathbb{R}^d$ is formally stated, ensuring topological invariance for convergence and continuity analyses. The text rigorously details properties of Banach and Hilbert spaces—critical for both existence and uniqueness results in optimization.

The treatment of sequences, compactness, and topology is mathematically meticulous. The Bolzano-Weierstrass theorem is proved constructively, providing a foundation for existence theorems for minimizers.

---

Continuity, Linear Operators, Duality

Key notions of continuity, Lipschitz continuity, and linear operator theory are introduced, with a focus on mapping structures and dual spaces. The Riesz representation theorem underlines the identification of a Hilbert space with its dual, and foundational duality relationships—central for formulating optimality conditions and dual ascent methods—are made explicit.

---

Lower Semi-Continuity, Coercivity, and Existence of Minima

The interplay between lower semi-continuity (lsc), closedness of functions, and coercivity is highlighted as a triad sufficient for asserting the existence of minimizers in finite-dimensional spaces. The direct method, combining compactness and lsc, is used to guarantee that the infimum of a proper, coercive, lsc function on $\mathbb{R}^d$ is achieved.

Figure 1: Lack of lower-semicontinuity may lead to infima not being attained; lsc is necessary for existence of minimizers.

---

Convex Sets and Functions

The notion of convex sets is axiomatized, with rigorous treatment of convex hulls, intersections, products, images, and pre-images under linear maps. Constructions preserving convexity are explicitly enumerated. Examples encompass norm balls (feasible in all $\ell^p$ norms with $p \geq 1$), affine subspaces, and polyhedra.

Figure 2: Top row: convex sets. Bottom row: non-convex sets. The figure distinguishes geometries that support convexity-based optimization results.

The structure and geometry of norm balls under various $p$-norms are examined to illustrate how the choice of norm influences convexity properties.

Figure 3: Norm balls for $\overline{B}_{\|\cdot\|_p, R}(0)$ for various $\ell^p$ norms, emphasizing convexity for $p \geq 1$.

Convex functions are then characterized via zero-order, first-order, and second-order conditions, establishing strict convexity and strong convexity when relevant. Jensen's inequality, the first-order condition (supporting hyperplane property), and the second-order condition ($\nabla^2 f(x) \succeq 0$) are all proved explicitly.

Figure 4: Example of the qualitative difference between convex and non-convex one-dimensional functions, highlighting global minimizer uniqueness and landscape.

The notes also elucidate the monotonicity of the gradient (Baillon-Haddad theorem for convex gradients) and detail powerful composition rules for preserving convexity under sums, affine changes of variables, maxima, and composition with monotone convex functions.

---

Subdifferential Calculus and Optimality

The calculus of subgradients is systematically presented. The subdifferential generalizes the gradient to non-differentiable convex functions, with the foundational result that subgradients exist at every point in the interior of the domain for proper, lsc, convex functions—a direct corollary of the supporting hyperplane theorem and Hahn-Banach separation.

Subdifferential calculus rules are provided for sums, compositions, affine transformations, and pointwise maxima. A key result is the characterization of optima: $\mathbb{R}^d$0 iff $\mathbb{R}^d$1—the necessary and sufficient condition for convex minimization.

---

First-Order Algorithms: Projected Subgradient and Proximal Methods

Comprehensive coverage is given to the basic projected subgradient method for constrained problems, including the foundational projection theorem and non-expansivity of projections in Hilbert spaces. The critical convergence rate of $\mathbb{R}^d$2 for subgradient descent (with Polyak or diminishing step sizes) is established, highlighting the performance gap relative to smooth gradient methods ($\mathbb{R}^d$3) and the challenge with non-smooth objectives.

Accelerated methods are then introduced:

• Proximal gradient methods for composite objectives $\mathbb{R}^d$4 (smooth + non-smooth), with detailed analysis of the Moreau envelope, fundamental descent inequalities, and $\mathbb{R}^d$5 convergence.
• Heavy ball and Nesterov acceleration are derived as discretizations of continuous-time differential equations, culminating in Fast Iterative Shrinkage-Thresholding Algorithm (FISTA), which achieves $\mathbb{R}^d$6 convergence for convex $\mathbb{R}^d$7.

---

Duality Theory and Primal-Dual Schemes

The notes formalize Fenchel duality, properties of convex conjugation, and their relation to subdifferential calculus (Fenchel's equality). The Moreau identity is proved, connecting primal and dual proximal operators. The Fenchel-Rockafellar theorem is stated and proved in detail, establishing conditions for strong duality in composite convex problems.

Modern primal-dual algorithms, particularly the primal-dual hybrid gradient (PDHG / Chambolle-Pock) and the Alternating Direction Method of Multipliers (ADMM), are discussed with explicit algorithmic updates and convergence proofs, including ergodic gap bounds.

---

Stochastic Optimization and Applications

The stochastic subgradient method is analyzed under unbiasedness and bounded variance assumptions, extending deterministic convergence results to expectation. Applications in empirical risk minimization, large-scale learning, and the modern context of data-driven optimization are suggested.

---

Excursion: Optimal Transport

The book concludes with a rigorous mini-course on optimal transport, from Monge’s formulation to the Kantorovich relaxation, existence theorems, Kantorovich duality (support function form), and entropy-regularized formulations (Sinkhorn algorithm). This material serves dual roles: offering an illustrative application of convex duality theory and connecting with contemporary topics in machine learning.

---

Implications and Future Directions

The strong focus on convex analysis as a language and toolset for modern optimization illuminates both theoretical and algorithmic frontiers. The lecture notes highlight how convexity endows optimization problems with powerful guarantees on existence, uniqueness, and efficient computability (polynomial-time first-order algorithms). As large-scale, high-dimensional learning problems proliferate in AI, the foundational guarantees of convexity continue to inform both the design of new algorithms (especially for composite and structured objectives) and the safe deployment of optimization primitives in sensitive applications.

On the practical side, the spectrum of optimization algorithms discussed—gradient-based, subgradient, proximity, and accelerating methods—forms the computational backbone of a significant share of current machine learning and signal processing pipelines. The text also prepares the ground for extensions: stochastic optimization for high-dimensional data, distributed optimization for federated systems, primal-dual frameworks for structured estimation and variational inference, and optimal transport for generative modeling and fairness.

Theoretically, the clarity of subdifferential theory and convex duality presented here enables the development of more sophisticated models, including saddle-point, min-max, and variational inequality problems. Given ongoing research in non-convex optimization (notably in deep learning), the mathematical machinery developed for convex problems remains indispensable for understanding approximation, relaxation, and tractable surrogates in increasingly complex model classes.

---

Conclusion

These notes constitute a robust and comprehensive reference for researchers in convex optimization, offering rigorous treatment of foundational material and advanced algorithms alike. The work bridges the gap between deep mathematical theory (functional analysis, duality, set-valued calculus) and concrete algorithmic developments (first-order and primal-dual schemes), providing both a reliable toolkit and a launching point for future research in optimization and its impact across modern computational science.