Convex Optimal Control Framework

• Convex optimal control framework is a class of methods that leverage convexity to ensure global optimality and robust numerical solvability.
• It exploits strict Hamiltonian convexity in continuous time and addresses discretization-induced convexity loss through refined numerical schemes.
• Algorithmic approaches including direct, indirect, and convex relaxation methods enable scalable, globally optimal control synthesis even under nonconvex constraints.

A convex optimal control framework is a class of theoretical and algorithmic methodologies that exploit convexity in the formulation or relaxation of optimal control problems, enabling global optimality, robust numerical solvability, and principled characterizations of solution structure. Convexity can arise intrinsically from the system and cost structure (e.g., via the Hamiltonian in continuous time), be induced through dualization or convex lifting, or be constructed through convex relaxation of nonconvex constraints. The development and analysis of such frameworks are central to modern control theory, where global optimality properties and computational efficiency are major objectives.

1. Fundamental Principles: Hamiltonian Convexity and Global Uniqueness

The cornerstone of convex optimal control in continuous time is the observation that, for general nonlinear control problems with control-affine dynamics

$\dot{x}(t) = F(x(t)) + G(x(t)) u(t),$

and cost functional

$$J[u(\cdot)] = \Phi(x(t_f)) + \int_0^{t_f} \left[ \mathcal{L}(x(t)) + \mathcal{R}(u(t)) \right]dt,$$

the property of strict convexity of the Hamiltonian with respect to the control variable $u$ is sufficient for unique global optimality. The Pontryagin Hamiltonian is given by

$$H(x, u, \lambda, t) = \mathcal{L}(x) + \mathcal{R}(u) + \lambda^\top [F(x) + G(x)u],$$

and strict convexity in $u$ (i.e., the Hessian $\partial^2 H / \partial u^2 \succ 0$) ensures that the minimization in the Pontryagin principle yields a unique $u^*$ at each time.

A central theorem asserts that, given smooth data and strict convexity of $H$ in $u$, the necessary conditions (the two-point boundary value problem arising from Pontryagin’s Maximum Principle) possess a unique solution, which globally minimizes $J$. Notably, nonlinearity or nonconvexity in state dynamics $F(x), G(x)$ or running cost $\mathcal{L}(x)$ is immaterial for global uniqueness; only convexity in $u$ is needed, since the minimization over $u$ is decoupled from system nonlinearity (Abhijeet et al., 12 Apr 2024).

2. Discrete-Time Convexity Loss and Conditions for Spurious Minima

When passing to discrete time, convexity in $u$ may be lost even if the continuous-time problem is globally convex. In forward-Euler discretization,

$x_{k+1} = x_k + \Delta t [f(x_k) + g(x_k) u_k],$

the update map $q(x_k, u_k)$ can become nonlinear in $u_k$ for non-infinitesimal $\Delta t$. The dynamic programming Q-function,

$Q_{k+1}(x, u) = \ell(x, u) + J_{k+1}(q(x, u)),$

may then develop multiple local minima in $u$, breaking convexity irrespective of the convexity of the original control cost $\mathcal{R}(u)$. Spurious optima arise unless $\Delta t$ is chosen sufficiently small, so that higher-order terms in $q$ are negligible, and the overall dependence on $u$ remains convex. The formal criterion is

$$\Delta t\, \max_\text{eig} \left( \frac{\partial^2}{\partial u^2} J_{k+1}(q(x, u)) \right) < \lambda_\text{min} \left( \frac{\partial^2 \mathcal{R}}{\partial u^2} \right),$$

which enforces a minimum grid resolution for global uniqueness in discretized problems (Abhijeet et al., 12 Apr 2024).

3. Algorithmic Approaches: Direct and Indirect Methods

Two principal classes of numerical solution methods for convex optimal control are direct methods such as Sequential Quadratic Programming (SQP) and indirect methods such as the Iterative Linear Quadratic Regulator (iLQR) or Differential Dynamic Programming (DDP).

• Direct methods (e.g., SQP): The problem is transcribed into a finite-dimensional nonlinear program (NLP) over discrete trajectories $\{x_k, u_k\}$. SQP linearizes the constraints and forms a quadratic approximation of the Lagrangian per iteration, solving a sequence of QP subproblems. However, under coarse discretization, these approaches are susceptible to spurious local minima due to the aforementioned loss of convexity.
• Indirect methods (e.g., iLQR): At each iteration, dynamics and costs are linearized/quadratized about the current trajectory, enabling backward-forward Riccati recursions to compute optimal control updates. iLQR is structurally immune to spurious minima from discretization, consistently converging to the global minimum for control-affine, strictly Hamiltonian-convex problems, unless initialized at a genuine stationary point corresponding to a spurious solution of the underlying discretized NLP (Abhijeet et al., 12 Apr 2024).

Empirical studies show that for dynamics such as pendulum and cart-pole, indirect methods reliably find the global minimizer across initializations and step sizes, whereas direct methods may converge to sub-optimal stationary solutions for insufficiently fine discretization.

4. Convex Variational and Lifting Frameworks

Alternative convex formulations include convex variational dualizations, extended convex lifting (ECL), and measure-theoretic relaxations.

• Convex variational principle: Functionals are constructed such that Euler-Lagrange equations, via auxiliary dual variables and strongly convex potentials, recover the Pontryagin extremals as unique minimizers. This enables time-dependent LQR and certain nonlinear optimal control problems to be solved by direct convex minimization over a set of dual fields, bypassing Riccati equations, with well-defined existence and uniqueness under coercivity (Acharya et al., 21 Feb 2025).
• Extended convex lifting (ECL): Many nonconvex policy optimization problems (e.g., LQR, LQG, $\mathcal{H}_\infty$ control) can be lifted via change of variables into convex semidefinite programs over auxiliary variables such as Lyapunov matrices. The existence of such a lift implies global minimality for all first-order nondegenerate stationary points and absence of spurious nondegenerate critical points. This applies to both centralized and distributed control under quadratic invariance conditions (Zheng et al., 6 Jun 2024).
• Moment/SOS and measure-theoretic relaxations: For hybrid systems and systems with polynomial dynamics and costs, the occupation measure approach encodes reachable densities as measures, yielding infinite-dimensional linear programs (LPs). Hierarchies of finite-dimensional semidefinite programs (SDPs) are constructed via truncation of moment sequences, converging from below to the global optimum (Zhao et al., 2017).

5. Convex Relaxation and Practical Implementation for Nonconvex Constraints

Several frameworks demonstrate that classes of nonconvex or integer-driven control problems admit practical convex relaxations (often second-order cone programs, SOCPs) that are lossless under technical conditions.

• Lossless convexification: For semi-continuous input constraints (e.g., inputs must be zero or within given bounds, only a subset active at once), introducing slack variables and convex surrogates (e.g., $\gamma_i \in [0,1]$), coupled with careful regularity assumptions (strong observability, pointwise normality), yields SOCPs whose solutions are globally optimal almost everywhere, with solutions certified to be integer-valued in the limit (Malyuta et al., 2019).
• Hybrid and switching systems: Convex relaxation of hybrid mode switching and binary-continuous controls can be analyzed via convex duality and Moreau-Yosida regularization, leading to Newton-type methods and tight optimality gap estimates for switching controls in PDE-constrained systems (Clason et al., 2017).
• Structured controller design: For finite-horizon problems with arbitrary convex constraints on feedback gains (sparsity, delays, decentralization), convex surrogates built on spectral functions of the affine-inverse closed-loop mapping enable global design with nontrivial suboptimality guarantees (Dvijotham et al., 2013).

6. Extensions: Stochasticity, Learning, and Data-Driven Control

Convex optimal control principles extend to stochastic systems, learning-enabled controllers, and operator-theoretic/data-driven settings.

• Sequential convex programming for stochastic OCP: Iterative linearization and convexification around nominal state-control trajectories enables convergence to stochastic Pontryagin extremals under suitable trust region scheduling (Bonalli et al., 2020).
• Convex control policies as meta-optimization: Embedding a (possibly complex or neural network-based) policy that computes optimal actions via internal convex optimization allows for automatic controller tuning via gradient-based meta-optimization, leveraging implicit differentiation through convex optimization layers (Agrawal et al., 2019).
• Data-driven operator convexification: Using data to approximate Koopman or Perron-Frobenius operators, optimal control problems over densities (occupation or transfer densities) are convexified into polynomial, sum-of-squares (SOS), or quadratic programs. Under compactness and stabilizability, this allows global stabilization of nonlinear continuous-state systems, with convergence as the approximation basis becomes dense (Huang et al., 2020, Moyalan et al., 2022, Vaidya et al., 2022).
• Convexity under uncertainty: Systems with monotone convex/concave dynamics and coherent risk measures admit convex lifted output-feedback synthesis problems via Q- or Youla-parameterizations, making scenario-based and receding-horizon model predictive control subproblems tractable and globally solvable (Kircher et al., 2019).

7. Practical Guidelines and Implications

• Enforce or choose strictly convex control costs to assure Hamiltonian convexity and global uniqueness in continuous time.
• In discretized formulations, use fine enough time steps to preserve convexity and avoid artificial local minima.
• Prefer solution methods leveraging the structure of the Pontryagin two-point boundary value problem, as indirect methods (e.g., iLQR) are more robust to discretization artifacts than generic NLP solvers.
• When tackling systems with integer, semi-continuous, switching, or hybrid constraints, utilize theoretically grounded convex relaxations, and verify technical conditions (observability, normality) for losslessness.
• For data-driven and learning-based controllers, design architectures (e.g., input-convex neural networks) that enable convex model predictive control, and use gradient-based tuning exploiting differentiability of the convex optimization layer.
• Where possible, exploit convex dualities and moment/SOS relaxations for hybrid and partially observed systems to globalize nonconvex optimal control problems and certify solution quality.

These frameworks collectively provide a rigorous and computationally tractable foundation for both classical and emerging optimal control applications, enabling globally optimal, robust, and scalable control synthesis across deterministic, stochastic, hybrid, and learning-augmented domains (Abhijeet et al., 12 Apr 2024, Acharya et al., 21 Feb 2025, Zheng et al., 6 Jun 2024, Malyuta et al., 2019, Huang et al., 2020, Kircher et al., 2019, Clason et al., 2017, Dvijotham et al., 2013, Agrawal et al., 2019).