Deterministic Optimal Control Overview

• Deterministic Optimal Control (DET-OCP) is defined by noise-free system dynamics with explicit state and control constraints and an objective functional for trajectory optimization.
• It leverages techniques such as one-shot optimization, dynamic programming, and convex reformulations to establish rigorous solution equivalence and guide feedback law synthesis.
• Innovative algorithms—including max-plus methods, data-driven predictive control, and rollout schemes—enable efficient, robust, and scalable implementation of control strategies.

Deterministic Optimal Control (DET-OCP) refers to the class of optimal control problems where the evolution of the system state is governed by deterministic (noise-free) dynamics and the objective is to optimize a given performance criterion over admissible control sequences or policies. DET-OCP forms the backbone of modern control theory, serving as the foundational setting for both the analysis of continuous and discrete-time systems, and for practical algorithms in feedback control, trajectory optimization, and reinforcement learning.

1. Mathematical Structures and Problem Formulation

A canonical deterministic optimal control problem (OCP) is specified by:

• State dynamics, either in discrete time $x_{k+1} = f(x_k, u_k)$ or continuous time $\dot{x}(t) = f(x(t), u(t))$;
• Admissible control sets $u_k \in A \subset \mathbb{R}^m$, often constrained to compact, convex sets;
• An objective functional, e.g., for finite horizon and discrete time:

$$J(u_{0:T-1}) = \phi(x_T) + \sum_{k=0}^{T-1} L(x_k, u_k)$$

or for infinite horizon, discounted or average costs.

Key well-posedness assumptions include continuity and Lipschitz properties of $f$, lower semi-continuity and boundedness of cost functions, and non-emptiness of admissible control sets for each state. The system may also include state and trajectory constraints, frequently handled via viability conditions or auxiliary augmented states (Yang, 2017).

DET-OCP covers both open-loop (control sequence optimization) and closed-loop (feedback policy) settings, with typical goals being the minimization of the cost for a given initial state or the synthesis of structure-preserving feedback laws.

2. Solution Approaches: Dynamic Programming, One-Shot Optimization, and Their Equivalence

DET-OCP is traditionally approached via two classical formulations:

• One-shot (Direct) Optimization: Minimize the objective over the entire sequence of control variables, resulting in a generally nonconvex, high-dimensional problem. For discrete finite horizon, this yields a nonlinear program in $mT$ variables (Kim et al., 2024).
• Dynamic Programming (DP): Leverages Bellman's principle to define value functions recursively:

$$V_T(x) = \phi(x), \quad V_k(x) = \min_{u \in A} \left\{L(x, u) + V_{k+1}(f(x, u))\right\}$$

resulting in a sequence of subproblems, typically lower-dimensional yet still potentially nonconvex or non-smooth.

A central theoretical result is the precise correspondence between local (and global) optima of these formulations. Specifically, for general nonlinear deterministic systems with compact action sets and $C^2$ regularity, every local minimizer of one formulation induces a local minimum in the other, and vice versa; if the DP “Q-Function” subproblems are strictly convex (unique minimizer), both approaches return the same solution (Kim et al., 2024). This extends to parameterized policy optimization, with equivalence under basis fullness and uniqueness of DP minimizers, thus providing a rigorous landscape correspondence for analysis and algorithm design.

Algorithmic implications are significant: gradient descent on the one-shot objective, or backward DP, converge to the same set of local minima and stationary points, affirming the transferability of optimization techniques across the two domains (e.g., policy gradient in RL vs. policy iteration) under mild conditions.

3. Linear and Convex Analytic Approaches

A recurring theme for DET-OCP—particularly for infinite-horizon, average, or discounted cost settings—is the reformulation as convex programs over measures or function spaces. Notably:

• Occupation Measure and LP Formulation: For both long-run average (Borkar et al., 2018) and infinite-horizon discounted (Gaitsgory et al., 2017) cost, the OCP is equivalently characterized by infinite-dimensional linear programs (LPs) over probability measures representing averaged state-control occupancy. The primal LP seeks to minimize the expected cost over this limiting measure, constrained by “balance” or invariance conditions derived from the system dynamics.
• Dual Formulation and Value-Function Approximation: The dual problem is cast over continuous (or lower-semicontinuous) functions resembling value or potential functions, leading to max–min or convex relaxations of the (generalized) Bellman equation. For example, the dual of the discounted-cost OCP reads

$$\sup_{\psi} \inf_{(y,u)} \left\{g(y,u) + \psi(f(y,u)) - \psi(y) + (1-\gamma)(\psi(y_0) - \psi(y))\right\}$$

with dual attainment at the value function (Gaitsgory et al., 2017).

• Optimality Conditions and Feedback Synthesis: Explicit necessary and sufficient optimality conditions are derived in dual variables. The synthesis of (approximately) optimal feedback laws is enabled via finite-dimensional LP or SOS relaxations, which converge to the true value function as the basis grows (Borkar et al., 2018, Gaitsgory et al., 2017, Abdalmoaty et al., 2012).

These convex analytic approaches underlie the development of systematic approximation schemes (including data-driven and reinforcement learning algorithms) and provide robust global characterizations absent in pointwise dynamic programming.

4. Special Classes: LQ, Affine-Quadratic, and Piecewise-Affine Systems

Several structured subclasses of DET-OCP have been extensively analyzed for their favorable properties and algorithmic tractability:

• Affine-Quadratic and LQ Problems (AQ-LQ OCP): For systems with affine-linear dynamics and quadratic costs,

$$\dot{x}(t) = A(t)x(t) + B(t)u(t) + f(t),\quad J(u) = \frac{1}{2}x(T)^\top G x(T) + \frac{1}{2}\int_0^T \left[ x(t)^\top Q(t)x(t) + 2x(t)^\top S(t)u(t) + u(t)^\top R(t)u(t) \right] dt$$

existence, differentiability of value functions, synthesis of optimal feedback, and solvability via the Hamilton–Jacobi–Bellman equation are guaranteed under mild regularity and coercivity. The solution reduces to integrating a generalized (quasi-)Riccati equation and constructing a closed-loop feedback law (Wang et al., 2013, Li et al., 2019).

• Piecewise-Affine (PWA) Systems: For continuous-time systems with PWA dynamics and polynomial cost/state-input constraints, one can reformulate the OCP as an infinite-dimensional LP over occupation measures and employ a hierarchy of LMI relaxations—each readily solvable—to obtain polynomial approximations of the value function that provably converge to the viscosity solution of the associated HJB equation (Abdalmoaty et al., 2012).

This exploitation of problem structure enables transparent, efficient, and robust controller synthesis across a wide range of deterministic control applications.

5. Numerical Algorithms, Data-Driven and Rollout Schemes

Advances in numerical algorithms for DET-OCP focus on the attenuation of computational complexity and practical tractability:

• Adaptive Max-Plus and Sparse-Grid Algorithms: For high-dimensional finite-horizon problems, multi-level adaptive max-plus schemes approximate viscosity solutions of HJB equations via basis/test function expansions, concentrating computational resources on neighborhoods of optimal trajectories. Complexity is reduced from $O(\varepsilon^{-ad})$ (grid-based) to $O(C^d/\varepsilon)$ for error tolerance $\varepsilon$, facilitating efficient solutions in moderate-dimension settings (Akian et al., 2023).
• Explicit Data-Driven Predictive Control (DPC): Canonical model-based MPC and data-driven DPC, when formulated for deterministic linear systems, admit dimension reduction via null-space elimination and yield strictly convex QPs of the same complexity. DPC and MPC thus share explicit piecewise-affine solution structure even in constraint and region-complexity (Klädtke et al., 2022).
• Rollout and Approximate Dynamic Programming: Data-driven rollout algorithms leverage cost-to-go approximations obtained from base policies to initialize value iteration, furnishing monotonic performance improvement and straightforward integration of trajectory constraints or multi-agent settings. Rollout is empirically competitive with standard MPC while offering stronger a priori performance guarantees under broad settings (Li et al., 2021).

6. Extensions: Probabilistic Inference, Statistical Estimation, and Reinforcement Learning

Frontiers of DET-OCP research are increasingly informed by a probabilistic and learning-theoretic perspective:

• Probabilistic Optimal Control and Expectation-Maximization: Finite-horizon deterministic trajectory optimization admits reformulation as probabilistic inference in an artificial graphical model (“probabilistic optimal control”), where maximization of likelihood for “optimality” events yields the original deterministic objective in the zero-variance limit. Variants of the EM algorithm generate sequences of stochastic policies converging to the deterministic optimum, with sigma-point methods employed to propagate moments through nonlinear dynamics for both exploration (in early iterations) and efficient convergence (Filabadi et al., 2024).
• Convex Q-Learning and Overparameterization: By recasting Bellman’s equation as a convex (infinite-dimensional) optimization problem with separated value and Q-function variables, one can construct batch learning algorithms for deterministic OCP with guaranteed global convergence, distinct from nonconvex fixed-point solvers such as DQN. This convex analytic view is central to algorithmic design in data-driven and reinforcement learning for deterministic systems (Mehta et al., 2020).
• Parameter Estimation as DET-OCP: The parameter and state estimation problem for partially observed ODEs can be formulated as a DET-OCP, where a control input is introduced to model discrepancy, and penalized for energy. The optimal profiling reduces to a sequence of LQ problems solvable by Riccati equations, yielding estimation procedures with established consistency and asymptotic normality under standard statistical assumptions (Clairon et al., 2014).

This integration of control, inference, and statistical learning yields robust methodologies applicable to high-dimensional, partially observed, and data-rich deterministic systems.

7. Constraint Handling, Approximation, and Theoretical Guarantees

Practical DET-OCP often involves challenging constraints and the need for certified approximations:

• State and Trajectory Constraints: Discrete-time enforcement of continuous-time state constraints yields near-optimal solutions that converge to the truly constrained optimum as discretization is refined, a crucial property for the viable and robust implementation of constrained control laws (Yang, 2017).
• Rates of Convergence and Error Bounds: Theoretical results provide explicit error rates for grid-based, LMI, and max-plus approximation schemes, as well as performance gap guarantees for rollout and data-driven methods under invariance and monotonicity properties (Akian et al., 2023, Li et al., 2021, Gaitsgory et al., 2017).
• Strong Duality and Optimality Certification: Measure-theoretic LP frameworks ensure strong duality under viability and regularity conditions, equipping practitioners with both lower and upper performance bounds and constructive optimality certificates for finite and infinite-horizon deterministic OCP (Borkar et al., 2018, Gaitsgory et al., 2017, Abdalmoaty et al., 2012).

Altogether, the deterministic optimal control paradigm provides a comprehensive and mathematically rigorous foundation for control synthesis, trajectory optimization, learning-based control, and estimation in noise-free dynamical systems, uniting classical approaches (Riccati/HJB) with modern convex analytic, data-driven, and inferential methodologies.