Stochastic Optimal Control Framework

• Stochastic optimal control is a framework that minimizes expected cumulative costs in systems governed by random dynamics.
• It leverages probabilistic inference, dynamic programming, and policy gradient methods to manage high-dimensional, nonlinear, and uncertain systems.
• Applications range from robotics and finance to energy systems, underscoring its relevance in robust decision-making under uncertainty.

Stochastic optimal control (SOC) is the mathematical framework for selecting control policies that minimize (or maximize) the expected value of a cumulative cost (or reward) in systems governed by stochastic dynamics. It extends deterministic optimal control by explicitly modeling uncertainties, typically via stochastic differential or difference equations. Modern SOC integrates perspectives from probabilistic inference, dynamic programming, policy gradient optimization, measure-theoretic relaxations, and distributed computation, enabling tractable control of high-dimensional, nonlinear, and uncertain systems across scientific and engineering domains.

1. Mathematical Formulation and Problem Classes

A canonical SOC problem considers a discrete- or continuous-time, fully observed Markov system:

• Discrete-time, nonlinear system:

$$x_{t+1} = f(x_t, u_t) + w_t, \qquad w_t \sim \mathcal N(0, Q_t)$$

The objective is to minimize the expected (possibly quadratic) cumulative cost:

$$J[u_{0:T-1}] = \mathbb E\left[(x_T - x_g)^T S_T (x_T - x_g) + \sum_{t=0}^{T-1} \left((x_t - x_g)^T S_{x,t} (x_t - x_g) + (u_t - u_g)^T S_{u,t} (u_t - u_g)\right)\right]$$

• Continuous-time, controlled SDE:

$dX_t = b(X_t, u_t)\,dt + \sigma(X_t, u_t)\,dW_t$

with cost

$$J[u] = \mathbb E\left[\int_0^T r(X_t, u_t)\,dt + h(X_T)\right]$$

A broad spectrum of settings is encompassed:

• Fully/partially observed systems, path-dependent or Markovian
• Terminal, infinite-horizon, minimum-time, or risk-sensitive costs
• Continuous, discrete, mixed-integer, or distributed control actions
• Constraints on state, control, and performance

The unified modeling paradigm is “optimize over policies” mapping system histories (or filtered beliefs) to controls (Powell, 2019).

2. Duality between Control, Inference, and Policy Optimization

Several frameworks cast stochastic optimal control as an inference, optimization, or saddle-point problem:

• Control as Input Inference: SOC may be posed as inferring the “most likely” input sequence in a probabilistic graphical model constructed from system dynamics and “virtual observations” representing costs. For quadratic costs, the negative log-likelihood coincides with the control objective, and control policies can be inferred via expectation-maximization (EM) and message passing on factor graphs (Watson et al., 2019).
• Policy Gradient Flows: A continuous-time dynamical system for the policy parameters (the “policy gradient flow”) minimizes expected cost via functional gradient descent. The global minimizer is reached under convexity assumptions, and local maximizers correspond to HJB optimal feedbacks. This gradient-flow view yields global convergence guarantees for policy-gradient-style algorithms in strongly concave Hamiltonian settings (Zhou et al., 2023).
• Occupation Measures and Measure-LP Relaxations: SOC may be equivalently reformulated as infinite-dimensional linear or convex programs over occupation measures on state-control trajectories. By leveraging duality (e.g., the link with HJB PDEs or value functions), one derives strong existence and optimality results and constructs hierarchies of semidefinite or linear programs for numerical approximation (Holtorf et al., 2022, Vaidya et al., 2022).
• Bayesian and Maximum Entropy Formulations: The inference-driven approaches automatically yield maximum-entropy (exploratory, robust) policies; for quadratic–linear systems, these recover the classical maximum-entropy LQG solutions (Watson et al., 2019, Hou et al., 10 Nov 2025).

3. Computational Methods and Algorithmic Realizations

The theoretical challenges of SOC—curse of dimensionality, nonconvexity, nonlinearity, and constraint handling—have led to diverse computational schemes:

a) Probabilistic Inference and Message Passing

The input-inference-for-control (i2c) framework constructs a Forney-style factor graph linking states, controls, and cost-observations. Iterative EM is used:

• E-step: Gaussian message-passing computes posterior marginals over state and input trajectories.
• M-step: Cost-scale and regularization hyperparameters are updated via evidence maximization, with KL-divergence-based trust-region control to prevent overfitting local linearizations (Watson et al., 2019).

The induced policies are time-varying linear-Gaussian feedbacks,

$$u_t | x_t \sim \mathcal N(K_t x_t + k_t, \Sigma_{u|x})$$

with closed-form expressions for the gain and covariance.

b) Policy-Gradient and Value-Gradient Approaches

Policy gradient methods operate via forward simulation of sample paths, estimating gradient flows in function space:

• Alternating policy evaluation (solving the linear HJB under current policy) and policy improvement (gradient update) yields global or exponential convergence under strong regularity (Zhou et al., 2023).
• Discretization and parameterization by neural networks or finite elements enable high-dimensional applications, circumventing the impracticality of explicit PDE solvers.

c) Sequential Convex Programming

Sequential Convex Programming (SCP) transcribes the nonlinear stochastic OCP into an iterative sequence of stochastic convex programs by linearizing dynamics and cost around the current trajectory. Each subproblem is solved efficiently via convex programming techniques, and under mild conditions, SCP iterates converge to stochastic PMP extremals (Bonalli et al., 2020).

d) Measure Relaxations and Sum-of-Squares Hierarchies

Occupation-measure or transfer-operator approaches encode the entire distribution of trajectories and compute globally optimal lower bounds via moment-SOS (sum-of-squares) or data-driven convex optimization. Local occupation measures and spatio-temporal partitioning enable precise control over approximation complexity and enable numerically stable SDPs that scale beyond classical global methods (Holtorf et al., 2022, Vaidya et al., 2022).

4. Structure of Optimal Policies and Solution Representations

a) Extraction of Feedback Controllers

The posterior over joint state-action variables in inference-based approaches is Gaussian, allowing the recovery of locally optimal feedback policies. For linear–quadratic systems, this exactly recovers the Riccati-based linear-Gaussian (LQG) feedback law extended to include entropy-regularization (exploration), with the policy covariance automatically balancing uncertainty and robustness (Watson et al., 2019).

b) Existence and Regularity Results

• Under convexity and compactness, measure-theoretic relaxations guarantee existence, strong duality, and attainment of optimal policies in either relaxed or original state spaces (Holtorf et al., 2022).
• For strongly concave Hamiltonians, critical points of the functional gradient flow coincide with globally optimal policies, and exponential rates can be established under mild “flatness” assumptions (Zhou et al., 2023).
• Maximum-entropy (soft) solutions arise naturally within Bayesian and inference-based frameworks.

c) Links to Classical and Modern RL Settings

The classical discrete or continuous-time Bellman—HJB equations remain the backbone of the theoretical structure (Powell, 2019). However, policy search (direct parameter optimization), value function approximation (approximate DP), cost function approximation, and direct lookahead all fit into a universal taxonomy of policies relevant to both SOC and RL (Powell, 2019).

5. Regularization, Uncertainty Quantification, and Bayesian Aspects

Bayesian perspectives in inference-driven SOC automatically endow the solution with several desirable features (Watson et al., 2019):

• The prior over control inputs regularizes the solution, prevents divergence, and ensures exploration when no trajectory is available.
• Control covariances $\Sigma_{u|x}$ directly quantify robustness and the “turn-off” of aggressive feedback under high process noise, offering systematic replacements for heuristic damping or trust-region adjustments required in trajectory-optimization methods (iLQR/DDP).
• Bayesian hyperparameters (input prior and cost-scaling) can be adapted via evidence maximization, integrating regularization directly into the probabilistic inference loop.

6. Connections, Generalizations, and Applications

The stochastic optimal control framework forms the rigorous bridge between deterministic optimal control, reinforcement learning, and approximate dynamic programming (Powell, 2019). It extends to:

• Distributed and multi-agent problems (e.g., distributed mixed-integer microgrid control, with stochastic MILP formulations and primal decomposition (Camisa et al., 2021))
• Partially observed and path-dependent systems, with value representation via backward SDEs driven by randomized exogenous signals (Bandini et al., 2015)
• High-dimensional, nonlinear, and hybrid (discrete/continuous) systems in robotics, energy, epidemiology, and more

Applications span from high-dimensional robot learning and motion planning to finance, supply chain management, power systems, stochastic thermodynamics, and molecular control (Watson et al., 2019, Yang, 9 Oct 2025, Mohite et al., 2 Nov 2025, Meneghello et al., 2017).

7. Unifying Principles, Practical Implications, and Current Challenges

The SOC framework is anchored by three unifying principles:

1. Duality of control and inference: Cost minimization, probabilistic inference, and occupancy measure optimization are formally equivalent under appropriate constructions (Watson et al., 2019, Holtorf et al., 2022).
2. Hierarchy of policy classes: All practical control or RL strategies fit into four universal meta-classes—policy-function approximations, cost-function approximations, value-function approximations, direct lookahead policies (Powell, 2019).
3. Algorithmic tractability: Sampling-based message passing, policy gradient flows, measure-LP relaxations, and neural FBSDE architectures enable efficient computation even in high dimensions (Zhou et al., 2023, Pereira et al., 2019).

Contemporary research continues to refine computational scalability, nonconvexity handling, distributional robustness, and principled regularization. Bayesian methods and dual inference-control perspectives are enabling steadier convergence, reduced reliance on heuristics, and robust performance under model uncertainty and noise (Watson et al., 2019). The framework’s flexibility allows systematic integration with reinforcement learning and data-driven approaches, further bridging the gap between theory and large-scale stochastic decision systems.