---
title: Density-Driven Optimal Control (D2OC)
url: https://www.emergentmind.com/topics/density-driven-optimal-control-d2oc
type: topic
---

# Density-Driven Optimal Control (D2OC)

Density-Driven Optimal Control (D2OC) is a class of optimal control methodologies that approach optimal policy synthesis by lifting nonlinear, stochastic, or mean-field system dynamics into the space of probability densities, rather than optimizing directly over trajectories or controls. By leveraging linear operator theory—chiefly the Perron-Frobenius (P-F) and Koopman operators—D2OC provides a principled, convex and often data-driven framework for solving controlled density evolution, including both pathwise and stationary objectives, safety constraints, and high-dimensional control synthesis, with algorithmic applicability to deterministic, stochastic, and hybrid dynamical systems.

## 1. Mathematical Foundations: System and Density Evolution

Density-Driven Optimal Control is fundamentally grounded in the evolution of probability densities governed by controlled dynamical systems. For a controlled diffusion process
\[
d x_t = f(x_t)\,dt + G(x_t)u_t\,dt + \sigma(x_t)\,dW_t
\]
the state density $\rho(x,t)$ evolves according to the Fokker-Planck (forward Kolmogorov) partial differential equation (PDE):
\[
\partial_t \rho + \nabla\cdot[-f_c \rho] + \frac{1}{2}\sum_{i,j} \partial_{x_ix_j}^2\left((\sigma \sigma^\top)_{ij}(x)\rho\right) = 0
\]
where $f_c(x) = f(x) + G(x)k(x)$ for a feedback law $u = k(x)$. The infinitesimal generator $A_{PF}^{f_c}$ expresses this linear evolution on densities:
\[
\partial_t \rho = A_{PF}^{f_c}[\rho]
\]
while the dual Koopman operator $A_K^{f_c}$ acts on observables $\phi(x)$ as:
\[
\partial_t \phi = A_K^{f_c}[\phi] = f_c\cdot \nabla \phi + \frac{1}{2}\operatorname{tr}[\sigma \sigma^\top \nabla^2 \phi]
\]
This operator-theoretic structure supports both the forward (density) and backward (value function) perspectives crucial to D2OC [2202.01387].

## 2. Convex Density Program: Infinite-Dimensional Formulation

D2OC reformulates the original stochastic optimal control problem (SOCP) as a convex optimization in the space of densities. For infinite-horizon running cost $\ell(x, u) = q(x) + r|u|^2$ and initial density $h_0$, the expected cost functional is:
\[
J = \int_{x} \left(q(x, k(x)) \rho(x)\right)dx, \qquad \rho(x) = \int_0^\infty [P_t^c h_0](x) dt
\]
Defining the flux variable $m(x) = \rho(x)k(x)$, the stationary P-F (or Liouville) equation is:
\[
A_{PF}^f[\rho] + \nabla \cdot [G(x)m(x)] = -h_0(x)
\]
The convex program becomes:
\[
\min_{\rho\geq 0, m} \int q(x)\rho(x) + r\frac{|m(x)|^2}{\rho(x)}dx 
\quad \text{subject to } A_{PF}^f[\rho] + \nabla \cdot (Gm) = -h_0
\]
This quadratic-over-linear control cost is jointly convex in $(\rho, m)$ for $\rho > 0$, and the PDE constraint is linear [2202.01387, 2210.00723, 2010.01742].

## 3. Data-Driven and Finite-Dimensional Approximation

Finite-dimensional approximation of the above infinite program is achieved by projecting densities and fluxes onto a dictionary of nonnegative basis functions (e.g. Gaussian radial basis or polynomials):
\[
\rho(x) \approx \Psi(x)^\top v,\quad m(x) \approx \Psi(x)^\top w,\quad h_0(x) \approx \Psi(x)^\top h
\]
The infinitesimal P-F and Koopman operators are identified from data by methods such as extended dynamic mode decomposition (EDMD) or naturally structured DMD (NSDMD). Operator learning exploits time-series data from uncontrolled and controlled system simulations:
\[
P \approx \exp(\Delta t A_{PF}^f),\qquad A_{PF}^f \approx (P-I)/\Delta t
\]
The discrete convex quadratic program then reads:
\[
\min_{v\geq 0, w} d^\top v + rw^\top D w
\quad \text{subject to } -(A_f v + A_G w) = h
\]
with $D = \int \Psi\Psi^\top dx$, $d = \int q(x)\Psi(x)dx$. This is solvable by standard convex solvers (e.g. CVX), with the feedback law recovered as $k^*(x) = [\Psi(x)^\top w^*]/[\Psi(x)^\top v^*]$ [2202.01387, 2210.00723, 2010.01742].

## 4. Duality: Koopman-HJB Formulation and Policy Iteration

The density-driven convex program is dual to a value-function approach posed in the space of observables. The corresponding Hamilton-Jacobi-Bellman (HJB) PDE, with generator $A_K^{f+G u}$, is:
\[
\sup_{u}\left[ A_K^{f+G u}V(x) + q(x) + r|u|^2 \right] = 0
\]
At the optimum $u = -\frac{1}{2r} G^\top \nabla V$, giving a closed-loop operator $A_K^{f+G k}$. Policy iteration proceeds by alternately solving:
1. Policy evaluation: linear PDE for $V_k$ under fixed $k_k$
2. Policy improvement: update $k_{k+1}(x) = -\frac{1}{2r}G(x)^\top \nabla V_k(x)$

Koopman and P-F operators are adjoint, and under technical conditions, the optimal flux $m^* = \rho^* k^*$ and value function $V$ are related through Sen-Sen duality [2202.01387].

## 5. Extensions: Constraints, Safety, and Dual Density-Driven Structures

The density-driven approach natively accommodates state and input constraints via linear or convex restrictions in density space:
- Hard state constraints: $\int_{X_u}\rho(x) dx = 0$ for obstacle avoidance
- Traversability or safety budgets: $\int B(x)\rho(x)dx \leq \gamma$

Maximum-entropy variants add differential entropy regularization, producing Gaussian control policies and connecting D2OC to Schrödinger Bridges—entropy-optimal interpolating processes between marginals [2204.05263]. Extensions to hybrid jump-diffusions, mean-field limits, and PDE-constrained swarm control rely on generalized Chapman-Kolmogorov or Fokker-Planck type equations, with first-order optimality conditions derived via infinite-dimensional Pontryagin or minimum principle frameworks [2009.07154, 2104.06373].

## 6. Convergence, Global Optimality, and Practical Algorithms

Convexity of the lifted density-control cost ensures global optimality of the computed control law within the chosen function space. As the number of basis functions increases and operator approximations improve with data, the solution converges to the infinite-dimensional optimum [2202.01387, 2210.00723]. Standard convex programming complexity applies, dominated by quadratic program sizing. For high-dimensional or nonlinear systems, neural-network parameterizations and automatic-differentiation enable particle-based saddle-point solvers that bypass state-space gridding [2307.13135].

## 7. Applications and Numerical Demonstrations

D2OC frameworks have been validated on a range of systems:
- Nonlinear polynomial systems matching analytic HJB feedback [2202.01856]
- Navigating Dubins car models on off-road terrains with traversability and obstacle constraints [2210.00723]
- Large-scale particle swarms with boundary actuators via PDE-constrained nonlinear optimal control [2104.06373]
- Stochastic jump-diffusions with ensemble control [2009.07154]
- Schrödinger bridge and MaxEnt steering in discrete-time linear systems [2204.05263]
- Safe controller synthesis with distributional constraints for adaptive cruise control [1909.11798]

Each application exploits the ability to encode distributional performance, uncertainty, hard constraints, and scalability via the convex density-driven lifting, yielding significant advantages over traditional trajectory-based or merely value-function-based approaches.

---

**References**  
- "Data-Driven Stochastic Optimal Control using Linear Transfer Operators" [2202.01387]  
- "Maximum entropy optimal density control of discrete-time linear systems and Schrödinger bridges" [2204.05263]  
- "Density control of large-scale particles swarm through PDE-constrained optimization" [2104.06373]  
- "Data-Driven Convex Approach to Off-road Navigation via Linear Transfer Operators" [2210.00723]  
- "Data-Driven Optimal Control via Linear Transfer Operators: A Convex Approach" [2202.01856]  
- "A Convex Approach to Data-driven Optimal Control via Perron-Frobenius and Koopman Operators" [2010.01742]  
- "Open-loop Deterministic Density Control of Marked Jump Diffusions" [2009.07154]  
- "Optimal Safe Controller Synthesis: A Density Function Approach" [1909.11798]  
- "High-dimensional Optimal Density Control with Wasserstein Metric Matching" [2307.13135]

Source: https://www.emergentmind.com/topics/density-driven-optimal-control-d2oc