Differential Dynamic Programming

• Differential Dynamic Programming is a trajectory optimization method that constructs local quadratic approximations to minimize a nonlinear cost function.
• It iteratively refines control policies through backward Riccati-like recursions and forward simulations, ensuring fast convergence in high-dimensional systems.
• Recent extensions incorporate stochastic, robust, and constraint-handling approaches to improve performance under uncertainty and complex system dynamics.

Differential Dynamic Programming (DDP) is a trajectory optimization technique for solving nonlinear optimal control problems by iteratively constructing local quadratic approximations of the cost-to-go function along a nominal trajectory and updating state and control sequences through a backward–forward (Riccati-like) recursion. DDP leverages Taylor expansion–based local models of dynamics and cost, providing fast convergence rates and enabling real-time optimal control in complex high-dimensional systems such as robotics. Over the past decade, a broad spectrum of DDP extensions has advanced the field to encompass stochastic systems, partially observable planning, nonzero-sum dynamic games, robust optimization under uncertainty, high-dimensional constrained control, uncertainty-aware trajectory planning, and inverse optimality for model learning and imitation.

1. Core Principles and Local Quadratic Expansion

At its core, DDP seeks to minimize a discrete-time cost functional of the form

$$J(x_0, \mathcal{U}) = \sum_{k=0}^{N-1} \ell(x_k, u_k) + \phi(x_N)$$

subject to nonlinear dynamics $x_{k+1} = f(x_k, u_k)$. The method proceeds by iteratively updating a nominal trajectory via:

• Backward pass: Recursively expanding the Bellman equation

$$V_k(x_k) = \min_{u_k} [\ell(x_k, u_k) + V_{k+1}(f(x_k, u_k))]$$

with a second-order Taylor expansion around $(x_k^0, u_k^0)$, yielding quadratic models of the Q-function. The quadratic form is parameterized as

$$Q(\delta x, \delta u) = Q_0 + Q_x^\top \delta x + Q_u^\top \delta u + \frac{1}{2} \begin{bmatrix} \delta x \ \delta u \end{bmatrix}^\top \begin{bmatrix} Q_{xx} & Q_{xu} \ Q_{ux} & Q_{uu} \end{bmatrix} \begin{bmatrix} \delta x \ \delta u \end{bmatrix}$$

with update laws

$$\delta u^* = k + K \delta x, \quad k = -Q_{uu}^{-1} Q_u, \quad K = -Q_{uu}^{-1} Q_{ux}$$

• Forward pass: Simulate the system under updated controls

$$u_k^{\text{new}} = u_k^0 + \alpha k_k + K_k (x_k^{\text{new}} - x_k^0), \quad x_{k+1}^{\text{new}} = f(x_k^{\text{new}}, u_k^{\text{new}})$$

and line search on $\alpha \in (0,1]$.

The recursion advances from $k = N$ to $0$, then propagates a new trajectory forward in time.

Local quadraticization, state–control feedback updates, and trajectory rollout are at the heart of DDP’s strong convergence and practical efficiency (Fan et al., 2017).

2. Generalizations: Delays, Uncertainty, and Robustness

Time-Delayed Dynamics

Classic DDP assumes Markovian, first-order recurrences. For time-delayed systems, the state update depends on the full history over k time steps: $x_{i+1} = f(x_i, x_{i-1}, \ldots, x_{i-k}, u_i)$ This is addressed by defining an augmented state vector $\bar{x}_i$ and applying the quadratic expansion and Bellman recursion in this higher-dimensional space. All derivative tensors (first and second order) are computed jointly over present and delayed states, and the Q-function expansion generalizes accordingly. Feedback control includes gains for each delayed state variable, and computational overhead scales as $\mathcal{O}(k^2)$ for delay order k (Fan et al., 2017).

Stochastic Systems and Covariance Control

The stochastic DDP (SDDP) framework performs a similar expansion around a nominal trajectory but now accounts for system noise, typically modeled as

$x(t+dt) = f(x, u) dt + F(x, u) dW_t$

Derivatives of the value function propagate not only the mean but also higher-order moments. In covariance steering, terminal constraints on the mean and covariance are imposed using Lagrange multipliers. A primal–dual dual-ascent algorithm alternates between SDDP updates (for a fixed multiplier pair) and gradient ascent on the multipliers, allowing regulation of both mean and covariance to a target at horizon (Yi et al., 2019).

Robust DDP

Robust DDP addresses adversarial or unknown disturbances by solving a minimax Bellman problem: $$V_k(x) = \min_{u_k} \max_{w_k \in \mathcal{W}_k} [\ell(x, u, w) + V_{k+1}(f(x, u, w))]$$ Convex relaxations (e.g., S-procedure, multiplier relaxations) and generalized plant representations translate the maximization over disturbances into matrix inequalities at each DDP iteration. The recursion yields affine state feedback policies guaranteeing upper bounds on the worst-case cost, and robust DDP displays numerical tractability for nonlinear planning under uncertainty (Gramlich et al., 2022).

3. Constraints and Interior Point Approaches

Primal–Dual Interior Point Differential Dynamic Programming

Constraints (both equality and inequality) are directly incorporated via barrier and slack variables:

• Inequality constraints $c(x, u) \leq 0$ are recast as $c(x, u) + s = 0$, $s > 0$, and penalized by barrier terms $-\tau \sum_i \log s_i$
• The cost is augmented, and the Bellman equation is solved for the barrier problem at fixed $\tau$
• The backward pass of DDP involves inexact Newton steps for state, control, multipliers, and slacks

$$\begin{bmatrix} Q_{uu}^t & Q_{us}^t \ S_t Q_{su}^t & C_t \end{bmatrix} \begin{bmatrix} \delta u \ \delta s \end{bmatrix} = -\begin{bmatrix} Q_u^t \ r_t \end{bmatrix} - \begin{bmatrix} Q_{ux}^t \ S_t Q_{sx}^t \end{bmatrix} \delta x$$

where $S_t = \text{diag}(s_t)$, $C_t = \text{diag}(c(x_t, u_t))$, and $r_t = S_t c(x_t, u_t) + \mu$ (Pavlov et al., 2020, Prabhu et al., 18 Sep 2024). Both feasible and infeasible variants (with additional slack variables) are possible. The iterates satisfy perturbed KKT conditions and, as $\mu \to 0$, converge locally quadratically to the true constrained optimum.

Augmented Lagrangian, Active Set, and Filter Methods

Augmented Lagrangian methods add penalty-Lagrangian terms to relax constraint violation and blend quadratic subproblem updates with incremental dual variable adjustment. Active set approaches guess active inequalities and enforce them as equalities in the expansion, but can struggle with combinatorial complexity in large or uncertain active sets. Modern DDP approaches such as IPDDP2 utilize a primal-dual interior point formulation with a filter linesearch to enforce both merit function decrease and constraint reduction, achieving robust, rapid convergence suitable for contact-implicit robotic trajectory optimization (Xu et al., 11 Apr 2025).

4. Non-Classical Problems: Partial Observability, Multi-Agent, and Parameterization

Planning in Belief Space

PODDP extends DDP to POMDPs with latent discrete states by propagating belief distributions and constructing a trajectory tree over possible observations. The backward pass includes derivatives of the belief update (with reparameterization to enforce positivity), computing feedback on both system state and latent belief variables. Hierarchical versions reduce exponential complexity by segmenting over coarse observation intervals (Qiu et al., 2019).

Dynamic Games and Nash-Optimality

For nonzero-sum, multi-agent systems, DDP generalizes to solve Bellman-type recursions for each agent's cost, recursively solving a local static quadratic game at each stage. Under invertibility of the players' Hessian block matrix, the unique Nash feedback is explicitly recovered. The DDP iterates are shown to be $O(\epsilon^2)$-close to a Newton step, and thus inherit local quadratic convergence to strict local Nash equilibrium (Di et al., 2018, Di et al., 2019).

Parameterized DDP

Parameterized DDP (PDDP) augments the optimization variables to include both time-varying controls and time-invariant parameters (e.g., unknown physical coefficients or switching times). The recursion is extended to compute and couple gradients across state, control, and parameter dimensions. The overall cost reduction per iteration is controlled, generalizing the classical Armijo reduction to multi-dimensional settings. Applications include joint adaptive MPC and moving horizon estimation, system identification, and optimal hybrid mode scheduling (Oshin et al., 2022).

5. Extensions: Exploration, Data-Driven, and Learning Frameworks

Maximum Entropy DDP

By regularizing the cost with an entropy term, DDP computes stochastic control distributions favoring exploration: $$J_\pi = \mathbb{E}_{u \sim \pi(\cdot|x)} [\Phi(x_T) + \sum_{t=0}^{T-1} (\ell(x_t, u_t) - \alpha H[\pi_t])]$$ The corresponding Bellman equation yields a softmax/Gibbs policy, and quadratic expansion recovers locally Gaussian policies with covariance proportional to the entropy weight. For nonconvex landscapes, multimodal (mixture) Gaussian extensions support explicit compositionality and improved escape from local minima (So et al., 2021).

Distributionally Robust DDP via Wasserstein Ambiguity

Advancing robust DDP to data-driven uncertainty quantification, distributionally robust DDP (DR-DDP) constructs a Wasserstein ambiguity set around empirical disturbance samples, reformulates the value function via Kantorovich duality, and derives explicit closed-form policies for both control and worst-case disturbance. This avoids heavy minimax optimization yet ensures robust out-of-sample performance and scalability in high-dimensional problems such as coupled oscillator networks and autonomous navigation (Hakobyan et al., 2023).

Differentiable Optimal Control via DDP

When DDP is used as the inner optimizer for inverse optimal control, model learning, or design optimization, efficient sensitivity computations (incorporating all second-order derivatives of dynamics and cost) are essential. Differentiable DDP leverages backpropagation through the full value and Q-function recursion, recovering the gradients needed for bilevel learning and ensuring that parameter updates converge even in highly nonlinear or underactuated designs (Dinev et al., 2022).

Inverse Reinforcement Learning via DDP

A DDP-based framework for inverse reinforcement learning (IRL) enables joint identification of cost, dynamical, and constraint parameters from demonstrations by differentiating DDP recursions with respect to these parameters. Closed-loop loss formulations matching feedback gains between demonstration and learned policies outperform standard open-loop trajectory losses, yielding robust parameter recovery even in the presence of feedback-structured demonstration noise. The approach is validated in robotic and aerial domains, with rank conditions guaranteeing identifiability under data richness assumptions (Cao et al., 29 Jul 2024).

6. Applications and Impact

DDP and its extensions are prominent in control and robotics for trajectory optimization, dynamic games (multi-robot and economic systems), planning under uncertainty, partially observable decision making, high-dimensional constrained motion (e.g., humanoid contact-rich tasks), stochastic and robust control, parameter learning, system identification, and optimal policy inference from demonstration.

Experiments consistently demonstrate DDP’s advantages in:

• Enabling real-time MPC with nonlinear dynamics and constraints
• Achieving fast, locally quadratic convergence (often reducing CPU times by orders of magnitude compared to full stochastic dynamic programming or nonlinear programming approaches)
• Integrating uncertainty sets (stochastic, robust, distributional) directly into planning and feedback synthesis
• Handling complex constraints (contact, collision, actuation, hybrid system logic) systematically via barrier, interior-point, or augmented Lagrangian methods
• Providing global convergence mechanisms (filter line-search, proximal terms) and reliable feasibility maintenance.

Statistical and numerical evidence from multi-contact walking, trajectory planning with obstacles, closed-loop IRL in flying robots, and distributionally robust navigation under ambiguous disturbances highlights both empirical effectiveness and computational tractability (Fan et al., 2017, Di et al., 2018, Budhiraja et al., 2019, Yi et al., 2019, Pavlov et al., 2020, Gramlich et al., 2022, Hakobyan et al., 2023, Cao et al., 29 Jul 2024, Xu et al., 11 Apr 2025).

7. Contemporary Challenges and Future Directions

While DDP has achieved wide applicability, current research addresses:

• Scalability to extremely high-dimensional systems and multi-modal uncertainty representation
• Integration with end-to-end differentiable optimization pipelines, particularly for model-based learning and co-design
• Extension to fully general nonlinear complementarity and contact-implicit formulations
• Efficient globalization strategies for highly nonconvex landscapes (combining stochastic search, mixture policies, and compositionality)
• Unification of DDP-based sensitivity analysis with reinforcement learning and adaptive optimal control in black-box and partially observed environments

Ongoing work explores hybridization with direct multiple-shooting methods, more sophisticated treatment of equality and inequality constraints, and more general uncertain, time-varying, and hybrid dynamical settings. The continued convergence between DDP and robust, distributionally informed, and learning-based approaches is central to the progress in both theoretical optimal control and practical robotics.