---
title: Differential Dynamic Programming (DDP)
url: https://www.emergentmind.com/topics/differential-dynamic-programming-ddp
type: topic
---

# Differential Dynamic Programming (DDP)

Differential Dynamic Programming (DDP) is a second-order, Bellman-based, Newton-type method for finite-horizon optimal control. In its standard discrete-time form, it solves nonlinear problems with dynamics \(x_{k+1}=f(x_k,u_k)\) and objective \(\ell_N(x_N)+\sum_{k=0}^{N-1}\ell_k(x_k,u_k)\) by iteratively improving a nominal state–control trajectory through local quadratic approximations of the Bellman recursion, a backward pass that computes affine local feedback laws, and a forward rollout that updates the trajectory [2211.12159] [2104.04044] [2010.00411].

## 1. Core formulation and dynamic-programming structure

A standard finite-horizon DDP problem is written as
\[
\min_{\{x_k,u_k\}} \; \ell_N(x_N)+\sum_{k=0}^{N-1}\ell_k(x_k,u_k)
\quad\text{s.t.}\quad
x_{k+1}=f(x_k,u_k),
\]
with an initial condition \(x_0=\tilde x_0\). The value function is
\[
V_k(x_k)=\min_{\{u_j\}_{j=k}^{N-1}}
\Big[\ell_N(x_N)+\sum_{j=k}^{N-1}\ell_j(x_j,u_j)\Big],
\]
and the Bellman recursion takes the form
\[
V_k(x_k)=\min_{u_k}\big[\ell_k(x_k,u_k)+V_{k+1}(f(x_k,u_k))\big].
\]
The corresponding local action-value function is
\[
Q_k(x_k,u_k)=\ell_k(x_k,u_k)+V_{k+1}(f(x_k,u_k)).
\]

The distinctive feature of DDP is that it preserves the dynamic-programming structure without tabulating the value function on a grid. Instead, it operates around a nominal trajectory \(\{(\bar x_k,\bar u_k)\}\) and treats the Bellman recursion locally. This contrasts with standard Dynamic Programming or Stochastic Dynamic Programming on discretized state grids, which incur substantial memory and computation costs and are subject to the curse of dimensionality [2211.12159] [2005.00985].

In continuous time, the same logic can be expressed through a local expansion of the Hamilton–Jacobi–Bellman equation. This viewpoint becomes especially important in infinite-dimensional generalizations, where the state can be a field rather than a finite-dimensional vector [2104.04044].

## 2. Local quadratic models, backward recursion, and feedback law

DDP introduces perturbations
\[
\delta x_k = x_k-\bar x_k,\qquad \delta u_k=u_k-\bar u_k,
\]
and approximates the next-step value function by a quadratic form,
\[
V_{k+1}(x_{k+1}) \approx V_{k+1}^0 + V_{x,k+1}^\top \delta x_{k+1}
+ \tfrac12 \delta x_{k+1}^\top V_{xx,k+1}\delta x_{k+1}.
\]
The dynamics are linearized as
\[
\delta x_{k+1}\approx f_{x,k}\delta x_k+f_{u,k}\delta u_k,
\]
and the local Q-function is quadratically expanded in \((\delta x_k,\delta u_k)\).

Minimizing that quadratic model with respect to \(\delta u_k\) yields the affine local control law
\[
\delta u_k^* = k_k + K_k\,\delta x_k,
\qquad
k_k=-Q_{uu,k}^{-1}Q_{u,k},
\qquad
K_k=-Q_{uu,k}^{-1}Q_{ux,k}.
\]
Substituting this law back into the local model gives a new quadratic approximation of \(V_k\), which defines the backward pass. The forward pass then rolls out
\[
\hat u_k=\bar u_k+\alpha k_k+K_k(\hat x_k-\bar x_k),
\qquad
\hat x_{k+1}=f(\hat x_k,\hat u_k),
\]
with a line-search parameter \(\alpha\in(0,1]\) to improve the nominal trajectory [2010.00411] [2104.04044].

Regularization is central. In practice, \(Q_{uu}\) must be positive definite, so DDP implementations add Levenberg–Marquardt-type terms to \(Q_{uu}\) or \(V_{xx}\) to stabilize the backward pass and interpolate between Newton-like and gradient-like behavior when necessary [2010.00411].

A standard distinction is between full DDP and iLQR. Full DDP includes second derivatives of the dynamics \(f_{xx}, f_{ux}, f_{uu}\) in the local model; iLQR linearizes the dynamics and quadratizes only the cost. In many robotics applications, “DDP” and “iLQR” are used almost interchangeably, but strictly speaking iLQR is the first-order-dynamics specialization, whereas DDP retains second-order dynamics terms [2006.08102] [1904.05072].

## 3. Extensions beyond fully observed deterministic finite-dimensional systems

The basic recursion has been extended in several directions. For systems with multiple state delays,
\[
x_{i+1}=f(x_i,x_{i-1},\dots,x_{i-k},u_i),
\]
DDP can be reformulated on the delayed state segment \(\bar x_i=\{x_i,\dots,x_{i-k}\}\). The local feedback law then depends on the current state and the last \(k\) states,
\[
\delta u^*=\mathbf{k}+\sum_{j=0}^k \mathbf{K}_j\,\delta x^j,
\]
and the per-iteration complexity becomes \(\mathcal{O}(k^2 N n^3)\) [1701.01882].

For partially observable problems with uncertainty over a discrete latent variable \(z\), PODDP augments the continuous state \(x\) with a belief parameter \(\beta\) such that
\[
b(z;\beta)=\frac{\exp(\beta(z))}{\sum_{z'}\exp(\beta(z'))}.
\]
The planning state becomes \(\langle x,\beta\rangle\), and dynamic programming is performed over a branching observation tree. The Q-function at each node is a belief-weighted sum over latent modes, which produces a contingency policy rather than a single nominal trajectory. This formulation is used for uncertainty over reward structure, dynamics modes, and interactive intentions [1912.06787].

For distributional uncertainty in stochastic disturbances, Distributionally Robust DDP embeds Wasserstein Distributionally Robust Control into DDP. The disturbance law is assumed unknown, but samples define an empirical distribution \(\mathbb Q_t\), and robustness is imposed through a Wasserstein penalty. Using Kantorovich duality, the Bellman update is decomposed into sample-wise worst-case Q-functions, enabling closed-form local robust control laws and closed-form worst-case disturbance policies without numerically solving inner minimax problems. One iteration has complexity
\[
O\big(T(n_x^3+n_u^3+(N+n_w)n_w^2)\big)
\]
[2305.09760].

At the infinite-dimensional end, Spatio-Temporal DDP develops DDP for nonlinear PDE systems in Hilbert spaces, covering both distributed and boundary control. The value functional is expanded with operator-valued first and second derivatives, the backward recursion becomes a system of Riccati-like PDEs, and the resulting STDDP framework generalizes both finite-dimensional DDP and spatio-temporal LQR. A proof of global convergence is given for the continuous-time forward–backward equations [2104.04044].

## 4. Constraints, feasibility restoration, and augmented-Lagrangian formulations

A large part of the modern DDP literature addresses the fact that classical DDP is fundamentally unconstrained. One route is to impose constraints directly in the stage-wise Bellman minimization. “Constrained Differential Dynamic Programming Revisited” introduces slack variables \(s_{i,k}\ge 0\) for path inequalities, complementarity conditions \(s_{i,k}\lambda_{i,k}=0\), and a stage-wise KKT system in \((\delta u_k,\delta s_k,\delta\lambda_k)\). The same work also proposes a smooth augmented-Lagrangian penalty \(\mathcal P(\lambda,\mu,g)\) chosen to preserve second-order differentiability, so that DDP’s quadratic expansions remain valid [2005.00985].

A more systematic construction is the primal–dual augmented Lagrangian approach. There the stage Bellman subproblem is replaced by a primal–dual augmented Q-function, and the backward pass solves a regularized KKT system whose saddle-point matrix contains the dynamics Jacobians, active inequality Jacobians, and the negative definite blocks \(-\mu_e I\) and \(-\mu_i I\). The resulting algorithm is explicitly described as Newton-like, and it accepts both equality and inequality constraints without switching to a different strategy near the solution [2210.15409].

Another line of work is feasibility-driven DDP. BOX-FDDP introduces explicit dynamic gaps
\[
\bar f_{k+1}=f(x_k,u_k)\ominus x_{k+1}
\]
and treats dynamic feasibility as a first-class quantity. Its backward pass relinearizes the value gradient by these gaps, and its forward pass contracts them according to \(\bar f_k\leftarrow(1-\alpha)\bar f_k\). The algorithm alternates between a feasibility-driven mode, which ignores box control bounds in the backward pass to restore dynamic feasibility, and a control-bounded mode, which solves box-constrained QPs once the trajectory is dynamically feasible. The dynamic feasibility error decreases monotonically, emulating direct multiple shooting while retaining DDP’s temporal structure [2010.00411].

A structurally different application appears in polynomial trajectory generation for differentially flat systems. DIRECT reformulates multiple polynomial segments as a discrete-time finite-horizon control system with fixed state dimension, so continuity conditions are automatically enforced by the state update. With inequality constraints, the resulting problem is solved by an interior-point DDP method; with fixed time and no constraints, it reduces exactly to a linear-quadratic tracking problem, so the solution is obtained in one DDP iteration [2109.04686].

## 5. Hybrid systems, impacts, contacts, and multiple shooting

DDP has also been extended to hybrid and contact-rich systems. In Hybrid Systems DDP, the continuous dynamics are supplemented by state-based switching, reset maps, and impact events. At impacts, the backward pass uses
\[
V(x^-)=\Phi(x^-)+V(x^+),
\]
and the value derivatives are propagated through the reset map derivative \(P_x\). The same framework adds an Augmented Lagrangian for switching constraints, a Relaxed Barrier method for inequality constraints such as torque and friction limits, and a Switching Time Optimization algorithm that updates mode durations by using value derivatives with respect to a timing state [2006.08102].

Multiple-shooting variants address the sensitivity of single-shooting DDP to initialization. A unified multiple-shooting perspective introduces explicit defects
\[
d_{k+1}=f(x_k,u_k)-x_{k+1}
\]
at shooting nodes and modifies the dynamics perturbation in the backward pass to account for them. This adds defect terms to the Q-function derivatives, yields an exact expected cost-change model, and supports an adaptive merit function together with an Armijo-type acceptance condition. The same derivation recovers SS-DDP, SS-iLQR, MS-iLQR, and MS-DDP as special cases; with second-order dynamics and nonlinear rollout, local quadratic convergence is retained [2309.07872].

For hybrid legged systems, Multi-Shooting DDP has been combined with analytical first- and second-order derivatives of rigid-body contact dynamics written in KKT form. The same work also proposes a Quasi-Newton approximation of the second-order state-transition information. In the reported bounding-gait example, that Quasi-Newton construction gives order-of-magnitude speedups in convergence compared with recomputing full second-order dynamics at every iteration [2307.12606].

A related rigid-contact formulation embeds the KKT contact model directly into the dynamics, so that accelerations and contact forces are obtained from
\[
\begin{bmatrix} M & J_c^\top \\ J_c & 0 \end{bmatrix}
\begin{bmatrix} a^* \\ \lambda \end{bmatrix}
=
\begin{bmatrix} \tau_b \\ -\dot J_c v \end{bmatrix}.
\]
In that setting, contact forces become implicit functions \(\lambda=g(x,u)\), and DDP optimizes whole-body motion with multi-phase rigid contacts while exploiting angular momentum to reduce forces and impacts [1904.05072].

## 6. Special cases, applications, and reported computational behavior

Several recent studies make the computational profile of DDP unusually explicit.

| Setting | Reported result | Citation |
|---|---|---|
| Stochastic GLOSA with uncertain signal switching time | DDP converges in 3–4 iterations, with CPU times \(0.0005\)–\(0.0007\) s per scenario; the same paper reports 631 s for SDP and 0.692–1.246 s for DDDP in the listed scenarios | [2211.12159] |
| Wasserstein distributionally robust car navigation | Out-of-sample cost: DR-DDP 172.8, GT-DDP 201.7, box-DDP 235.2; DR-DDP and GT-DDP both take about 25 seconds for 1000 simulations | [2305.09760] |
| Switching-time optimization for quadruped bounding | HS-DDP reduces total cycle time to \(0.2335\,\text{s}\), a 21.1% reduction; the comparison method reaches only 6.3% under the same iteration budget | [2006.08102] |
| Feasibility-driven MPC for ANYmal C | The MPC example runs at \(>100\) Hz, and Box-FDDP solves each OC problem in \(\sim 10\) ms without code generation | [2010.00411] |
| Polynomial trajectory generation for quadrotors | DIRECT-based algorithms have linear complexity with respect to the number of segments, and the fixed-time unconstrained case is solved by an LQT recursion | [2109.04686] |

These results show that DDP is not a single algorithm but a family of structure-exploiting second-order methods whose central Bellman recursion survives substantial changes in modeling assumptions. In special cases it reduces to classical linear-quadratic machinery: the finite-dimensional fixed-time unconstrained trajectory-generation problem in DIRECT is a linear-quadratic tracking problem, and the infinite-dimensional STDDP framework reduces to spatio-temporal LQR for linear PDEs with quadratic cost [2109.04686] [2104.04044].

A plausible implication is that the enduring importance of DDP lies less in any single formulation than in the stability of its local dynamic-programming template: quadratic value approximations, affine feedback laws, and forward–backward sweeps continue to organize methods for delayed dynamics, belief-space planning, distributional robustness, hybrid systems, rigid contact, multiple shooting, and constrained trajectory optimization.

Source: https://www.emergentmind.com/topics/differential-dynamic-programming-ddp