Differential Dynamic Programming (DDP)

• DDP is a second-order optimal control method that uses local quadratic approximations of the Bellman recursion to iteratively improve a nominal state-control trajectory.
• It computes affine feedback laws through a backward pass and updates trajectories via a forward rollout, incorporating regularization for robust performance.
• DDP has versatile extensions such as iLQR, constraint handling, hybrid systems, and distributionally robust control, making it highly applicable in robotics and advanced control scenarios.

Differential Dynamic Programming (DDP) is a second-order, Bellman-based, Newton-type method for finite-horizon optimal control. In its standard discrete-time form, it solves nonlinear problems with dynamics $x_{k+1}=f(x_k,u_k)$ and objective $\ell_N(x_N)+\sum_{k=0}^{N-1}\ell_k(x_k,u_k)$ by iteratively improving a nominal state–control trajectory through local quadratic approximations of the Bellman recursion, a backward pass that computes affine local feedback laws, and a forward rollout that updates the trajectory (Typaldos et al., 2022, Evans et al., 2021, Mastalli et al., 2020).

1. Core formulation and dynamic-programming structure

A standard finite-horizon DDP problem is written as

$$\min_{\{x_k,u_k\}} \; \ell_N(x_N)+\sum_{k=0}^{N-1}\ell_k(x_k,u_k) \quad\text{s.t.}\quad x_{k+1}=f(x_k,u_k),$$

with an initial condition $x_0=\tilde x_0$. The value function is

$$V_k(x_k)=\min_{\{u_j\}_{j=k}^{N-1}} \Big[\ell_N(x_N)+\sum_{j=k}^{N-1}\ell_j(x_j,u_j)\Big],$$

and the Bellman recursion takes the form

$$V_k(x_k)=\min_{u_k}\big[\ell_k(x_k,u_k)+V_{k+1}(f(x_k,u_k))\big].$$

The corresponding local action-value function is

$Q_k(x_k,u_k)=\ell_k(x_k,u_k)+V_{k+1}(f(x_k,u_k)).$

The distinctive feature of DDP is that it preserves the dynamic-programming structure without tabulating the value function on a grid. Instead, it operates around a nominal trajectory $\{(\bar x_k,\bar u_k)\}$ and treats the Bellman recursion locally. This contrasts with standard Dynamic Programming or Stochastic Dynamic Programming on discretized state grids, which incur substantial memory and computation costs and are subject to the curse of dimensionality (Typaldos et al., 2022, Aoyama et al., 2020).

In continuous time, the same logic can be expressed through a local expansion of the Hamilton–Jacobi–Bellman equation. This viewpoint becomes especially important in infinite-dimensional generalizations, where the state can be a field rather than a finite-dimensional vector (Evans et al., 2021).

2. Local quadratic models, backward recursion, and feedback law

DDP introduces perturbations

$$\delta x_k = x_k-\bar x_k,\qquad \delta u_k=u_k-\bar u_k,$$

and approximates the next-step value function by a quadratic form,

$$V_{k+1}(x_{k+1}) \approx V_{k+1}^0 + V_{x,k+1}^\top \delta x_{k+1} + \tfrac12 \delta x_{k+1}^\top V_{xx,k+1}\delta x_{k+1}.$$

The dynamics are linearized as

$\ell_N(x_N)+\sum_{k=0}^{N-1}\ell_k(x_k,u_k)$0

and the local Q-function is quadratically expanded in $\ell_N(x_N)+\sum_{k=0}^{N-1}\ell_k(x_k,u_k)$1.

Minimizing that quadratic model with respect to $\ell_N(x_N)+\sum_{k=0}^{N-1}\ell_k(x_k,u_k)$2 yields the affine local control law

$\ell_N(x_N)+\sum_{k=0}^{N-1}\ell_k(x_k,u_k)$3

Substituting this law back into the local model gives a new quadratic approximation of $\ell_N(x_N)+\sum_{k=0}^{N-1}\ell_k(x_k,u_k)$4, which defines the backward pass. The forward pass then rolls out

$\ell_N(x_N)+\sum_{k=0}^{N-1}\ell_k(x_k,u_k)$5

with a line-search parameter $\ell_N(x_N)+\sum_{k=0}^{N-1}\ell_k(x_k,u_k)$6 to improve the nominal trajectory (Mastalli et al., 2020, Evans et al., 2021).

Regularization is central. In practice, $\ell_N(x_N)+\sum_{k=0}^{N-1}\ell_k(x_k,u_k)$7 must be positive definite, so DDP implementations add Levenberg–Marquardt-type terms to $\ell_N(x_N)+\sum_{k=0}^{N-1}\ell_k(x_k,u_k)$8 or $\ell_N(x_N)+\sum_{k=0}^{N-1}\ell_k(x_k,u_k)$9 to stabilize the backward pass and interpolate between Newton-like and gradient-like behavior when necessary (Mastalli et al., 2020).

A standard distinction is between full DDP and iLQR. Full DDP includes second derivatives of the dynamics $$\min_{\{x_k,u_k\}} \; \ell_N(x_N)+\sum_{k=0}^{N-1}\ell_k(x_k,u_k) \quad\text{s.t.}\quad x_{k+1}=f(x_k,u_k),$$0 in the local model; iLQR linearizes the dynamics and quadratizes only the cost. In many robotics applications, “DDP” and “iLQR” are used almost interchangeably, but strictly speaking iLQR is the first-order-dynamics specialization, whereas DDP retains second-order dynamics terms (Li et al., 2020, Budhiraja et al., 2019).

3. Extensions beyond fully observed deterministic finite-dimensional systems

The basic recursion has been extended in several directions. For systems with multiple state delays,

$$\min_{\{x_k,u_k\}} \; \ell_N(x_N)+\sum_{k=0}^{N-1}\ell_k(x_k,u_k) \quad\text{s.t.}\quad x_{k+1}=f(x_k,u_k),$$1

DDP can be reformulated on the delayed state segment $$\min_{\{x_k,u_k\}} \; \ell_N(x_N)+\sum_{k=0}^{N-1}\ell_k(x_k,u_k) \quad\text{s.t.}\quad x_{k+1}=f(x_k,u_k),$$2. The local feedback law then depends on the current state and the last $$\min_{\{x_k,u_k\}} \; \ell_N(x_N)+\sum_{k=0}^{N-1}\ell_k(x_k,u_k) \quad\text{s.t.}\quad x_{k+1}=f(x_k,u_k),$$3 states,

$$\min_{\{x_k,u_k\}} \; \ell_N(x_N)+\sum_{k=0}^{N-1}\ell_k(x_k,u_k) \quad\text{s.t.}\quad x_{k+1}=f(x_k,u_k),$$4

and the per-iteration complexity becomes $$\min_{\{x_k,u_k\}} \; \ell_N(x_N)+\sum_{k=0}^{N-1}\ell_k(x_k,u_k) \quad\text{s.t.}\quad x_{k+1}=f(x_k,u_k),$$5 (Fan et al., 2017).

For partially observable problems with uncertainty over a discrete latent variable $$\min_{\{x_k,u_k\}} \; \ell_N(x_N)+\sum_{k=0}^{N-1}\ell_k(x_k,u_k) \quad\text{s.t.}\quad x_{k+1}=f(x_k,u_k),$$6, PODDP augments the continuous state $$\min_{\{x_k,u_k\}} \; \ell_N(x_N)+\sum_{k=0}^{N-1}\ell_k(x_k,u_k) \quad\text{s.t.}\quad x_{k+1}=f(x_k,u_k),$$7 with a belief parameter $$\min_{\{x_k,u_k\}} \; \ell_N(x_N)+\sum_{k=0}^{N-1}\ell_k(x_k,u_k) \quad\text{s.t.}\quad x_{k+1}=f(x_k,u_k),$$8 such that

$$\min_{\{x_k,u_k\}} \; \ell_N(x_N)+\sum_{k=0}^{N-1}\ell_k(x_k,u_k) \quad\text{s.t.}\quad x_{k+1}=f(x_k,u_k),$$9

The planning state becomes $x_0=\tilde x_0$0, and dynamic programming is performed over a branching observation tree. The Q-function at each node is a belief-weighted sum over latent modes, which produces a contingency policy rather than a single nominal trajectory. This formulation is used for uncertainty over reward structure, dynamics modes, and interactive intentions (Qiu et al., 2019).

For distributional uncertainty in stochastic disturbances, Distributionally Robust DDP embeds Wasserstein Distributionally Robust Control into DDP. The disturbance law is assumed unknown, but samples define an empirical distribution $x_0=\tilde x_0$1, and robustness is imposed through a Wasserstein penalty. Using Kantorovich duality, the Bellman update is decomposed into sample-wise worst-case Q-functions, enabling closed-form local robust control laws and closed-form worst-case disturbance policies without numerically solving inner minimax problems. One iteration has complexity

$x_0=\tilde x_0$2

(Hakobyan et al., 2023).

At the infinite-dimensional end, Spatio-Temporal DDP develops DDP for nonlinear PDE systems in Hilbert spaces, covering both distributed and boundary control. The value functional is expanded with operator-valued first and second derivatives, the backward recursion becomes a system of Riccati-like PDEs, and the resulting STDDP framework generalizes both finite-dimensional DDP and spatio-temporal LQR. A proof of global convergence is given for the continuous-time forward–backward equations (Evans et al., 2021).

4. Constraints, feasibility restoration, and augmented-Lagrangian formulations

A large part of the modern DDP literature addresses the fact that classical DDP is fundamentally unconstrained. One route is to impose constraints directly in the stage-wise Bellman minimization. “Constrained Differential Dynamic Programming Revisited” introduces slack variables $x_0=\tilde x_0$3 for path inequalities, complementarity conditions $x_0=\tilde x_0$4, and a stage-wise KKT system in $x_0=\tilde x_0$5. The same work also proposes a smooth augmented-Lagrangian penalty $x_0=\tilde x_0$6 chosen to preserve second-order differentiability, so that DDP’s quadratic expansions remain valid (Aoyama et al., 2020).

A more systematic construction is the primal–dual augmented Lagrangian approach. There the stage Bellman subproblem is replaced by a primal–dual augmented Q-function, and the backward pass solves a regularized KKT system whose saddle-point matrix contains the dynamics Jacobians, active inequality Jacobians, and the negative definite blocks $x_0=\tilde x_0$7 and $x_0=\tilde x_0$8. The resulting algorithm is explicitly described as Newton-like, and it accepts both equality and inequality constraints without switching to a different strategy near the solution (Jallet et al., 2022).

Another line of work is feasibility-driven DDP. BOX-FDDP introduces explicit dynamic gaps

$x_0=\tilde x_0$9

and treats dynamic feasibility as a first-class quantity. Its backward pass relinearizes the value gradient by these gaps, and its forward pass contracts them according to $$V_k(x_k)=\min_{\{u_j\}_{j=k}^{N-1}} \Big[\ell_N(x_N)+\sum_{j=k}^{N-1}\ell_j(x_j,u_j)\Big],$$0. The algorithm alternates between a feasibility-driven mode, which ignores box control bounds in the backward pass to restore dynamic feasibility, and a control-bounded mode, which solves box-constrained QPs once the trajectory is dynamically feasible. The dynamic feasibility error decreases monotonically, emulating direct multiple shooting while retaining DDP’s temporal structure (Mastalli et al., 2020).

A structurally different application appears in polynomial trajectory generation for differentially flat systems. DIRECT reformulates multiple polynomial segments as a discrete-time finite-horizon control system with fixed state dimension, so continuity conditions are automatically enforced by the state update. With inequality constraints, the resulting problem is solved by an interior-point DDP method; with fixed time and no constraints, it reduces exactly to a linear-quadratic tracking problem, so the solution is obtained in one DDP iteration (Cao et al., 2021).

5. Hybrid systems, impacts, contacts, and multiple shooting

DDP has also been extended to hybrid and contact-rich systems. In Hybrid Systems DDP, the continuous dynamics are supplemented by state-based switching, reset maps, and impact events. At impacts, the backward pass uses

$$V_k(x_k)=\min_{\{u_j\}_{j=k}^{N-1}} \Big[\ell_N(x_N)+\sum_{j=k}^{N-1}\ell_j(x_j,u_j)\Big],$$1

and the value derivatives are propagated through the reset map derivative $$V_k(x_k)=\min_{\{u_j\}_{j=k}^{N-1}} \Big[\ell_N(x_N)+\sum_{j=k}^{N-1}\ell_j(x_j,u_j)\Big],$$2. The same framework adds an Augmented Lagrangian for switching constraints, a Relaxed Barrier method for inequality constraints such as torque and friction limits, and a Switching Time Optimization algorithm that updates mode durations by using value derivatives with respect to a timing state (Li et al., 2020).

Multiple-shooting variants address the sensitivity of single-shooting DDP to initialization. A unified multiple-shooting perspective introduces explicit defects

$$V_k(x_k)=\min_{\{u_j\}_{j=k}^{N-1}} \Big[\ell_N(x_N)+\sum_{j=k}^{N-1}\ell_j(x_j,u_j)\Big],$$3

at shooting nodes and modifies the dynamics perturbation in the backward pass to account for them. This adds defect terms to the Q-function derivatives, yields an exact expected cost-change model, and supports an adaptive merit function together with an Armijo-type acceptance condition. The same derivation recovers SS-DDP, SS-iLQR, MS-iLQR, and MS-DDP as special cases; with second-order dynamics and nonlinear rollout, local quadratic convergence is retained (Li et al., 2023).

For hybrid legged systems, Multi-Shooting DDP has been combined with analytical first- and second-order derivatives of rigid-body contact dynamics written in KKT form. The same work also proposes a Quasi-Newton approximation of the second-order state-transition information. In the reported bounding-gait example, that Quasi-Newton construction gives order-of-magnitude speedups in convergence compared with recomputing full second-order dynamics at every iteration (Singh et al., 2023).

A related rigid-contact formulation embeds the KKT contact model directly into the dynamics, so that accelerations and contact forces are obtained from

$$V_k(x_k)=\min_{\{u_j\}_{j=k}^{N-1}} \Big[\ell_N(x_N)+\sum_{j=k}^{N-1}\ell_j(x_j,u_j)\Big],$$4

In that setting, contact forces become implicit functions $$V_k(x_k)=\min_{\{u_j\}_{j=k}^{N-1}} \Big[\ell_N(x_N)+\sum_{j=k}^{N-1}\ell_j(x_j,u_j)\Big],$$5, and DDP optimizes whole-body motion with multi-phase rigid contacts while exploiting angular momentum to reduce forces and impacts (Budhiraja et al., 2019).

6. Special cases, applications, and reported computational behavior

Several recent studies make the computational profile of DDP unusually explicit.

Setting	Reported result	Citation
Stochastic GLOSA with uncertain signal switching time	DDP converges in 3–4 iterations, with CPU times $$V_k(x_k)=\min_{\{u_j\}_{j=k}^{N-1}} \Big[\ell_N(x_N)+\sum_{j=k}^{N-1}\ell_j(x_j,u_j)\Big],$$6–$$V_k(x_k)=\min_{\{u_j\}_{j=k}^{N-1}} \Big[\ell_N(x_N)+\sum_{j=k}^{N-1}\ell_j(x_j,u_j)\Big],$$7 s per scenario; the same paper reports 631 s for SDP and 0.692–1.246 s for DDDP in the listed scenarios	(Typaldos et al., 2022)
Wasserstein distributionally robust car navigation	Out-of-sample cost: DR-DDP 172.8, GT-DDP 201.7, box-DDP 235.2; DR-DDP and GT-DDP both take about 25 seconds for 1000 simulations	(Hakobyan et al., 2023)
Switching-time optimization for quadruped bounding	HS-DDP reduces total cycle time to $$V_k(x_k)=\min_{\{u_j\}_{j=k}^{N-1}} \Big[\ell_N(x_N)+\sum_{j=k}^{N-1}\ell_j(x_j,u_j)\Big],$$8, a 21.1% reduction; the comparison method reaches only 6.3% under the same iteration budget	(Li et al., 2020)
Feasibility-driven MPC for ANYmal C	The MPC example runs at $$V_k(x_k)=\min_{\{u_j\}_{j=k}^{N-1}} \Big[\ell_N(x_N)+\sum_{j=k}^{N-1}\ell_j(x_j,u_j)\Big],$$9 Hz, and Box-FDDP solves each OC problem in $$V_k(x_k)=\min_{u_k}\big[\ell_k(x_k,u_k)+V_{k+1}(f(x_k,u_k))\big].$$0 ms without code generation	(Mastalli et al., 2020)
Polynomial trajectory generation for quadrotors	DIRECT-based algorithms have linear complexity with respect to the number of segments, and the fixed-time unconstrained case is solved by an LQT recursion	(Cao et al., 2021)

These results show that DDP is not a single algorithm but a family of structure-exploiting second-order methods whose central Bellman recursion survives substantial changes in modeling assumptions. In special cases it reduces to classical linear-quadratic machinery: the finite-dimensional fixed-time unconstrained trajectory-generation problem in DIRECT is a linear-quadratic tracking problem, and the infinite-dimensional STDDP framework reduces to spatio-temporal LQR for linear PDEs with quadratic cost (Cao et al., 2021, Evans et al., 2021).

A plausible implication is that the enduring importance of DDP lies less in any single formulation than in the stability of its local dynamic-programming template: quadratic value approximations, affine feedback laws, and forward–backward sweeps continue to organize methods for delayed dynamics, belief-space planning, distributional robustness, hybrid systems, rigid contact, multiple shooting, and constrained trajectory optimization.