Continuous-Time Differential Dynamic Programming

• Continuous-time DDP is a second-order optimal control method that locally approximates the Bellman equation to solve nonlinear continuous-time problems.
• It applies both finite-dimensional H∞ control via the HJI equation and infinite-dimensional PDE control via the HJB equation for distributed and boundary systems.
• The method employs iterative forward-backward recursions to update nominal trajectories, demonstrating robust performance in applications like quadcopter dynamics and heat equation control.

Continuous-time differential dynamic programming (DDP) is a family of local second-order optimal control methods that solve nonlinear continuous-time optimal control problems by expanding the Bellman equation around a nominal trajectory, propagating value-function derivatives backward in time, and updating the control law through a forward-backward iteration. In the recent arXiv literature considered here, the method appears explicitly in two continuous-time forms: a finite-dimensional min-max formulation based on the Hamilton-Jacobi-Isaacs (HJI) equation for nonlinear $H_\infty$ control, and an infinite-dimensional Hilbert-space formulation based on the Hamilton-Jacobi-Bellman (HJB) equation for distributed and boundary control of PDEs (Sun et al., 2021, Evans et al., 2021).

1. Bellman foundations in continuous time

The continuous-time DDP viewpoint begins from a value function or value functional defined over a state at time $t$ and a finite horizon ending at $t_f$. For finite-dimensional nonlinear systems with disturbance, the robust formulation is a zero-sum differential game,

$$\frac{dx(t)}{dt}=F(x(t),u(t),v(t)), \qquad x(t_0)=x_0,$$

with value

$$V(x_0,t_0)=\min_{u(\cdot)}\max_{v(\cdot)} J_\gamma(u(\cdot),v(\cdot)).$$

The associated HJI equation is

$$-\frac{\partial V(x,t)}{\partial t} = \min_u \max_v Q(x,u,v,t), \qquad Q(x,u,v,t)=\mathcal{L}_\gamma(x,u,v)+V_x^\top F(x,u,v),$$

with terminal condition $V(x(t_f),t_f)=\phi(x(t_f),t_f)$ (Sun et al., 2021).

For spatio-temporal PDE control, the state is treated as an element of a Hilbert space such as $L_2^n(D)$, and the continuous-time optimal control problem is written directly in infinite dimensions. The value functional

$$V\big(X(t), t \big) := \min_{U_d, U_b} \Big[ J\big( X(t), U_d(t), U_b(t) \big) \Big]$$

satisfies the HJB equation

$$-\partial_t V\big(X(t),t\big) = \min_{U_d, U_b} \Big[ L(t,X,U_d,U_b) + V_X(t,X), F(t, X, U_d) \Big],$$

with terminal condition $t$0 (Evans et al., 2021).

These two formulations establish the central distinction within continuous-time DDP. In nominal optimal control the governing equation is HJB; in minimax robust control it is HJI. In both cases the method does not attempt to solve the nonlinear Bellman equation exactly. Instead, it constructs a local second-order approximation around a nominal state-control trajectory and iteratively refines that trajectory.

2. Local quadratic structure and forward-backward recursions

The defining mechanism of continuous-time DDP is a local expansion of the value object and the Bellman integrand around a nominal trajectory. In continuous-time game-theoretic DDP, the expansion is taken around $t$1, and the HJI equation is written to second order in deviations $t$2. The right-hand side becomes a quadratic form with derivatives

$t$3

so that the local problem at each time is a quadratic min-max subproblem (Sun et al., 2021).

Setting the gradients of this quadratic model with respect to $t$4 and $t$5 to zero yields implicit optimality conditions,

$t$6

which are rearranged into explicit affine laws

$t$7

Substituting these into the HJI expansion produces backward ODEs for $t$8, $t$9, and $t_f$0, propagated from the terminal conditions at $t_f$1 (Sun et al., 2021).

In infinite-dimensional STDDP, the same logic is expressed through a Volterra-Taylor expansion of the value functional,

$t_f$2

together with second-order expansions of the running cost and first-order expansions of the dynamics and boundary operator (Evans et al., 2021). The resulting minimization over distributed and boundary control increments is again quadratic, now in function spaces rather than Euclidean spaces.

Algorithmically, both formulations use the same structural loop. A nominal trajectory is initialized, a forward pass simulates the current dynamics, a backward pass propagates the derivatives of the value object and computes local gains, and a policy update generates the next trajectory. The difference lies in the mathematical setting: finite-dimensional ODE differential games in one case, infinite-dimensional PDE control in the other.

3. Finite-dimensional continuous-time GT-DDP and nonlinear $t_f$3 control

A concrete continuous-time DDP instantiation is the learning-based nonlinear $t_f$4 controller of “Game-Theoretic Differential Dynamic Programming,” where Gaussian Process (GP) regression is combined with a continuous-time min-max DDP solver (Sun et al., 2021). The real dynamics are modeled as

$t_f$5

with $t_f$6 the nominal model and $t_f$7 an unknown term capturing model error, disturbance, and unmodeled effects. GP regression is used to learn $t_f$8 from state-control data $t_f$9 with a squared exponential kernel. The posterior mean

$$\frac{dx(t)}{dt}=F(x(t),u(t),v(t)), \qquad x(t_0)=x_0,$$0

is used as a disturbance estimate, while the posterior uncertainty is converted into a multiplicative disturbance scale

$$\frac{dx(t)}{dt}=F(x(t),u(t),v(t)), \qquad x(t_0)=x_0,$$1

The learned robust dynamics are then written as

$$\frac{dx(t)}{dt}=F(x(t),u(t),v(t)), \qquad x(t_0)=x_0,$$2

This decomposition is important because it separates nominal dynamics, learned bias, and uncertainty-shaped disturbance injection. If disturbance enters through the same channel as control in a control-affine system, $$\frac{dx(t)}{dt}=F(x(t),u(t),v(t)), \qquad x(t_0)=x_0,$$3 can be chosen accordingly; if no structure is known, one may take $$\frac{dx(t)}{dt}=F(x(t),u(t),v(t)), \qquad x(t_0)=x_0,$$4 and $$\frac{dx(t)}{dt}=F(x(t),u(t),v(t)), \qquad x(t_0)=x_0,$$5 (Sun et al., 2021).

The performance criterion is a soft-constrained nonlinear $$\frac{dx(t)}{dt}=F(x(t),u(t),v(t)), \qquad x(t_0)=x_0,$$6 objective,

$$\frac{dx(t)}{dt}=F(x(t),u(t),v(t)), \qquad x(t_0)=x_0,$$7

The term $$\frac{dx(t)}{dt}=F(x(t),u(t),v(t)), \qquad x(t_0)=x_0,$$8 penalizes disturbance energy within the maximizing player’s objective and yields the disturbance attenuation bound

$$\frac{dx(t)}{dt}=F(x(t),u(t),v(t)), \qquad x(t_0)=x_0,$$9

for some positive definite $$V(x_0,t_0)=\min_{u(\cdot)}\max_{v(\cdot)} J_\gamma(u(\cdot),v(\cdot)).$$0 (Sun et al., 2021). Under the stated regularity assumptions and the Isaacs condition,

$$V(x_0,t_0)=\min_{u(\cdot)}\max_{v(\cdot)} J_\gamma(u(\cdot),v(\cdot)).$$1

the game value exists and the min/max order can be interchanged.

After convergence, the online feedback controller takes the local form

$$V(x_0,t_0)=\min_{u(\cdot)}\max_{v(\cdot)} J_\gamma(u(\cdot),v(\cdot)).$$2

combining a feedforward optimized trajectory with linear state feedback (Sun et al., 2021). The reported test case is a quadcopter with 16 states, including position, Euler angles, linear velocity, angular velocity, and rotor speeds. Training data are generated from a perturbed stochastic simulation with altered parameters such as inertia and arm length. The paper reports that the cost per GT-DDP iteration decreases over iterations and that the mean of the state trajectories over 100 Monte Carlo simulations converges to the desired terminal state despite stochastic disturbances (Sun et al., 2021).

4. Infinite-dimensional STDDP for spatio-temporal PDE control

A second explicit continuous-time DDP line is “Spatio-Temporal Differential Dynamic Programming,” which develops STDDP directly in infinite-dimensional Hilbert spaces for nonlinear PDEs with distributed and boundary control (Evans et al., 2021). The starting point is the field system

$$V(x_0,t_0)=\min_{u(\cdot)}\max_{v(\cdot)} J_\gamma(u(\cdot),v(\cdot)).$$3

$$V(x_0,t_0)=\min_{u(\cdot)}\max_{v(\cdot)} J_\gamma(u(\cdot),v(\cdot)).$$4

with interior control $$V(x_0,t_0)=\min_{u(\cdot)}\max_{v(\cdot)} J_\gamma(u(\cdot),v(\cdot)).$$5 over the spatial domain and boundary control $$V(x_0,t_0)=\min_{u(\cdot)}\max_{v(\cdot)} J_\gamma(u(\cdot),v(\cdot)).$$6 on $$V(x_0,t_0)=\min_{u(\cdot)}\max_{v(\cdot)} J_\gamma(u(\cdot),v(\cdot)).$$7.

A distinctive feature of this formulation is that boundary actuation is not appended heuristically. The derivation uses a Green’s-theorem-based adjoint identity in Hilbert spaces, which introduces the correct boundary sensitivity terms into the HJB expansion. This yields separate continuous-time DDP updates for distributed and boundary control increments,

$$V(x_0,t_0)=\min_{u(\cdot)}\max_{v(\cdot)} J_\gamma(u(\cdot),v(\cdot)).$$8

$$V(x_0,t_0)=\min_{u(\cdot)}\max_{v(\cdot)} J_\gamma(u(\cdot),v(\cdot)).$$9

which are the infinite-dimensional analogues of the standard finite-dimensional DDP update (Evans et al., 2021).

The backward pass propagates the scalar value, the first functional derivative $$-\frac{\partial V(x,t)}{\partial t} = \min_u \max_v Q(x,u,v,t), \qquad Q(x,u,v,t)=\mathcal{L}_\gamma(x,u,v)+V_x^\top F(x,u,v),$$0, and the second functional derivative $$-\frac{\partial V(x,t)}{\partial t} = \min_u \max_v Q(x,u,v,t), \qquad Q(x,u,v,t)=\mathcal{L}_\gamma(x,u,v)+V_x^\top F(x,u,v),$$1. The equations for $$-\frac{\partial V(x,t)}{\partial t} = \min_u \max_v Q(x,u,v,t), \qquad Q(x,u,v,t)=\mathcal{L}_\gamma(x,u,v)+V_x^\top F(x,u,v),$$2 and $$-\frac{\partial V(x,t)}{\partial t} = \min_u \max_v Q(x,u,v,t), \qquad Q(x,u,v,t)=\mathcal{L}_\gamma(x,u,v)+V_x^\top F(x,u,v),$$3 are continuous-time Riccati-type equations with distributed and boundary terms. The forward pass propagates the variation dynamics, and the control is updated by

$$-\frac{\partial V(x,t)}{\partial t} = \min_u \max_v Q(x,u,v,t), \qquad Q(x,u,v,t)=\mathcal{L}_\gamma(x,u,v)+V_x^\top F(x,u,v),$$4

with fixed or line-searched learning rates (Evans et al., 2021).

The paper identifies two classical limits. If there is no spatial coordinate and no boundary, the equations reduce to standard continuous-time DDP in finite dimensions. If the dynamics are linear and the cost is purely quadratic, the STDDP equations reduce to spatio-temporal LQR, with operator identifications such as

$$-\frac{\partial V(x,t)}{\partial t} = \min_u \max_v Q(x,u,v,t), \qquad Q(x,u,v,t)=\mathcal{L}_\gamma(x,u,v)+V_x^\top F(x,u,v),$$5

(Evans et al., 2021).

The reported numerical experiments involve a 1D heat equation and a nonlinear Burgers equation. For the heat equation, the setup uses 1200 time steps, 64 spatial grid points, and weights $$-\frac{\partial V(x,t)}{\partial t} = \min_u \max_v Q(x,u,v,t), \qquad Q(x,u,v,t)=\mathcal{L}_\gamma(x,u,v)+V_x^\top F(x,u,v),$$6, $$-\frac{\partial V(x,t)}{\partial t} = \min_u \max_v Q(x,u,v,t), \qquad Q(x,u,v,t)=\mathcal{L}_\gamma(x,u,v)+V_x^\top F(x,u,v),$$7, $$-\frac{\partial V(x,t)}{\partial t} = \min_u \max_v Q(x,u,v,t), \qquad Q(x,u,v,t)=\mathcal{L}_\gamma(x,u,v)+V_x^\top F(x,u,v),$$8; STDDP reaches the target field and exhibits super-quadratic early convergence in the value integral over the first approximately 50 iterations. For the Burgers equation, the setup uses 5 actuators with $$-\frac{\partial V(x,t)}{\partial t} = \min_u \max_v Q(x,u,v,t), \qquad Q(x,u,v,t)=\mathcal{L}_\gamma(x,u,v)+V_x^\top F(x,u,v),$$9, $V(x(t_f),t_f)=\phi(x(t_f),t_f)$0, $V(x(t_f),t_f)=\phi(x(t_f),t_f)$1, and a simulated annealing scheme that enables a very large state-to-control weight ratio $V(x(t_f),t_f)=\phi(x(t_f),t_f)$2 (Evans et al., 2021).

5. Relation to adjacent DDP literatures

The recent DDP literature contains many variants that are directly relevant to continuous-time DDP conceptually, but that are formulated in discrete time. The distinction is explicit in the papers themselves.

Formulation	Primary setting	Continuous-time status
GT-DDP	Nonlinear $V(x(t_f),t_f)=\phi(x(t_f),t_f)$3 differential game	Continuous-time HJI derivation
STDDP	PDE control in Hilbert spaces	Continuous-time HJB derivation
DBaS-DDP	Safety-embedded trajectory optimization	Discrete-time extension of BaS
PDDP	Joint control-parameter optimization	Discrete-time only
Robust DDP	Generalized-plant uncertainty	Discrete-time only
Optimal-horizon MPC-DDP	Free-horizon MPC	Discrete-time only

“Safety Embedded Differential Dynamic Programming Using Discrete Barrier States” extends an earlier continuous-time barrier state concept to discrete-time systems, but its DDP development is explicitly discrete-time: safety is embedded into augmented dynamics through the discrete barrier state

$V(x(t_f),t_f)=\phi(x(t_f),t_f)$4

and standard discrete-time DDP is then applied to the augmented system (Almubarak et al., 2021).

“Parameterized Differential Dynamic Programming” generalizes standard discrete-time DDP to include a time-invariant parameter vector $V(x(t_f),t_f)=\phi(x(t_f),t_f)$5, introduces a parameter feedback gain $V(x(t_f),t_f)=\phi(x(t_f),t_f)$6 in the control law, and proves convergence properties for the discrete-time recursion. Its connection to continuous-time DDP is indirect: continuous-time hybrid OCPs are reparameterized, discretized, and then solved as parameterized discrete-time problems (Oshin et al., 2022).

“Robust Differential Dynamic Programming” develops a discrete-time robust DDP based on generalized plants and multiplier relaxations. Its derivation is discrete-time throughout; the only continuous-time content is an inverted-pendulum example that is discretized before DDP is applied (Gramlich et al., 2022). “Optimal-Horizon Model-Predictive Control with Differential Dynamic Programming” is likewise a discrete-time optimal-horizon extension of DDP/iLQR for MPC, not a continuous-time derivation, although it discusses earlier continuous-time final-time optimization work as prior context (Stachowicz et al., 2021).

The literature therefore suggests a clear boundary: continuous-time DDP currently appears as a genuine Bellman-based continuous-time theory in specific settings, whereas many important robustness, safety, estimation, and MPC extensions are still developed in discrete time.

6. Guarantees, misconceptions, and limitations

A common misconception is that DDP is intrinsically a discrete-time algorithm. The cited literature does not support that claim. STDDP derives a continuous-time forward-backward scheme directly from the HJB equation in infinite-dimensional Hilbert spaces, and GT-DDP derives a continuous-time min-max recursion directly from the HJI equation for nonlinear $V(x(t_f),t_f)=\phi(x(t_f),t_f)$7 control (Evans et al., 2021, Sun et al., 2021).

A second misconception is that continuous-time DDP is confined to finite-dimensional ODEs. STDDP shows that the framework can be carried into function spaces and applied to distributed and boundary control of PDEs without first reducing the PDE to a ROM. The paper explicitly describes this as an optimize-then-discretize, discretization-agnostic framework (Evans et al., 2021).

The guarantees are formulation-dependent. In GT-DDP, the existence of the game value relies on standard regularity assumptions, piecewise continuity in time and Lipschitz dependence in state for both players, perfect knowledge of dynamics, cost, constraints, and current state, and the Isaacs condition (Sun et al., 2021). In STDDP, the derivation assumes sufficient differentiability in the Gateaux sense, and the convergence result is global convergence to a stationary point of the continuous-time forward-backward system under the stated assumptions; it is not a proof of convergence to a global optimum (Evans et al., 2021).

The principal numerical limitation reported for continuous-time PDE DDP is sensitivity in the backward Riccati-like equation for $V(x(t_f),t_f)=\phi(x(t_f),t_f)$8, which is often stiff and highly sensitive to the time step $V(x(t_f),t_f)=\phi(x(t_f),t_f)$9, to the ratio of state-weight to control-weight, and to underactuation. Explicit Euler is fastest per iteration but most sensitive to step size, Runge-Kutta is more accurate but more expensive, and semi-implicit schemes are less sensitive to $L_2^n(D)$0 but require large sparse matrix inversions (Evans et al., 2021). In the finite-dimensional game-theoretic setting, the method remains a local second-order approximation around a nominal trajectory, so convergence and performance are governed by the quality of the local model, including the learned GP disturbance estimate and the local quadratic HJI expansion (Sun et al., 2021).

Taken together, these results position continuous-time DDP as a Bellman-based local second-order methodology rather than a single algorithmic template. In finite dimensions it can operate as a continuous-time solver for nonlinear differential games with learned uncertainty; in infinite dimensions it can operate as a continuous-time forward-backward method for PDE field control. The surrounding DDP ecosystem—barrier-state safety, parameter optimization, generalized-plant robustness, and optimal-horizon MPC—remains closely related, but in the current literature it is predominantly discrete-time.