---
title: HJ-Based Tracking Control
url: https://www.emergentmind.com/topics/hj-based-tracking-control-algorithm
type: topic
---

# HJ-Based Tracking Control

A Hamilton–Jacobi (HJ)-based tracking control algorithm refers to a class of control synthesis and analysis methods which leverage Hamilton–Jacobi partial differential equations (PDEs) to achieve provable tracking error guarantees, optimality, or robustness for high-dimensional or uncertain dynamical systems. These techniques are foundational across several domains, including safe real-time planning under model mismatch, data-driven optimal tracking in multi-agent systems, robust nonlinear tracking under input constraints, and model-free $H_{\infty}$ optimal tracking via reinforcement learning.

## 1. Fundamental Principles and Problem Setup

HJ-based tracking control unifies the concept of tracking a reference trajectory under uncertainty, disturbances, or model mismatch by minimizing a value function dictated by the system’s dynamic evolution. Common setups involve:

- **Model mismatch planning and tracking**: Fast trajectory planning via a low-dimensional “planner” model $\dot{x}_p = f_p(x_p, u_p)$ paired with physical execution via a high-dimensional “tracker” model $\dot{x}_t = f_t(x_t, u_t)$, with $x_p$ embedded in $x_t$ and the relative state $r = x_t - Qx_p$ [1808.00649].
- **Consensus tracking for multi-agent systems**: Each agent observes its own and neighbors’ states, computes a tracking error $e_i(k)$ relative to both neighbors and leader under heterogeneous linear dynamics, and seeks Nash-optimal consensus tracking [1711.11422].
- **Optimal tracking with robustness and actuator constraints**: Nonlinear or uncertain affine-in-control plants augmented for error and reference, optimizing discounted cost subject to input and mismatch bounds [1911.04157].
- **Nonlinear $H_{\infty}$ tracking via differential game**: Affine nonlinear systems with unknown dynamics and disturbances, minimizing the $L_2$-gain from disturbance to tracking error/control output via policy iteration [2401.12882].

In all cases, the tracking error evolution and optimal control synthesis derive from the HJ (Hamilton–Jacobi–Bellman or Isaacs) equations associated to a dynamic programming or differential game viewpoint.

## 2. Hamilton–Jacobi Reachability and Tracking Error Bounds

The canonical HJ-based framework formulates tracking control as a reachability problem or dynamic game, generating error bounds via PDE solutions:

- **HJ Differential Game**: For planner–tracker systems, the reachability value function $V(\tau, r)$ solves
  $$
  \frac{\partial V}{\partial \tau}(\tau, r) + H(r, \nabla_r V(\tau, r)) = 0
  $$
  with Hamiltonian
  $$
  H(r, p) = \max_{u_p \in \mathcal{U}_p} \min_{u_t \in \mathcal{U}_t} p^T (f_t(r, u_t) - Q f_p(Q^+ r, u_p))
  $$
  characterizing the worst-case model mismatch. The zero sublevel set of the infinite-horizon $V$ is the smallest forward-invariant Tracking Error Bound (TEB) [1808.00649].

- **Multi-agent Nash-equilibrium tracking**: The coupled HJ equations describe optimal consensus via the value function $V_i$ for each agent, given as
  $$
  V_i(e_i) = \min_{u_i} \{ e_i^T Q_i e_i + u_i^T R_i u_i + V_i(\bar{A}_i e_i + F_i u_i + \sum_{j \in N_i} E_{ij} u_j) \}
  $$
  yielding local control policies synthesized from the Riccati-like equations [1711.11422].

- **Nonlinear optimal and robust tracking**: The HJB residual $\mathcal{H}(z, u, V_z)$ encapsulates costs, system dynamics, bounds on uncertainty, and input constraints, enabling robust tracking [1911.04157].

- **$H_{\infty}$ Tracking**: The Isaacs equation for min–max dynamic games is 
  $$
  X^T Q_r X  - a V^{*}(X) + \nabla V^{*}(X)^T F(X) - \frac{1}{4} \nabla V^{*T} G R^{-1} G^T \nabla V^{*} + \frac{1}{4\gamma^2} \nabla V^{*T} K K^T \nabla V^{*} = 0
  $$
  the solution $V^*$ yields the state-feedback policy that ensures $L_2$ attenuation at level $\gamma$ [2401.12882].

## 3. Computational Methodologies: Relaxations, Approximations, and Data-Driven Algorithms

Direct solution of HJ PDEs is generally intractable for high-dimensional systems. Significant algorithmic advances include:

- **Sum-of-Squares (SOS) Relaxation**: For planner–tracker systems, a polynomial certificate $V(r)$ and controller $K(r, u_p)$ are constructed. The forward invariance conditions and input constraints are relaxed via SOS programming on semialgebraic sets, employing polynomial multipliers and alternating bilinear optimization over $(V, K, E)$ via convex subproblems [1808.00649].

- **Quadratic Value Approximation for Multi-agent Tracking**: In multi-agent scenarios, each agent’s value function $V_i(e_i)$ is quadratically approximated, resulting in Riccati-like equations for $P_i$. These are iteratively updated using input/output Q-learning, leveraging sliding window data, least-squares regression, and value iteration entirely from measured I/O signals and without explicit models [1711.11422].

- **Critic Neural Network for Uncertain Nonlinear Systems**: The value function $V^*(z)$ is parameterized by a neural network with weights $\widehat{W}$ and regressor $\vartheta(z)$, trained via gradient descent on the HJB residual error using a variable-gain law $\Gamma(e)$ adaptive to error magnitude, together with Lyapunov-stabilizing and bounding terms [1911.04157].

- **Damped Newton and $\delta$–Policy Iteration**: Model-free HJ-based $H_{\infty}$ tracking is achieved by recasting the HJI PDE as a generalized Bellman equation solvable by damped-Newton iteration. Both on-policy and off-policy $\delta$–policy iteration (with $\delta \in (0,1]$ controlling step size and region of convergence) admit least-squares solutions for network weights using only recorded state–input–disturbance trajectories. Actor–critic neural networks implement control policies and value functions without requiring system model knowledge [2401.12882].

## 4. Tracking Error Guarantees, Safety Margins, and Integration into Planning

A defining property of HJ-based tracking control is the production of explicit error bounds or invariant sets:

- **Tracking Error Bound (TEB)**: Once $V(r)$ and ellipsoid matrix $E$ are synthesized, the set $\{r: V(r) \leq 1\}$ is guaranteed forward invariant, projecting to the error domain as $\mathcal{E} = \{e: V(c^{-1}(e)) \leq 1\}$ or ellipsoid $\{z: z^T E z \leq 1\}$, used as a safety margin in real-time planning [1808.00649].

- **Inflated Obstacles in Planner Space**: Offline computation of controller $K$, invariant $V$, and ellipsoid $E$ enables real-time inflation of planner obstacles, ensuring any low-dimensional plan that avoids these inflated regions will be trackable by the actual (high-dimensional) system [1808.00649].

- **Consensus for Multi-agent Networks**: Nash-optimal controller updates drive all agent and leader states to consensus, with tracking errors $e_i(k) \rightarrow 0$ as value iteration converges. Asymptotic convergence is established under persistent excitation and topology constraints [1711.11422].

- **Uniform Ultimate Boundedness (UUB)**: Variable gain laws can achieve strictly tighter residual sets (lower tracking error oscillations and faster convergence) than fixed-rate gradient descent, with explicit Lyapunov analysis confirming the boundedness of both error and critic weights [1911.04157].

- **$H_{\infty}$ Performance**: Solution of the Isaacs equation and implementation of optimal policy $u^*$ guarantee closed-loop $L_2$-gain from disturbance $d$ to $[e, u]$ is less than prescribed level $\gamma$ [2401.12882].

## 5. Numerical Experiments and Trade-Offs

HJ-based algorithms have been validated across domains, with extensive trade-off analysis:

| System Scenario                | HJ Method         | SOS/ADP/NN-based Methods          |
|-------------------------------|------------------|-----------------------------------|
| 5D car tracking 3D Dubins     | $\sim$0.2m radius, 25h CPU | $\sim$0.48m radius, 5min CPU [1808.00649] |
| 8D airplane tracking 4D Dubins| HJ intractable    | $\sim$6m$\times$4m$\times$4.5m in <2h [1808.00649] |
| 3-agent consensus (discrete)  | Not considered    | I/O Q-learning VI, 20–25 iterations [1711.11422] |
| 2D/6D nonlinear, UAV tracking | Constant gain: slow, larger oscillation | Variable gain: 5–10$\times$ faster, 3–5$\times$ smaller oscillation [1911.04157] |
| Nonlinear $H_{\infty}$ (off-policy)| Requires model | NN-based, model-free, $\leq$100 iterations [2401.12882] |

These results demonstrate that relaxations (SOS, ADP, RL, VI) substantially improve scalability to high-dimensional or uncertain systems at some cost in conservativeness (larger error sets or bounds), while providing rigorous guarantees and efficient learning.

## 6. Convergence, Stability, and Theoretical Guarantees

Convergence analysis and stability results underpin the reliability of HJ-based tracking control:

- **Alternating Bilinear SOS Optimization**: Guaranteed monotonic improvement via slack minimization and trust regions, with certificates of forward invariance when all SOS constraints are tight [1808.00649].
- **Bellman contraction and value iteration**: Proven asymptotic convergence in multi-agent settings under strong connectivity, observability, reachability, and persistent excitation conditions [1711.11422].
- **Adaptive critic learning**: Variable-gain descent provably tightens ultimate boundedness and enhances convergence speed. Lyapunov-based analysis guarantees UUB of system trajectories and weights [1911.04157].
- **Newton-type policy iteration**: Kantorovich’s theorem ensures convergence of $\delta$–policy iteration for HJI equations when appropriate regularity and step-size conditions are met, with global stability for small $\delta$ and quadratic local convergence for $\delta = 1$ [2401.12882].
- **$H_{\infty}$-guaranteed bounds**: Solution of Isaacs PDEs yields policies with certified $L_2$-attenuation of disturbances, as validated both theoretically and in simulation [2401.12882].

## 7. Technological Impact and Extensions

HJ-based tracking control algorithms maintain safety and performance guarantees for systems that were previously intractable due to dimensionality or uncertainty. SOS programming, data-based RL, and policy iteration permit the extension of reachability-theoretic guarantees to multi-agent, nonlinear, and model-free settings with scalability, thereby enabling real-time planning, robust consensus, and resilient tracking across autonomous vehicles, robot teams, and uncertain nonlinear platforms.

A plausible implication is that further improvements in expressive polynomial/SOS representation, deep function approximation in RL/NN methods, and tailored data-driven PDE solvers will continue to expand the applicability of HJ-based tracking control into complex, safety-critical domains, while maintaining rigorous error bounds and optimality certificates.

Source: https://www.emergentmind.com/topics/hj-based-tracking-control-algorithm