---
title: Hamilton-Jacobi-Bellman Equations on Graphs
url: https://www.emergentmind.com/topics/hamilton-jacobi-bellman-equations-on-graphs
type: topic
---

# Hamilton-Jacobi-Bellman Equations on Graphs

Hamilton-Jacobi-Bellman (HJB) equations on graphs are discrete analogues of continuous Hamilton-Jacobi (HJ), Hamilton-Jacobi-Bellman-Isaacs, and more general (possibly nonlocal) partial differential equations (PDEs), formulated for functions defined on the vertices of a finite or countable graph. They encode optimal control, dynamic programming, and geodesic/distance-type problems in settings with inherently discrete, network, or data-driven structure. Recently, rigorous theoretical foundations, convergence results, min–max structural representations, and robust computational frameworks for these discrete HJB equations have been established, enabling systematic treatment across applications including data science, Markov decision processes, large deviation theory, and quantum controls [2511.07653][2202.08789][2501.12505][1809.03872][1902.08926][2509.25965][1203.0522].

## 1. Discrete Formulations: Operators and Bellman-Type Updates

Let $G = (V, E, w)$ be a finite (directed or undirected) weighted graph with vertex set $V = \{x_1, ..., x_N\}$, edge set $E$, and weights $w_{ij}$.

### First-Order (Hamilton-Jacobi) Form
Discrete HJB equations often take the form
$$
H(u)(x_i) = \sup_{\alpha \in A} \left\{ F^\alpha\left( \{u(x_j)-u(x_i)\}_{x_j \sim x_i} \right) \right\} + f(x_i),
$$
where $F^\alpha$ is a monotone function of the neighbor differences, and $f$ is a given source term [2511.07653].

### Second-Order / Bellman–Isaacs Form
For Markov decision processes or stochastic control,
$$
H(u)(x_i) = \sup_{\alpha \in A} \inf_{\beta \in B} \left\{ \sum_{x_j} a^{\alpha\beta}(x_i, x_j)[u(x_j) - u(x_i)] + c^{\alpha\beta}(x_i) \right\}.
$$

### The $p$-Eikonal Equation
A central class for distance-type problems is the $p$-eikonal operator [2202.08789], defined as
$$
A_{G,p}u(x_i) = \sum_{j=1}^N w_{ji} (u(x_i) - u(x_j))_+^p,
$$
with $(a)_+ = \max(a,0)$.

Special cases:
- $p=1$: Bellman-type update defines a robust (non-shortest-path) distance, satisfying a fixed-point/minimum recursion that admits provable Lipschitz stability under edge perturbations.
- $p\to \infty$: Recovers the shortest-path (min-plus) distance; the operator becomes a max/min recursion corresponding to classical dynamic programming.

## 2. Existence, Uniqueness, and Structural Properties

### Global Comparison Property (GCP)
A key requirement for well-posedness is monotonicity in local differences—if $u\leq v$ everywhere and $u(x_0)=v(x_0)$, then $I(u,x_0)\leq I(v,x_0)$. Any operator satisfying GCP, plus certain semicontinuity and “subtract-constant” properties, admits global comparison and Perron-type existence theorems [2511.07653].

#### Main Theorem: Comparison and Existence
- If $I$ has the GCP and related properties, any subsolution $u$ and supersolution $v$ satisfy $\max_{V}(u-v)_+ \leq \max_{\Gamma}(u-v)_+$ (for prescribed boundary $\Gamma$).
- Perron’s method constructs a unique solution sandwiched between sub- and supersolutions.

### Bellman–Isaacs Min–Max Representation
Every operator $I:C(G)\to C(G)$ (locally Lipschitz, with GCP) admits a min–max decomposition in terms of graph Laplacians:
$$
I(u,x) = \min_{v\in Y} \max_{L\in \partial I(Y)} \left\{ f_{L,v}(x) + c_L(x) u(x) + \sum_{j} K_L(x, x_j)[u(x_j)-u(x)] \right\},
$$
with suitable nonnegative kernels and coefficients [2511.07653].

### Viscosity Solutions
Definitions for sub- and supersolutions follow a comparison-based, inequality-formulation at each node (and—when applicable—at boundaries or junctions in network-type graphs), precisely paralleling the viscosity principles in the continuum [2501.12505][1809.03872].

## 3. Applications: Robust Distance Functions, Optimal Control, and Large Deviations

### Robust Distance and Learning
The $p$-eikonal equation provides a robust alternative to shortest-path metrics, minimizing sensitivity to edge corruption. Applications include graph-based data depth, geometric medians, and semi-supervised learning [2202.08789]:
- Graph-based data depth: Solve $A_{G,p} D_{\Gamma}^{p,\alpha}(x) = \rho(x)^{-\alpha}$, $D=0$ on a set $\Gamma$; the minimizer gives a robust median and depth ranking.
- SSL: For labeled classes $\Gamma_1, \dots, \Gamma_k$, solve one $p$-eikonal per class and classify via the smallest $u_j(x)$.

### Markov Decision Processes on Graphs
Continuous-time Markov chains with graph-structured state spaces yield systems of HJB ODEs, with
$$
\frac{d}{dt}V^r(i, t) - r V^r(i, t) + H\left(i, \left[V^r(j, t) - V^r(i, t)\right]_{j \in V(i)}\right) = 0,
$$
where $H$ encodes optimal control over jump intensities. Ergodic limits and long-term (average-cost) HJB equations are characterized by bias/relative value solutions [1902.08926].

### Large Deviations and Discrete Weak KAM
For Markov chains with exponentially small transition rates, Friedlin–Wentzell theory leads to discrete HJ equations for quasipotentials:
$$
W(x) = \min_{y:(y,x)\in E} \left\{ W(y) + \Delta(y,x) \right\},
$$
where $\Delta(y,x)$ is the large deviation cost. The space of solutions is a polyhedron determined by cycle-based quasipotentials, and the vanishing viscosity/Friedlin–Wentzell solution corresponds to a minimal arborescence selection principle [2501.12505].

## 4. Analytical Techniques and Discrete-to-Continuum Limits

### Consistency and Convergence
Frameworks have been established for discrete-to-continuum convergence. For example, the $p$-eikonal equation on random geometric graphs, as $n\to\infty$, converges (with quantitative rates) to density-weighted geodesic PDEs:
$$
\rho|\nabla u|^p = f \quad \text{in } \Omega \setminus \Gamma, \quad u|_\Gamma = 0,
$$
and unique viscosity solutions transfer from the discrete to continuum domains under appropriate scaling [2202.08789].

### Network PDEs and State-Constraint Coupling
On spatial networks, arced-parameter PDEs coupled via special “state-constraint” or “junction” conditions at vertices induce discrete functional equations, e.g.,
$$
U(x) = \min_{e \in E : t(e)=x} p(U(o(e)), e),
$$
where $p$ solves a local Dirichlet problem along edges; solutions are constructed pathwise (Hopf–Lax) and the “Aubry set” provides a discrete weak KAM backbone [1809.03872].

## 5. Algebraic Structure: Idempotent/Tropical Analysis and Universal Algorithms

Hamilton-Jacobi and Bellman equations on graphs are, in essence, linear over idempotent semirings (“tropical” algebra). For instance, shortest path computation is a linear equation in the min-plus semiring:
$$
x_j = \min_i \{ a_{ji} + x_i \} \wedge b_j,
$$
enabling universal algorithms (e.g., Bellman–Ford, LDM factorization) for such recurrent equations [1203.0522].

Interval versions of these linear-algebraic algorithms (over idempotent intervals) yield rigorous enclosures for optimization under interval uncertainty, remaining tractable due to monotonicity and the max/min structure.

## 6. Extensions: Stochastic Optimal Control, Wasserstein Geometry, and Open Problems

Recent developments include HJB equations related to stochastic Wasserstein–Hamiltonian systems over graphs. Here, the value function $U(t, \rho, x)$ on the joint phase space $\mathcal{P}(G) \times \mathbb{R}^n$ satisfies a time-dependent HJB involving both Wasserstein and Euclidean gradients:
$$
\frac{\partial U}{\partial t} + \inf_{\mathbb{V} \in B_\ell} \left\{ \langle \partial_\rho U, D_x \mathcal{H}_0^\mathbb{V} \rangle - \langle D_x U, D_\rho \mathcal{H}_0^\mathbb{V} \rangle + \tfrac{1}{2} \operatorname{tr}[\sigma \sigma^\top D_x^2 U] + F \right\} = 0.
$$
Existence and uniqueness of viscosity solutions have been proved using energy truncation and doubling-of-variables arguments adapted to graph Wasserstein space. Applications include quantum control systems (e.g., stochastic nonlinear/discrete Schrödinger equations), with extensions to infinite graphs and general kinetic energies as open technical challenges [2509.25965].

## 7. Structural Connections to Continuum and Nonlocal PDEs

The discrete HJB frameworks mirror and unify methodologies from continuum viscosity PDE theory, nonlocal elliptic equations, and geometric control:
- Discrete gradient differences replace spatial derivatives.
- Graph Laplacians and jump kernels parallel divergence operators and nonlocal integrals.
- Perron’s method, comparison principles, and stability analysis carry over from PDE to network settings [2511.07653].
- Discrete-to-continuum limits under mesh refinement recover classical PDEs, including classical HJ, eikonal, nonlocal/fractional, and geodesic equations.
- Cycle-based “Aubry sets” and Hopf–Lax path minimizations replicate weak KAM and action-minimization principles from the continuum [1809.03872][2501.12505].

---

**References:**
- [2511.07653] Forcillo, Kitagawa, Schwab, "Hamilton-Jacobi-Bellman equations on graphs"
- [2202.08789] Calder, Slepčev et al., "Hamilton-Jacobi equations on graphs with applications to semi-supervised learning and data depth"
- [1809.03872] Pozza, Siconolfi, "Discounted Hamilton-Jacobi equations on networks and asymptotic analysis"
- [2501.12505] Aleandri, Gabrielli, Pallotta, "Friedlin-Wentzell solutions of discrete Hamilton Jacobi equations"
- [1902.08926] Guéant, Manziukt, "Optimal control on graphs: existence, uniqueness, and long-term behavior"
- [2509.25965] He, Huang, Lu, Schwab, "Hamilton--Jacobi--Bellman equation for optimal control of stochastic Wasserstein--Hamiltonian system on graphs"
- [1203.0522] Litvinov, Maslov, Shpiz, "Idempotent/tropical analysis, the Hamilton-Jacobi and Bellman equations"

Source: https://www.emergentmind.com/topics/hamilton-jacobi-bellman-equations-on-graphs