Hamilton-Jacobi-Bellman Equations on Graphs

• Hamilton-Jacobi-Bellman Equations on Graphs are discrete analogs of continuous HJB equations that model optimal control, dynamic programming, and distance-type problems on networks.
• They employ discrete operators, min–max structural representations, and viscosity solutions to ensure existence, uniqueness, and convergence from discrete graphs to continuum PDEs.
• Applications include robust distance functions, Markov decision processes, and large deviation principles, with significant impacts in data science, quantum controls, and network analysis.

Hamilton-Jacobi-Bellman (HJB) equations on graphs are discrete analogues of continuous Hamilton-Jacobi (HJ), Hamilton-Jacobi-Bellman-Isaacs, and more general (possibly nonlocal) partial differential equations (PDEs), formulated for functions defined on the vertices of a finite or countable graph. They encode optimal control, dynamic programming, and geodesic/distance-type problems in settings with inherently discrete, network, or data-driven structure. Recently, rigorous theoretical foundations, convergence results, min–max structural representations, and robust computational frameworks for these discrete HJB equations have been established, enabling systematic treatment across applications including data science, Markov decision processes, large deviation theory, and quantum controls (Forcillo et al., 10 Nov 2025, Calder et al., 2022, Aleandri et al., 21 Jan 2025, Pozza et al., 2018, Guéant et al., 2019, Cui et al., 30 Sep 2025, Litvinov, 2012).

1. Discrete Formulations: Operators and Bellman-Type Updates

Let $G = (V, E, w)$ be a finite (directed or undirected) weighted graph with vertex set $V = \{x_1, ..., x_N\}$, edge set $E$, and weights $w_{ij}$.

First-Order (Hamilton-Jacobi) Form

Discrete HJB equations often take the form

$$H(u)(x_i) = \sup_{\alpha \in A} \left\{ F^\alpha\left( \{u(x_j)-u(x_i)\}_{x_j \sim x_i} \right) \right\} + f(x_i),$$

where $F^\alpha$ is a monotone function of the neighbor differences, and $f$ is a given source term (Forcillo et al., 10 Nov 2025).

Second-Order / Bellman–Isaacs Form

For Markov decision processes or stochastic control,

$$H(u)(x_i) = \sup_{\alpha \in A} \inf_{\beta \in B} \left\{ \sum_{x_j} a^{\alpha\beta}(x_i, x_j)[u(x_j) - u(x_i)] + c^{\alpha\beta}(x_i) \right\}.$$

The $p$-Eikonal Equation

A central class for distance-type problems is the $p$-eikonal operator (Calder et al., 2022), defined as

$$A_{G,p}u(x_i) = \sum_{j=1}^N w_{ji} (u(x_i) - u(x_j))_+^p,$$

with $(a)_+ = \max(a,0)$.

Special cases:

• $p=1$: Bellman-type update defines a robust (non-shortest-path) distance, satisfying a fixed-point/minimum recursion that admits provable Lipschitz stability under edge perturbations.
• $p\to \infty$: Recovers the shortest-path (min-plus) distance; the operator becomes a max/min recursion corresponding to classical dynamic programming.

2. Existence, Uniqueness, and Structural Properties

Global Comparison Property (GCP)

A key requirement for well-posedness is monotonicity in local differences—if $u\leq v$ everywhere and $u(x_0)=v(x_0)$, then $I(u,x_0)\leq I(v,x_0)$. Any operator satisfying GCP, plus certain semicontinuity and “subtract-constant” properties, admits global comparison and Perron-type existence theorems (Forcillo et al., 10 Nov 2025).

Main Theorem: Comparison and Existence

• If $I$ has the GCP and related properties, any subsolution $u$ and supersolution $v$ satisfy $\max_{V}(u-v)_+ \leq \max_{\Gamma}(u-v)_+$ (for prescribed boundary $\Gamma$).
• Perron’s method constructs a unique solution sandwiched between sub- and supersolutions.

Bellman–Isaacs Min–Max Representation

Every operator $I:C(G)\to C(G)$ (locally Lipschitz, with GCP) admits a min–max decomposition in terms of graph Laplacians:

$$I(u,x) = \min_{v\in Y} \max_{L\in \partial I(Y)} \left\{ f_{L,v}(x) + c_L(x) u(x) + \sum_{j} K_L(x, x_j)[u(x_j)-u(x)] \right\},$$

with suitable nonnegative kernels and coefficients (Forcillo et al., 10 Nov 2025).

Viscosity Solutions

Definitions for sub- and supersolutions follow a comparison-based, inequality-formulation at each node (and—when applicable—at boundaries or junctions in network-type graphs), precisely paralleling the viscosity principles in the continuum (Aleandri et al., 21 Jan 2025, Pozza et al., 2018).

3. Applications: Robust Distance Functions, Optimal Control, and Large Deviations

Robust Distance and Learning

The $p$-eikonal equation provides a robust alternative to shortest-path metrics, minimizing sensitivity to edge corruption. Applications include graph-based data depth, geometric medians, and semi-supervised learning (Calder et al., 2022):

• Graph-based data depth: Solve $$A_{G,p} D_{\Gamma}^{p,\alpha}(x) = \rho(x)^{-\alpha}$$, $D=0$ on a set $\Gamma$; the minimizer gives a robust median and depth ranking.
• SSL: For labeled classes $\Gamma_1, \dots, \Gamma_k$, solve one $p$-eikonal per class and classify via the smallest $u_j(x)$.

Markov Decision Processes on Graphs

Continuous-time Markov chains with graph-structured state spaces yield systems of HJB ODEs, with

$$\frac{d}{dt}V^r(i, t) - r V^r(i, t) + H\left(i, \left[V^r(j, t) - V^r(i, t)\right]_{j \in V(i)}\right) = 0,$$

where $H$ encodes optimal control over jump intensities. Ergodic limits and long-term (average-cost) HJB equations are characterized by bias/relative value solutions (Guéant et al., 2019).

Large Deviations and Discrete Weak KAM

For Markov chains with exponentially small transition rates, Friedlin–Wentzell theory leads to discrete HJ equations for quasipotentials:

$$W(x) = \min_{y:(y,x)\in E} \left\{ W(y) + \Delta(y,x) \right\},$$

where $\Delta(y,x)$ is the large deviation cost. The space of solutions is a polyhedron determined by cycle-based quasipotentials, and the vanishing viscosity/Friedlin–Wentzell solution corresponds to a minimal arborescence selection principle (Aleandri et al., 21 Jan 2025).

4. Analytical Techniques and Discrete-to-Continuum Limits

Consistency and Convergence

Frameworks have been established for discrete-to-continuum convergence. For example, the $p$-eikonal equation on random geometric graphs, as $n\to\infty$, converges (with quantitative rates) to density-weighted geodesic PDEs:

$$\rho|\nabla u|^p = f \quad \text{in } \Omega \setminus \Gamma, \quad u|_\Gamma = 0,$$

and unique viscosity solutions transfer from the discrete to continuum domains under appropriate scaling (Calder et al., 2022).

Network PDEs and State-Constraint Coupling

On spatial networks, arced-parameter PDEs coupled via special “state-constraint” or “junction” conditions at vertices induce discrete functional equations, e.g.,

$U(x) = \min_{e \in E : t(e)=x} p(U(o(e)), e),$

where $p$ solves a local Dirichlet problem along edges; solutions are constructed pathwise (Hopf–Lax) and the “Aubry set” provides a discrete weak KAM backbone (Pozza et al., 2018).

5. Algebraic Structure: Idempotent/Tropical Analysis and Universal Algorithms

Hamilton-Jacobi and Bellman equations on graphs are, in essence, linear over idempotent semirings (“tropical” algebra). For instance, shortest path computation is a linear equation in the min-plus semiring:

$x_j = \min_i \{ a_{ji} + x_i \} \wedge b_j,$

enabling universal algorithms (e.g., Bellman–Ford, LDM factorization) for such recurrent equations (Litvinov, 2012).

Interval versions of these linear-algebraic algorithms (over idempotent intervals) yield rigorous enclosures for optimization under interval uncertainty, remaining tractable due to monotonicity and the max/min structure.

6. Extensions: Stochastic Optimal Control, Wasserstein Geometry, and Open Problems

Recent developments include HJB equations related to stochastic Wasserstein–Hamiltonian systems over graphs. Here, the value function $U(t, \rho, x)$ on the joint phase space $\mathcal{P}(G) \times \mathbb{R}^n$ satisfies a time-dependent HJB involving both Wasserstein and Euclidean gradients:

$$\frac{\partial U}{\partial t} + \inf_{\mathbb{V} \in B_\ell} \left\{ \langle \partial_\rho U, D_x \mathcal{H}_0^\mathbb{V} \rangle - \langle D_x U, D_\rho \mathcal{H}_0^\mathbb{V} \rangle + \tfrac{1}{2} \operatorname{tr}[\sigma \sigma^\top D_x^2 U] + F \right\} = 0.$$

Existence and uniqueness of viscosity solutions have been proved using energy truncation and doubling-of-variables arguments adapted to graph Wasserstein space. Applications include quantum control systems (e.g., stochastic nonlinear/discrete Schrödinger equations), with extensions to infinite graphs and general kinetic energies as open technical challenges (Cui et al., 30 Sep 2025).

7. Structural Connections to Continuum and Nonlocal PDEs

The discrete HJB frameworks mirror and unify methodologies from continuum viscosity PDE theory, nonlocal elliptic equations, and geometric control:

• Discrete gradient differences replace spatial derivatives.
• Graph Laplacians and jump kernels parallel divergence operators and nonlocal integrals.
• Perron’s method, comparison principles, and stability analysis carry over from PDE to network settings (Forcillo et al., 10 Nov 2025).
• Discrete-to-continuum limits under mesh refinement recover classical PDEs, including classical HJ, eikonal, nonlocal/fractional, and geodesic equations.
• Cycle-based “Aubry sets” and Hopf–Lax path minimizations replicate weak KAM and action-minimization principles from the continuum (Pozza et al., 2018, Aleandri et al., 21 Jan 2025).

---

References:

• (Forcillo et al., 10 Nov 2025) Forcillo, Kitagawa, Schwab, "Hamilton-Jacobi-Bellman equations on graphs"
• (Calder et al., 2022) Calder, Slepčev et al., "Hamilton-Jacobi equations on graphs with applications to semi-supervised learning and data depth"
• (Pozza et al., 2018) Pozza, Siconolfi, "Discounted Hamilton-Jacobi equations on networks and asymptotic analysis"
• (Aleandri et al., 21 Jan 2025) Aleandri, Gabrielli, Pallotta, "Friedlin-Wentzell solutions of discrete Hamilton Jacobi equations"
• (Guéant et al., 2019) Guéant, Manziukt, "Optimal control on graphs: existence, uniqueness, and long-term behavior"
• (Cui et al., 30 Sep 2025) He, Huang, Lu, Schwab, "Hamilton--Jacobi--Bellman equation for optimal control of stochastic Wasserstein--Hamiltonian system on graphs"
• (Litvinov, 2012) Litvinov, Maslov, Shpiz, "Idempotent/tropical analysis, the Hamilton-Jacobi and Bellman equations"