Discrete Hamilton–Jacobi–Bellman Equation

• The discrete Hamilton–Jacobi–Bellman equation is a framework for analyzing optimal control and nonlinear PDEs on discrete spaces such as graphs, meshes, and tropical algebra settings.
• It incorporates various formulations, including graph-based, finite element, and tropical Bellman equations, to enable robust numerical methods like policy iteration and IMEX schemes.
• Its application spans network control, stochastic games, and approximation of degenerate PDEs, ensuring well-posedness through monotonicity and comparison principles.

The discrete Hamilton–Jacobi–Bellman (HJB) equation provides a rigorous framework for the analysis and numerical solution of optimal control, dynamic programming, and related nonlinear elliptic and parabolic PDEs on discrete spaces such as finite graphs, simplicial meshes, and, in tropical mathematics, algebraic structures like the max–plus semiring. Discrete HJB theory extends the continuum viscosity-solution analysis to fully nonlinear difference and graph Laplacian operators, underpins tropical (idempotent) algebraic methods, and forms the foundation for robust finite element, finite difference, and policy iteration schemes. Applications include optimal control on networks, stochastic games, shortest path problems, and the numerical approximation of possibly degenerate second-order PDEs with mixed boundary conditions.

1. Structural Forms of the Discrete HJB Equation

Discrete HJB equations manifest in a variety of structural forms depending on their context:

• Graph-Based HJB: On a finite graph $G = (V, E)$, the HJB operator acts on functions $u \in C(G) \cong \mathbb{R}^N$, with boundary data on a subset $\Gamma \subset V$, and takes the form:

$$\begin{cases} I(u, x) = f(x), & x \in V \setminus \Gamma, \ u(x) = g(x), & x \in \Gamma, \end{cases}$$

where

$$I(u, x) = \min_{\alpha \in \mathcal{A}} \max_{\beta \in \mathcal{B}_\alpha} \left\{ f^{\alpha\beta}(x) + c^{\alpha\beta}(x) u(x) + \sum_{y \in V} a_{xy}^{\alpha\beta} (u(y) - u(x)) \right\}.$$

The coefficients $a_{xy}^{\alpha\beta} \geq 0$ are edge weights, and the operator encompasses local and nonlocal (interaction kernel) generalizations (Forcillo et al., 10 Nov 2025).

• Finite Element (FE) and IMEX Discretization: For parabolic (time-dependent) isotropic HJB equations on bounded polyhedral domains $\Omega \subset \mathbb{R}^d$, using P1 finite element spaces on acute triangulations:

$$-d_i v_i^k + \sup_{\alpha \in \mathcal{A}} \left\{ \mathcal{E}_i^\alpha v_i^{k+1} + \mathcal{I}_i^\alpha v_i^k - F_i^\alpha \right\} = 0,$$

where $\mathcal{E}_i^\alpha$ and $\mathcal{I}_i^\alpha$ are explicit and implicit matrix–vector actions, and $u \in C(G) \cong \mathbb{R}^N$0 encodes the time-stepping via backward differences (Jaroszkowski et al., 2021).

• Matrix (Tropical) Bellman Equation: In idempotent analysis using max–plus algebra, the stationary Bellman equation takes the form:

$u \in C(G) \cong \mathbb{R}^N$1

with $u \in C(G) \cong \mathbb{R}^N$2, $u \in C(G) \cong \mathbb{R}^N$3 a max–plus matrix, and $u \in C(G) \cong \mathbb{R}^N$4, $u \in C(G) \cong \mathbb{R}^N$5 corresponding to $u \in C(G) \cong \mathbb{R}^N$6, $u \in C(G) \cong \mathbb{R}^N$7 respectively (Litvinov, 2012).

2. Boundary Conditions and Mixed-Type Treatment

Discrete HJB problems require careful specification of boundary data:

• Dirichlet Boundary ($u \in C(G) \cong \mathbb{R}^N$8, graph: $u \in C(G) \cong \mathbb{R}^N$9): Prescribed values $\Gamma \subset V$0.
• Robin/Neumann Boundary ($\Gamma \subset V$1): On finite element meshes, directional derivatives along outflow vectors $\Gamma \subset V$2 are approximated via lower Dini differences:

$\Gamma \subset V$3

with $\Gamma \subset V$4 small enough for nodal confinement (Jaroszkowski et al., 2021). Discontinuities of coefficients across faces are admissible; the discrete operator accesses only local face data.

For discrete HJB on graphs, the "boundary" is any prescribed subset $\Gamma \subset V$5, and data is set directly on the corresponding nodes (Forcillo et al., 10 Nov 2025).

3. Monotonicity, Comparison, and Well-Posedness

Crucial well-posedness results for discrete HJB equations hinge on discrete analogues of monotonicity and the comparison principle:

• Global Comparison Property (GCP): If $\Gamma \subset V$6 everywhere and $\Gamma \subset V$7, then $\Gamma \subset V$8 (Forcillo et al., 10 Nov 2025).
• Properness: The operator is order-preserving under subtraction of constants.
• Local Monotonicity Property (LMP): For monotone FE schemes, if $\Gamma \subset V$9 attains a minimum at node $$\begin{cases} I(u, x) = f(x), & x \in V \setminus \Gamma, \ u(x) = g(x), & x \in \Gamma, \end{cases}$$0, then both explicit and implicit matrix entries at $$\begin{cases} I(u, x) = f(x), & x \in V \setminus \Gamma, \ u(x) = g(x), & x \in \Gamma, \end{cases}$$1 direction satisfy $$\begin{cases} I(u, x) = f(x), & x \in V \setminus \Gamma, \ u(x) = g(x), & x \in \Gamma, \end{cases}$$2, $$\begin{cases} I(u, x) = f(x), & x \in V \setminus \Gamma, \ u(x) = g(x), & x \in \Gamma, \end{cases}$$3 (Jaroszkowski et al., 2021).
• Discrete Maximum Principle: The minimum of a discrete subsolution is controlled by the Dirichlet data.

These properties guarantee the validity of a comparison theorem: any subsolution is below any supersolution throughout the domain, ensuring uniqueness of the solution.

4. Numerical Schemes and Policy Iteration

Discrete HJB equations necessitate iterative solution algorithms owing to their nonlinearity and high-dimensional nature:

• FE–IMEX–Howard’s Algorithm: For each time step, the policy–iteration (Howard's) scheme alternates between:
  1. Policy evaluation: Solve the linearized Bellman equation with coefficients frozen at the current policy to obtain the value update.
  2. Policy improvement: Update the control policy at each spatial node to maximize the Bellman payoff.

The monotonicity of the stiffness matrices (M-matrix property) ensures convergence in a finite number of superlinear steps (Jaroszkowski et al., 2021).

• "Tropical""/Idempotent Algorithms: In max–plus algebra, numerical solvers correspond to tropical analogues of classical linear algebra:
  • Bellman–Ford iteration: Repeated sweeps to compute the least solution, convergent in at most $$\begin{cases} I(u, x) = f(x), & x \in V \setminus \Gamma, \ u(x) = g(x), & x \in \Gamma, \end{cases}$$4 iterations in absence of positive cycles.
  • Floyd–Warshall closure: Computes all-pairs solutions, corresponding to matrix closure $$\begin{cases} I(u, x) = f(x), & x \in V \setminus \Gamma, \ u(x) = g(x), & x \in \Gamma, \end{cases}$$5.
  • LDM-factorization: Analog of LU decomposition for max–plus systems (Litvinov, 2012).

These algorithms inherit linear complexity properties from their classical counterparts.

5. Existence, Uniqueness, and Convergence Results

Theoretical guarantees for discrete HJB systems are established via:

• Comparison and Perron's Method: Existence is obtained as the supremum of all subsolutions below a given supersolution, under upper semicontinuity and a suitable lower-bump lemma (Forcillo et al., 10 Nov 2025).
• Consistency and Convergence: In FE settings, provided mesh regularity, monotonicity, and time step restrictions $$\begin{cases} I(u, x) = f(x), & x \in V \setminus \Gamma, \ u(x) = g(x), & x \in \Gamma, \end{cases}$$6, the sequence of numerical solutions converges uniformly to the unique viscosity solution in $$\begin{cases} I(u, x) = f(x), & x \in V \setminus \Gamma, \ u(x) = g(x), & x \in \Gamma, \end{cases}$$7 norm (Jaroszkowski et al., 2021).
• General Representation: Any monotone, locally Lipschitz operator satisfying comparison admits a minimal Bellman–Isaacs representation, i.e., a min–max over graph-type Laplacians plus lower order terms (Forcillo et al., 10 Nov 2025).
• Barles–Souganidis Argument: The convergence follows from monotonicity, stability, and consistency as originally established in the context of viscosity solutions (Jaroszkowski et al., 2021).

6. Idempotent (Tropical) Perspective and Universal Linearity

The idempotent analysis reveals that, while nonlinear in the classical sense, the (discrete) HJB/Bellman equations become linear equations over semirings such as $$\begin{cases} I(u, x) = f(x), & x \in V \setminus \Gamma, \ u(x) = g(x), & x \in \Gamma, \end{cases}$$8 (max–plus algebra):

• Maslov Dequantization: The passage from the (imaginary-time) Schrödinger equation to Hamilton–Jacobi via exponential change and $$\begin{cases} I(u, x) = f(x), & x \in V \setminus \Gamma, \ u(x) = g(x), & x \in \Gamma, \end{cases}$$9 limit converts superposition and dynamics to tropical algebra operations (Litvinov, 2012).
• Idempotent Superposition Principle: Solutions of the discrete Bellman equation enjoy linearity over $$I(u, x) = \min_{\alpha \in \mathcal{A}} \max_{\beta \in \mathcal{B}_\alpha} \left\{ f^{\alpha\beta}(x) + c^{\alpha\beta}(x) u(x) + \sum_{y \in V} a_{xy}^{\alpha\beta} (u(y) - u(x)) \right\}.$$0 and can be constructed as max-plus combinations of other solutions.
• Correspondence Principle: All standard linear algebraic algorithms, including interval analysis, transfer to the tropical setting via algebraic substitutions, preserving their operational complexity.

This structure is central for efficient implementation of algorithms in discrete optimization, control, and network analysis.

7. Examples and Extensions

Discrete HJB theory encompasses a wide variety of operators and settings, including but not limited to:

Example	Operator Structure	Reference
Linear Graph Laplacian	$$I(u, x) = \min_{\alpha \in \mathcal{A}} \max_{\beta \in \mathcal{B}_\alpha} \left\{ f^{\alpha\beta}(x) + c^{\alpha\beta}(x) u(x) + \sum_{y \in V} a_{xy}^{\alpha\beta} (u(y) - u(x)) \right\}.$$1	(Forcillo et al., 10 Nov 2025)
Bellman/Isaacs Control on Graphs	$$I(u, x) = \min_{\alpha \in \mathcal{A}} \max_{\beta \in \mathcal{B}_\alpha} \left\{ f^{\alpha\beta}(x) + c^{\alpha\beta}(x) u(x) + \sum_{y \in V} a_{xy}^{\alpha\beta} (u(y) - u(x)) \right\}.$$2	(Forcillo et al., 10 Nov 2025)
Eikonal/Energy Minimization	$$I(u, x) = \min_{\alpha \in \mathcal{A}} \max_{\beta \in \mathcal{B}_\alpha} \left\{ f^{\alpha\beta}(x) + c^{\alpha\beta}(x) u(x) + \sum_{y \in V} a_{xy}^{\alpha\beta} (u(y) - u(x)) \right\}.$$3	(Forcillo et al., 10 Nov 2025)
Tropical Matrix Bellman	$$I(u, x) = \min_{\alpha \in \mathcal{A}} \max_{\beta \in \mathcal{B}_\alpha} \left\{ f^{\alpha\beta}(x) + c^{\alpha\beta}(x) u(x) + \sum_{y \in V} a_{xy}^{\alpha\beta} (u(y) - u(x)) \right\}.$$4	(Litvinov, 2012)
FE HJB with Mixed Boundary	FE–IMEX structure with artificial diffusion, Dini derivatives for Robin boundaries	(Jaroszkowski et al., 2021)

The framework extends to fully nonlinear, nonlocal, and fractional Laplacian analogues on discrete structures by modifying the edge weights and interaction kernels accordingly (Forcillo et al., 10 Nov 2025).

Discrete HJB systems thereby provide a comprehensive and unifying methodology for a broad class of control, optimization, and nonlinear PDE problems in discrete and tropical mathematical settings.