GU-HJBI Equation with Gradient Uncertainty

• GU-HJBI is a robust-control PDE model that incorporates adversarial perturbations of both system dynamics and the value function gradient.
• It extends classical HJB and robust HJBI formulations by modeling gradient uncertainty via bounded perturbations, leading to novel Isaacs equations.
• The framework has significant implications for reinforcement learning and challenges standard quadratic LQ theory by inducing nonlinearity even for small uncertainties.

Searching arXiv for recent and foundational papers on GU-HJBI, gradient constraints, and related HJBI structure. The Hamilton–Jacobi–Bellman–Isaacs equation with Gradient Uncertainty (GU-HJBI) is a robust-control PDE introduced to model adversarial perturbations not only of system dynamics but also of the value function gradient itself. In the formulation developed in “Robust Control with Gradient Uncertainty” (Qi, 20 Jul 2025), the controller minimizes cost while an adversary perturbs both the drift through a standard robust-control channel and the local sensitivity $\nabla V(x)$ through a bounded perturbation $\delta$. This produces a zero-sum dynamic game with a fully nonlinear Isaacs operator and leads to a new PDE distinct from both the classical HJB equation and the standard robust HJBI equation. The topic sits at the intersection of robust stochastic control, differential games, viscosity solutions, and approximate dynamic programming, and it is especially motivated by applications in reinforcement learning, where value gradients are typically estimated rather than known exactly (Qi, 20 Jul 2025).

1. Definition and problem formulation

In the time-homogeneous setting of (Qi, 20 Jul 2025), the controlled state $X_t\in\mathbb R^n$ satisfies

$dX_t = f(X_t,u_t)\,dt + \sigma(X_t,u_t)\,dB_t,$

with control $u_t\in U\subset\mathbb R^k$, admissible controls $u\in A$, running cost $L(x,u)$, and discounted objective

$$J(x;u) \coloneqq E_x\left[\int_0^\infty e^{-\rho t} L(X_t,u_t)\,dt\right], \qquad V(x) \coloneqq \inf_{u\in A} J(x;u).$$

The classical HJB equation is

$$\rho V(x)=\inf_{u\in U}\left\{L(x,u)+\mathcal{L}^u V(x)\right\},$$

where

$$\mathcal{L}^u \phi(x)\coloneqq \nabla \phi(x)^\top f(x,u)+\frac12 \operatorname{Tr}\!\left(\sigma(x,u)\sigma(x,u)^\top D^2\phi(x)\right).$$

Standard robust control augments the drift by an adversarial distortion $\delta$0,

$\delta$1

penalizing $\delta$2 quadratically and yielding the standard robust HJBI equation

$\delta$3

GU-HJBI adds a second adversarial channel by replacing the exact gradient with a perturbed gradient

$\delta$4

where $\delta$5 belongs to the uncertainty set

$\delta$6

The full GU-HJBI PDE is

$\delta$7

This formulation is specific to (Qi, 20 Jul 2025). It differs from singular-control HJBs with gradient constraints, such as

$\delta$8

or

$\delta$9

because there the gradient term encodes a constraint arising from singular control rather than an uncertainty set over gradients (Hynd, 2011, Moreno-Franco, 2016).

2. Reduced equation and Hamiltonian structure

For fixed $X_t\in\mathbb R^n$0, with $X_t\in\mathbb R^n$1, the inner optimization over $X_t\in\mathbb R^n$2 in (Qi, 20 Jul 2025) is

$X_t\in\mathbb R^n$3

whose maximizer is

$X_t\in\mathbb R^n$4

Substituting this into the full game yields the reduced GU-HJBI equation

$X_t\in\mathbb R^n$5

This is Proposition 3.1 of (Qi, 20 Jul 2025).

The corresponding Hamiltonian is

$X_t\in\mathbb R^n$6

and the PDE is written as

$X_t\in\mathbb R^n$7

The paper states that this operator is proper / degenerate elliptic because increasing $X_t\in\mathbb R^n$8 increases $X_t\in\mathbb R^n$9, hence decreases $dX_t = f(X_t,u_t)\,dt + \sigma(X_t,u_t)\,dB_t,$0 (Qi, 20 Jul 2025).

This Isaacs structure is explicit. By contrast, graph HJBI formulations such as

$dX_t = f(X_t,u_t)\,dt + \sigma(X_t,u_t)\,dB_t,$1

encode uncertainty through coefficients acting on discrete gradients $dX_t = f(X_t,u_t)\,dt + \sigma(X_t,u_t)\,dB_t,$2, which is structurally related but not identical to perturbing a continuum gradient variable directly (Forcillo et al., 10 Nov 2025). Similarly, classical Bellman–Isaacs operators of the form

$dX_t = f(X_t,u_t)\,dt + \sigma(X_t,u_t)\,dB_t,$3

capture adversarial uncertainty in the coefficient multiplying $dX_t = f(X_t,u_t)\,dt + \sigma(X_t,u_t)\,dB_t,$4, not in $dX_t = f(X_t,u_t)\,dt + \sigma(X_t,u_t)\,dB_t,$5 itself (Marchi, 2010).

3. Interpretation of gradient uncertainty

The motivation of (Qi, 20 Jul 2025) is that in approximate dynamic programming and reinforcement learning, the value function is estimated from data, so $dX_t = f(X_t,u_t)\,dt + \sigma(X_t,u_t)\,dB_t,$6 is itself uncertain. The paper models this by allowing the adversary to perturb the controller’s local state valuation through $dX_t = f(X_t,u_t)\,dt + \sigma(X_t,u_t)\,dB_t,$7. A central point is that, in standard robust control, the adversary’s optimal model distortion is proportional to $dX_t = f(X_t,u_t)\,dt + \sigma(X_t,u_t)\,dB_t,$8, so errors in $dX_t = f(X_t,u_t)\,dt + \sigma(X_t,u_t)\,dB_t,$9 directly affect the robustness term (Qi, 20 Jul 2025).

This makes GU-HJBI conceptually different from several nearby PDE classes.

First, it is not a singular-control gradient-constraint equation. In the HJBs studied in (Hynd, 2011) and (Moreno-Franco, 2016), the gradient term enters as a constraint such as $u_t\in U\subset\mathbb R^k$0 or $u_t\in U\subset\mathbb R^k$1, derived from control costs or singular interventions, not from uncertainty in the gradient variable. The structural resemblance is real, but the semantics differ.

Second, it is not merely a Bellman–Isaacs equation with uncertain drift. In works such as (Marchi, 2010) and (Kawecki et al., 2021), gradient dependence appears through affine terms like $u_t\in U\subset\mathbb R^k$2 or $u_t\in U\subset\mathbb R^k$3. This means the adversary chooses a coefficient acting on the gradient. GU-HJBI goes further by perturbing the gradient argument itself (Qi, 20 Jul 2025).

Third, it is not equivalent to the recursive stochastic HJBI equations arising from BSDE games with random coefficients or non-Lipschitz generators. Those equations can involve gradient-sensitive quantities such as $u_t\in U\subset\mathbb R^k$4 or $u_t\in U\subset\mathbb R^k$5, but the uncertainty is attached to controls, coefficients, or the BSDE driver, rather than to a separate gradient ambiguity set (Qiu et al., 2020, Wang et al., 2024).

A plausible implication is that GU-HJBI should be viewed as an internal-uncertainty robust control model, complementing classical external model misspecification. That interpretation is explicit in (Qi, 20 Jul 2025).

4. Small-$u_t\in U\subset\mathbb R^k$6 asymptotics and induced nonlinearity

Let

$u_t\in U\subset\mathbb R^k$7

Proposition 3.2 of (Qi, 20 Jul 2025) states that for small $u_t\in U\subset\mathbb R^k$8,

$u_t\in U\subset\mathbb R^k$9

Hence the approximate GU-HJBI equation is

$u\in A$0

The first-order correction is therefore a norm penalty in the vector

$u\in A$1

This asymptotic structure is one of the key technical features of GU-HJBI. It shows that even arbitrarily small gradient uncertainty adds a non-polynomial first-order term. The paper further states that the geometry of the uncertainty set changes this penalty through dual norms. For small $u\in A$2, if

$u\in A$3

then:

Uncertainty set	First-order correction
$u\in A$4-ball	$u\in A$5
$u\in A$6-box	$u\in A$7
$u\in A$8	$u\in A$9

These are stated in Proposition 6.1 of (Qi, 20 Jul 2025). This suggests that GU-HJBI is naturally sensitive to the dual geometry of the uncertainty model.

5. Well-posedness in viscosity sense

The paper (Qi, 20 Jul 2025) imposes the following assumptions. Assumption 2.1 requires continuity of $L(x,u)$0 in all arguments, uniform Lipschitz continuity in $L(x,u)$1,

$L(x,u)$2

and linear growth

$L(x,u)$3

Assumption 4.1 imposes uniform ellipticity: $L(x,u)$4 for some $L(x,u)$5.

With the PDE written as

$L(x,u)$6

the viscosity definition is standard: a locally bounded $L(x,u)$7 is a viscosity subsolution if for every $L(x,u)$8 touching from above at $L(x,u)$9,

$$J(x;u) \coloneqq E_x\left[\int_0^\infty e^{-\rho t} L(X_t,u_t)\,dt\right], \qquad V(x) \coloneqq \inf_{u\in A} J(x;u).$$0

and analogously for supersolutions (Qi, 20 Jul 2025).

The main result is Theorem 4.1: under Assumptions 2.1 and 4.1, if $$J(x;u) \coloneqq E_x\left[\int_0^\infty e^{-\rho t} L(X_t,u_t)\,dt\right], \qquad V(x) \coloneqq \inf_{u\in A} J(x;u).$$1 is a bounded viscosity subsolution and $$J(x;u) \coloneqq E_x\left[\int_0^\infty e^{-\rho t} L(X_t,u_t)\,dt\right], \qquad V(x) \coloneqq \inf_{u\in A} J(x;u).$$2 is a bounded viscosity supersolution of the GU-HJBI equation, then

$$J(x;u) \coloneqq E_x\left[\int_0^\infty e^{-\rho t} L(X_t,u_t)\,dt\right], \qquad V(x) \coloneqq \inf_{u\in A} J(x;u).$$3

The paper then states uniqueness of bounded continuous viscosity solutions, and also states existence of a unique bounded and continuous viscosity solution under the same assumptions (Qi, 20 Jul 2025).

The proof uses the doubling-of-variables method with

$$J(x;u) \coloneqq E_x\left[\int_0^\infty e^{-\rho t} L(X_t,u_t)\,dt\right], \qquad V(x) \coloneqq \inf_{u\in A} J(x;u).$$4

Ishii’s lemma, and continuity of the compactly maximized $$J(x;u) \coloneqq E_x\left[\int_0^\infty e^{-\rho t} L(X_t,u_t)\,dt\right], \qquad V(x) \coloneqq \inf_{u\in A} J(x;u).$$5-Hamiltonian (Qi, 20 Jul 2025). The paper emphasizes that extension to the degenerate case $$J(x;u) \coloneqq E_x\left[\int_0^\infty e^{-\rho t} L(X_t,u_t)\,dt\right], \qquad V(x) \coloneqq \inf_{u\in A} J(x;u).$$6 is open because the term

$$J(x;u) \coloneqq E_x\left[\int_0^\infty e^{-\rho t} L(X_t,u_t)\,dt\right], \qquad V(x) \coloneqq \inf_{u\in A} J(x;u).$$7

couples adversarial gradient perturbations with diffusion geometry.

This well-posedness result places GU-HJBI alongside other viscosity-based Isaacs theories, but under a specific nonlinear Hamiltonian. Earlier Bellman–Isaacs comparison and continuous dependence results, such as those for parabolic HJBI operators in $$J(x;u) \coloneqq E_x\left[\int_0^\infty e^{-\rho t} L(X_t,u_t)\,dt\right], \qquad V(x) \coloneqq \inf_{u\in A} J(x;u).$$8, concern affine gradient dependence and do not directly address the GU term (Marchi, 2010). Likewise, stochastic HJBI equations with random coefficients establish viscosity characterizations in a random-field or BSPDE setting, but not this specific gradient-uncertainty Hamiltonian (Qiu et al., 2020).

6. Linear-quadratic case and structural breakdown of Riccati theory

The LQ specialization in (Qi, 20 Jul 2025) uses

$$J(x;u) \coloneqq E_x\left[\int_0^\infty e^{-\rho t} L(X_t,u_t)\,dt\right], \qquad V(x) \coloneqq \inf_{u\in A} J(x;u).$$9

with $$\rho V(x)=\inf_{u\in U}\left\{L(x,u)+\mathcal{L}^u V(x)\right\},$$0, $$\rho V(x)=\inf_{u\in U}\left\{L(x,u)+\mathcal{L}^u V(x)\right\},$$1, and Assumption 5.1 that $$\rho V(x)=\inf_{u\in U}\left\{L(x,u)+\mathcal{L}^u V(x)\right\},$$2 is stabilizable and $$\rho V(x)=\inf_{u\in U}\left\{L(x,u)+\mathcal{L}^u V(x)\right\},$$3 is detectable.

At $$\rho V(x)=\inf_{u\in U}\left\{L(x,u)+\mathcal{L}^u V(x)\right\},$$4, one recovers standard robust LQ control. The quadratic ansatz

$$\rho V(x)=\inf_{u\in U}\left\{L(x,u)+\mathcal{L}^u V(x)\right\},$$5

leads to the robust algebraic Riccati equation

$$\rho V(x)=\inf_{u\in U}\left\{L(x,u)+\mathcal{L}^u V(x)\right\},$$6

with optimal control

$$\rho V(x)=\inf_{u\in U}\left\{L(x,u)+\mathcal{L}^u V(x)\right\},$$7

Define

$$\rho V(x)=\inf_{u\in U}\left\{L(x,u)+\mathcal{L}^u V(x)\right\},$$8

The paper’s central LQ insight is Proposition 5.1: for any $$\rho V(x)=\inf_{u\in U}\left\{L(x,u)+\mathcal{L}^u V(x)\right\},$$9 and any non-degenerate problem data, the value function is not quadratic. The contradiction argument assumes

$$\mathcal{L}^u \phi(x)\coloneqq \nabla \phi(x)^\top f(x,u)+\frac12 \operatorname{Tr}\!\left(\sigma(x,u)\sigma(x,u)^\top D^2\phi(x)\right).$$0

and shows that the approximate GU-HJBI then contains the term

$$\mathcal{L}^u \phi(x)\coloneqq \nabla \phi(x)^\top f(x,u)+\frac12 \operatorname{Tr}\!\left(\sigma(x,u)\sigma(x,u)^\top D^2\phi(x)\right).$$1

so the PDE’s right-hand side is not a quadratic polynomial in $$\mathcal{L}^u \phi(x)\coloneqq \nabla \phi(x)^\top f(x,u)+\frac12 \operatorname{Tr}\!\left(\sigma(x,u)\sigma(x,u)^\top D^2\phi(x)\right).$$2 unless $$\mathcal{L}^u \phi(x)\coloneqq \nabla \phi(x)^\top f(x,u)+\frac12 \operatorname{Tr}\!\left(\sigma(x,u)\sigma(x,u)^\top D^2\phi(x)\right).$$3 (Qi, 20 Jul 2025).

This is a sharp structural difference from standard robust LQ control. A plausible implication is that GU-HJBI removes the finite-dimensional Riccati closure even in the simplest quadratic setting. That implication is directly supported by the paper’s statement that the classical quadratic value function assumption fails for any non-zero gradient uncertainty (Qi, 20 Jul 2025).

7. Perturbation theory, numerical evidence, and RL connection

The paper develops the expansion

$$\mathcal{L}^u \phi(x)\coloneqq \nabla \phi(x)^\top f(x,u)+\frac12 \operatorname{Tr}\!\left(\sigma(x,u)\sigma(x,u)^\top D^2\phi(x)\right).$$4

The first-order correction satisfies

$$\mathcal{L}^u \phi(x)\coloneqq \nabla \phi(x)^\top f(x,u)+\frac12 \operatorname{Tr}\!\left(\sigma(x,u)\sigma(x,u)^\top D^2\phi(x)\right).$$5

with Feynman–Kac representation

$$\mathcal{L}^u \phi(x)\coloneqq \nabla \phi(x)^\top f(x,u)+\frac12 \operatorname{Tr}\!\left(\sigma(x,u)\sigma(x,u)^\top D^2\phi(x)\right).$$6

The main-text first-order control correction is

$$\mathcal{L}^u \phi(x)\coloneqq \nabla \phi(x)^\top f(x,u)+\frac12 \operatorname{Tr}\!\left(\sigma(x,u)\sigma(x,u)^\top D^2\phi(x)\right).$$7

The second-order correction solves

$$\mathcal{L}^u \phi(x)\coloneqq \nabla \phi(x)^\top f(x,u)+\frac12 \operatorname{Tr}\!\left(\sigma(x,u)\sigma(x,u)^\top D^2\phi(x)\right).$$8

where

$$\mathcal{L}^u \phi(x)\coloneqq \nabla \phi(x)^\top f(x,u)+\frac12 \operatorname{Tr}\!\left(\sigma(x,u)\sigma(x,u)^\top D^2\phi(x)\right).$$9

These formulas show that the non-polynomial structure propagates through the perturbation hierarchy (Qi, 20 Jul 2025).

The numerical section of (Qi, 20 Jul 2025) treats 1D and 2D LQ examples by solving the $\delta$00-equation numerically. The paper reports that the approximate value function visibly deviates from the quadratic baseline, that the approximate control becomes nonlinear, and that in two dimensions the contour lines of $\delta$01 are not elliptical. These observations are presented as validation of the theoretical claim that gradient uncertainty destroys quadratic/linear structure (Qi, 20 Jul 2025).

The same paper proposes the Gradient-Uncertainty-Robust Actor-Critic algorithm, instantiated as GURAC-TD3. It uses the small-$\delta$02 GU-HJBI penalty

$\delta$03

as motivation for an actor regularizer based on critic state-gradients. With

$\delta$04

the actor loss is modified to

$\delta$05

where

$\delta$06

On Pendulum-v1, the paper reports improved training stability relative to standard TD3, with smoother learning curves and tighter variance across seeds (Qi, 20 Jul 2025).

This suggests a concrete bridge from PDE-level GU-HJBI theory to robust approximate dynamic programming. By contrast, algebraic or idempotent approaches to HJB interpret dynamic programming operators through min-plus or max-plus linearity, but do not formulate a gradient-uncertainty Isaacs PDE of this kind (Litvinov, 2012).

8. Relation to adjacent theories and limitations

GU-HJBI belongs to a broader family of HJB, HJBI, and nonlocal or graph-based Isaacs equations, but it is not reducible to any one of them.

Gradient-constraint HJBs such as

$\delta$07

and

$\delta$08

provide rigorous techniques for convex gradient terms, penalization, and regularity, but they model singular-control constraints rather than uncertainty sets over $\delta$09 (Hynd, 2011, Moreno-Franco, 2016). This suggests that some analytic tools may transfer, but the game-theoretic interpretation does not.

Standard Bellman–Isaacs PDEs and their ergodic or homogenized versions incorporate adversarial drift and diffusion through min–max operators over linear coefficients acting on $\delta$10 and $\delta$11 (Marchi, 2010, Kawecki et al., 2021). GU-HJBI extends this by inserting a second adversarial optimization over perturbations of the gradient argument itself (Qi, 20 Jul 2025).

Graph HJBI operators give a discrete analogue in which uncertainty acts through coefficients multiplying edge differences $\delta$12, that is, discrete gradients (Forcillo et al., 10 Nov 2025). This is structurally close, but GU-HJBI is a continuum PDE with explicit uncertainty set $\delta$13.

Stochastic HJBI equations with random coefficients and BSPDE structure develop zero-sum differential games in random environments, including Hamiltonians depending on $\delta$14 and $\delta$15, but they do not isolate an explicit gradient-uncertainty ambiguity set (Qiu et al., 2020). Recursive stochastic differential games with non-Lipschitz generators likewise give useful Isaacs and BSDE machinery, but their viscosity theory does not directly cover explicit gradient perturbation channels (Wang et al., 2024).

The main limitation stated in (Qi, 20 Jul 2025) is the reliance on uniform ellipticity. Extension to degenerate diffusions is open. The paper also notes the computational difficulty of solving GU-HJBI in high dimensions and leaves broader theory for other uncertainty models and solvers to future work. This suggests that the current theory is foundational rather than exhaustive.

9. Representative formulas

For reference, the principal formulas associated with GU-HJBI are collected below.

Object	Formula
Gradient uncertainty set	$\delta$16
Full GU-HJBI	$\delta$17 as in (GU-HJBI)
Worst-case drift distortion	$\delta$18
Reduced GU-HJBI	$\delta$19
Hamiltonian	$\delta$20
PDE operator	$\delta$21
Small-$\delta$22 correction	$\delta$23
Robust ARE at $\delta$24	$\delta$25
First-order correction PDE	$\delta$26

GU-HJBI therefore designates a specific class of Isaacs equations in which the adversary perturbs both the model and the controller’s gradient information. In the formulation currently available, it is characterized by an inner supremum over a compact gradient-uncertainty set, a reduced Hamiltonian after elimination of the drift distortion, a viscosity comparison principle under uniform ellipticity, and a perturbative structure that already invalidates classical quadratic LQ theory for any $\delta$27 (Qi, 20 Jul 2025).