Gradient-Uncertainty-Robust Actor-Critic

• The paper introduces a robust control framework that explicitly addresses uncertainty in the value function gradient for actor-critic methods.
• The algorithm incorporates an adversarial penalty term into the actor update, effectively mitigating spurious policy shifts under gradient noise.
• Empirical results and scaling analyses demonstrate improved training stability and convergence compared to standard TD3 algorithms.

Gradient-Uncertainty-Robust Actor-Critic (GURAC) denotes an actor-critic formulation that explicitly addresses uncertainty in the gradient information used for policy improvement, especially uncertainty in the value function gradient under function approximation. The explicit term and a concrete algorithmic instantiation appear in "Robust Control with Gradient Uncertainty" (Qi, 20 Jul 2025), where gradient perturbations are incorporated into a zero-sum control game and induce a modified actor objective. A broader usage is suggested by adjacent work in which gradient uncertainty is quantified through finite-width asymptotics of neural actor-critic estimators, posterior covariance of policy gradients, or robust Bellman and policy-gradient constructions under transition ambiguity (Georgoudios et al., 25 Jan 2026).

1. Definition and conceptual scope

In the explicit robust-control formulation, gradient uncertainty means uncertainty about the gradient of the value function with respect to the state, $\nabla V(x)$. The motivating premise is that in reinforcement learning the critic is almost always approximated, so $\nabla_x V_\theta(x)$ or $\nabla_x Q_\theta(x,u)$ is noisy or biased, and policy updates can therefore be driven by spurious directions. This uncertainty is modeled as Knightian uncertainty: the effective gradient is treated as $\nabla V(x)+\delta$, where an adversary chooses $\delta$ from a bounded set (Qi, 20 Jul 2025).

This notion is narrower than generic stochastic-gradient noise. Related work shows that the same phrase can also encompass finite-width stochastic fluctuations in actor and critic outputs, posterior uncertainty in gradient estimates, and uncertainty induced by model misspecification. In the scaling analysis of neural actor-critic, uncertainty quantification means treating the network outputs as statistical estimators and separating deterministic bias terms from random fluctuation terms in asymptotic expansions (Georgoudios et al., 25 Jan 2026). In Bayesian policy-gradient work, the corresponding uncertainty object is the posterior covariance of $\nabla_\theta \eta(\theta)$ rather than an adversarial perturbation set (Ghavamzadeh et al., 30 Apr 2026).

A persistent source of ambiguity is that not all “robust” actor-critic methods robustify the same object. Some robustify transition kernels, some robustify value-gradient information, and some quantify uncertainty in the gradient estimator itself. GURAC, in the strict sense introduced in (Qi, 20 Jul 2025), concerns robustness to uncertainty in the critic’s value gradient.

2. Control-theoretic formulation

The control-theoretic foundation of GURAC is a zero-sum dynamic game in which the controller chooses a policy while nature perturbs both the system dynamics and the value-gradient information. With nominal diffusion

$dX_t = f(X_t, u_t)\,dt + \sigma(X_t,u_t)\,dB_t,$

running cost $L(x,u)$, discount rate $\rho$, adversarial drift perturbation $h_t$, and gradient perturbation $\nabla_x V_\theta(x)$0, the dynamic-programming equation becomes the Hamilton–Jacobi–Bellman–Isaacs equation with Gradient Uncertainty (GU-HJBI):

$\nabla_x V_\theta(x)$1

Maximizing over $\nabla_x V_\theta(x)$2 yields the reduced GU-HJBI

$\nabla_x V_\theta(x)$3

with $\nabla_x V_\theta(x)$4 (Qi, 20 Jul 2025).

For small $\nabla_x V_\theta(x)$5, the inner maximization admits the first-order expansion

$\nabla_x V_\theta(x)$6

so the approximate GU-HJBI contains a non-smooth norm penalty

$\nabla_x V_\theta(x)$7

This term is the direct source of the penalty later used in the actor update (Qi, 20 Jul 2025).

The linear-quadratic case is structurally revealing. For any non-zero gradient uncertainty, the classical quadratic value-function ansatz fails: the right-hand side contains a norm of a linear function of $\nabla_x V_\theta(x)$8, so the value function is no longer quadratic and the optimal control law is no longer linear. The first-order correction satisfies a linear PDE with source term $\nabla_x V_\theta(x)$9, which is non-polynomial; accordingly, the first-order control correction is nonlinear in $\nabla_x Q_\theta(x,u)$0 (Qi, 20 Jul 2025). This result establishes that gradient uncertainty is not a minor perturbation of standard robust control, but a change in problem class.

3. Algorithmic realization

The explicit GURAC algorithm is implemented as a TD3-like off-policy actor-critic. Its critic side remains standard TD3: double $\nabla_x Q_\theta(x,u)$1 networks, target networks, target policy smoothing, and delayed policy updates. The robust modification is concentrated in the actor objective. If $\nabla_x Q_\theta(x,u)$2 denotes the first critic, $\nabla_x Q_\theta(x,u)$3 the deterministic actor, $\nabla_x Q_\theta(x,u)$4 an estimated drift, and $\nabla_x Q_\theta(x,u)$5 a known or assumed diffusion term, the actor optimizes

$\nabla_x Q_\theta(x,u)$6

Equivalently, the implemented loss is

$\nabla_x Q_\theta(x,u)$7

with

$\nabla_x Q_\theta(x,u)$8

The continuous-time inner maximization over $\nabla_x Q_\theta(x,u)$9 is therefore not solved explicitly during learning; instead, GURAC uses the first-order norm penalty corresponding to the adversary’s best response (Qi, 20 Jul 2025).

This architecture makes the algorithm asymmetric. The critic is trained by an ordinary TD3 Bellman loss, whereas the actor is regularized by a term that depends on the critic’s state-gradient. The resulting update dampens policy moves that rely on state-action regions where the critic gradient is highly sensitive. In the reported implementation on Pendulum-v1, actor and critics are MLPs with two hidden layers of 256 units, the robustness weight is $\nabla V(x)+\delta$0, the model-uncertainty parameter is $\nabla V(x)+\delta$1, and training runs for 200,000 environment steps (Qi, 20 Jul 2025).

The available theoretical guarantees are indirect. The PDE analysis establishes existence and uniqueness of a bounded continuous viscosity solution to GU-HJBI under uniform ellipticity and shows that the Hamiltonian-based penalty is structurally justified. It does not provide a direct convergence theorem for the discrete-time, function-approximation GURAC-TD3 algorithm (Qi, 20 Jul 2025).

4. Scaling laws and uncertainty quantification in neural actor-critic

A second line of work supplies a distinct route to GURAC-like design by quantifying how actor and critic uncertainty scales with network width. In the shallow two-layer actor-critic studied in "Scaling Effects and Uncertainty Quantification in Neural Actor Critic Algorithms" (Georgoudios et al., 25 Jan 2026), both policy and critic use the parametrization

$\nabla V(x)+\delta$2

with scaling exponent $\nabla V(x)+\delta$3. The leading deterministic limit is shared across all such $\nabla V(x)+\delta$4, but the fluctuation structure is not.

The central result is a multi-scale expansion of $\nabla V(x)+\delta$5 and $\nabla V(x)+\delta$6 as deterministic bias terms plus a random fluctuation term. The leading fluctuation amplitude scales as $\nabla V(x)+\delta$7, so the variance scales as $\nabla V(x)+\delta$8. As $\nabla V(x)+\delta$9, variance decays like $\delta$0; as $\delta$1, the decay becomes slow. The same analysis yields finite-time bounds

$\delta$2

and similarly for $\delta$3, making the bias-variance tradeoff explicit (Georgoudios et al., 25 Jan 2026).

This suggests a scaling-based form of gradient-uncertainty robustness. If the goal is low variance in policy and value estimates, then $\delta$4 should be chosen close to $\delta$5, with learning rates scaled as

$\delta$6

and exploration rate chosen, for example, as

$\delta$7

The same work notes that the asymptotic total leading error is minimized at $\delta$8, while the variance alone is minimized as $\delta$9, and numerical experiments on a forest-management MDP with width $\nabla_\theta \eta(\theta)$0 show that larger $\nabla_\theta \eta(\theta)$1 reduces standard deviation and accelerates convergence (Georgoudios et al., 25 Jan 2026).

In this scaling-based interpretation, GURAC is not a new loss term but a hyperparameter regime: large width, $\nabla_\theta \eta(\theta)$2 near mean-field scaling, and width-matched learning and exploration schedules.

5. Related formulations and neighboring lineages

Several neighboring strands contribute pieces of the broader GURAC picture without using the same adversarial value-gradient formulation.

Under Wasserstein distributional robustness, policy optimization becomes a max-min problem over policies and transition laws. The robust Bellman operator is dualized into a scalar optimization over $\nabla_\theta \eta(\theta)$3 and an inner transport selector $\nabla_\theta \eta(\theta)$4, leading to a robust policy gradient with two terms:

$\nabla_\theta \eta(\theta)$5

The associated actor-critic introduces a value critic $\nabla_\theta \eta(\theta)$6, a sensitivity critic $\nabla_\theta \eta(\theta)$7, and an actor updated from $\nabla_\theta \eta(\theta)$8. Here robustness is to transition-model ambiguity, and the additional sensitivity term propagates gradients through the worst-case transport mapping rather than through nominal next states (Hafsi et al., 25 Jun 2026).

Soft-robust actor-critic replaces worst-case model selection by averaging over a distribution $\nabla_\theta \eta(\theta)$9 on an uncertainty set. With average transition model $dX_t = f(X_t, u_t)\,dt + \sigma(X_t,u_t)\,dB_t,$0, the soft-robust policy gradient is

$dX_t = f(X_t, u_t)\,dt + \sigma(X_t,u_t)\,dB_t,$1

and the critic uses a TD error built from $dX_t = f(X_t, u_t)\,dt + \sigma(X_t,u_t)\,dB_t,$2 rather than a single nominal kernel. This is robust to model uncertainty but avoids the full conservativeness of worst-case optimization (Derman et al., 2018).

In robust average-reward reinforcement learning, the critic estimates a robust $dX_t = f(X_t, u_t)\,dt + \sigma(X_t,u_t)\,dB_t,$3 function via a robust Bellman operator involving support functions over contamination, TV, or Wasserstein ambiguity sets, and the actor performs a KL mirror-descent step

$dX_t = f(X_t, u_t)\,dt + \sigma(X_t,u_t)\,dB_t,$4

This setting admits non-asymptotic sample complexity $dX_t = f(X_t, u_t)\,dt + \sigma(X_t,u_t)\,dB_t,$5 for obtaining an $dX_t = f(X_t, u_t)\,dt + \sigma(X_t,u_t)\,dB_t,$6-optimal robust policy, but the guarantees are tabular and specific to average reward (Xu et al., 8 Jun 2025). For discounted robust MDPs with non-rectangular uncertainty, actor-critic can still be formulated by using an approximate worst-case kernel as critic and then taking projected policy-gradient steps under that kernel; the resulting scheme achieves an $dX_t = f(X_t, u_t)\,dt + \sigma(X_t,u_t)\,dB_t,$7-optimal solution in $dX_t = f(X_t, u_t)\,dt + \sigma(X_t,u_t)\,dB_t,$8 iterations, with approximation error tied to a measure of non-rectangularity (Li et al., 2023).

A different branch quantifies gradient uncertainty probabilistically. Bayesian policy gradient and Bayesian actor-critic model either the gradient itself or the action-value function as a Gaussian process and derive a posterior mean and covariance for $dX_t = f(X_t, u_t)\,dt + \sigma(X_t,u_t)\,dB_t,$9. In the actor-critic version,

$L(x,u)$0

so uncertainty is a full covariance matrix rather than an adversarial set. The variance-aware BPG-var method then scales the learning rate by a function of posterior gradient variance (Ghavamzadeh et al., 30 Apr 2026). In continuous control, UA-DDPG supplies yet another perspective: aleatoric uncertainty is represented by quantile critics, epistemic uncertainty by critic ensembles, and the actor optimizes a risk-sensitive weighted sum of quantiles while exploration follows the gradient of an ensemble-variance score (Kanazawa et al., 2022).

6. Empirical behavior, trade-offs, and open issues

The empirical profile of the explicit GURAC algorithm is primarily one of training stabilization. On Pendulum-v1, GURAC-TD3 exhibits smooth, monotone-ish improvement, narrow confidence bands across seeds, and no catastrophic collapses in performance, whereas TD3 often reaches high return quickly but shows high variance across seeds and repeated performance collapses. Under injected actuation noise at test time, GURAC starts from higher mean return at zero noise and degrades smoothly with relatively narrow confidence intervals, while TD3 can display a slightly less steep decline in mean return but much wider variance across seeds (Qi, 20 Jul 2025).

The scaling-based actor-critic results point in a compatible direction. In the forest-management MDP experiments, larger $L(x,u)$1—especially $L(x,u)$2—reduces the standard deviation of both rewards and policy outputs and leads to faster decrease of actor MSE and faster reward convergence. The theoretical fluctuation scaling $L(x,u)$3 matches the observed ordering across $L(x,u)$4 values (Georgoudios et al., 25 Jan 2026). This is not the same mechanism as the GU-HJBI penalty, but it supports the more general thesis that actor-critic stability can be improved by explicitly designing for gradient or estimator uncertainty.

Two technical caveats recur across the literature. First, robustness is not uniform across formulations. The GU-HJBI-based GURAC robustifies internal value-gradient ambiguity (Qi, 20 Jul 2025); Wasserstein actor-critic robustifies transition kernels (Hafsi et al., 25 Jun 2026); Bayesian actor-critic quantifies posterior uncertainty in the policy gradient (Ghavamzadeh et al., 30 Apr 2026); and ensemble-distributional methods separate epistemic from aleatoric uncertainty in critic outputs (Kanazawa et al., 2022). These are related but non-equivalent uncertainty models. Second, the strongest convergence guarantees are typically available either at the PDE level, in tabular robust MDPs, or for stylized shallow-network scaling limits. Direct convergence theory for deep, off-policy, function-approximation GURAC remains limited: the GU-HJBI work justifies the penalty structurally but not the full TD3-like algorithm, and the Wasserstein robust actor-critic work explicitly notes the absence of a full convergence proof with function approximation (Qi, 20 Jul 2025, Hafsi et al., 25 Jun 2026).

The resulting picture is technically coherent but not yet unified. GURAC is explicit as a robust-control-derived actor-critic with an actor penalty based on critic state-gradients (Qi, 20 Jul 2025). At the same time, adjacent research suggests a broader research program in which gradient uncertainty can be handled by adversarial perturbation models, finite-width asymptotics, robust Bellman recursions, posterior covariance estimation, or ensemble-distributional critics. A plausible implication is that future GURAC formulations will combine these strands rather than choose only one.