Bellman Gradient Iteration in RL and Control

• Bellman Gradient Iteration (BGI) is a framework that computes explicit gradients from Bellman residuals, enabling stable and scalable optimization in control and reinforcement learning.
• It employs smooth approximations, fixed-point recursions, and closed-form gradient flows—such as through Lyapunov equations—to ensure convergence and enhanced stability compared to classical methods.
• BGI is applied across continuous-time LQR, online IRL, and gradient TD learning, providing sample-efficient performance and scalability in high-dimensional decision-making tasks.

Bellman Gradient Iteration (BGI) refers to a class of iterative methods that enable exact or approximate policy and value optimization in control, reinforcement learning (RL), and inverse reinforcement learning (IRL) by computing explicit gradients of Bellman or Bellman-like residuals with respect to key parameters. These methods utilize differentiable approximations of the Bellman operator, smooth fixed-point recursions, and ODE-based flows to facilitate gradient-based optimization, offering convergence guarantees and improved stability relative to classical semigradient approaches. BGI has emerged independently in continuous-time optimal control, policy-gradient RL, (online) IRL, and gradient temporal-difference (TD) learning.

1. Continuous-Time Formulation: LQR Bellman Gradient Iteration

In continuous-time infinite-horizon Linear Quadratic Regulation (LQR), BGI provides a direct route to computing the optimal feedback gain by recasting the Hamilton-Jacobi-Bellman (HJB) residual as a parametric, differentiable Bellman error in terms of the feedback matrix $K$ (Gießler et al., 11 Jun 2025). For a system $\dot{x} = Ax + Bu$ and quadratic cost $\int_0^\infty [x^{\top}Qx + u^{\top}Ru]dt$, the candidate value $V(x)=x^\top P x$ leads to the algebraic Riccati equation. Instead of iterating on $P$, the BGI method proceeds as follows:

• Bellman Error Definition: For any stabilizing $K$ (i.e., $A-BK$ Hurwitz), the unique $P_K$ solves

$A_K^\top P_K + P_K A_K + Q + K^\top R K = 0$

and the scalar Bellman error is set as $E(K) := -\operatorname{tr}[M_K] \ge 0$ with

$\dot{x} = Ax + Bu$0

$\dot{x} = Ax + Bu$1 if and only if $\dot{x} = Ax + Bu$2 is the unique optimal gain.

• Closed-Form Gradient: The gradient of $\dot{x} = Ax + Bu$3 with respect to $\dot{x} = Ax + Bu$4 can be expressed as

$\dot{x} = Ax + Bu$5

where $\dot{x} = Ax + Bu$6 solves a Lyapunov equation coupled to $\dot{x} = Ax + Bu$7.

• Gradient Flow: The ODE $\dot{x} = Ax + Bu$8 generates a trajectory staying entirely within the stabilizing set and converging globally and asymptotically to $\dot{x} = Ax + Bu$9.
• Analytic Properties: $\int_0^\infty [x^{\top}Qx + u^{\top}Ru]dt$0 is real-analytic, coercive, and has a unique global minimum in the stabilizing region. This re-parametrization leverages Lyapunov equations to circumvent infinite-horizon challenges, providing a globally convergent, actor-gradient policy improvement flow entirely within the stabilizing set.

The BGI framework in this context unifies LQR and continuous-time reinforcement learning perspectives, providing a state-independent, trace-based Bellman error as the central optimization object (Gießler et al., 11 Jun 2025).

2. Differentiable Bellman Approximations in IRL

BGI was first formalized in the IRL literature to address the need for exact gradients of the Q-function with respect to the reward parameters, circumventing the nondifferentiability of the Bellman "max" operator (Li et al., 2017, Li et al., 2017). The principal methods employ:

• Smooth Approximations of Bellman Optimality: The hard maximum is replaced by either
  • $\int_0^\infty [x^{\top}Qx + u^{\top}Ru]dt$1-norm approximation: $\int_0^\infty [x^{\top}Qx + u^{\top}Ru]dt$2
  • $\int_0^\infty [x^{\top}Qx + u^{\top}Ru]dt$3-softmax: $\int_0^\infty [x^{\top}Qx + u^{\top}Ru]dt$4
  • These approximations yield smooth surrogate Bellman operators suitable for gradient computation.
• Bellman Gradient Fixed-Point Recursion: The Q- and V-functions are first computed via standard value iteration under the approximate max. Then, by differentiating the approximate Bellman equations, one forms new, coupled fixed-point equations for $\int_0^\infty [x^{\top}Qx + u^{\top}Ru]dt$5 and $\int_0^\infty [x^{\top}Qx + u^{\top}Ru]dt$6, which are solved with a second round of value-iteration-style sweeps ("Bellman Gradient Iteration").
• Algorithmic Structure: BGI proceeds as a two-stage process:
  1. Compute smoothed Q, V via standard approximate value iteration,
  2. Compute their derivatives with respect to the reward parameters through the derived gradient fixed-points. This machinery enables gradient ascent in the IRL log-likelihood for reward learning.

• Action Preference Handling: The $\int_0^\infty [x^{\top}Qx + u^{\top}Ru]dt$7 parameter of the approximation controls the sharpness of the approximation—low $\int_0^\infty [x^{\top}Qx + u^{\top}Ru]dt$8 blends near-optimal actions, enabling modeling of stochastic or non-deterministic agents; $\int_0^\infty [x^{\top}Qx + u^{\top}Ru]dt$9 recovers the hard max (Li et al., 2017, Li et al., 2017).

• Computational Complexity: Each BGI pass runs in $V(x)=x^\top P x$0 per iteration (per value sweep), with empirical times described for state sizes up to 1600 on CPU and GPU hardware.

3. Online IRL and Real-Time BGI

In the online IRL setting, BGI enables immediate policy update and reward learning from sequentially arriving state-action pairs. The method only stores the current reward estimate and the latest observation. At each new expert demonstration, the gradient of the policy log-likelihood with respect to reward parameters is efficiently computed via BGI, rendering it feasible for deployment in robotic and persistent monitoring scenarios (Li et al., 2017).

Experimentally, BGI-based online IRL grows correlation with the ground-truth reward monotonically with increasing samples and demonstrates competitive or superior sample-efficiency in both linear and nonlinear reward settings (NN parameterization) compared to alternative approaches.

4. Temporal-Difference Learning: Gradient Iterated TD

BGI as instantiated in "Gradient Iterated Temporal-Difference learning" (Gi-TD) introduces a class of algorithms that maintain a sequence (chain) of $V(x)=x^\top P x$1 action-value functions. Each head $V(x)=x^\top P x$2 tracks the $V(x)=x^\top P x$3-fold application of the Bellman operator to an initial target network $V(x)=x^\top P x$4 (Vincent et al., 8 Mar 2026).

• Iterated Bellman Chain: $V(x)=x^\top P x$5 propagates reward information $V(x)=x^\top P x$6 steps deep per gradient update, enhancing the speed of reward propagation relative to classical TD-learning.
• Objective Function: BGI minimizes the sum of squared Bellman errors across the chain:

$V(x)=x^\top P x$7

Saddle-point optimization with auxiliary networks $V(x)=x^\top P x$8 for each $V(x)=x^\top P x$9 permits unbiased, stable gradient estimation.

• Algorithmic Details: At each step, all $P$0 and $P$1 heads are updated in parallel with SGD. Regularized corrections and chain shifting are crucial for stability.
• Theoretical and Empirical Properties: BGI admits policy suboptimality bounds derived from controlling the sum of Bellman errors. Empirically, BGI (Gi-TD) achieves sample-efficiency on par with semi-gradient methods but with provable convergence guarantees, delivering robust performance under high data-reuse settings and large unrolled iterations ($P$2).

5. Connections, Distinctives, and Practical Considerations

The BGI formalism offers several distinctive features and practical advantages across RL, IRL, and control:

• Unified Gradient Architecture: BGI subsumes classic TD and policy iteration for both policy evaluation (TD) and direct policy optimization (policy gradient), including continuous-time generalizations.
• Stability and Convergence: By maintaining all gradients with respect to moving targets (including recursive Bellman updates) and by leveraging smooth approximations, BGI achieves stability that is elusive for semi-gradient TD and similar methods. In continuous-time LQR, the entire optimization remains within the stabilizing gain set.
• Full Differentiability: The closed-form (or efficiently computable) gradients admit modern optimization techniques (Adam, RMSProp) and enable integration into neural and differentiable programming pipelines.
• Scalability: Empirical benchmarks report efficient CPU and (especially) GPU implementations, enabling BGI application to large MDPs and high-dimensional function approximators.
• Limitations: BGI's performance and accuracy for IRL and online IRL depend on the smoothness parameter $P$3, tuning of learning rates, and managing local optima (addressed via multi-start and convexification at high $P$4). High sample complexity can arise in very large state-action domains. For TD variants, architectural strategies and regularization of auxiliary heads are necessary for best performance (Gießler et al., 11 Jun 2025, Li et al., 2017, Vincent et al., 8 Mar 2026, Li et al., 2017).

6. Comparative Summary and Empirical Results

Empirical findings across LQR, small MDPs, offline/online IRL, and deep RL benchmarks validate the core advantages of BGI:

Domain	BGI Variant	Key Metrics/Findings
LQR (continuous-time)	Bellman-gradient ODE	Linear convergence to $P$5; faster contraction vs. classical LQR gradient flow
IRL (batch)	Value-gradient BGI	Policy accuracy: matches or exceeds MaxEnt/Bayes IRL (linear); approaches DeepMaxEnt (NN)
IRL (online)	Online BGI	Monotonic reward correlation growth; efficient (~$P$6 per sample)
RL (deep TD)	Gi-TD BGI	Sample-efficiency matches semi-gradient methods; stability in high-update regimes

In gridworld/Objectworld IRL, p-norm $P$7 (gridworld) or $P$8-soft $P$9 suffices for near-optimal value recovery. For Gi-TD, robust performance is observed for $K$0. BGI-based cleaning robot in simulation attains near-optimal energy usage when reward is learned via nonlinear BGI-NN (Gießler et al., 11 Jun 2025, Li et al., 2017, Vincent et al., 8 Mar 2026, Li et al., 2017).

7. Extensions and Potential Research Directions

BGI provides a modular and extensible foundation for optimization in RL and control:

• Extensions to continuous state/action spaces via function approximation.
• Actor–critic variants in IRL to reduce per-step computational burden.
• Adaptation to partially observable settings and stochastic/robust control.
• Further integration with neural network approximators for deep RL and IRL.
• Exploration of alternative smooth approximations beyond p-norm/g-soft for specialized tasks.

BGI's principled approach to differentiable Bellman errors and stability-aware gradient propagation continues to draw interest across RL, IRL, and optimal control research (Gießler et al., 11 Jun 2025, Li et al., 2017, Vincent et al., 8 Mar 2026, Li et al., 2017).