Policy Iteration for Stationary Discounted Hamilton--Jacobi--Bellman Equations: A Viscosity Approach

Abstract: We study policy iteration (PI) for deterministic infinite-horizon discounted optimal control problems, whose value function is characterized by a stationary Hamilton--Jacobi--Bellman (HJB) equation. At the PDE level, PI is fundamentally ill-posed: the improvement step requires pointwise evaluation of $\nabla V$, which is not well defined for viscosity solutions, and thus the associated nonlinear operator cannot be interpreted in a stable functional sense. We develop a monotone semi-discrete formulation for the stationary discounted setting by introducing a space-discrete scheme with artificial viscosity of order $O(h)$. This regularization restores comparison, ensures monotonicity of the discrete operator, and yields a well-defined pointwise policy improvement via discrete gradients. Our analysis reveals a convergence mechanism fundamentally different from the finite-horizon case. For each fixed mesh size $h>0$, we prove that the semi-discrete PI sequence converges monotonically and geometrically to the unique discrete solution, where the contraction is induced by the resolvent structure of the discounted operator. We further establish the sharp vanishing-viscosity estimate $$|V^h - V|_{L^\infty} \leq C\sqrt{h}$$, and derive a quantitative error decomposition that separates policy iteration error from discretization error, exhibiting a nontrivial coupling between iteration count and mesh size. Numerical experiments in nonlinear one and two-dimensional control problems confirm the theoretical predictions, including geometric convergence and the characteristic decay-then-plateau behavior of the total error.

• The paper introduces a viscosity-based policy iteration method that resolves classical ill-posedness by using a monotone finite difference scheme.
• It employs artificial viscosity, ensuring geometric convergence with a contraction rate tied to discretization and a vanishing-viscosity error bound of order √h.
• Decoupling iteration and discretization errors, the approach provides precise error bounds and actionable insights for deterministic infinite-horizon optimal control.

Policy Iteration for Stationary Discounted Hamilton–Jacobi–Bellman Equations: A Viscosity Approach

Introduction and Motivation

This paper provides a rigorous analysis and practical scheme for solving deterministic infinite-horizon discounted optimal control problems via policy iteration (PI), using viscosity solution theory and monotone semi-discrete approximation methods (2604.10191). In such continuous-state control problems, the value function satisfies a stationary discounted Hamilton–Jacobi–Bellman (HJB) partial differential equation (PDE). However, the classical PDE-level interpretation of PI—repeatedly improving a policy using gradients of the value function—fails due to the lack of sufficient regularity: viscosity solutions generally have only Lipschitz continuity, so their gradients are not pointwise defined, rendering the core policy improvement step ill-posed.

To resolve this, the paper introduces a monotone discretization incorporating artificial viscosity of order $O(h)$ and analyzes both convergence of the resulting PI scheme and discretization error. The authors obtain several strong results, notably geometric (linear) convergence of the PI scheme for fixed grid spacing, a sharp vanishing-viscosity error bound of order $\sqrt{h}$, and quantitative decoupling of policy-iteration and discretization errors.

Ill-Posedness in Continuous-Space Policy Iteration

The core difficulty with PI for continuous-time, deterministic control arises at the analytical level: although the value function $V$ is a viscosity solution to the stationary HJB equation,

$\lambda V(x) + H(x, \nabla V(x)) = 0,$

regularity is limited to Lipschitz continuity. The classical PI approach requires a well-defined gradient, since the policy improvement update is pointwise in $\nabla V$. This is not possible in general viscosity solutions, so the policy map cannot be defined robustly or iterated reliably. Thus, convergence and even well-posedness for continuous PI fail without addressing this regularity gap.

Monotone Viscosity Regularization: Restoring Well-Posedness

To overcome nonexistence of pointwise gradients, the authors pursue a monotone finite difference scheme with an added artificial viscosity term proportional to $h$, the spatial discretization width. This semi-discrete operator takes the form:

$$\lambda V^h(x) + H(x, \nabla_h V^h(x)) = N h \Delta_h V^h(x),$$

where $\nabla_h$ and $\Delta_h$ are the discrete gradient and Laplace operators, and $N$ is chosen large enough to ensure monotonicity. This structure guarantees several properties:

• Comparison principle by monotonicity,
• Pointwise policy improvement via discrete gradients,
• Sufficient coercivity for stable fixed-point iteration,
• Compatibility with viscosity solution theory.

With this setup, policy evaluation and improvement steps are well-defined at the discrete level for each grid size $\sqrt{h}$0.

Semi-Discrete Policy Iteration Algorithm

The discrete PI scheme comprises two alternating steps for mesh size $\sqrt{h}$1:

1. Policy evaluation: Given a policy $\sqrt{h}$2, solve the linear system

$\sqrt{h}$3

2. Policy improvement: Update the policy using

$\sqrt{h}$4

The discrete operator is contractive in the $\sqrt{h}$5 norm due to the stationarity and the discount factor, and the sequence of iterates converges monotonically.

Convergence Analysis

Geometric Convergence for Fixed Mesh Size

For each fixed $\sqrt{h}$6, the authors prove that the policy iteration sequence $\sqrt{h}$7 converges geometrically to the unique solution $\sqrt{h}$8 of the semi-discrete HJB equation. The contraction rate,

$\sqrt{h}$9

is governed by the artificial viscosity $V$0, spatial dimension $V$1, and the discount factor $V$2. Notably, as $V$3, $V$4, representing a trade-off between spatial resolution and iteration speed.

Figure 1: Value function iterates from policy iteration versus the analytic solution, confirming geometric convergence and the effectiveness of the monotone regularization.

Sharp Vanishing-Viscosity Discretization Error

The scheme achieves a strong discretization error bound. Specifically, the semi-discrete solution $V$5 converges to the viscosity solution $V$6 of the original HJB equation as $V$7 with rate:

$V$8

This rate is optimal for first-order Hamilton–Jacobi equations with monotone, viscosity-type regularization.

Error Decomposition and Coupling

A key insight is the decoupling of iteration and discretization errors:

$V$9

where the first term decays geometrically with the iteration number $\lambda V(x) + H(x, \nabla V(x)) = 0,$0 and the second is the asymptotic discretization error. To achieve error $\lambda V(x) + H(x, \nabla V(x)) = 0,$1, the number of PI steps must scale like $\lambda V(x) + H(x, \nabla V(x)) = 0,$2, quantifying the iteration-discretization trade-off.

Numerical Validation and Strong Claims

The theoretical results are substantiated by detailed numerical experiments in one and two dimensions. Highlights include:

• Geometric decay of the policy iteration error with respect to $\lambda V(x) + H(x, \nabla V(x)) = 0,$3 for fixed $\lambda V(x) + H(x, \nabla V(x)) = 0,$4,
• Error plateau once the discretization limit is reached,
• Monotone convergence in nonlinear 2D benchmarks,
• Consistency with reference (analytic or manufactured) solutions.

Figure 2: Policy iteration value function $\lambda V(x) + H(x, \nabla V(x)) = 0,$5 versus manufactured reference $\lambda V(x) + H(x, \nabla V(x)) = 0,$6 for a representative fixed $\lambda V(x) + H(x, \nabla V(x)) = 0,$7, showing precise matching and error decay.

Figure 3: Physics-informed neural network (PINN) prediction of the value function on a slice $\lambda V(x) + H(x, \nabla V(x)) = 0,$8 at $\lambda V(x) + H(x, \nabla V(x)) = 0,$9, illustrating the viability of neural approximations in the semi-discrete setting.

Theoretical and Practical Implications

The analysis establishes a robust PDE-based framework for applying dynamical programming and PI methods to deterministic infinite-horizon discounted control problems in continuous state spaces. By resolving the ill-posedness via monotonic viscosity regularization, the scheme offers a clearly justified, convergent alternative to ad hoc regularizations in reinforcement learning and control.

Practically, the approach guides numerical implementation in reinforcement learning, optimal control, and PDE-constrained optimization. The discrete contraction mechanism emphasizes the importance of the discount factor and demonstrates that iterative solvers can be tuned in tandem with discretization parameters for computational efficiency.

Theoretically, this work complements recent results for stochastic control, entropy-regularized, and exploratory HJB settings, as well as operator- and neural-based PI variants. The precise decoupling of iteration and discretization errors clarifies the reliability and limitations of PI in deterministic settings, especially as the discount vanishes or dimensionality increases.

Future Directions

Several promising research avenues are suggested:

• Extension to undiscounted (ergodic) control where no resolvent term is present, raising challenges for uniqueness and contraction.
• Integration with high-dimensional and neural-based solvers for scalable control and reinforcement learning—PINN and operator learning approaches are suggested as compatible.
• Analysis of policy iteration in the presence of additional structure, such as constraints or stochasticity, leveraging viscosity solution techniques and monotone schemes.

Conclusion

By formulating a monotone, viscosity-regularized policy iteration scheme and analyzing its convergence properties for stationary discounted HJB equations, the paper (2604.10191) systematically addresses a core analytical and numerical challenge in continuous deterministic optimal control. The division of iteration and discretization errors, along with geometric convergence and optimal discretization rates, are rigorously established and thoroughly validated numerically. These results both consolidate the theoretical foundation for PI in deterministic control and inform practical algorithm design for high-accuracy numerical and machine learning applications in control.