---
title: 'PINN-PI: Neural Policy Iteration in Control'
url: https://www.emergentmind.com/topics/physics-informed-neural-network-policy-iteration-pinn-pi
type: topic
---

# PINN-PI: Neural Policy Iteration in Control

Searching arXiv for recent papers on Physics-Informed Neural Network Policy Iteration to ground the article in the cited literature.
Physics-Informed Neural Network Policy Iteration (PINN-PI) denotes a family of model-based algorithms in which policy iteration is carried out through neural solution of control PDEs. Across the published formulations, the defining pattern is to freeze a policy or feedback law, solve the resulting linearized HJB-, HJI-, or Zubov-type PDE with a physics-informed neural network, and then update the policy by pointwise optimization of the corresponding Hamiltonian. The framework appears in deterministic infinite-horizon optimal control, second-order stochastic control, entropy-regularized control, nonconvex stochastic differential games, PDE control, and robust region-of-attraction computation [2402.10119], [2508.01718], [2508.01720], [2507.15455], [2305.10952], [2508.19398].

## 1. Mathematical scope and problem classes

In deterministic nonlinear optimal control, PINN-PI is formulated for control-affine systems
\[
\dot x = f(x) + g(x)\,u,
\]
with infinite-horizon cost
\[
J(x_0,u)=\int_0^\infty [Q(x(t)) + u(t)^T R\,u(t)]\,dt.
\]
The value function is the viscosity solution of the HJB equation, and exact policy iteration replaces the fully nonlinear HJB by a sequence of linear generalized HJB equations
\[
Q(x) + u_i(x)^T R\,u_i(x)+\nabla V_i(x)\cdot[f(x)+g(x)u_i(x)] = 0,
\]
followed by the update
\[
u_{i+1}(x)= -\tfrac12 R^{-1}g(x)^T\nabla V_i(x)^T.
\]
This deterministic template is the basis of the algorithms in "Physics-Informed Neural Network Policy Iteration: Algorithms, Convergence, and Verification" [2402.10119].

In stochastic control, the same logic is applied to second-order elliptic HJB equations. For
\[
dX_t = b(X_t,a_t)\,dt + \sigma\,dW_t,
\]
with discounted running cost \(L(x,a)\), the value function satisfies
\[
\lambda V(x) - \tfrac12 \operatorname{tr}[\sigma\sigma^T D^2V(x)] - \sup_{a\in A}\{b(x,a)\cdot \nabla V(x)+L(x,a)\}=0.
\]
When a Markov policy \(\pi_k\) is frozen, the policy-evaluation step becomes the linear PDE
\[
\tfrac12 \operatorname{tr}[\sigma\sigma^T D^2V] + b(x,\pi_k(x))\cdot \nabla V + L(x,\pi_k(x))=0,
\]
and the policy-improvement step is
\[
\pi_{k+1}(x)=\arg\min_{u\in A}\{L(x,u)+\nabla V^{\pi_k}(x)\cdot b(x,u)\}.
\]
This second-order setting is developed in "Neural Policy Iteration for Stochastic Optimal Control: A Physics-Informed Approach" [2508.01718].

Entropy-regularized stochastic control introduces randomized policies \(\pi(x,\cdot)\in \mathcal P(U)\) and the regularized objective
\[
J^\pi(x)=\E_x\!\Bigl[\int_0^\infty e^{-\rho t}\!\int_U \bigl(r(X_t,u)-\lambda\ln \pi(X_t,u)\bigr)\pi(X_t,u)\,du\,dt\Bigr].
\]
Its soft HJB takes the form
\[
\rho V(x)=\tfrac12\operatorname{tr}(\sigma\sigma^\top D^2V(x))
+\lambda\ln\!\Bigl(\int_U e^{\frac{b(x,u)\cdot\nabla V(x)+r(x,u)}{\lambda}}\,du\Bigr),
\]
and soft policy iteration alternates a linear evaluation PDE with the analytical improvement rule
\[
\pi^{n+1}(x,u)\propto \exp\!\Bigl(\tfrac{b(x,u)\cdot \nabla v^n(x)+r(x,u)}{\lambda}\Bigr).
\]
This formulation is given in "Physics-informed approach for exploratory Hamilton--Jacobi--Bellman equations via policy iterations" [2508.01720].

PINN-PI has also been extended to nonconvex Hamilton--Jacobi--Isaacs equations for zero-sum stochastic differential games. There the value
\[
v(t,x)=\sup_{\beta\in \Gamma_t}\inf_{a\in \mathcal A_t}\mathbb E\Bigl[\int_t^T c(X(s),a(s),\beta[a](s))\,ds + g(X(T))\Bigr]
\]
solves a nonconvex viscous HJI with minimax Hamiltonian
\[
H(t,x,p)=\sup_{b\in B}\inf_{a\in A}[c(t,x,a,b)+p\cdot f(t,x,a,b)],
\]
and PINN-based policy iteration alternates linear PDE solves under fixed feedback pairs \((\alpha_n,\beta_n)\) with pointwise minimax updates [2507.15455].

A further specialization targets robust stability rather than optimal regulation. For perturbed dynamics
\[
\dot x(t)=f(x(t),\delta(t)),\qquad \delta(t)\in \Delta,
\]
the robust region of attraction is
\[
\mathcal A=\{x\in \mathbb R^n:\forall\,\delta(\cdot),\ \phi(t;x,\delta)\to 0\text{ as }t\to\infty\},
\]
and the generalized maximal Lyapunov function
\[
V(x)=\sup_{\delta(\cdot)\in \Delta}\int_0^\infty g(\phi(t;x,\delta),\delta(t))\,dt
\]
induces the bounded transform
\[
v(x)=1-e^{-\alpha V(x)}.
\]
The corresponding generalized Zubov PDE is
\[
\sup_{\delta\in \Delta}\Bigl\{\nabla v(x)^\top f(x,\delta)+\alpha(1-v(x))g(x,\delta)\Bigr\}=0,
\]
with \(v(0)=0\), \(\lim_{\|x\|\to\infty}v(x)=1\), and \(\mathcal A=\{x:v(x)<1\}\). This robust-region formulation underlies "Learning Robust Regions of Attraction Using Rollout-Enhanced Physics-Informed Neural Networks with Policy Iteration" [2508.19398].

## 2. Canonical algorithmic structure

The common algorithmic structure is an alternation between policy evaluation and policy improvement. In the evaluation phase, a neural approximant \(V_\theta\), \(v_\theta\), or \(v_n(\cdot;\theta_n)\) is trained so that its derivatives satisfy the PDE associated with the current policy. Because the policy is frozen, the evaluation equation is linear in the unknown value function in the deterministic generalized HJB, the stochastic elliptic HJB, the soft-policy evaluation PDE, and the fixed-policy HJI formulations [2402.10119], [2508.01718], [2508.01720], [2507.15455].

In the improvement phase, the trained value network is differentiated with automatic differentiation, and its gradient is inserted into a Hamiltonian optimizer. The specific optimizer depends on the problem class. Deterministic control uses the closed-form update
\[
u_{i+1}(x)= -\tfrac12 R^{-1}g(x)^T\nabla V_i(x)^T
\]
[2402.10119]. Stochastic optimal control uses a greedy Hamiltonian minimization
\[
\pi_{k+1}(x)=\arg\min_{u\in A}\{L(x,u)+\nabla V_\theta(x)\cdot b(x,u)\}
\]
[2508.01718]. Entropy-regularized control uses the softmax policy induced by \(\nabla V_\theta\) [2508.01720]. Zero-sum games use the pointwise minimax pair
\[
\beta_{n+1}(t,x)=\arg\max_{b\in B}\inf_{a\in A}L(t,x,p_n(t,x))(a,b),\qquad
\alpha_{n+1}(t,x)=\arg\min_{a\in A}L(t,x,p_n(t,x))(a,\beta_{n+1}(t,x))
\]
with \(p_n=\nabla_x v_n\) [2507.15455].

The robust-region variant replaces policy improvement by disturbance improvement. The disturbance field is updated through
\[
\delta^*(x)=\arg\max_{\delta\in \Delta}\Bigl\{\nabla v_\theta(x)^\top f(x,\delta)+\alpha(1-v_\theta(x))g(x,\delta)\Bigr\},
\]
and rollout-generated anchor values are then computed under the current disturbance sequence to stabilize the next evaluation step [2508.19398].

A hybrid actor-critic version appears in PDE control of fluid-cooled battery packs. There, the value network is a PINN for the continuous-time HJB, while the actor is updated with PPO. During rollout, actions are selected with probability \(1/2\) from the PINN-derived controller and otherwise from the Gaussian policy network. This formulation is denoted HJBPPO in the reported experiments [2305.10952].

## 3. Neural parameterizations, residual losses, and sampling

Reported PINN-PI implementations use standard fully connected feedforward architectures, but the architecture is tuned to the PDE class. The deterministic control formulation in [2402.10119] uses a deep feed-forward network \(V_i(x;\theta)=NN(x;\theta)\) for PINN-PI and a one-layer random-feature model for ELM-PI. The stochastic-control formulation in [2508.01718] reports 4–6 hidden layers with 128–256 neurons and tanh or SiLU activations. The soft HJB formulation in [2508.01720] reports 3–5 hidden layers with 64–128 neurons and \(\tanh\) or ReLU activations. The HJI game solver in [2507.15455] uses 3–4 hidden layers with 64 neurons per layer and sinusoidal or tanh activations. The generalized Zubov solver in [2508.19398] uses 5 hidden layers of width \(w=50\), sigmoid activations, and a linear output layer. The battery-pack HJB value network is described as a typical MLP with 3–5 hidden layers, 64–128 neurons, and tanh activations [2305.10952].

The training objective is always centered on a PDE residual, but the auxiliary terms vary by formulation. Deterministic PINN-PI uses the residual of the generalized HJB together with \(V_i(0)=0\) and a stability-preserving term
\[
\lambda_s\|\nabla V_i(0;\theta)-K_i\|_F^2
\]
that enforces matching of the linearization at the origin [2402.10119]. The battery-pack setting adds a Dirichlet condition \(V_\phi(0,-R)=0\) and a Neumann condition \(\nabla V_\phi(0,-R)=0\) to the residual loss [2305.10952]. The generalized Zubov formulation decomposes the loss into boundary, residual, and data-anchor components,
\[
\hat\ell(\theta)=\hat\ell_{\rm bnd}(\theta)+\lambda_r \hat\ell_{\rm res}(\theta)+\lambda_d \hat\ell_{\rm data}(\theta),
\]
where the anchor targets \(\hat v_i\) come from rollout estimates [2508.19398]. The soft HJB formulation separates value and policy training, using a PINN residual loss for \(V_\theta\) and a KL or cross-entropy loss to fit the policy network \(\pi_\omega\) to the analytical softmax update [2508.01720].

Sampling is likewise problem-dependent. The stochastic-control paper reports uniform or quasi-Monte Carlo collocation in bounded domains, with \(N\approx 10^4\)–\(10^5\) points and truncation of \(\mathbb R^d\) to a large ball when necessary [2508.01718]. The HJI paper refreshes 1,000–2,000 interior collocation points every 100 epochs and warm-starts each policy-evaluation step from the previous network parameters [2507.15455]. The generalized Zubov method samples separate sets on \(\partial \Omega\), in the interior for the PDE residual, and in the interior for data anchors; in the reported 2D experiments \(M_b=M_r=2\times 10^4\) and \(M_d=2\times 10^3\), while the 10D case uses \(M_b=M_r=5\times 10^4\) and \(M_d=3\times 10^4\) [2508.19398]. Across the surveyed works, optimization is performed with Adam, sometimes followed by L-BFGS or accompanied by periodic resampling [2508.01718], [2507.15455], [2508.19398].

## 4. Convergence theory and error control

A central motivation for PINN-PI is that freezing the policy converts a nonlinear control PDE into a sequence of linear evaluation problems, which are more amenable to both training and analysis. In the deterministic control-affine setting, exact policy iteration yields a sequence \(V_0>V_1>\cdots>V^*\) pointwise, and \(V_i\to V^*\) uniformly on compact sets; the generalized HJB at each step has a unique positive-definite viscosity solution \(V_i\in C(\mathcal Q)\cap C^1(\mathcal Q\setminus\{0\})\) [2402.10119]. In stochastic optimal control, exact policy iteration inherits the global exponential contraction
\[
\|V^{\pi_n}-V^*\|_{L^2}\le C\kappa^n,\qquad \kappa\in (0,1),
\]
under Assumptions (A1)–(A4) including compact convex \(A\), Lipschitz \(b,L\), strong convexity in \(a\), and uniform ellipticity of \(\sigma\sigma^\top\) [2508.01718]. In entropy-regularized control, the exact soft policy-iteration operator yields iteration error \(\|v^n-V\|_{L^2}\le C_{\mathcal X}\kappa^n\) [2508.01720]. In nonconvex HJI, exact value iterates converge locally uniformly to the unique bounded continuous viscosity solution, and an exponential \(L^2\) rate
\[
\sup_{t\in[0,T]}\|v_n(t,\cdot)-v(t,\cdot)\|_{L^2(\mathbb R^d)}\le C\rho^n
\]
is proved under compactness, strong convexity-concavity, Lipschitz continuity, and uniform ellipticity [2507.15455].

The practical PINN-PI analyses quantify how neural approximation error enters these classical policy-iteration dynamics. In [2508.01718], if \(p=\sup_n \|\text{residual}\|_{L^2}\), then
\[
\|\tilde V_n-V^*\|_{L^2}\le C[p+\kappa^n+\tilde \kappa^n],
\]
where \(\tilde \kappa<1\) comes from gradient-error propagation. The same paper proves a Lipschitz-type policy-gradient bound:
\[
\|\pi_{k+1}-\pi_{k+1}^{\mathrm{exact}}\|_{L^2}\le \theta \|\nabla V_\theta-\nabla V^{\pi_k}\|_{L^2}.
\]
In [2508.01720], the learned value error is decomposed into PDE residual error, policy-network error, and iteration error, and the main theorem states
\[
\|\tilde v^n-V\|_{L^2(\mathcal X)}\le C(r+q)+C_{\mathcal X}\kappa^n,
\]
with \(r=\sup_n\|r_n\|_{L^2}\) and \(q=\sup_n\|q_n\|_{L^2}\), so the total error is uniformly bounded by the approximation accuracies. In [2507.15455], the saddle-point selector is \(\kappa\)-Lipschitz in the co-state variable \(p\), and the practical PINN-PI error satisfies
\[
\sup_{t\in[0,T]}\|\tilde v_n(t,\cdot)-v(t,\cdot)\|_{L^2}\le C(p_n+\rho^n).
\]

The deterministic paper adds a convergence statement tailored to neural residual minimization: if the PINN or ELM residual can be driven to zero and the neural class approximates \(C^1\)-functions with small residual, then
\[
\sup_{x\in \mathcal Q\setminus U_\varepsilon}|\widehat V_i(x)-V_i(x)|+\sup_{x\in \mathcal Q\setminus U_\varepsilon}\|\widehat u_{i+1}(x)-u_{i+1}(x)\|\to 0
\]
as the number of collocation points grows and training converges, for any excluded ball \(U_\varepsilon\) around the origin [2402.10119]. This body of theory distinguishes PINN-PI from direct one-shot neural solution of the fully nonlinear PDE: the error analysis is organized around policy-evaluation residuals and policy-update stability rather than around a single nonlinear residual objective.

## 5. Robust stability, regions of attraction, and verification

The robust-region-of-attraction variant reframes PINN-PI as Lyapunov-function learning for perturbed systems. The generalized Zubov equation produces a bounded viscosity solution \(v\) whose strict sublevel set \(\{x:v(x)<1\}\) equals the robust region of attraction. The reported method alternates between training \(v_\theta\) against the Zubov residual and updating the disturbance field through pointwise maximization of the generalized Hamiltonian. A distinctive feature is the rollout-enhanced anchor strategy:
\[
\hat V_i=\sum_{k=0}^K \|x_{i,k}\|^2\,\Delta t,\qquad
x_{i,k+1}=x_{i,k}+f(x_{i,k},\delta^*_{i,k})\,\Delta t,
\]
followed by
\[
\hat v_i=1-\exp(-\alpha \hat V_i).
\]
These anchors are used as supervised labels in the loss to prevent the trivial singular solution \(v\equiv 1\) outside a tiny neighborhood of the origin and to guide training toward the true viscosity solution [2508.19398].

A different stability-oriented thread appears in the deterministic verification framework. There the learned value and policy are checked against a Lyapunov decrease condition
\[
\nabla \widehat V(x)\cdot [f(x)+g(x)\widehat u(x)]\le -p
\]
on \(\mathcal Q\setminus U_\varepsilon\), together with nested level-set conditions \(L_{c_2}\subset \mathcal Q\) and \(L_{c_1}\subset U_\varepsilon\). Because exact checking is undecidable in general, the procedure uses a \(\delta\)-complete SMT solver, dReal, to certify the decrease condition up to \(\delta\) precision or produce a counterexample. The reported outcome is a verifiable region of attraction around the origin [2402.10119].

These two strands are closely related but not identical. The generalized Zubov approach computes a robust region of attraction directly from a PDE whose solution is itself a Lyapunov-like function, whereas the verification pipeline begins with a learned optimal-control solution and then certifies a Lyapunov decrease condition for the induced closed loop. This suggests two distinct stability uses of PINN-PI: direct robust-attraction computation and post hoc certification of synthesized controllers.

## 6. Reported applications and empirical results

The published applications span ODE control, stochastic control, PDE control, differential games, and robust-attraction analysis.

| Setting | Representative task | Reported outcome |
|---|---|---|
| [2305.10952] | 1D PDE model for fluid-cooled battery packs | HJB Value Iteration: final reward \(\simeq -7.3\times 10^3\); PPO: \(\simeq -3.97\times 10^3\); HJBPPO: \(\simeq -8.8\times 10^2\) over 5 seeds and \(10^6\) timesteps each |
| [2508.01718] | Stochastic LQR, cartpole, pendulum | Value-error \(\sim 10^{-2}\) and policy-error \(\sim 10^{-2}\) in 20 PI iterations for 5D and 10D LQR; stable control in 5–10 PI steps on cartpole and pendulum |
| [2508.01720] | Entropy-regularized 5D/10D LQR, pendulum, cartpole | \(>98\%\) of optimal discounted reward in \(\sim 50\) policy iterations; SAC stalls below \(85\%\) in 10D |
| [2507.15455] | 2D stochastic path planning; 5D/10D publisher-subscriber game | 2D relative \(L^2\)-errors \(\le 10^{-3}\); 5D isotropic relative \(L^2\) at \(t=0\): \(1.17\times 10^{-2}\) vs direct \(1.12\times 10^{-1}\); 10D isotropic: \(5.80\times 10^{-2}\) vs direct \(1.96\times 10^{-1}\) |
| [2508.19398] | Perturbed Van der Pol, inverted pendulum, 10D decoupled system | Without rollout, PINN collapses to \(v\equiv 1\) almost everywhere; with rollout-anchors, matches finite-difference baselines in 2D and approximates true RROA \([-1,1]^{10}\) with only small distortions |
| [2402.10119] | Synthetic nonlinear benchmark, inverted pendulum, Lorenz, cartpole, 2D/3D quadrotor | ELM-PI attains \(10^{-3}\)–\(10^{-5}\) accuracy in seconds for \(n\le 3\); PINN-PI attains \(10^{-2}\)–\(10^{-3}\) accuracy in \(O(10^2 s)\) up to \(n=10\); Lorenz solved in \(\approx 7\,s\); SGA took \(>800\,s\) for the same accuracy |

The application record exhibits several recurring empirical patterns. First, PINN-PI is repeatedly positioned against direct nonlinear PDE solvers, classical grid-based methods, or model-free RL baselines. The differential-game study explicitly contrasts PINN-PI with direct PINN minimization of the one-shot Hamiltonian residual, and the deterministic control study compares against Galerkin methods and PPO-like baselines [2507.15455], [2402.10119]. Second, the advantage often comes from decomposition rather than from network architecture alone: the battery-pack study attributes the poor performance of pure HJB value iteration to poor exploration and the better performance of HJBPPO to the combination of physics-exploiting value updates with PPO exploration [2305.10952]. Third, the current empirical frontier in the surveyed literature is “moderately high dimensional”: multiple papers report successful 10D experiments, while also emphasizing mesh-free operation and the impracticality of finite differences beyond about three dimensions [2508.01718], [2508.01720], [2507.15455], [2508.19398].

## 7. Distinctions, limitations, and future directions

PINN-PI is not a single fixed algorithm. The surveyed literature includes deterministic generalized-HJB policy iteration, stochastic second-order HJB policy iteration, entropy-regularized soft policy iteration, minimax HJI policy iteration, rollout-enhanced Zubov learning, and actor-critic hybrids in which only the value update is physics-informed [2402.10119], [2508.01718], [2508.01720], [2507.15455], [2508.19398], [2305.10952]. A common misconception is to treat these methods as model-free RL with a PINN regularizer. In the reported formulations, the dynamics and PDE operators are explicit ingredients of the loss; even the battery-pack actor-critic method retains a continuous-time HJB residual for the critic [2305.10952], [2508.01720].

The limitations are equally formulation-specific. The entropy-regularized framework states that it requires smooth soft-max policies and does not directly handle bang-bang or discrete actions; it also relies on full knowledge of \(b,\sigma,r\) [2508.01720]. The HJI game solver assumes nondegenerate diffusion through uniform ellipticity and notes PINN sensitivities to initialization and collocation sampling [2507.15455]. The battery-pack study reports that pure HJB-PINN control converges to a suboptimal bang-bang strategy with poor exploration [2305.10952]. The generalized Zubov study reports collapse to the singular solution \(v\equiv 1\) when rollout anchors are removed [2508.19398]. The deterministic comparison between ELM-PI and PINN-PI indicates that the random-feature ELM variant is very fast and highly accurate in low dimensions but becomes impractical as dimension grows, whereas the deep PINN variant scales better to \(n=10\) [2402.10119].

The listed future directions are also diverse. The HJI work points to first-order HJI, adaptive sampling, operator learning through DeepONet/FNO, and integration with model-based RL [2507.15455]. The soft HJB work identifies joint learning of dynamics and value, risk-sensitive or constrained extensions via barrier/PINN coupling, and distributional or multi-agent objectives [2508.01720]. A plausible implication is that future PINN-PI research will be shaped by two competing requirements: preserving the linearized policy-evaluation structure that enables residual-based analysis, while enlarging the class of admissible policies, uncertainty models, and observation regimes.

Source: https://www.emergentmind.com/topics/physics-informed-neural-network-policy-iteration-pinn-pi