PINN-PI: Neural Policy Iteration in Control

• The paper introduces a framework that freezes policies to solve linearized control PDEs via physics-informed neural networks, enabling rigorous convergence guarantees.
• PINN-PI is applicable to deterministic, stochastic, entropy-regularized, and differential game scenarios, providing a versatile tool for high-dimensional control problems.
• Empirical results show that this approach achieves fast convergence, robust stability, and competitive performance against direct PDE solvers and model-free RL algorithms.

Searching arXiv for papers on Physics-Informed Neural Network Policy Iteration to ground the article in the cited literature. Physics-Informed Neural Network Policy Iteration (PINN-PI) denotes a family of model-based algorithms in which policy iteration is carried out through neural solution of control PDEs. Across the published formulations, the defining pattern is to freeze a policy or feedback law, solve the resulting linearized HJB-, HJI-, or Zubov-type PDE with a physics-informed neural network, and then update the policy by pointwise optimization of the corresponding Hamiltonian. The framework appears in deterministic infinite-horizon optimal control, second-order stochastic control, entropy-regularized control, nonconvex stochastic differential games, PDE control, and robust region-of-attraction computation (Meng et al., 2024, Kim et al., 3 Aug 2025, Kim et al., 3 Aug 2025, Yang et al., 21 Jul 2025, Mukherjee et al., 2023, Wang et al., 26 Aug 2025).

1. Mathematical scope and problem classes

In deterministic nonlinear optimal control, PINN-PI is formulated for control-affine systems

$\dot x = f(x) + g(x)\,u,$

with infinite-horizon cost

$$J(x_0,u)=\int_0^\infty [Q(x(t)) + u(t)^T R\,u(t)]\,dt.$$

The value function is the viscosity solution of the HJB equation, and exact policy iteration replaces the fully nonlinear HJB by a sequence of linear generalized HJB equations

$$Q(x) + u_i(x)^T R\,u_i(x)+\nabla V_i(x)\cdot[f(x)+g(x)u_i(x)] = 0,$$

followed by the update

$u_{i+1}(x)= -\tfrac12 R^{-1}g(x)^T\nabla V_i(x)^T.$

This deterministic template is the basis of the algorithms in "Physics-Informed Neural Network Policy Iteration: Algorithms, Convergence, and Verification" (Meng et al., 2024).

In stochastic control, the same logic is applied to second-order elliptic HJB equations. For

$dX_t = b(X_t,a_t)\,dt + \sigma\,dW_t,$

with discounted running cost $L(x,a)$, the value function satisfies

$$\lambda V(x) - \tfrac12 \operatorname{tr}[\sigma\sigma^T D^2V(x)] - \sup_{a\in A}\{b(x,a)\cdot \nabla V(x)+L(x,a)\}=0.$$

When a Markov policy $\pi_k$ is frozen, the policy-evaluation step becomes the linear PDE

$$\tfrac12 \operatorname{tr}[\sigma\sigma^T D^2V] + b(x,\pi_k(x))\cdot \nabla V + L(x,\pi_k(x))=0,$$

and the policy-improvement step is

$$\pi_{k+1}(x)=\arg\min_{u\in A}\{L(x,u)+\nabla V^{\pi_k}(x)\cdot b(x,u)\}.$$

This second-order setting is developed in "Neural Policy Iteration for Stochastic Optimal Control: A Physics-Informed Approach" (Kim et al., 3 Aug 2025).

Entropy-regularized stochastic control introduces randomized policies $$J(x_0,u)=\int_0^\infty [Q(x(t)) + u(t)^T R\,u(t)]\,dt.$$0 and the regularized objective

$$J(x_0,u)=\int_0^\infty [Q(x(t)) + u(t)^T R\,u(t)]\,dt.$$1

Its soft HJB takes the form

$$J(x_0,u)=\int_0^\infty [Q(x(t)) + u(t)^T R\,u(t)]\,dt.$$2

and soft policy iteration alternates a linear evaluation PDE with the analytical improvement rule

$$J(x_0,u)=\int_0^\infty [Q(x(t)) + u(t)^T R\,u(t)]\,dt.$$3

This formulation is given in "Physics-informed approach for exploratory Hamilton--Jacobi--Bellman equations via policy iterations" (Kim et al., 3 Aug 2025).

PINN-PI has also been extended to nonconvex Hamilton--Jacobi--Isaacs equations for zero-sum stochastic differential games. There the value

$$J(x_0,u)=\int_0^\infty [Q(x(t)) + u(t)^T R\,u(t)]\,dt.$$4

solves a nonconvex viscous HJI with minimax Hamiltonian

$$J(x_0,u)=\int_0^\infty [Q(x(t)) + u(t)^T R\,u(t)]\,dt.$$5

and PINN-based policy iteration alternates linear PDE solves under fixed feedback pairs $$J(x_0,u)=\int_0^\infty [Q(x(t)) + u(t)^T R\,u(t)]\,dt.$$6 with pointwise minimax updates (Yang et al., 21 Jul 2025).

A further specialization targets robust stability rather than optimal regulation. For perturbed dynamics

$$J(x_0,u)=\int_0^\infty [Q(x(t)) + u(t)^T R\,u(t)]\,dt.$$7

the robust region of attraction is

$$J(x_0,u)=\int_0^\infty [Q(x(t)) + u(t)^T R\,u(t)]\,dt.$$8

and the generalized maximal Lyapunov function

$$J(x_0,u)=\int_0^\infty [Q(x(t)) + u(t)^T R\,u(t)]\,dt.$$9

induces the bounded transform

$$Q(x) + u_i(x)^T R\,u_i(x)+\nabla V_i(x)\cdot[f(x)+g(x)u_i(x)] = 0,$$0

The corresponding generalized Zubov PDE is

$$Q(x) + u_i(x)^T R\,u_i(x)+\nabla V_i(x)\cdot[f(x)+g(x)u_i(x)] = 0,$$1

with $$Q(x) + u_i(x)^T R\,u_i(x)+\nabla V_i(x)\cdot[f(x)+g(x)u_i(x)] = 0,$$2, $$Q(x) + u_i(x)^T R\,u_i(x)+\nabla V_i(x)\cdot[f(x)+g(x)u_i(x)] = 0,$$3, and $$Q(x) + u_i(x)^T R\,u_i(x)+\nabla V_i(x)\cdot[f(x)+g(x)u_i(x)] = 0,$$4. This robust-region formulation underlies "Learning Robust Regions of Attraction Using Rollout-Enhanced Physics-Informed Neural Networks with Policy Iteration" (Wang et al., 26 Aug 2025).

2. Canonical algorithmic structure

The common algorithmic structure is an alternation between policy evaluation and policy improvement. In the evaluation phase, a neural approximant $$Q(x) + u_i(x)^T R\,u_i(x)+\nabla V_i(x)\cdot[f(x)+g(x)u_i(x)] = 0,$$5, $$Q(x) + u_i(x)^T R\,u_i(x)+\nabla V_i(x)\cdot[f(x)+g(x)u_i(x)] = 0,$$6, or $$Q(x) + u_i(x)^T R\,u_i(x)+\nabla V_i(x)\cdot[f(x)+g(x)u_i(x)] = 0,$$7 is trained so that its derivatives satisfy the PDE associated with the current policy. Because the policy is frozen, the evaluation equation is linear in the unknown value function in the deterministic generalized HJB, the stochastic elliptic HJB, the soft-policy evaluation PDE, and the fixed-policy HJI formulations (Meng et al., 2024, Kim et al., 3 Aug 2025, Kim et al., 3 Aug 2025, Yang et al., 21 Jul 2025).

In the improvement phase, the trained value network is differentiated with automatic differentiation, and its gradient is inserted into a Hamiltonian optimizer. The specific optimizer depends on the problem class. Deterministic control uses the closed-form update

$$Q(x) + u_i(x)^T R\,u_i(x)+\nabla V_i(x)\cdot[f(x)+g(x)u_i(x)] = 0,$$8

(Meng et al., 2024). Stochastic optimal control uses a greedy Hamiltonian minimization

$$Q(x) + u_i(x)^T R\,u_i(x)+\nabla V_i(x)\cdot[f(x)+g(x)u_i(x)] = 0,$$9

(Kim et al., 3 Aug 2025). Entropy-regularized control uses the softmax policy induced by $u_{i+1}(x)= -\tfrac12 R^{-1}g(x)^T\nabla V_i(x)^T.$0 (Kim et al., 3 Aug 2025). Zero-sum games use the pointwise minimax pair

$u_{i+1}(x)= -\tfrac12 R^{-1}g(x)^T\nabla V_i(x)^T.$1

with $u_{i+1}(x)= -\tfrac12 R^{-1}g(x)^T\nabla V_i(x)^T.$2 (Yang et al., 21 Jul 2025).

The robust-region variant replaces policy improvement by disturbance improvement. The disturbance field is updated through

$u_{i+1}(x)= -\tfrac12 R^{-1}g(x)^T\nabla V_i(x)^T.$3

and rollout-generated anchor values are then computed under the current disturbance sequence to stabilize the next evaluation step (Wang et al., 26 Aug 2025).

A hybrid actor-critic version appears in PDE control of fluid-cooled battery packs. There, the value network is a PINN for the continuous-time HJB, while the actor is updated with PPO. During rollout, actions are selected with probability $u_{i+1}(x)= -\tfrac12 R^{-1}g(x)^T\nabla V_i(x)^T.$4 from the PINN-derived controller and otherwise from the Gaussian policy network. This formulation is denoted HJBPPO in the reported experiments (Mukherjee et al., 2023).

3. Neural parameterizations, residual losses, and sampling

Reported PINN-PI implementations use standard fully connected feedforward architectures, but the architecture is tuned to the PDE class. The deterministic control formulation in (Meng et al., 2024) uses a deep feed-forward network $u_{i+1}(x)= -\tfrac12 R^{-1}g(x)^T\nabla V_i(x)^T.$5 for PINN-PI and a one-layer random-feature model for ELM-PI. The stochastic-control formulation in (Kim et al., 3 Aug 2025) reports 4–6 hidden layers with 128–256 neurons and tanh or SiLU activations. The soft HJB formulation in (Kim et al., 3 Aug 2025) reports 3–5 hidden layers with 64–128 neurons and $u_{i+1}(x)= -\tfrac12 R^{-1}g(x)^T\nabla V_i(x)^T.$6 or ReLU activations. The HJI game solver in (Yang et al., 21 Jul 2025) uses 3–4 hidden layers with 64 neurons per layer and sinusoidal or tanh activations. The generalized Zubov solver in (Wang et al., 26 Aug 2025) uses 5 hidden layers of width $u_{i+1}(x)= -\tfrac12 R^{-1}g(x)^T\nabla V_i(x)^T.$7, sigmoid activations, and a linear output layer. The battery-pack HJB value network is described as a typical MLP with 3–5 hidden layers, 64–128 neurons, and tanh activations (Mukherjee et al., 2023).

The training objective is always centered on a PDE residual, but the auxiliary terms vary by formulation. Deterministic PINN-PI uses the residual of the generalized HJB together with $u_{i+1}(x)= -\tfrac12 R^{-1}g(x)^T\nabla V_i(x)^T.$8 and a stability-preserving term

$u_{i+1}(x)= -\tfrac12 R^{-1}g(x)^T\nabla V_i(x)^T.$9

that enforces matching of the linearization at the origin (Meng et al., 2024). The battery-pack setting adds a Dirichlet condition $dX_t = b(X_t,a_t)\,dt + \sigma\,dW_t,$0 and a Neumann condition $dX_t = b(X_t,a_t)\,dt + \sigma\,dW_t,$1 to the residual loss (Mukherjee et al., 2023). The generalized Zubov formulation decomposes the loss into boundary, residual, and data-anchor components,

$dX_t = b(X_t,a_t)\,dt + \sigma\,dW_t,$2

where the anchor targets $dX_t = b(X_t,a_t)\,dt + \sigma\,dW_t,$3 come from rollout estimates (Wang et al., 26 Aug 2025). The soft HJB formulation separates value and policy training, using a PINN residual loss for $dX_t = b(X_t,a_t)\,dt + \sigma\,dW_t,$4 and a KL or cross-entropy loss to fit the policy network $dX_t = b(X_t,a_t)\,dt + \sigma\,dW_t,$5 to the analytical softmax update (Kim et al., 3 Aug 2025).

Sampling is likewise problem-dependent. The stochastic-control paper reports uniform or quasi-Monte Carlo collocation in bounded domains, with $dX_t = b(X_t,a_t)\,dt + \sigma\,dW_t,$6–$dX_t = b(X_t,a_t)\,dt + \sigma\,dW_t,$7 points and truncation of $dX_t = b(X_t,a_t)\,dt + \sigma\,dW_t,$8 to a large ball when necessary (Kim et al., 3 Aug 2025). The HJI paper refreshes 1,000–2,000 interior collocation points every 100 epochs and warm-starts each policy-evaluation step from the previous network parameters (Yang et al., 21 Jul 2025). The generalized Zubov method samples separate sets on $dX_t = b(X_t,a_t)\,dt + \sigma\,dW_t,$9, in the interior for the PDE residual, and in the interior for data anchors; in the reported 2D experiments $L(x,a)$0 and $L(x,a)$1, while the 10D case uses $L(x,a)$2 and $L(x,a)$3 (Wang et al., 26 Aug 2025). Across the surveyed works, optimization is performed with Adam, sometimes followed by L-BFGS or accompanied by periodic resampling (Kim et al., 3 Aug 2025, Yang et al., 21 Jul 2025, Wang et al., 26 Aug 2025).

4. Convergence theory and error control

A central motivation for PINN-PI is that freezing the policy converts a nonlinear control PDE into a sequence of linear evaluation problems, which are more amenable to both training and analysis. In the deterministic control-affine setting, exact policy iteration yields a sequence $L(x,a)$4 pointwise, and $L(x,a)$5 uniformly on compact sets; the generalized HJB at each step has a unique positive-definite viscosity solution $L(x,a)$6 (Meng et al., 2024). In stochastic optimal control, exact policy iteration inherits the global exponential contraction

$L(x,a)$7

under Assumptions (A1)–(A4) including compact convex $L(x,a)$8, Lipschitz $L(x,a)$9, strong convexity in $$\lambda V(x) - \tfrac12 \operatorname{tr}[\sigma\sigma^T D^2V(x)] - \sup_{a\in A}\{b(x,a)\cdot \nabla V(x)+L(x,a)\}=0.$$0, and uniform ellipticity of $$\lambda V(x) - \tfrac12 \operatorname{tr}[\sigma\sigma^T D^2V(x)] - \sup_{a\in A}\{b(x,a)\cdot \nabla V(x)+L(x,a)\}=0.$$1 (Kim et al., 3 Aug 2025). In entropy-regularized control, the exact soft policy-iteration operator yields iteration error $$\lambda V(x) - \tfrac12 \operatorname{tr}[\sigma\sigma^T D^2V(x)] - \sup_{a\in A}\{b(x,a)\cdot \nabla V(x)+L(x,a)\}=0.$$2 (Kim et al., 3 Aug 2025). In nonconvex HJI, exact value iterates converge locally uniformly to the unique bounded continuous viscosity solution, and an exponential $$\lambda V(x) - \tfrac12 \operatorname{tr}[\sigma\sigma^T D^2V(x)] - \sup_{a\in A}\{b(x,a)\cdot \nabla V(x)+L(x,a)\}=0.$$3 rate

$$\lambda V(x) - \tfrac12 \operatorname{tr}[\sigma\sigma^T D^2V(x)] - \sup_{a\in A}\{b(x,a)\cdot \nabla V(x)+L(x,a)\}=0.$$4

is proved under compactness, strong convexity-concavity, Lipschitz continuity, and uniform ellipticity (Yang et al., 21 Jul 2025).

The practical PINN-PI analyses quantify how neural approximation error enters these classical policy-iteration dynamics. In (Kim et al., 3 Aug 2025), if $$\lambda V(x) - \tfrac12 \operatorname{tr}[\sigma\sigma^T D^2V(x)] - \sup_{a\in A}\{b(x,a)\cdot \nabla V(x)+L(x,a)\}=0.$$5, then

$$\lambda V(x) - \tfrac12 \operatorname{tr}[\sigma\sigma^T D^2V(x)] - \sup_{a\in A}\{b(x,a)\cdot \nabla V(x)+L(x,a)\}=0.$$6

where $$\lambda V(x) - \tfrac12 \operatorname{tr}[\sigma\sigma^T D^2V(x)] - \sup_{a\in A}\{b(x,a)\cdot \nabla V(x)+L(x,a)\}=0.$$7 comes from gradient-error propagation. The same paper proves a Lipschitz-type policy-gradient bound: $$\lambda V(x) - \tfrac12 \operatorname{tr}[\sigma\sigma^T D^2V(x)] - \sup_{a\in A}\{b(x,a)\cdot \nabla V(x)+L(x,a)\}=0.$$8 In (Kim et al., 3 Aug 2025), the learned value error is decomposed into PDE residual error, policy-network error, and iteration error, and the main theorem states

$$\lambda V(x) - \tfrac12 \operatorname{tr}[\sigma\sigma^T D^2V(x)] - \sup_{a\in A}\{b(x,a)\cdot \nabla V(x)+L(x,a)\}=0.$$9

with $\pi_k$0 and $\pi_k$1, so the total error is uniformly bounded by the approximation accuracies. In (Yang et al., 21 Jul 2025), the saddle-point selector is $\pi_k$2-Lipschitz in the co-state variable $\pi_k$3, and the practical PINN-PI error satisfies

$\pi_k$4

The deterministic paper adds a convergence statement tailored to neural residual minimization: if the PINN or ELM residual can be driven to zero and the neural class approximates $\pi_k$5-functions with small residual, then

$\pi_k$6

as the number of collocation points grows and training converges, for any excluded ball $\pi_k$7 around the origin (Meng et al., 2024). This body of theory distinguishes PINN-PI from direct one-shot neural solution of the fully nonlinear PDE: the error analysis is organized around policy-evaluation residuals and policy-update stability rather than around a single nonlinear residual objective.

5. Robust stability, regions of attraction, and verification

The robust-region-of-attraction variant reframes PINN-PI as Lyapunov-function learning for perturbed systems. The generalized Zubov equation produces a bounded viscosity solution $\pi_k$8 whose strict sublevel set $\pi_k$9 equals the robust region of attraction. The reported method alternates between training $$\tfrac12 \operatorname{tr}[\sigma\sigma^T D^2V] + b(x,\pi_k(x))\cdot \nabla V + L(x,\pi_k(x))=0,$$0 against the Zubov residual and updating the disturbance field through pointwise maximization of the generalized Hamiltonian. A distinctive feature is the rollout-enhanced anchor strategy: $$\tfrac12 \operatorname{tr}[\sigma\sigma^T D^2V] + b(x,\pi_k(x))\cdot \nabla V + L(x,\pi_k(x))=0,$$1 followed by

$$\tfrac12 \operatorname{tr}[\sigma\sigma^T D^2V] + b(x,\pi_k(x))\cdot \nabla V + L(x,\pi_k(x))=0,$$2

These anchors are used as supervised labels in the loss to prevent the trivial singular solution $$\tfrac12 \operatorname{tr}[\sigma\sigma^T D^2V] + b(x,\pi_k(x))\cdot \nabla V + L(x,\pi_k(x))=0,$$3 outside a tiny neighborhood of the origin and to guide training toward the true viscosity solution (Wang et al., 26 Aug 2025).

A different stability-oriented thread appears in the deterministic verification framework. There the learned value and policy are checked against a Lyapunov decrease condition

$$\tfrac12 \operatorname{tr}[\sigma\sigma^T D^2V] + b(x,\pi_k(x))\cdot \nabla V + L(x,\pi_k(x))=0,$$4

on $$\tfrac12 \operatorname{tr}[\sigma\sigma^T D^2V] + b(x,\pi_k(x))\cdot \nabla V + L(x,\pi_k(x))=0,$$5, together with nested level-set conditions $$\tfrac12 \operatorname{tr}[\sigma\sigma^T D^2V] + b(x,\pi_k(x))\cdot \nabla V + L(x,\pi_k(x))=0,$$6 and $$\tfrac12 \operatorname{tr}[\sigma\sigma^T D^2V] + b(x,\pi_k(x))\cdot \nabla V + L(x,\pi_k(x))=0,$$7. Because exact checking is undecidable in general, the procedure uses a $$\tfrac12 \operatorname{tr}[\sigma\sigma^T D^2V] + b(x,\pi_k(x))\cdot \nabla V + L(x,\pi_k(x))=0,$$8-complete SMT solver, dReal, to certify the decrease condition up to $$\tfrac12 \operatorname{tr}[\sigma\sigma^T D^2V] + b(x,\pi_k(x))\cdot \nabla V + L(x,\pi_k(x))=0,$$9 precision or produce a counterexample. The reported outcome is a verifiable region of attraction around the origin (Meng et al., 2024).

These two strands are closely related but not identical. The generalized Zubov approach computes a robust region of attraction directly from a PDE whose solution is itself a Lyapunov-like function, whereas the verification pipeline begins with a learned optimal-control solution and then certifies a Lyapunov decrease condition for the induced closed loop. This suggests two distinct stability uses of PINN-PI: direct robust-attraction computation and post hoc certification of synthesized controllers.

6. Reported applications and empirical results

The published applications span ODE control, stochastic control, PDE control, differential games, and robust-attraction analysis.

Setting	Representative task	Reported outcome
(Mukherjee et al., 2023)	1D PDE model for fluid-cooled battery packs	HJB Value Iteration: final reward $$\pi_{k+1}(x)=\arg\min_{u\in A}\{L(x,u)+\nabla V^{\pi_k}(x)\cdot b(x,u)\}.$$0; PPO: $$\pi_{k+1}(x)=\arg\min_{u\in A}\{L(x,u)+\nabla V^{\pi_k}(x)\cdot b(x,u)\}.$$1; HJBPPO: $$\pi_{k+1}(x)=\arg\min_{u\in A}\{L(x,u)+\nabla V^{\pi_k}(x)\cdot b(x,u)\}.$$2 over 5 seeds and $$\pi_{k+1}(x)=\arg\min_{u\in A}\{L(x,u)+\nabla V^{\pi_k}(x)\cdot b(x,u)\}.$$3 timesteps each
(Kim et al., 3 Aug 2025)	Stochastic LQR, cartpole, pendulum	Value-error $$\pi_{k+1}(x)=\arg\min_{u\in A}\{L(x,u)+\nabla V^{\pi_k}(x)\cdot b(x,u)\}.$$4 and policy-error $$\pi_{k+1}(x)=\arg\min_{u\in A}\{L(x,u)+\nabla V^{\pi_k}(x)\cdot b(x,u)\}.$$5 in 20 PI iterations for 5D and 10D LQR; stable control in 5–10 PI steps on cartpole and pendulum
(Kim et al., 3 Aug 2025)	Entropy-regularized 5D/10D LQR, pendulum, cartpole	$$\pi_{k+1}(x)=\arg\min_{u\in A}\{L(x,u)+\nabla V^{\pi_k}(x)\cdot b(x,u)\}.$$6 of optimal discounted reward in $$\pi_{k+1}(x)=\arg\min_{u\in A}\{L(x,u)+\nabla V^{\pi_k}(x)\cdot b(x,u)\}.$$7 policy iterations; SAC stalls below $$\pi_{k+1}(x)=\arg\min_{u\in A}\{L(x,u)+\nabla V^{\pi_k}(x)\cdot b(x,u)\}.$$8 in 10D
(Yang et al., 21 Jul 2025)	2D stochastic path planning; 5D/10D publisher-subscriber game	2D relative $$\pi_{k+1}(x)=\arg\min_{u\in A}\{L(x,u)+\nabla V^{\pi_k}(x)\cdot b(x,u)\}.$$9-errors $$J(x_0,u)=\int_0^\infty [Q(x(t)) + u(t)^T R\,u(t)]\,dt.$$00; 5D isotropic relative $$J(x_0,u)=\int_0^\infty [Q(x(t)) + u(t)^T R\,u(t)]\,dt.$$01 at $$J(x_0,u)=\int_0^\infty [Q(x(t)) + u(t)^T R\,u(t)]\,dt.$$02: $$J(x_0,u)=\int_0^\infty [Q(x(t)) + u(t)^T R\,u(t)]\,dt.$$03 vs direct $$J(x_0,u)=\int_0^\infty [Q(x(t)) + u(t)^T R\,u(t)]\,dt.$$04; 10D isotropic: $$J(x_0,u)=\int_0^\infty [Q(x(t)) + u(t)^T R\,u(t)]\,dt.$$05 vs direct $$J(x_0,u)=\int_0^\infty [Q(x(t)) + u(t)^T R\,u(t)]\,dt.$$06
(Wang et al., 26 Aug 2025)	Perturbed Van der Pol, inverted pendulum, 10D decoupled system	Without rollout, PINN collapses to $$J(x_0,u)=\int_0^\infty [Q(x(t)) + u(t)^T R\,u(t)]\,dt.$$07 almost everywhere; with rollout-anchors, matches finite-difference baselines in 2D and approximates true RROA $$J(x_0,u)=\int_0^\infty [Q(x(t)) + u(t)^T R\,u(t)]\,dt.$$08 with only small distortions
(Meng et al., 2024)	Synthetic nonlinear benchmark, inverted pendulum, Lorenz, cartpole, 2D/3D quadrotor	ELM-PI attains $$J(x_0,u)=\int_0^\infty [Q(x(t)) + u(t)^T R\,u(t)]\,dt.$$09–$$J(x_0,u)=\int_0^\infty [Q(x(t)) + u(t)^T R\,u(t)]\,dt.$$10 accuracy in seconds for $$J(x_0,u)=\int_0^\infty [Q(x(t)) + u(t)^T R\,u(t)]\,dt.$$11; PINN-PI attains $$J(x_0,u)=\int_0^\infty [Q(x(t)) + u(t)^T R\,u(t)]\,dt.$$12–$$J(x_0,u)=\int_0^\infty [Q(x(t)) + u(t)^T R\,u(t)]\,dt.$$13 accuracy in $$J(x_0,u)=\int_0^\infty [Q(x(t)) + u(t)^T R\,u(t)]\,dt.$$14 up to $$J(x_0,u)=\int_0^\infty [Q(x(t)) + u(t)^T R\,u(t)]\,dt.$$15; Lorenz solved in $$J(x_0,u)=\int_0^\infty [Q(x(t)) + u(t)^T R\,u(t)]\,dt.$$16; SGA took $$J(x_0,u)=\int_0^\infty [Q(x(t)) + u(t)^T R\,u(t)]\,dt.$$17 for the same accuracy

The application record exhibits several recurring empirical patterns. First, PINN-PI is repeatedly positioned against direct nonlinear PDE solvers, classical grid-based methods, or model-free RL baselines. The differential-game study explicitly contrasts PINN-PI with direct PINN minimization of the one-shot Hamiltonian residual, and the deterministic control study compares against Galerkin methods and PPO-like baselines (Yang et al., 21 Jul 2025, Meng et al., 2024). Second, the advantage often comes from decomposition rather than from network architecture alone: the battery-pack study attributes the poor performance of pure HJB value iteration to poor exploration and the better performance of HJBPPO to the combination of physics-exploiting value updates with PPO exploration (Mukherjee et al., 2023). Third, the current empirical frontier in the surveyed literature is “moderately high dimensional”: multiple papers report successful 10D experiments, while also emphasizing mesh-free operation and the impracticality of finite differences beyond about three dimensions (Kim et al., 3 Aug 2025, Kim et al., 3 Aug 2025, Yang et al., 21 Jul 2025, Wang et al., 26 Aug 2025).

7. Distinctions, limitations, and future directions

PINN-PI is not a single fixed algorithm. The surveyed literature includes deterministic generalized-HJB policy iteration, stochastic second-order HJB policy iteration, entropy-regularized soft policy iteration, minimax HJI policy iteration, rollout-enhanced Zubov learning, and actor-critic hybrids in which only the value update is physics-informed (Meng et al., 2024, Kim et al., 3 Aug 2025, Kim et al., 3 Aug 2025, Yang et al., 21 Jul 2025, Wang et al., 26 Aug 2025, Mukherjee et al., 2023). A common misconception is to treat these methods as model-free RL with a PINN regularizer. In the reported formulations, the dynamics and PDE operators are explicit ingredients of the loss; even the battery-pack actor-critic method retains a continuous-time HJB residual for the critic (Mukherjee et al., 2023, Kim et al., 3 Aug 2025).

The limitations are equally formulation-specific. The entropy-regularized framework states that it requires smooth soft-max policies and does not directly handle bang-bang or discrete actions; it also relies on full knowledge of $$J(x_0,u)=\int_0^\infty [Q(x(t)) + u(t)^T R\,u(t)]\,dt.$$18 (Kim et al., 3 Aug 2025). The HJI game solver assumes nondegenerate diffusion through uniform ellipticity and notes PINN sensitivities to initialization and collocation sampling (Yang et al., 21 Jul 2025). The battery-pack study reports that pure HJB-PINN control converges to a suboptimal bang-bang strategy with poor exploration (Mukherjee et al., 2023). The generalized Zubov study reports collapse to the singular solution $$J(x_0,u)=\int_0^\infty [Q(x(t)) + u(t)^T R\,u(t)]\,dt.$$19 when rollout anchors are removed (Wang et al., 26 Aug 2025). The deterministic comparison between ELM-PI and PINN-PI indicates that the random-feature ELM variant is very fast and highly accurate in low dimensions but becomes impractical as dimension grows, whereas the deep PINN variant scales better to $$J(x_0,u)=\int_0^\infty [Q(x(t)) + u(t)^T R\,u(t)]\,dt.$$20 (Meng et al., 2024).

The listed future directions are also diverse. The HJI work points to first-order HJI, adaptive sampling, operator learning through DeepONet/FNO, and integration with model-based RL (Yang et al., 21 Jul 2025). The soft HJB work identifies joint learning of dynamics and value, risk-sensitive or constrained extensions via barrier/PINN coupling, and distributional or multi-agent objectives (Kim et al., 3 Aug 2025). A plausible implication is that future PINN-PI research will be shaped by two competing requirements: preserving the linearized policy-evaluation structure that enables residual-based analysis, while enlarging the class of admissible policies, uncertainty models, and observation regimes.