Residual Conditioned Policy Iteration (RCPI)

• RCPI is a family of policy iteration methods that use residual errors—from PDE or Bellman equations—as active conditions to certify policy improvement.
• In the stochastic optimal control formulation, a physics-informed neural network minimizes a PDE residual to control both value and gradient errors, leading to exponential convergence.
• In the Markov game and robust MDP setting, RCPI employs residual-guarded policy evaluation to ensure monotonic progress toward an ε-saddle point even in nonsmooth search landscapes.

Searching arXiv for the specified RCPI papers and closely related policy-iteration context. Residual Conditioned Policy Iteration (RCPI) denotes two distinct algorithmic constructions that share a common organizing principle: policy iteration is conditioned on a residual criterion that controls or certifies improvement. In one formulation, RCPI appears as a physics-informed neural network policy iteration framework for stochastic optimal control governed by second-order Hamilton–Jacobi–Bellman (HJB) equations, where policy evaluation is performed by minimizing a linear PDE residual under a fixed policy (Kim et al., 3 Aug 2025). In another formulation, RCPI is introduced for zero-sum Markov games and s-rectangular robust MDPs as a residual-guarded variant of fast policy iteration, designed to guarantee convergence to a saddle point while retaining the practical speed of exact policy-evaluation steps (Badger et al., 8 Aug 2025). Across both settings, the defining motif is the use of a residual—not merely as a stopping diagnostic, but as a structural condition that determines when an evaluation or improvement step is accepted.

1. Conceptual Scope and Naming

The name “Residual Conditioned Policy Iteration” is used in two technically different contexts in the 2025 literature.

In the stochastic optimal control setting, the framework described in “Neural Policy Iteration for Stochastic Optimal Control: A Physics-Informed Approach” uses a neural approximation of the value function and performs policy evaluation by minimizing the residual of a linear elliptic PDE induced by a fixed Markov policy (Kim et al., 3 Aug 2025). The conditioning mechanism is explicit in the analysis: the residual controls both the value-function error and the gradient error through an energy estimate, and the gradient error in turn affects the greedy policy update. This makes the residual a quantitative proxy for policy quality during training.

In the Markov-game and robust-MDP setting, “Convergence of Fast Policy Iteration in Markov Games and Robust MDPs” introduces RCPI as a correction to the Filar–Tolwinski (FT) algorithm (Badger et al., 8 Aug 2025). Here the conditioning mechanism is operational rather than variational: an exact policy-evaluation candidate is accepted only if the residual can be reduced sufficiently, and otherwise the method reverts to a standard Bellman backup. The resulting algorithm enforces monotonic residual reduction and a linear contraction in the $\infty$-norm residual up to the approximation floor.

These two uses of RCPI are not the same algorithm. They are united by a shared methodological pattern: residuals are promoted from passive error indicators to active acceptance criteria or training objectives. This suggests a broader interpretation of RCPI as a residual-governed family of policy-iteration procedures rather than a single canonical method.

2. RCPI for Stochastic Optimal Control and Second-Order HJB Equations

In the stochastic control formulation, the infinite-horizon problem is given by

$$dX_t = f(X_t,u_t)\,dt + \sigma(X_t)\,dW_t,\qquad X_0=x,$$

with state $x\in\mathbb{R}^d$, control $u\in A\subset\mathbb{R}^m$, drift $f(x,u)\in\mathbb{R}^d$, diffusion matrix $\sigma(x)\in\mathbb{R}^{d\times k}$, running cost $L(x,u)\ge 0$, value function $V(x)$, and discount $\lambda>0$ (Kim et al., 3 Aug 2025). The value is

$$V(x)=\sup_{u(\cdot)} \mathbb{E}_x\!\left[\int_0^\infty e^{-\lambda s}L(X_s,u_s)\,ds\right].$$

Its dynamic programming equation is the second-order HJB PDE

$$dX_t = f(X_t,u_t)\,dt + \sigma(X_t)\,dW_t,\qquad X_0=x,$$0

in viscosity sense (Kim et al., 3 Aug 2025).

The essential simplification arises when a Markov policy $$dX_t = f(X_t,u_t)\,dt + \sigma(X_t)\,dW_t,\qquad X_0=x,$$1 is fixed. The supremum is then removed, yielding a linear elliptic PDE for the policy value $$dX_t = f(X_t,u_t)\,dt + \sigma(X_t)\,dW_t,\qquad X_0=x,$$2:

$$dX_t = f(X_t,u_t)\,dt + \sigma(X_t)\,dW_t,\qquad X_0=x,$$3

This linearization is the basis of the PINN-based evaluation step. Rather than solving the nonlinear HJB directly, the method alternates between solving linear PDEs induced by current policies and updating those policies greedily.

The paper states that this framework extends recent deterministic PINN-based approaches to stochastic settings and inherits the global exponential convergence guarantees of classical policy iteration under mild conditions (Kim et al., 3 Aug 2025). It also demonstrates benchmark performance on stochastic cartpole, pendulum problems, and high-dimensional linear quadratic regulation problems in up to 10D.

3. Residual-Based Policy Evaluation in the PINN-PI Framework

Given a fixed policy $$dX_t = f(X_t,u_t)\,dt + \sigma(X_t)\,dW_t,\qquad X_0=x,$$4, the value function is approximated by a neural network $$dX_t = f(X_t,u_t)\,dt + \sigma(X_t)\,dW_t,\qquad X_0=x,$$5, and the PDE residual is defined as

$$dX_t = f(X_t,u_t)\,dt + \sigma(X_t)\,dW_t,\qquad X_0=x,$$6

The evaluation loss is the squared $$dX_t = f(X_t,u_t)\,dt + \sigma(X_t)\,dW_t,\qquad X_0=x,$$7 residual, either as

$$dX_t = f(X_t,u_t)\,dt + \sigma(X_t)\,dW_t,\qquad X_0=x,$$8

or empirically over collocation points $$dX_t = f(X_t,u_t)\,dt + \sigma(X_t)\,dW_t,\qquad X_0=x,$$9,

$x\in\mathbb{R}^d$0

(Kim et al., 3 Aug 2025).

The core analytical device is the energy estimate stated as Proposition 4.1:

$x\in\mathbb{R}^d$1

where $x\in\mathbb{R}^d$2 and the constants $x\in\mathbb{R}^d$3 come from Assumption (A4) (Kim et al., 3 Aug 2025). By Cauchy–Schwarz and Young, controlling $x\in\mathbb{R}^d$4 forces both $x\in\mathbb{R}^d$5 and $x\in\mathbb{R}^d$6 to be small. This is a defining feature of the framework: the residual is not merely minimized heuristically, but linked to systematic $x\in\mathbb{R}^d$7 error control at each policy-evaluation step.

The policy improvement step uses the learned gradient:

$x\in\mathbb{R}^d$8

The quantity $x\in\mathbb{R}^d$9 enters as a parameter in a pointwise convex optimization in $u\in A\subset\mathbb{R}^m$0 (Kim et al., 3 Aug 2025). The paper further provides a policy-error Lipschitz bound: if $u\in A\subset\mathbb{R}^m$1 with $u\in A\subset\mathbb{R}^m$2, then the selector

$u\in A\subset\mathbb{R}^m$3

satisfies

$u\in A\subset\mathbb{R}^m$4

This quantifies how gradient approximation errors propagate into policy updates.

The exact policy-iteration iterates satisfy a global exponential convergence theorem under assumptions (A1–A4): if

$u\in A\subset\mathbb{R}^m$5

then

$u\in A\subset\mathbb{R}^m$6

(Kim et al., 3 Aug 2025). The residual-conditioned character of the method is therefore theoretical as well as algorithmic: the PDE residual controls evaluation error, the evaluation error controls gradient error, and the gradient error controls policy quality.

4. RCPI as a Convergent Alternative to Fast Policy Iteration in Markov Games and Robust MDPs

In zero-sum Markov games, the state space is $u\in A\subset\mathbb{R}^m$7, the maximizing agent’s actions are $u\in A\subset\mathbb{R}^m$8, the minimizing agent’s actions are $u\in A\subset\mathbb{R}^m$9, the reward function is $f(x,u)\in\mathbb{R}^d$0, the transition kernel is $f(x,u)\in\mathbb{R}^d$1, and the discount factor is $f(x,u)\in\mathbb{R}^d$2 (Badger et al., 8 Aug 2025). For stationary randomized policies $f(x,u)\in\mathbb{R}^d$3 and $f(x,u)\in\mathbb{R}^d$4, the value $f(x,u)\in\mathbb{R}^d$5 solves

$f(x,u)\in\mathbb{R}^d$6

and the saddle-point value satisfies

$f(x,u)\in\mathbb{R}^d$7

The Bellman evaluation operator is

$f(x,u)\in\mathbb{R}^d$8

and the equilibrium Bellman operator is

$f(x,u)\in\mathbb{R}^d$9

The policy-improvement operator is

$\sigma(x)\in\mathbb{R}^{d\times k}$0

(Badger et al., 8 Aug 2025). The paper defines $\sigma(x)\in\mathbb{R}^{d\times k}$1-saddle points and extends the same structure analogously to s-rectangular robust MDPs, replacing the adversarial policy by an uncertain transition kernel $\sigma(x)\in\mathbb{R}^{d\times k}$2 in an ambiguity set.

Within this setting, the paper first analyzes the FT algorithm. FT starts from an initial value $\sigma(x)\in\mathbb{R}^{d\times k}$3, alternates greedy policy improvement and exact policy evaluation, forms the search direction $\sigma(x)\in\mathbb{R}^{d\times k}$4, and performs a backtracking Armijo line search to decrease the squared $\sigma(x)\in\mathbb{R}^{d\times k}$5 Bellman residual $\sigma(x)\in\mathbb{R}^{d\times k}$6 (Badger et al., 8 Aug 2025). Its failure mode is central to the motivation for RCPI. The paper states that FT relies on differentiability of $\sigma(x)\in\mathbb{R}^{d\times k}$7, but $\sigma(x)\in\mathbb{R}^{d\times k}$8 is not smooth everywhere because the min/max operations create kinks. A 3-state Markov-game counterexample shows that for $\sigma(x)\in\mathbb{R}^{d\times k}$9, any $L(x,u)\ge 0$0 along the first search direction increases $L(x,u)\ge 0$1, so Armijo’s condition can never be met and FT loops in its line search:

$L(x,u)\ge 0$2

(Badger et al., 8 Aug 2025).

RCPI is introduced as a remedy. It combines one-step exact policy evaluation with a guard that reverts to a standard Bellman backup if the candidate value does not sufficiently reduce the residual. The stated consequence is that RCPI is guaranteed to converge to a saddle point, whereas FT may fail to converge and may loop indefinitely even in small games (Badger et al., 8 Aug 2025).

5. Algorithmic Mechanisms and Residual Contraction

The RCPI algorithm for Markov games and robust MDPs is formulated with approximate Bellman and policy-improvement oracles $L(x,u)\ge 0$3 and $L(x,u)\ge 0$4, satisfying $L(x,u)\ge 0$5 and $L(x,u)\ge 0$6-saddle points (Badger et al., 8 Aug 2025). Its inputs are $L(x,u)\ge 0$7, target $L(x,u)\ge 0$8, backup tolerance $L(x,u)\ge 0$9 with

$V(x)$0

and a maximum number of fixes $V(x)$1.

At iteration $V(x)$2, the algorithm first computes a greedy or approximately greedy saddle-point policy pair $V(x)$3, then forms the exact-evaluation candidate

$V(x)$4

If

$V(x)$5

the method immediately falls back to the Bellman backup

$V(x)$6

Otherwise it applies up to $V(x)$7 successive $V(x)$8 backups to the candidate while

$V(x)$9

and then sets $\lambda>0$0 (Badger et al., 8 Aug 2025).

The paper’s summary of the intuition is precise. The candidate $\lambda>0$1 is accepted only if in at most $\lambda>0$2 successive $\lambda>0$3 backups the residual will drop below $\lambda>0$4; otherwise the algorithm falls back immediately to one Bellman backup. Every outer iteration guarantees

$\lambda>0$5

which is a linear contraction up to the $\lambda>0$6-floor (Badger et al., 8 Aug 2025). Unlike FT, there is no need for line search or differentiability, because monotonic residual reduction is enforced directly.

The stochastic-control RCPI/PINN-PI framework has a different algorithmic form but a related logic. Its pseudocode takes as input an initial policy $\lambda>0$7, network $\lambda>0$8 with $\lambda>0$9, sampler $$V(x)=\sup_{u(\cdot)} \mathbb{E}_x\!\left[\int_0^\infty e^{-\lambda s}L(X_s,u_s)\,ds\right].$$0, and tolerance $$V(x)=\sup_{u(\cdot)} \mathbb{E}_x\!\left[\int_0^\infty e^{-\lambda s}L(X_s,u_s)\,ds\right].$$1. At each iteration it samples collocation points, minimizes the empirical squared PDE residual, updates the policy greedily through

$$V(x)=\sup_{u(\cdot)} \mathbb{E}_x\!\left[\int_0^\infty e^{-\lambda s}L(X_s,u_s)\,ds\right].$$2

and stops when $$V(x)=\sup_{u(\cdot)} \mathbb{E}_x\!\left[\int_0^\infty e^{-\lambda s}L(X_s,u_s)\,ds\right].$$3 (Kim et al., 3 Aug 2025). Here the residual conditioning takes the form of residual-driven policy evaluation rather than residual-guarded acceptance, but in both cases the residual determines whether policy iteration is considered reliable.

6. Complexity, Empirical Results, and Practical Parameters

The two RCPI formulations differ substantially in computational profile.

For Markov games and robust MDPs, each RCPI outer loop costs

$$V(x)=\sup_{u(\cdot)} \mathbb{E}_x\!\left[\int_0^\infty e^{-\lambda s}L(X_s,u_s)\,ds\right].$$4

where $$V(x)=\sup_{u(\cdot)} \mathbb{E}_x\!\left[\int_0^\infty e^{-\lambda s}L(X_s,u_s)\,ds\right].$$5 is the cost of one approximate-oracle Bellman-policy computation (Badger et al., 8 Aug 2025). The runtime theorem states that if $$V(x)=\sup_{u(\cdot)} \mathbb{E}_x\!\left[\int_0^\infty e^{-\lambda s}L(X_s,u_s)\,ds\right].$$6 is small enough that

$$V(x)=\sup_{u(\cdot)} \mathbb{E}_x\!\left[\int_0^\infty e^{-\lambda s}L(X_s,u_s)\,ds\right].$$7

and

$$V(x)=\sup_{u(\cdot)} \mathbb{E}_x\!\left[\int_0^\infty e^{-\lambda s}L(X_s,u_s)\,ds\right].$$8

then RCPI stops in at most $$V(x)=\sup_{u(\cdot)} \mathbb{E}_x\!\left[\int_0^\infty e^{-\lambda s}L(X_s,u_s)\,ds\right].$$9 iterations and returns $$dX_t = f(X_t,u_t)\,dt + \sigma(X_t)\,dW_t,\qquad X_0=x,$$00, an $$dX_t = f(X_t,u_t)\,dt + \sigma(X_t)\,dW_t,\qquad X_0=x,$$01-saddle point, with total arithmetic cost

$$dX_t = f(X_t,u_t)\,dt + \sigma(X_t)\,dW_t,\qquad X_0=x,$$02

(Badger et al., 8 Aug 2025). The paper further specifies

$$dX_t = f(X_t,u_t)\,dt + \sigma(X_t)\,dW_t,\qquad X_0=x,$$03

in robust MDPs and

$$dX_t = f(X_t,u_t)\,dt + \sigma(X_t)\,dW_t,\qquad X_0=x,$$04

in Markov games.

Empirically, the paper reports four domains—Markov-games, inventory, gambler’s-ruin, and gridworld, each in “small” and “large” versions. RCPI$$dX_t = f(X_t,u_t)\,dt + \sigma(X_t)\,dW_t,\qquad X_0=x,$$05 and RCPI$$dX_t = f(X_t,u_t)\,dt + \sigma(X_t)\,dW_t,\qquad X_0=x,$$06 solve in seconds even on large problems such as 1000 states, whereas value iteration requires minutes to hours (Badger et al., 8 Aug 2025). PAI/FT often cycle or slow; HK, WS, and PPI are convergent but 1–3 orders of magnitude slower than RCPI; Bellman-residual versus time plots show super-linear convergence for RCPI (Badger et al., 8 Aug 2025).

For the stochastic-control PINN-PI framework, the practical considerations listed in the paper include network depth and width, activation function such as tanh or SiLU, learning rate, optimizer (Adam), number of residual epochs, residual penalty, collocation size $$dX_t = f(X_t,u_t)\,dt + \sigma(X_t)\,dW_t,\qquad X_0=x,$$07, sampling measure $$dX_t = f(X_t,u_t)\,dt + \sigma(X_t)\,dW_t,\qquad X_0=x,$$08, discount $$dX_t = f(X_t,u_t)\,dt + \sigma(X_t)\,dW_t,\qquad X_0=x,$$09, and domain radius $$dX_t = f(X_t,u_t)\,dt + \sigma(X_t)\,dW_t,\qquad X_0=x,$$10 (Kim et al., 3 Aug 2025). In the 10D LQR example with compact control, the setup uses random stable $$dX_t = f(X_t,u_t)\,dt + \sigma(X_t)\,dW_t,\qquad X_0=x,$$11, $$dX_t = f(X_t,u_t)\,dt + \sigma(X_t)\,dW_t,\qquad X_0=x,$$12, diffusion $$dX_t = f(X_t,u_t)\,dt + \sigma(X_t)\,dW_t,\qquad X_0=x,$$13, cost $$dX_t = f(X_t,u_t)\,dt + \sigma(X_t)\,dW_t,\qquad X_0=x,$$14 with $$dX_t = f(X_t,u_t)\,dt + \sigma(X_t)\,dW_t,\qquad X_0=x,$$15, a PINN architecture of 4 hidden layers $$dX_t = f(X_t,u_t)\,dt + \sigma(X_t)\,dW_t,\qquad X_0=x,$$16 neurons with tanh activations, $$dX_t = f(X_t,u_t)\,dt + \sigma(X_t)\,dW_t,\qquad X_0=x,$$17 collocation points uniform in $$dX_t = f(X_t,u_t)\,dt + \sigma(X_t)\,dW_t,\qquad X_0=x,$$18, Adam with learning rate $$dX_t = f(X_t,u_t)\,dt + \sigma(X_t)\,dW_t,\qquad X_0=x,$$19, and $$dX_t = f(X_t,u_t)\,dt + \sigma(X_t)\,dW_t,\qquad X_0=x,$$20 gradient steps per iteration (Kim et al., 3 Aug 2025). The reported results are that the residual $$dX_t = f(X_t,u_t)\,dt + \sigma(X_t)\,dW_t,\qquad X_0=x,$$21 falls below $$dX_t = f(X_t,u_t)\,dt + \sigma(X_t)\,dW_t,\qquad X_0=x,$$22 by iteration 5, $$dX_t = f(X_t,u_t)\,dt + \sigma(X_t)\,dW_t,\qquad X_0=x,$$23 after 8 policy iterations, the policy error $$dX_t = f(X_t,u_t)\,dt + \sigma(X_t)\,dW_t,\qquad X_0=x,$$24 decays exponentially, and the method outperforms SAC in reward and sample-efficiency (Kim et al., 3 Aug 2025).

A compact comparison is useful.

Setting	Residual role	Stated outcome
Stochastic optimal control (Kim et al., 3 Aug 2025)	Squared $$dX_t = f(X_t,u_t)\,dt + \sigma(X_t)\,dW_t,\qquad X_0=x,$$25 PDE residual used for policy evaluation	$$dX_t = f(X_t,u_t)\,dt + \sigma(X_t)\,dW_t,\qquad X_0=x,$$26 error control, gradient-error propagation bounds, exponential convergence of exact PI
Markov games / robust MDPs (Badger et al., 8 Aug 2025)	$$dX_t = f(X_t,u_t)\,dt + \sigma(X_t)\,dW_t,\qquad X_0=x,$$27-norm Bellman residual used as acceptance and fallback criterion	Guaranteed convergence to an $$dX_t = f(X_t,u_t)\,dt + \sigma(X_t)\,dW_t,\qquad X_0=x,$$28-saddle point with residual contraction each outer iteration

7. Interpretation, Misconceptions, and Open Directions

A common misconception would be to treat RCPI as a single universally defined algorithm. The 2025 literature uses the same name for two different procedures: one is a PINN-based policy-iteration method for second-order HJB equations, and the other is a residual-guarded fast policy-iteration method for saddle-point dynamic programming in Markov games and robust MDPs (Kim et al., 3 Aug 2025, Badger et al., 8 Aug 2025). The overlap lies in the methodological role of residuals, not in a shared state-action model, update rule, or computational primitive.

Another possible confusion concerns what the residual certifies. In the stochastic-control formulation, the residual is a PDE residual for a linear policy-evaluation equation, and the analysis shows that controlling it in $$dX_t = f(X_t,u_t)\,dt + \sigma(X_t)\,dW_t,\qquad X_0=x,$$29 controls both value and gradient errors (Kim et al., 3 Aug 2025). In the Markov-game formulation, the residual is a Bellman residual in $$dX_t = f(X_t,u_t)\,dt + \sigma(X_t)\,dW_t,\qquad X_0=x,$$30 and is used to guarantee monotonic progress toward a saddle point despite failures of differentiability-based line search (Badger et al., 8 Aug 2025). These are distinct error objects with different norms, different operators, and different theoretical roles.

The limitations stated in the Markov-game paper are also important for delimiting the concept. RCPI there assumes full model knowledge and exact computations, and the paper identifies sample-based, model-free, or approximate versions as an important direction, along with extensions to average-reward, risk-sensitive, or multi-player general-sum games, and adaptive tuning of $$dX_t = f(X_t,u_t)\,dt + \sigma(X_t)\,dW_t,\qquad X_0=x,$$31 and $$dX_t = f(X_t,u_t)\,dt + \sigma(X_t)\,dW_t,\qquad X_0=x,$$32 (Badger et al., 8 Aug 2025). In the stochastic-control work, the abstract emphasizes that the method provides a theoretical basis for evaluating policy quality during training and extends deterministic PINN-based approaches to stochastic settings (Kim et al., 3 Aug 2025). This suggests that residual-conditioned policy iteration is especially relevant where direct solution of Bellman or HJB equations is either analytically intractable or numerically delicate, and where residuals can serve as robust surrogates for trust in policy updates.

Taken together, these works position RCPI as a residual-centric design pattern in modern policy iteration. In one branch, residual minimization supplies a principled neural policy-evaluation mechanism for second-order HJB problems; in the other, residual thresholds furnish a convergence-preserving safeguard for policy iteration in minimax and robust dynamic programming.