---
title: Residual Conditioned Policy Iteration (RCPI)
url: https://www.emergentmind.com/topics/residual-conditioned-policy-iteration-rcpi
type: topic
---

# Residual Conditioned Policy Iteration (RCPI)

Searching arXiv for the specified RCPI papers and closely related policy-iteration context.
Residual Conditioned Policy Iteration (RCPI) denotes two distinct algorithmic constructions that share a common organizing principle: policy iteration is conditioned on a residual criterion that controls or certifies improvement. In one formulation, RCPI appears as a physics-informed neural network policy iteration framework for stochastic optimal control governed by second-order Hamilton–Jacobi–Bellman (HJB) equations, where policy evaluation is performed by minimizing a linear PDE residual under a fixed policy [2508.01718]. In another formulation, RCPI is introduced for zero-sum Markov games and s-rectangular robust MDPs as a residual-guarded variant of fast policy iteration, designed to guarantee convergence to a saddle point while retaining the practical speed of exact policy-evaluation steps [2508.06661]. Across both settings, the defining motif is the use of a residual—not merely as a stopping diagnostic, but as a structural condition that determines when an evaluation or improvement step is accepted.

## 1. Conceptual Scope and Naming

The name “Residual Conditioned Policy Iteration” is used in two technically different contexts in the 2025 literature.

In the stochastic optimal control setting, the framework described in “Neural Policy Iteration for Stochastic Optimal Control: A Physics-Informed Approach” uses a neural approximation of the value function and performs policy evaluation by minimizing the residual of a linear elliptic PDE induced by a fixed Markov policy [2508.01718]. The conditioning mechanism is explicit in the analysis: the residual controls both the value-function error and the gradient error through an energy estimate, and the gradient error in turn affects the greedy policy update. This makes the residual a quantitative proxy for policy quality during training.

In the Markov-game and robust-MDP setting, “Convergence of Fast Policy Iteration in Markov Games and Robust MDPs” introduces RCPI as a correction to the Filar–Tolwinski (FT) algorithm [2508.06661]. Here the conditioning mechanism is operational rather than variational: an exact policy-evaluation candidate is accepted only if the residual can be reduced sufficiently, and otherwise the method reverts to a standard Bellman backup. The resulting algorithm enforces monotonic residual reduction and a linear contraction in the $\infty$-norm residual up to the approximation floor.

These two uses of RCPI are not the same algorithm. They are united by a shared methodological pattern: residuals are promoted from passive error indicators to active acceptance criteria or training objectives. This suggests a broader interpretation of RCPI as a residual-governed family of policy-iteration procedures rather than a single canonical method.

## 2. RCPI for Stochastic Optimal Control and Second-Order HJB Equations

In the stochastic control formulation, the infinite-horizon problem is given by
$$
dX_t = f(X_t,u_t)\,dt + \sigma(X_t)\,dW_t,\qquad X_0=x,
$$
with state $x\in\mathbb{R}^d$, control $u\in A\subset\mathbb{R}^m$, drift $f(x,u)\in\mathbb{R}^d$, diffusion matrix $\sigma(x)\in\mathbb{R}^{d\times k}$, running cost $L(x,u)\ge 0$, value function $V(x)$, and discount $\lambda>0$ [2508.01718]. The value is
$$
V(x)=\sup_{u(\cdot)} \mathbb{E}_x\!\left[\int_0^\infty e^{-\lambda s}L(X_s,u_s)\,ds\right].
$$
Its dynamic programming equation is the second-order HJB PDE
$$
\lambda V(x)-\tfrac12\,\mathrm{tr}\bigl[\sigma(x)\sigma(x)^\top D_{xx}^2V(x)\bigr]
-\sup_{u\in A}\bigl\{f(x,u)\cdot\nabla_xV(x)+L(x,u)\bigr\}=0
$$
in viscosity sense [2508.01718].

The essential simplification arises when a Markov policy $\pi_n(x)$ is fixed. The supremum is then removed, yielding a linear elliptic PDE for the policy value $v_n(x)=V^{\pi_n}(x)$:
$$
\lambda v_n(x)-\tfrac12\,\mathrm{Tr}[\sigma(x)\sigma(x)^\top D^2v_n(x)]
-f(x,\pi_n(x))\cdot\nabla v_n(x)=L(x,\pi_n(x)).
$$
This linearization is the basis of the PINN-based evaluation step. Rather than solving the nonlinear HJB directly, the method alternates between solving linear PDEs induced by current policies and updating those policies greedily.

The paper states that this framework extends recent deterministic PINN-based approaches to stochastic settings and inherits the global exponential convergence guarantees of classical policy iteration under mild conditions [2508.01718]. It also demonstrates benchmark performance on stochastic cartpole, pendulum problems, and high-dimensional linear quadratic regulation problems in up to 10D.

## 3. Residual-Based Policy Evaluation in the PINN-PI Framework

Given a fixed policy $\pi_n$, the value function is approximated by a neural network $v_n(\cdot;\theta)$, and the PDE residual is defined as
$$
R_n(x;\theta):=\lambda v_n(x;\theta)
-\tfrac12\,\mathrm{Tr}[\sigma(x)\sigma(x)^\top D^2v_n(x;\theta)]
-f\bigl(x,\pi_n(x)\bigr)\cdot\nabla v_n(x;\theta)
-L\bigl(x,\pi_n(x)\bigr).
$$
The evaluation loss is the squared $L^2$ residual, either as
$$
L_{\mathrm{eval}}(\theta)=\int_\Omega |R_n(x;\theta)|^2\,d\mu(x),
$$
or empirically over collocation points $\{x_i\}$,
$$
L_{\mathrm{eval}}(\theta)=\frac1N\sum_{i=1}^N |R_n(x_i;\theta)|^2
$$
[2508.01718].

The core analytical device is the energy estimate stated as Proposition 4.1:
$$
(\lambda-B/2)\|e\|_{L^2}^2 + (\nu/2)\|\nabla e\|_{L^2}^2 \le (R_n,e),
$$
where $e=v_n(\cdot;\theta)-v_n^\star$ and the constants $B,\nu$ come from Assumption (A4) [2508.01718]. By Cauchy–Schwarz and Young, controlling $\|R_n\|_{L^2}$ forces both $\|e\|_{L^2}$ and $\|\nabla e\|_{L^2}$ to be small. This is a defining feature of the framework: the residual is not merely minimized heuristically, but linked to systematic $L^2$ error control at each policy-evaluation step.

The policy improvement step uses the learned gradient:
$$
\pi_{n+1}(x)=\arg\min_{u\in A}\Bigl\{f(x,u)\cdot\nabla v_n(x;\theta_n^\ast)+L(x,u)\Bigr\}.
$$
The quantity $\nabla_x v_n$ enters as a parameter in a pointwise convex optimization in $u$ [2508.01718]. The paper further provides a policy-error Lipschitz bound: if $z,z'\in\mathbb{R}^d$ with $|z|,|z'|\le M<\mu_a/\tilde V$, then the selector
$$
a^\ast(x,z)=\arg\max_{a\in A}\{L(x,a)+b(x,a)\cdot z\}
$$
satisfies
$$
|a^\ast(x,z)-a^\ast(x,z')|\le \theta |z-z'|.
$$
This quantifies how gradient approximation errors propagate into policy updates.

The exact policy-iteration iterates satisfy a global exponential convergence theorem under assumptions (A1–A4): if
$$
\tilde\kappa=\sqrt{\frac{C_R^2}{\nu(\lambda-B/2)}}\in(0,1),
$$
then
$$
\|v_n-V\|_{L^2}\le C\cdot \tilde\kappa^n
$$
[2508.01718]. The residual-conditioned character of the method is therefore theoretical as well as algorithmic: the PDE residual controls evaluation error, the evaluation error controls gradient error, and the gradient error controls policy quality.

## 4. RCPI as a Convergent Alternative to Fast Policy Iteration in Markov Games and Robust MDPs

In zero-sum Markov games, the state space is $S=\{1,\dots,S\}$, the maximizing agent’s actions are $A=\{1,\dots,A\}$, the minimizing agent’s actions are $B=\{1,\dots,B\}$, the reward function is $r:S\times A\times B\times S\to[-r_{\max},r_{\max}]$, the transition kernel is $p(s,a,b,s')=P(s'|s,a,b)$, and the discount factor is $\gamma\in(0,1)$ [2508.06661]. For stationary randomized policies $\pi:S\to\Delta^A$ and $\sigma:S\to\Delta^B$, the value $v^{\pi,\sigma}\in\mathbb{R}^S$ solves
$$
v^{\pi,\sigma}(s)=\mathbb{E}_{\pi,\sigma}\!\left[\sum_{t\ge 0}\gamma^t r(s_t,a_t,b_t,s_{t+1})\mid s_0=s\right],
$$
and the saddle-point value satisfies
$$
v_s^\ast=\max_\pi\min_\sigma v^{\pi,\sigma}(s).
$$
The Bellman evaluation operator is
$$
(T^{\pi,\sigma}v)(s)=r^{\pi,\sigma}(s)+\gamma\sum_{s'}P^{\pi,\sigma}(s,s')v(s'),
$$
and the equilibrium Bellman operator is
$$
(Tv)(s)=\max_\pi\min_\sigma (T^{\pi,\sigma}v)(s).
$$
The policy-improvement operator is
$$
B(v)=\arg\max_\pi\arg\min_\sigma\,T^{\pi,\sigma}v
$$
[2508.06661]. The paper defines $\epsilon$-saddle points and extends the same structure analogously to s-rectangular robust MDPs, replacing the adversarial policy by an uncertain transition kernel $p$ in an ambiguity set.

Within this setting, the paper first analyzes the FT algorithm. FT starts from an initial value $v^0$, alternates greedy policy improvement and exact policy evaluation, forms the search direction $d^k=u^k-v^{k-1}$, and performs a backtracking Armijo line search to decrease the squared $L_2$ Bellman residual $\phi(v)=\psi_2(v)^2$ [2508.06661]. Its failure mode is central to the motivation for RCPI. The paper states that FT relies on differentiability of $\psi_2^2$, but $\psi_2$ is not smooth everywhere because the min/max operations create kinks. A 3-state Markov-game counterexample shows that for $v^0=0$, any $\alpha\in(0,1]$ along the first search direction increases $\psi_2^2$, so Armijo’s condition can never be met and FT loops in its line search:
$$
\Delta(\alpha)=\psi_2(v^0+\alpha d^1)^2-\psi_2(v^0)^2=C_1\alpha+C_2\alpha^2>0\qquad \forall \alpha>0
$$
[2508.06661].

RCPI is introduced as a remedy. It combines one-step exact policy evaluation with a guard that reverts to a standard Bellman backup if the candidate value does not sufficiently reduce the residual. The stated consequence is that RCPI is guaranteed to converge to a saddle point, whereas FT may fail to converge and may loop indefinitely even in small games [2508.06661].

## 5. Algorithmic Mechanisms and Residual Contraction

The RCPI algorithm for Markov games and robust MDPs is formulated with approximate Bellman and policy-improvement oracles $T^\delta$ and $B^\delta$, satisfying $\|T^\delta v-Tv\|_\infty\le \delta$ and $B^\delta(v)\subseteq \epsilon$-saddle points [2508.06661]. Its inputs are $v^0$, target $\epsilon$, backup tolerance $\delta$ with
$$
\delta<\frac{\epsilon(1-\gamma)^2}{2\gamma(3+\gamma)},
$$
and a maximum number of fixes $m$.

At iteration $k$, the algorithm first computes a greedy or approximately greedy saddle-point policy pair $(\pi^k,\sigma^k)\in B^\delta(v^{k-1})$, then forms the exact-evaluation candidate
$$
u^{k,0}=(I-\gamma P^{\pi^k,\sigma^k})^{-1}r^{\pi^k,\sigma^k}.
$$
If
$$
\gamma^{m-1}\psi_\infty^\delta(u^{k,0})+\frac{2(1+\gamma)}{1-\gamma}\delta>\psi_\infty^\delta(v^{k-1}),
$$
the method immediately falls back to the Bellman backup
$$
v^k\leftarrow T^\delta(v^{k-1}).
$$
Otherwise it applies up to $m$ successive $T^\delta$ backups to the candidate while
$$
\psi_\infty^\delta(u^{k,l})>\gamma\cdot\psi_\infty^\delta(v^{k-1})+2(1+\gamma)\delta,
$$
and then sets $v^k=u^{k,l}$ [2508.06661].

The paper’s summary of the intuition is precise. The candidate $u^{k,0}$ is accepted only if in at most $m$ successive $T^\delta$ backups the residual will drop below $\gamma\cdot\psi_\infty^\delta(v^{k-1})+O(\delta)$; otherwise the algorithm falls back immediately to one Bellman backup. Every outer iteration guarantees
$$
\psi_\infty^\delta(v^k)\le \gamma\cdot\psi_\infty^\delta(v^{k-1})+2(1+\gamma)\delta,
$$
which is a linear contraction up to the $\delta$-floor [2508.06661]. Unlike FT, there is no need for line search or differentiability, because monotonic residual reduction is enforced directly.

The stochastic-control RCPI/PINN-PI framework has a different algorithmic form but a related logic. Its pseudocode takes as input an initial policy $\pi_0(x)$, network $v(\cdot;\theta)$ with $\theta_0$, sampler $\mu$, and tolerance $\epsilon$. At each iteration it samples collocation points, minimizes the empirical squared PDE residual, updates the policy greedily through
$$
\pi_{n+1}(x)\leftarrow \arg\min_{u\in A}\{f(x,u)\cdot\nabla v(x;\theta_n)+L(x,u)\},
$$
and stops when $\|\pi_{n+1}-\pi_n\|_\infty<\epsilon$ [2508.01718]. Here the residual conditioning takes the form of residual-driven policy evaluation rather than residual-guarded acceptance, but in both cases the residual determines whether policy iteration is considered reliable.

## 6. Complexity, Empirical Results, and Practical Parameters

The two RCPI formulations differ substantially in computational profile.

For Markov games and robust MDPs, each RCPI outer loop costs
$$
O((1+m)\cdot T + S^2AB + S^3),
$$
where $T$ is the cost of one approximate-oracle Bellman-policy computation [2508.06661]. The runtime theorem states that if $\delta>0$ is small enough that
$$
\epsilon>\frac{2(1+\gamma)\delta}{(1-\gamma)^2},
$$
and
$$
Z=\left\lceil \frac{\ln\!\left(\left(\frac{1-\gamma}{2\gamma}\epsilon-\frac{3+\gamma}{1-\gamma}\delta\right)/(r_{\max}+\delta)\right)}{\ln \gamma}\right\rceil,
$$
then RCPI stops in at most $Z$ iterations and returns $(\pi,\sigma)$, an $\epsilon$-saddle point, with total arithmetic cost
$$
O\!\left(Z\cdot[T\cdot(1+m)+S^2AB+S^3]\right)
$$
[2508.06661]. The paper further specifies
$$
T_R=O(S^{4.5}A^{4.5})
$$
in robust MDPs and
$$
T_G=O(S^2AB + S(A+B)^{1.5}A^2\log(\delta^{-1}))
$$
in Markov games.

Empirically, the paper reports four domains—Markov-games, inventory, gambler’s-ruin, and gridworld, each in “small” and “large” versions. RCPI$_0$ and RCPI$_\infty$ solve in seconds even on large problems such as 1000 states, whereas value iteration requires minutes to hours [2508.06661]. PAI/FT often cycle or slow; HK, WS, and PPI are convergent but 1–3 orders of magnitude slower than RCPI; Bellman-residual versus time plots show super-linear convergence for RCPI [2508.06661].

For the stochastic-control PINN-PI framework, the practical considerations listed in the paper include network depth and width, activation function such as tanh or SiLU, learning rate, optimizer (Adam), number of residual epochs, residual penalty, collocation size $N$, sampling measure $\mu$, discount $\lambda$, and domain radius $R$ [2508.01718]. In the 10D LQR example with compact control, the setup uses random stable $A\in\mathbb{R}^{10\times 10}$, $B\in\mathbb{R}^{10\times m}$, diffusion $\sigma=0.1I_{10}$, cost $L(x,u)=x^\top Qx+u^\top Ru$ with $\|u\|_\infty\le u_{\max}$, a PINN architecture of 4 hidden layers $\times 128$ neurons with tanh activations, $N=20\,000$ collocation points uniform in $[-5,5]^{10}$, Adam with learning rate $10^{-3}$, and $10\,000$ gradient steps per iteration [2508.01718]. The reported results are that the residual $L_2$ falls below $10^{-4}$ by iteration 5, $\|v_n-V_{\mathrm{true}}\|_\infty\approx 10^{-3}$ after 8 policy iterations, the policy error $\|\pi_n-\pi^\ast\|_{L^2}$ decays exponentially, and the method outperforms SAC in reward and sample-efficiency [2508.01718].

A compact comparison is useful.

| Setting | Residual role | Stated outcome |
|---|---|---|
| Stochastic optimal control [2508.01718] | Squared $L^2$ PDE residual used for policy evaluation | $L^2$ error control, gradient-error propagation bounds, exponential convergence of exact PI |
| Markov games / robust MDPs [2508.06661] | $\infty$-norm Bellman residual used as acceptance and fallback criterion | Guaranteed convergence to an $\epsilon$-saddle point with residual contraction each outer iteration |

## 7. Interpretation, Misconceptions, and Open Directions

A common misconception would be to treat RCPI as a single universally defined algorithm. The 2025 literature uses the same name for two different procedures: one is a PINN-based policy-iteration method for second-order HJB equations, and the other is a residual-guarded fast policy-iteration method for saddle-point dynamic programming in Markov games and robust MDPs [2508.01718; 2508.06661]. The overlap lies in the methodological role of residuals, not in a shared state-action model, update rule, or computational primitive.

Another possible confusion concerns what the residual certifies. In the stochastic-control formulation, the residual is a PDE residual for a linear policy-evaluation equation, and the analysis shows that controlling it in $L^2$ controls both value and gradient errors [2508.01718]. In the Markov-game formulation, the residual is a Bellman residual in $\ell_\infty$ and is used to guarantee monotonic progress toward a saddle point despite failures of differentiability-based line search [2508.06661]. These are distinct error objects with different norms, different operators, and different theoretical roles.

The limitations stated in the Markov-game paper are also important for delimiting the concept. RCPI there assumes full model knowledge and exact computations, and the paper identifies sample-based, model-free, or approximate versions as an important direction, along with extensions to average-reward, risk-sensitive, or multi-player general-sum games, and adaptive tuning of $m$ and $\delta$ [2508.06661]. In the stochastic-control work, the abstract emphasizes that the method provides a theoretical basis for evaluating policy quality during training and extends deterministic PINN-based approaches to stochastic settings [2508.01718]. This suggests that residual-conditioned policy iteration is especially relevant where direct solution of Bellman or HJB equations is either analytically intractable or numerically delicate, and where residuals can serve as robust surrogates for trust in policy updates.

Taken together, these works position RCPI as a residual-centric design pattern in modern policy iteration. In one branch, residual minimization supplies a principled neural policy-evaluation mechanism for second-order HJB problems; in the other, residual thresholds furnish a convergence-preserving safeguard for policy iteration in minimax and robust dynamic programming.

Source: https://www.emergentmind.com/topics/residual-conditioned-policy-iteration-rcpi