Conservative Policy Iteration

• Conservative Policy Iteration is a policy improvement scheme that updates policies using a conservative mixture to ensure safe and monotonic performance gains.
• It strategically integrates a current policy with a greedy improvement to maintain critic accuracy and control occupancy measure drift.
• Recent extensions, including variance-reduced and offline variants, highlight its broad utility in enhancing stability and sample efficiency in RL.

Conservative Policy Iteration (CPI) is a policy-improvement scheme for discounted Markov decision processes in which a current policy is not replaced outright by a greedy or approximately greedy improvement policy, but is instead updated through a conservative mixture. In its classical form, CPI seeks safe, monotonic policy improvement by controlling the size of the policy change, typically through an update of the form $\pi_{k+1}=(1-\alpha)\pi_k+\alpha \tilde{\pi}$ with $\alpha\in[0,1]$. This conservative step is designed to keep the successor occupancy measure close to that of the incumbent policy, so that the critic or value estimate remains relevant on the states that determine the next update. Subsequent work has recast CPI as a reductions-based procedure over policy classes, analyzed its sample complexity, compared it to other approximate policy iteration schemes, and extended its logic to offline reinforcement learning, safe exploration, structured state abstraction, and decentralized partially observable control (Agarwal et al., 2022, Scherrer, 2014).

1. Classical formulation and objective

In the discounted setting, CPI is defined on an MDP with state space $S$, action space $A$, transition kernel $P$, reward function $r$, and discount factor $\gamma\in[0,1)$. For a stationary policy $\pi$, the value and action-value functions are

$$Q^{\pi}(s,a) \doteq r(s,a) + \gamma\,\mathbb{E}_{s' \sim P(\cdot|s,a)}\mathbb{E}_{a' \sim \pi(\cdot|s')} Q^{\pi}(s',a'),$$

and

$$V^{\pi}(s) \doteq \mathbb{E}_{a \sim \pi(\cdot|s)} Q^{\pi}(s,a),$$

with advantage $\alpha\in[0,1]$0. The performance objective is $\alpha\in[0,1]$1, typically the discounted return from a designated start-state distribution or start state (Sandhu et al., 6 May 2026, Agarwal et al., 2022).

The distinctive operation of CPI is the conservative mixing update. Given an improved policy $\alpha\in[0,1]$2—for example, greedy or approximately greedy with respect to the current critic—CPI forms

$\alpha\in[0,1]$3

or, in function-space notation,

$\alpha\in[0,1]$4

The purpose of the mixture is not merely heuristic damping. It is tied to the observation that the performance difference depends on the advantage of the old policy evaluated under the visitation distribution of the new policy, whereas the critic is usually accurate only on the distribution induced by the old policy. Conservative updates mitigate that mismatch by shrinking the occupancy shift (Sandhu et al., 6 May 2026).

Later comparative analyses place CPI within the broader family of approximate policy iteration schemes. In that literature, CPI is distinguished from standard API by using the discounted occupancy measure of the current policy in the greedy step and by explicitly maintaining a stochastic mixture over policies. This can yield stronger guarantees in terms of concentrability constants, although it may require substantially more iterations and memory proportional to the number of iterations (Scherrer, 2014).

2. Monotonic improvement and the geometry of conservatism

The theoretical core of CPI is the Performance Difference Lemma,

$\alpha\in[0,1]$5

or equivalently

$\alpha\in[0,1]$6

This identity makes clear why a fully greedy replacement can be problematic under approximation: the relevant expectation is taken under $\alpha\in[0,1]$7, not under $\alpha\in[0,1]$8. CPI addresses this by choosing $\alpha\in[0,1]$9 small enough that $S$0 remains close to $S$1 (Sandhu et al., 6 May 2026).

A standard monotonic improvement bound restated in recent work has the form

$S$2

for the CPI update

$S$3

The quadratic penalty captures the effect of distribution shift; the role of conservatism is precisely to keep that penalty small enough that the right-hand side stays positive (Sandhu et al., 6 May 2026).

Function-space analyses sharpen this picture. In the reductions-based treatment of CPI, the policy is viewed as an element of $S$4, and the update is a Frank–Wolfe step in function space. The performance functional is smooth in the $S$5 geometry:

$S$6

Under conservative mixing this yields

$S$7

and since $S$8, a sufficiently small $S$9 guarantees monotonic improvement whenever the ERM step yields positive advantage (Agarwal et al., 2022).

This function-space viewpoint also clarifies a common point of confusion. CPI does not fundamentally optimize in parameter space. In reductions-based formulations, a policy $A$0 is an $A$1-functional local optimum if

$A$2

This notion is independent of how the class $A$3 is parameterized. The cited analysis states that functional local optimality implies a small parameter gradient when a smooth parameterization exists, but not conversely; one can have $A$4 even when the policy is not locally optimal in function space, especially under non-linear or discontinuous parameterizations such as decision trees or ReLU nets. This is one reason reductions-based CPI is described as avoiding parameterization-induced non-convexities (Agarwal et al., 2022).

3. Reductions-based CPI and variance-reduced sample complexity

A modern reformulation treats policy improvement as empirical risk minimization over the policy space. An ERM oracle is given a dataset of state–loss pairs $A$5 with $A$6 and returns a policy $A$7 that approximately maximizes the linear objective induced by the losses. The construction is chosen so that the empirical linear objective estimates the functional gradient $A$8, and CPI then performs a conservative mixture step with the ERM output (Agarwal et al., 2022).

In classical CPI, each iteration requires a fresh estimate of the ERM objective. The cited analysis summarizes the resulting rates as follows: classical CPI runs for $A$9 iterations, each ERM requires $P$0 samples, and the total sample requirement to reach an $P$1-functional local optimum is therefore $P$2. Under restart-state coverage assumptions, classical global $P$3-optimality required $P$4 samples (Agarwal et al., 2022).

“Variance-Reduced Conservative Policy Iteration” replaces the per-iteration re-estimation strategy with a STORM-like estimator in function space. The estimator combines a fresh unbiased $P$5-sample, a Hessian-vector correction based on a future-advantage $P$6-sampler, and momentum on the previous dataset. Concretely, the method maintains a dataset $P$7 whose linear objective realizes an estimator $P$8 satisfying

$P$9

then solves

$r$0

followed by

$r$1

The $r$2-sampler returns $r$3 and an unbiased estimate of $r$4; the $r$5-sampler provides unbiased linear forms for the functional Hessian via the future advantage $r$6 (Agarwal et al., 2022).

The main local result is a high-probability guarantee that the algorithm outputs a policy $r$7 with

$r$8

after at most

$r$9

episodes of expected length $\gamma\in[0,1)$0. Under a restart distribution $\gamma\in[0,1)$1 with coverage coefficient $\gamma\in[0,1)$2, the same analysis yields

$\gamma\in[0,1)$3

after

$\gamma\in[0,1)$4

episodes. In the episodic model with exploratory policy class coefficient $\gamma\in[0,1)$5, the rate improves to

$\gamma\in[0,1)$6

episodes for global $\gamma\in[0,1)$7-optimality up to the policy-completeness residual $\gamma\in[0,1)$8 (Agarwal et al., 2022).

The conceptual claim of the variance-reduced analysis is that CPI can retain its function-space robustness while improving the sample complexity of local optimization from $\gamma\in[0,1)$9 to $\pi$0, and global optimization under exploratory classes from the previously established $\pi$1 to $\pi$2 (Agarwal et al., 2022).

4. Approximation, concentrability, and structural stability

Comparative analyses of approximate policy iteration emphasize that the quality of CPI bounds depends on concentrability constants. One such study states that “the performance guarantee of CPI can be arbitrarily better than that of API/API($\pi$3), but this comes at the cost of a relative—exponential in $\pi$4—increase of the number of iterations.” In the same comparison, CPI is associated with memory proportional to the number of iterations because the algorithm maintains stochastic mixtures over successive policies. This is contrasted with API, which uses constant memory, and with PSDP$\pi$5 and NSPI$\pi$6, which trade memory against performance guarantees (Scherrer, 2014).

A separate line of work links conservative updates to state abstraction via bisimulation metrics. In “Approximate Policy Iteration with Bisimulation Metrics,” conservative API($\pi$7) is motivated by the fact that the $\pi$8-bisimulation metric changes smoothly with the policy:

$\pi$9

Because the metric used for aggregation depends on the current policy, small policy changes stabilize metric learning itself. The resulting API($$Q^{\pi}(s,a) \doteq r(s,a) + \gamma\,\mathbb{E}_{s' \sim P(\cdot|s,a)}\mathbb{E}_{a' \sim \pi(\cdot|s')} Q^{\pi}(s',a'),$$0) scheme uses warm-started metric updates and the mixture

$$Q^{\pi}(s,a) \doteq r(s,a) + \gamma\,\mathbb{E}_{s' \sim P(\cdot|s,a)}\mathbb{E}_{a' \sim \pi(\cdot|s')} Q^{\pi}(s',a'),$$1

yielding the asymptotic bound

$$Q^{\pi}(s,a) \doteq r(s,a) + \gamma\,\mathbb{E}_{s' \sim P(\cdot|s,a)}\mathbb{E}_{a' \sim \pi(\cdot|s')} Q^{\pi}(s',a'),$$2

This suggests a broader interpretation of conservatism: the step size is controlling not only value-estimation drift but also representation drift (Kemertas et al., 2022).

In deterministic discounted control, related results show that exact policy iteration can sometimes achieve CPI’s intended goals without explicit mixing. For discrete-time systems with exact evaluation and exact greedy improvement, the cited work proves monotone improvement,

$$Q^{\pi}(s,a) \doteq r(s,a) + \gamma\,\mathbb{E}_{s' \sim P(\cdot|s,a)}\mathbb{E}_{a' \sim \pi(\cdot|s')} Q^{\pi}(s',a'),$$3

and derives a near-optimality bound

$$Q^{\pi}(s,a) \doteq r(s,a) + \gamma\,\mathbb{E}_{s' \sim P(\cdot|s,a)}\mathbb{E}_{a' \sim \pi(\cdot|s')} Q^{\pi}(s',a'),$$4

Under detectability and optimal-policy stability assumptions, it further establishes semiglobal practical stability after a finite, explicitly bounded number of iterations, and global exponential stability under additional linear bounds. The authors explicitly note that CPI is not analyzed in that paper, but the results illuminate when full greedy policy iteration is already “conservative enough” in the sense of monotonicity and stability (Brusse et al., 2024).

5. Extensions, analogues, and alternatives

CPI’s core logic—restrict the improvement step so that performance remains controlled—has been extended well beyond the original online discounted MDP setting. The following variants preserve that logic while changing what is meant by “conservative.”

Variant	Mechanism	Reported guarantee
Variance-Reduced CPI	ERM over policies with momentum dataset and Hessian-vector correction	$$Q^{\pi}(s,a) \doteq r(s,a) + \gamma\,\mathbb{E}_{s' \sim P(\cdot|s,a)}\mathbb{E}_{a' \sim \pi(\cdot|s')} Q^{\pi}(s',a'),$$5-functional local optimality in $$Q^{\pi}(s,a) \doteq r(s,a) + \gamma\,\mathbb{E}_{s' \sim P(\cdot|s,a)}\mathbb{E}_{a' \sim \pi(\cdot|s')} Q^{\pi}(s',a'),$$6
BCPO	Credible lower bound on $$Q^{\pi}(s,a) \doteq r(s,a) + \gamma\,\mathbb{E}_{s' \sim P(\cdot|s,a)}\mathbb{E}_{a' \sim \pi(\cdot|s')} Q^{\pi}(s',a'),$$7 and KL regularization toward behavior	Computable return lower bound improves up to explicit shift terms
SV-API / SV-PPO	Approximate Next Policy Sampling instead of shrinking target updates	Safety bound based on bounded critic error and next-policy alignment
RS-CPI for Dec-POMDPs	Conservative mixture over action–memory transition rules with entropic risk shaping	Monotonic improvement of a risk-aware objective and convergence to a person-by-person local optimum

In offline reinforcement learning, “Bayesian Conservative Policy Optimization (BCPO)” is described as an uncertainty-calibrated analogue of CPI. Rather than mixing toward a greedy policy under on-policy occupancy, BCPO constructs a lower credible bound

$$Q^{\pi}(s,a) \doteq r(s,a) + \gamma\,\mathbb{E}_{s' \sim P(\cdot|s,a)}\mathbb{E}_{a' \sim \pi(\cdot|s')} Q^{\pi}(s',a'),$$8

optimizes a pessimistic surrogate under the empirical state marginal of the dataset, and regularizes policy updates by KL divergence toward an estimated behavior policy, optionally with a trust region around the previous policy. In the finite-MDP theory, the pessimistic Bellman operator is a $$Q^{\pi}(s,a) \doteq r(s,a) + \gamma\,\mathbb{E}_{s' \sim P(\cdot|s,a)}\mathbb{E}_{a' \sim \pi(\cdot|s')} Q^{\pi}(s',a'),$$9-contraction with a unique fixed point $$V^{\pi}(s) \doteq \mathbb{E}_{a \sim \pi(\cdot|s)} Q^{\pi}(s,a),$$0, and on a high-probability event this fixed point lower-bounds the true value function pointwise. The paper’s monotonic improvement theorem states that if the update improves the BCPO surrogate and remains within the trust region, then

$$V^{\pi}(s) \doteq \mathbb{E}_{a \sim \pi(\cdot|s)} Q^{\pi}(s,a),$$1

where the shift term is explicitly bounded in terms of KL drift and critic magnitude. Here “conservative” refers to pessimistic value estimation and behavior anchoring rather than convex policy mixing (Chatterjee, 6 Mar 2026).

A different response to the same “chicken-and-egg” problem is Approximate Next Policy Sampling (ANPS). “Approximate Next Policy Sampling: Replacing Conservative Target Policy Updates in Deep RL” argues that conservative target updates are only one way to keep the critic accurate on the distribution that matters. ANPS instead modifies the training distribution so that it approximates the next policy’s occupancy. In the SV-API analysis, if the training distribution $$V^{\pi}(s) \doteq \mathbb{E}_{a \sim \pi(\cdot|s)} Q^{\pi}(s,a),$$2 satisfies $$V^{\pi}(s) \doteq \mathbb{E}_{a \sim \pi(\cdot|s)} Q^{\pi}(s,a),$$3-Next Policy Alignment,

$$V^{\pi}(s) \doteq \mathbb{E}_{a \sim \pi(\cdot|s)} Q^{\pi}(s,a),$$4

then critic error on $$V^{\pi}(s) \doteq \mathbb{E}_{a \sim \pi(\cdot|s)} Q^{\pi}(s,a),$$5 can be controlled by critic error on $$V^{\pi}(s) \doteq \mathbb{E}_{a \sim \pi(\cdot|s)} Q^{\pi}(s,a),$$6 plus a distribution-mismatch term. The resulting improvement theorem for SV-API states

$$V^{\pi}(s) \doteq \mathbb{E}_{a \sim \pi(\cdot|s)} Q^{\pi}(s,a),$$7

under bounded training error and small per-state TV drift of the behavioral policy. The explicit contrast with CPI is that the advantage term is not multiplied by a small target-update step size; conservatism is moved from the target update to the behavioral-policy evolution and stability gate (Sandhu et al., 6 May 2026).

Safe policy optimization also appears in online model-free learning with baseline constraints. “Conservative Optimistic Policy Optimization via Multiple Importance Sampling” imposes the cumulative requirement

$$V^{\pi}(s) \doteq \mathbb{E}_{a \sim \pi(\cdot|s)} Q^{\pi}(s,a),$$8

where $$V^{\pi}(s) \doteq \mathbb{E}_{a \sim \pi(\cdot|s)} Q^{\pi}(s,a),$$9 is a known baseline policy parameter. The method combines upper and lower confidence bounds from truncated balance-heuristic multiple importance sampling with a conservative safety check or safe set. The result is a high-probability guarantee that the baseline constraint is never violated, together with $\alpha\in[0,1]$00 regret in both discrete and compact continuous parameter spaces. This is not classical CPI, but it is explicitly framed as conservative exploration in policy optimization and preserves the central CPI theme that intermediate deployed policies must remain safe relative to a reference policy (Azize et al., 2021).

In decentralized partially observable control, “Risk-seeking conservative policy iteration with agent-state based policies for Dec-POMDPs with guaranteed convergence” adapts the CPI mixture idea to finite-memory agent-state policies. For each agent and time step, a greedy update is computed with respect to a local risk-averaged $\alpha\in[0,1]$01 function, and then mixed conservatively:

$\alpha\in[0,1]$02

The paper proves monotonic improvement for a risk-aware objective based on an entropic risk map and shows that the resulting sequence converges to a person-by-person local optimum in the finite-memory policy class. The stated motivation for the conservative mix is to prevent support collapse in partially observed multi-agent settings, where fully greedy updates can render continued improvement impossible on zero-probability histories (Sinha et al., 10 Apr 2026).

6. Misconceptions, limitations, and contemporary perspective

One recurrent misconception is that CPI is simply “small-step policy gradient.” Modern analyses contradict that characterization. Reductions-based CPI operates directly in policy function space, can be analyzed as Frank–Wolfe over $\alpha\in[0,1]$03, and targets functional local optimality rather than small parameter gradients. This distinction matters when the policy class is non-linear, non-smooth, or discontinuously parameterized (Agarwal et al., 2022).

A second misconception is that conservatism is synonymous with KL trust regions. Classical CPI uses convex mixing of policies; TRPO and PPO constrain policy change with KL-based surrogates; BCPO regularizes by KL toward the behavior policy and the previous policy; SV-API argues that the same safety objective can be achieved by aligning the critic’s training distribution with the next policy instead of shrinking the target update. These are related but non-identical instantiations of conservative improvement (Sandhu et al., 6 May 2026, Chatterjee, 6 Mar 2026).

The principal limitations of CPI-type analyses are also recurrent across the literature. Finite action spaces, bounded rewards, manageable policy-class complexity, and some form of coverage or concentrability are standard assumptions. In the variance-reduced theory, global guarantees depend on either a restart distribution with finite $\alpha\in[0,1]$04 or an exploratory policy class with finite $\alpha\in[0,1]$05, and residual terms $\alpha\in[0,1]$06 or $\alpha\in[0,1]$07 quantify policy misspecification. In comparative analyses, poor concentrability can make API-type guarantees vacuous, whereas CPI can still retain a meaningful bound if a stronger constant such as $\alpha\in[0,1]$08 is finite. In bisimulation-based API($\alpha\in[0,1]$09), conservative updates improve asymptotic constants but slow the one-step contraction rate. In offline variants, over-conservatism can collapse learning toward behavior cloning, while insufficient conservatism leads to drift outside dataset support (Agarwal et al., 2022, Scherrer, 2014, Kemertas et al., 2022, Chatterjee, 6 Mar 2026).

A final practical point is that CPI remains as much a design principle as a single algorithm. Deep-RL work around PPO, TRPO, trust-region methods, replay buffers, offline pessimism, and safe policy improvement often inherits CPI’s central insight: policy improvement is limited not only by what appears greedy under the current value estimate, but by where that estimate is valid and by how much policy-induced distribution shift can be tolerated. Theoretical work on variance reduction, alternative sampling distributions, conservative optimism, and finite-memory decentralized control indicates that CPI is best understood as a general doctrine of controlled policy improvement rather than a single fixed update rule (Sandhu et al., 6 May 2026, Agarwal et al., 2022).