Frozen Policy Iteration in RL & Control

• Frozen Policy Iteration is a framework that fixes the current policy during evaluation phases to enhance computational efficiency in MDPs and control systems.
• It integrates techniques like inexact evaluation, semismooth Newton methods, and partial evaluation to balance error attenuation and policy improvement.
• The approach underpins diverse applications from large-scale reinforcement learning and deterministic control to static analysis and language model alignment.

to=arxiv_search.search ,最新高清无码专区 万亚json {"query":"\"Frozen Policy Iteration\" arXiv", "max_results": 10} to=arxiv_search.search оШjson {"query":"\"Inexact Policy Iteration Methods for Large-Scale Markov Decision Processes\" (Gargiani et al., 2024)", "max_results": 5} to=arxiv_search.search 大发快三豹子json {"query":"\"Convergence Analysis of Policy Iteration\" (Heydari, 2015) OR \"Approximate Modified Policy Iteration\" (Scherrer et al., 2012) OR \"Easy Monotonic Policy Iteration\" (Achiam, 2016) OR \"Frozen Policy Iteration: Computationally Efficient RL under Linear Q^π Realizability for Deterministic Dynamics\" (Ke et al., 28 Feb 2026)", "max_results": 10} Frozen Policy Iteration (FPI) denotes a class of schemes in which a current policy is held fixed during a policy-evaluation phase and changed only in a subsequent improvement phase. In the cited literature, that template appears in several technically distinct forms: exact or inexact policy evaluation for discounted finite Markov decision processes (MDPs), admissibility-preserving policy iteration for deterministic undiscounted nonlinear control, partial evaluation in modified policy iteration, online reinforcement learning with statewise freezing under linear $Q^\pi$ realizability, test-time alignment of frozen LLMs through value-guided reweighting, and branch-freezing procedures for monotone max-min equation systems in static analysis (Gargiani et al., 2024, Heydari, 2015, Scherrer et al., 2012, Ke et al., 28 Feb 2026, Zhang et al., 21 Jun 2025, Monniaux et al., 9 Jun 2026).

1. Core scheme and formal structure

In discounted finite MDPs, the canonical setup is a 5-tuple $\{S,A,P,r,\gamma\}$ with finite state set $S=\{1,\ldots,n\}$, finite action set $A=\{1,\ldots,m\}$, transition matrices $P^a\in\mathbb{R}^{n\times n}$, reward vectors $r^a\in\mathbb{R}^n$, and discount $\gamma\in(0,1)$. A deterministic stationary policy $\pi:S\to A$ induces $r^\pi$ and $P^\pi$, and its Bellman operator is

$\{S,A,P,r,\gamma\}$0

Exact policy evaluation solves

$\{S,A,P,r,\gamma\}$1

while policy improvement selects a greedy policy

$\{S,A,P,r,\gamma\}$2

In this setting, freezing means that $\{S,A,P,r,\gamma\}$3 is fixed while its value is computed, whether exactly or inexactly (Gargiani et al., 2024).

In deterministic undiscounted nonlinear control, the same separation appears with different objects. For dynamics

$\{S,A,P,r,\gamma\}$4

stage cost $\{S,A,P,r,\gamma\}$5, and infinite-horizon performance index

$\{S,A,P,r,\gamma\}$6

a stationary feedback policy $\{S,A,P,r,\gamma\}$7 has value

$\{S,A,P,r,\gamma\}$8

Frozen evaluation solves

$\{S,A,P,r,\gamma\}$9

and improvement chooses

$S=\{1,\ldots,n\}$0

Here the frozen object is again the current policy, but the analysis does not rely on discounting or contraction (Heydari, 2015).

Setting	Frozen object	Evaluation mechanism
Discounted finite MDPs	Current greedy policy	Exact linear solve or inexact inner solver
Deterministic undiscounted control	Current stabilizing feedback law	Fixed-point solution of $S=\{1,\ldots,n\}$1
Modified / approximate PI	Current policy for $S=\{1,\ldots,n\}$2 steps	Partial evaluation $S=\{1,\ldots,n\}$3
Online RL under linear $S=\{1,\ldots,n\}$4 realizability	Policy at well-explored states	Least squares on on-policy trajectory suffixes
Frozen-base-model alignment	Base model weights	Value-guided decoding distribution
Max-policy iteration	Branch selections in max/min operators	Value iteration on $S=\{1,\ldots,n\}$5

This common structure does not imply a single convergence mechanism. In some works the key tool is Bellman contraction, in others semismooth Newton theory, comparison lemmas, self-normalized confidence bounds, or finite-height order structure. A plausible implication is that “frozen policy iteration” is best understood as a design pattern rather than a single algorithmic identity.

2. Discounted finite MDPs: inexact frozen evaluation and semismooth Newton structure

A particularly explicit formalization appears in the inexact policy iteration framework for large-scale discounted finite MDPs. There, FPI is interpreted as policy iteration in which the greedy policy is frozen during evaluation, but the policy-evaluation linear system is solved only approximately. Given $S=\{1,\ldots,n\}$6, one sets

$S=\{1,\ldots,n\}$7

then approximately solves

$S=\{1,\ldots,n\}$8

and finally sets $S=\{1,\ldots,n\}$9 to the last inner iterate (Gargiani et al., 2024).

The semismooth Newton interpretation is based on the Bellman residual

$A=\{1,\ldots,m\}$0

For any $A=\{1,\ldots,m\}$1, $A=\{1,\ldots,m\}$2 returns argmax selections of $A=\{1,\ldots,m\}$3, and the generalized Jacobian of $A=\{1,\ldots,m\}$4 is

$A=\{1,\ldots,m\}$5

Each $A=\{1,\ldots,m\}$6 is nonsingular. At the optimal value $A=\{1,\ldots,m\}$7, the residual is globally Lipschitz and strongly semismooth, with

$A=\{1,\ldots,m\}$8

for $A=\{1,\ldots,m\}$9 and any $P^a\in\mathbb{R}^{n\times n}$0.

The inexact evaluation step uses the residual condition

$P^a\in\mathbb{R}^{n\times n}$1

with $P^a\in\mathbb{R}^{n\times n}$2. In reward form this becomes

$P^a\in\mathbb{R}^{n\times n}$3

The practical local-convergence range identified in the paper is

$P^a\in\mathbb{R}^{n\times n}$4

Within that range, local convergence satisfies

$P^a\in\mathbb{R}^{n\times n}$5

Hence the method converges Q-linearly with asymptotic rate

$P^a\in\mathbb{R}^{n\times n}$6

and becomes Q-superlinear if $P^a\in\mathbb{R}^{n\times n}$7. The same work also gives a global contraction regime: if $P^a\in\mathbb{R}^{n\times n}$8 and $P^a\in\mathbb{R}^{n\times n}$9, then

$r^a\in\mathbb{R}^n$0

For arbitrary $r^a\in\mathbb{R}^n$1, the scheme converges globally to a ball $r^a\in\mathbb{R}^n$2 with

$r^a\in\mathbb{R}^n$3

This formulation strictly generalizes optimistic or modified policy iteration. If the inner solver is value iteration under the frozen policy and is stopped after a fixed number $r^a\in\mathbb{R}^n$4 of inner steps, then

$r^a\in\mathbb{R}^n$5

which is the optimistic/modified policy iteration form. The inexact framework replaces fixed $r^a\in\mathbb{R}^n$6 by a residual-based rule and allows alternative inner solvers, including Richardson, steepest descent, MinRes, and GMRES. The paper emphasizes that the choice of inner solver is fundamental. GMRES is particularly effective because policy-evaluation matrices have eigenvalues in the disk centered at $r^a\in\mathbb{R}^n$7 of radius $r^a\in\mathbb{R}^n$8, and the stopping condition is guaranteed to be met after finitely many inner iterations whenever the inner solver contracts linearly in some norm. On epidemiological discounted MDPs for dynamic SIS control, the numerical study reports that with $r^a\in\mathbb{R}^n$9, iGMRES-PI consistently outperforms exact PI in CPU time, with speedups of $\gamma\in(0,1)$0 at $\gamma\in(0,1)$1, $\gamma\in(0,1)$2 at $\gamma\in(0,1)$3, and up to $\gamma\in(0,1)$4 on the largest tested problem (Gargiani et al., 2024).

3. Deterministic undiscounted control: admissibility, uniqueness, and monotone convergence

In A. Heydari’s analysis of policy iteration for deterministic nonlinear systems, frozen policy iteration is studied on a compact, connected domain $\gamma\in(0,1)$5 containing the origin. The admissible-policy set $\gamma\in(0,1)$6 consists of policies that asymptotically stabilize the system in $\gamma\in(0,1)$7 and whose value functions are finite and upper-bounded on $\gamma\in(0,1)$8 by a continuous function $\gamma\in(0,1)$9 with $\pi:S\to A$0. The standing assumptions are the existence of at least one admissible policy and the condition that the intersection of $\pi:S\to A$1 with the invariant set of the autonomous dynamics $\pi:S\to A$2 is $\pi:S\to A$3 (Heydari, 2015).

The Bellman optimality equation is

$\pi:S\to A$4

No contraction argument is used. Instead, uniqueness of the Bellman solution on $\pi:S\to A$5 is derived from two comparison lemmas. The first states that if, for admissible $\pi:S\to A$6 and $\pi:S\to A$7,

$\pi:S\to A$8

then $\pi:S\to A$9 on $r^\pi$0. The second states that if $r^\pi$1 at some $r^\pi$2, then there exists some state where

$r^\pi$3

These lemmas yield the theorem that the Bellman equation has a unique solution on $r^\pi$4.

The frozen policy iteration algorithm begins from an initial admissible stabilizing policy $r^\pi$5, computes $r^\pi$6, and then alternates frozen evaluation

$r^\pi$7

with greedy improvement

$r^\pi$8

The improvement step implies

$r^\pi$9

and, through the comparison lemmas, yields the monotonicity property

$P^\pi$0

Thus $P^\pi$1 is pointwise nonincreasing and bounded below by $P^\pi$2, so it converges pointwise to some $P^\pi$3. At the limit,

$P^\pi$4

hence $P^\pi$5 solves the Bellman equation and, by uniqueness, $P^\pi$6 on $P^\pi$7. The paper further shows that all intermediate policies remain admissible.

A central comparative statement is that policy iteration is not slower than value iteration when both are initialized with the same admissible value function $P^\pi$8. For all $P^\pi$9 and all $\{S,A,P,r,\gamma\}$00,

$\{S,A,P,r,\gamma\}$01

The same framework is extended to multi-step look-ahead policy iteration, with improvement

$\{S,A,P,r,\gamma\}$02

and monotone convergence to $\{S,A,P,r,\gamma\}$03 is preserved. The limiting case $\{S,A,P,r,\gamma\}$04 selects the optimal policy in one improvement step. This suggests that, in the undiscounted deterministic setting, the frozen-evaluation viewpoint is compatible with strong monotonicity and stabilization guarantees even though contraction is unavailable.

4. Partial evaluation, approximation, and conservative improvement

Modified policy iteration (MPI) makes the degree of freezing explicit through an integer parameter $\{S,A,P,r,\gamma\}$05. Starting from $\{S,A,P,r,\gamma\}$06, it computes a greedy policy $\{S,A,P,r,\gamma\}$07 and then performs a frozen partial evaluation

$\{S,A,P,r,\gamma\}$08

The extreme cases recover value iteration at $\{S,A,P,r,\gamma\}$09 and policy iteration at $\{S,A,P,r,\gamma\}$10. In that sense, frozen policy iteration can be viewed as the general MPI pattern in which the policy remains fixed during $\{S,A,P,r,\gamma\}$11 Bellman updates (Scherrer et al., 2012).

The approximate MPI framework abstracts approximation through an evaluation error $\{S,A,P,r,\gamma\}$12 and a greedy or classification error $\{S,A,P,r,\gamma\}$13: $\{S,A,P,r,\gamma\}$14 Three implementations are analyzed: AMPI-V, AMPI-Q, and classification-based MPI (CBMPI). The paper’s key structural result is that for $\{S,A,P,r,\gamma\}$15, the operator $\{S,A,P,r,\gamma\}$16 is not a contraction in any norm and is not monotone. The analysis therefore proceeds through Bellman residual, distance-to-optimality, and shift-to-policy-value sequences. In CBMPI, the main $\{S,A,P,r,\gamma\}$17 bound shows that the evaluation-error term is multiplied by $\{S,A,P,r,\gamma\}$18, revealing an explicit trade-off: larger $\{S,A,P,r,\gamma\}$19 attenuates evaluation error but reduces how often policies are improved and, under fixed sample budget, reduces the data available to the classifier. The finite-sample CBMPI result summarizes this as

$\{S,A,P,r,\gamma\}$20

up to constants and log factors (Scherrer et al., 2012).

A different but related notion of freezing appears in Easy Monotonic Policy Iteration. There the emphasis is not frozen evaluation of a fixed Bellman operator, but conservative policy improvement controlled by an average-divergence penalty. The central lower bound is

$\{S,A,P,r,\gamma\}$21

which replaces a sup-norm divergence penalty by an average total-variation term. With $\{S,A,P,r,\gamma\}$22, this yields

$\{S,A,P,r,\gamma\}$23

The resulting update maximizes a certified lower bound on improvement, and because $\{S,A,P,r,\gamma\}$24 is always feasible with objective value $\{S,A,P,r,\gamma\}$25, the generated returns are guaranteed to be non-decreasing under exact expectations and a sufficiently conservative penalty. In the vocabulary of frozen policy iteration, this is best read as a conservative-update analogue: large deviations from the current policy are “frozen” by a trust-region-style penalty rather than by extended evaluation of a fixed policy (Achiam, 2016).

5. Modern reinterpretations: online RL and frozen-base-model alignment

In online RL under linear $\{S,A,P,r,\gamma\}$26 realizability, Frozen Policy Iteration becomes a specific computationally efficient algorithm for episodic deterministic MDPs with stochastic initial states and stochastic rewards. The setting is a horizon-$\{S,A,P,r,\gamma\}$27 MDP $\{S,A,P,r,\gamma\}$28 with deterministic transition maps $\{S,A,P,r,\gamma\}$29, rewards in $\{S,A,P,r,\gamma\}$30, and approximate linear realizability

$\{S,A,P,r,\gamma\}$31

with $\{S,A,P,r,\gamma\}$32 and $\{S,A,P,r,\gamma\}$33. The decisive mechanism is statewise freezing: once a state is well-explored, the algorithm freezes the policy there and computes $\{S,A,P,r,\gamma\}$34 using only the first $\{S,A,P,r,\gamma\}$35 samples in the stage-$\{S,A,P,r,\gamma\}$36 dataset. Coverage is defined through the self-normalized norm

$\{S,A,P,r,\gamma\}$37

where

$\{S,A,P,r,\gamma\}$38

Only the last non-covered step of each trajectory is added to the data. Determinism ensures that, after freezing, all returns used in regression remain effectively on-policy. The main guarantee is

$\{S,A,P,r,\gamma\}$39

and in the exact case $\{S,A,P,r,\gamma\}$40,

$\{S,A,P,r,\gamma\}$41

For $\{S,A,P,r,\gamma\}$42, this reduces to the optimal linear-bandit order $\{S,A,P,r,\gamma\}$43. The same framework yields a Uniform-PAC statement and extends to function classes with bounded eluder dimension (Ke et al., 28 Feb 2026).

A distinct reinterpretation arises in alignment of frozen LLMs through Iterative Reweight-then-Optimize (IRO). Here the base model $\{S,A,P,r,\gamma\}$44 remains frozen in its parameters, and policy iteration is carried out in the space of decoding distributions. Generation is modeled as a token-level MDP with state $\{S,A,P,r,\gamma\}$45, action $\{S,A,P,r,\gamma\}$46, deterministic transition $\{S,A,P,r,\gamma\}$47, and terminal reward $\{S,A,P,r,\gamma\}$48. At iteration $\{S,A,P,r,\gamma\}$49, IRO alternates: sampling trajectories using the current guided policy, fitting a lightweight value function by Monte Carlo regression,

$\{S,A,P,r,\gamma\}$50

and constructing an improved implicit decoding policy

$\{S,A,P,r,\gamma\}$51

The update is derived from a TRPO-style KL-constrained policy-improvement step,

$\{S,A,P,r,\gamma\}$52

At test time, the learned values guide chunked beam-search-like decoding, and the final candidate is selected by the external reward model. The paper states that IRO is policy iteration “over the sampling distribution,” not policy-gradient training of the base model. Under bounded function class and concentrability assumptions, Theorem 1 gives convergence toward an optimal policy at $\{S,A,P,r,\gamma\}$53 with adaptive $\{S,A,P,r,\gamma\}$54. Empirically, on AlpacaEval 2.0, Meta-Llama-3-8B-Instruct moves from a base length-controlled win rate of $\{S,A,P,r,\gamma\}$55 to $\{S,A,P,r,\gamma\}$56 by iteration 3, while Meta-Llama-3-70B-Instruct moves from $\{S,A,P,r,\gamma\}$57 to $\{S,A,P,r,\gamma\}$58 (Zhang et al., 21 Jun 2025).

Taken together, these two works show that modern FPI-style algorithms need not freeze an entire policy uniformly. One may freeze it only on well-explored states, or freeze the underlying model weights while iterating over auxiliary value-guided policies.

6. Frozen branch selection beyond reinforcement learning: max-policy iteration and fixpoint computation

In static analysis and invariant generation, frozen policy iteration no longer refers to an MDP policy over actions, but to branch selections inside monotone equation systems. The problem class is a system

$\{S,A,P,r,\gamma\}$59

with $\{S,A,P,r,\gamma\}$60 monotone and each component built from maxima and minima of affine forms, such as

$\{S,A,P,r,\gamma\}$61

or

$\{S,A,P,r,\gamma\}$62

A policy selects one branch for each operator occurrence. Freezing that policy yields an induced affine system $\{S,A,P,r,\gamma\}$63, and evaluation is performed by plain value iteration

$\{S,A,P,r,\gamma\}$64

starting from an appropriate lower bound (Monniaux et al., 9 Jun 2026).

The decisive observation is that mathematical optimization in max-policy iteration can be replaced by value iteration while preserving termination. If $\{S,A,P,r,\gamma\}$65 is a finite-height poset and $\{S,A,P,r,\gamma\}$66 is monotone, then the value-iteration sequence is monotone and stabilizes in finitely many steps at the least fixpoint of $\{S,A,P,r,\gamma\}$67 above the start point. This applies directly to IEEE 754 floating-point domains, since the domain is finite, and to bounded integer systems. The overall frozen max-policy or max-min policy iteration terminates after finitely many improvements because each improvement step either strictly increases the current value componentwise or leaves the policy unchanged.

For bounded systems, the algorithm returns the least fixpoint $\{S,A,P,r,\gamma\}$68. At termination there exists a policy $\{S,A,P,r,\gamma\}$69 such that

$\{S,A,P,r,\gamma\}$70

and $\{S,A,P,r,\gamma\}$71 is the least fixpoint of both $\{S,A,P,r,\gamma\}$72 and $\{S,A,P,r,\gamma\}$73. The framework also handles unbounded systems by propagating $\{S,A,P,r,\gamma\}$74 and detecting divergence. A further contribution is the construction of soundness and optimality certificates: a vector $\{S,A,P,r,\gamma\}$75 together with a witness policy showing that the selected branch is tight at each component and that $\{S,A,P,r,\gamma\}$76 or $\{S,A,P,r,\gamma\}$77, depending on the certificate type. In abstract-interpretation terms, this provides precise bound analysis for integer or floating-point variables while avoiding widening operators altogether.

This non-RL usage is structurally faithful to the broader FPI idea. A policy is frozen, an induced monotone system is evaluated exactly by iterative refinement, and a subsequent improvement step changes only the branch choices that are suboptimal at the evaluated point. The result is a transfer of policy-iteration logic from dynamic programming to program analysis, with monotonicity and finite-height order replacing Bellman contraction as the underlying convergence mechanism (Monniaux et al., 9 Jun 2026).