--- title: Frozen Policy Iteration in RL & Control url: https://www.emergentmind.com/topics/frozen-policy-iteration type: topic --- # Frozen Policy Iteration in RL & Control to=arxiv_search.search ,最新高清无码专区 万亚json {"query":"\"Frozen Policy Iteration\" arXiv", "max_results": 10} to=arxiv_search.search оШjson {"query":"\"Inexact Policy Iteration Methods for Large-Scale Markov Decision Processes\" 2404.06136", "max_results": 5} to=arxiv_search.search 大发快三豹子json {"query":"\"Convergence Analysis of Policy Iteration\" 1505.05216 OR \"Approximate Modified Policy Iteration\" 1205.3054 OR \"Easy Monotonic Policy Iteration\" 1602.09118 OR \"Frozen Policy Iteration: Computationally Efficient RL under Linear Q^π Realizability for Deterministic Dynamics\" 2603.00716", "max_results": 10} Frozen Policy Iteration (FPI) denotes a class of schemes in which a current policy is held fixed during a policy-evaluation phase and changed only in a subsequent improvement phase. In the cited literature, that template appears in several technically distinct forms: exact or inexact policy evaluation for discounted finite Markov decision processes (MDPs), admissibility-preserving policy iteration for deterministic undiscounted nonlinear control, partial evaluation in modified policy iteration, online reinforcement learning with statewise freezing under linear \(Q^\pi\) realizability, test-time alignment of frozen language models through value-guided reweighting, and branch-freezing procedures for monotone max-min equation systems in static analysis [2404.06136] [1505.05216] [1205.3054] [2603.00716] [2506.17828] [2606.10717]. ## 1. Core scheme and formal structure In discounted finite MDPs, the canonical setup is a 5-tuple \(\{S,A,P,r,\gamma\}\) with finite state set \(S=\{1,\ldots,n\}\), finite action set \(A=\{1,\ldots,m\}\), transition matrices \(P^a\in\mathbb{R}^{n\times n}\), reward vectors \(r^a\in\mathbb{R}^n\), and discount \(\gamma\in(0,1)\). A deterministic stationary policy \(\pi:S\to A\) induces \(r^\pi\) and \(P^\pi\), and its Bellman operator is \[ T^\pi(v)=r^\pi+\gamma P^\pi v, \qquad T(v)=\max_{a\in A}\{r^a+\gamma P^a v\}. \] Exact policy evaluation solves \[ (I-\gamma P^\pi)v^\pi=r^\pi, \] while policy improvement selects a greedy policy \[ \pi'(s)\in\arg\max_{a\in A}\big[r^a(s)+\gamma(P^a v)(s)\big]. \] In this setting, freezing means that \(\pi\) is fixed while its value is computed, whether exactly or inexactly [2404.06136]. In deterministic undiscounted nonlinear control, the same separation appears with different objects. For dynamics \[ x_{k+1}=f(x_k,u_k), \qquad f(0,0)=0, \] stage cost \(U(x,u)\), and infinite-horizon performance index \[ J=\sum_{k=0}^{\infty} U(x_k,u_k), \] a stationary feedback policy \(h\) has value \[ V_h(x_0)=\sum_{k=0}^{\infty} U(x_k^h,h(x_k^h)). \] Frozen evaluation solves \[ V_h(x)=U(x,h(x))+V_h(f(x,h(x))), \] and improvement chooses \[ h^{+}(x)\in\arg\min_{u\in U}\{U(x,u)+V_h(f(x,u))\}. \] Here the frozen object is again the current policy, but the analysis does not rely on discounting or contraction [1505.05216]. | Setting | Frozen object | Evaluation mechanism | |---|---|---| | Discounted finite MDPs | Current greedy policy | Exact linear solve or inexact inner solver | | Deterministic undiscounted control | Current stabilizing feedback law | Fixed-point solution of \(V=T^hV\) | | Modified / approximate PI | Current policy for \(m\) steps | Partial evaluation \((T^\pi)^m\) | | Online RL under linear \(Q^\pi\) realizability | Policy at well-explored states | Least squares on on-policy trajectory suffixes | | Frozen-base-model alignment | Base model weights | Value-guided decoding distribution | | Max-policy iteration | Branch selections in max/min operators | Value iteration on \(F_\pi\) | This common structure does not imply a single convergence mechanism. In some works the key tool is Bellman contraction, in others semismooth Newton theory, comparison lemmas, self-normalized confidence bounds, or finite-height order structure. A plausible implication is that “frozen policy iteration” is best understood as a design pattern rather than a single algorithmic identity. ## 2. Discounted finite MDPs: inexact frozen evaluation and semismooth Newton structure A particularly explicit formalization appears in the inexact policy iteration framework for large-scale discounted finite MDPs. There, FPI is interpreted as policy iteration in which the greedy policy is frozen during evaluation, but the policy-evaluation linear system is solved only approximately. Given \(v_k\), one sets \[ \pi_{k+1}\leftarrow \mathrm{GreedyPolicy}(v_k), \] then approximately solves \[ (I-\gamma P^{\pi_{k+1}})v \approx r^{\pi_{k+1}}, \] and finally sets \(v_{k+1}\) to the last inner iterate [2404.06136]. The semismooth Newton interpretation is based on the Bellman residual \[ F(v)=T(v)-v, \qquad r(v)=v-T(v). \] For any \(v\), \(\mathrm{GreedyPolicy}(v)\) returns argmax selections of \(T^\pi(v)\), and the generalized Jacobian of \(r\) is \[ \partial r(v)=\mathrm{Conv}\{I-\gamma P^\pi:\pi\in \mathrm{GreedyPolicy}(v)\}. \] Each \(J\in\partial r(v)\) is nonsingular. At the optimal value \(v^*\), the residual is globally Lipschitz and strongly semismooth, with \[ \| r(v)-r(v^*)-(I-\gamma P^\pi)(v-v^*) \|_\infty \le C\|v-v^*\|_\infty^2, \] for \(\pi\in \mathrm{GreedyPolicy}(v)\) and any \(C\ge C_{\min}:=\frac{2R}{(1-\gamma)\delta^2}\). The inexact evaluation step uses the residual condition \[ \| r(v_k)+J_k(v_{k+1}-v_k)\|_\infty \le \alpha_k \|r(v_k)\|_\infty, \qquad \alpha_k\in[0,1), \] with \(J_k=I-\gamma P^{\pi_{k+1}}\). In reward form this becomes \[ \| r^{\pi_{k+1}}-(I-\gamma P^{\pi_{k+1}})v_{k+1}\|_\infty \le \alpha_k \| r^{\pi_{k+1}}-(I-\gamma P^{\pi_{k+1}})v_k\|_\infty. \] The practical local-convergence range identified in the paper is \[ \alpha\in\left(0,\frac{1-\gamma}{1+\gamma}\right). \] Within that range, local convergence satisfies \[ \| v_{k+1}-v^*\|_\infty \le \frac{1+\gamma}{1-\gamma}\alpha \|v_k-v^*\|_\infty + \frac{C_{\min}}{1-\gamma}\|v_k-v^*\|_\infty^2. \] Hence the method converges Q-linearly with asymptotic rate \[ \beta_{\mathrm{loc}}=\frac{1+\gamma}{1-\gamma}\alpha, \] and becomes Q-superlinear if \(\alpha_k\to 0\). The same work also gives a global contraction regime: if \(\gamma\in(0,-1+\sqrt{2})\) and \(\alpha\in\left(0,\frac{1-2\gamma-\gamma^2}{1+\gamma}\right)\), then \[ \|v_{k+1}-v^*\|_\infty \le \beta_{\mathrm{glob}}\|v_k-v^*\|_\infty, \qquad \beta_{\mathrm{glob}}=((\alpha+\gamma)(1+\gamma)+\gamma)<1. \] For arbitrary \(\gamma\in(0,1)\), the scheme converges globally to a ball \(\mathcal{B}(v^*,\rho)\) with \[ \rho=\frac{2R}{(1-\gamma)(1-\zeta)}, \qquad \zeta=\frac{1+\gamma}{1-\gamma}\alpha. \] This formulation strictly generalizes optimistic or modified policy iteration. If the inner solver is value iteration under the frozen policy and is stopped after a fixed number \(w\) of inner steps, then \[ v_{k+1}=(T^{\pi_{k+1}})^w v_k, \] which is the optimistic/modified policy iteration form. The inexact framework replaces fixed \(w\) by a residual-based rule and allows alternative inner solvers, including Richardson, steepest descent, MinRes, and GMRES. The paper emphasizes that the choice of inner solver is fundamental. GMRES is particularly effective because policy-evaluation matrices have eigenvalues in the disk centered at \(1\) of radius \(\gamma\), and the stopping condition is guaranteed to be met after finitely many inner iterations whenever the inner solver contracts linearly in some norm. On epidemiological discounted MDPs for dynamic SIS control, the numerical study reports that with \(\alpha=0.1\), iGMRES-PI consistently outperforms exact PI in CPU time, with speedups of \(\approx 1.46\times\) at \(\gamma=0.1\), \(\approx 1.56\times\) at \(\gamma=0.9\), and up to \(\approx 3.7\times\) on the largest tested problem [2404.06136]. ## 3. Deterministic undiscounted control: admissibility, uniqueness, and monotone convergence In A. Heydari’s analysis of policy iteration for deterministic nonlinear systems, frozen policy iteration is studied on a compact, connected domain \(\Omega\subset\mathbb{R}^n\) containing the origin. The admissible-policy set \(H\) consists of policies that asymptotically stabilize the system in \(\Omega\) and whose value functions are finite and upper-bounded on \(\Omega\) by a continuous function \(\bar V\) with \(\bar V(0)=0\). The standing assumptions are the existence of at least one admissible policy and the condition that the intersection of \(\{x:U(x,0)=0\}\) with the invariant set of the autonomous dynamics \(x_{k+1}=f(x_k,0)\) is \(\{0\}\) [1505.05216]. The Bellman optimality equation is \[ V^*(x)=\min_{u\in U}\{U(x,u)+V^*(f(x,u))\}. \] No contraction argument is used. Instead, uniqueness of the Bellman solution on \(\Omega\) is derived from two comparison lemmas. The first states that if, for admissible \(h\) and \(g\), \[ U(x,h(x))+V_g(f(x,h(x))) \le V_g(x) \quad \forall x\in\Omega, \] then \(V_h(x)\le V_g(x)\) on \(\Omega\). The second states that if \(V_h(x)1\), the operator \(v\mapsto (T_{G(v)})^m v\) is not a contraction in any norm and is not monotone. The analysis therefore proceeds through Bellman residual, distance-to-optimality, and shift-to-policy-value sequences. In CBMPI, the main \(L_p\) bound shows that the evaluation-error term is multiplied by \(\gamma^m\), revealing an explicit trade-off: larger \(m\) attenuates evaluation error but reduces how often policies are improved and, under fixed sample budget, reduces the data available to the classifier. The finite-sample CBMPI result summarizes this as \[ \|l_k\|_{1,\mu} = O\!\left( \gamma^m\left(d_m+\sqrt{m/B}\right) + d' + \sqrt{M|A|m/B} \right), \] up to constants and log factors [1205.3054]. A different but related notion of freezing appears in Easy Monotonic Policy Iteration. There the emphasis is not frozen evaluation of a fixed Bellman operator, but conservative policy improvement controlled by an average-divergence penalty. The central lower bound is \[ J(\pi')-J(\pi) \ge \frac{1}{1-\gamma} \left( L_{\pi,f}(\pi') - \frac{2\gamma \epsilon_f^{\pi'}}{1-\gamma} \mathbb{E}_{s\sim d^\pi}[D_{\mathrm{TV}}(\pi'\|\pi)[s]] \right), \] which replaces a sup-norm divergence penalty by an average total-variation term. With \(f=V^\pi\), this yields \[ J(\pi')-J(\pi) \ge \frac{1}{1-\gamma} \mathbb{E}_{s\sim d^\pi,a\sim\pi'} \left[ A^\pi(s,a) - \frac{2\gamma\epsilon^{\pi'}}{1-\gamma}D_{\mathrm{TV}}(\pi'\|\pi)[s] \right]. \] The resulting update maximizes a certified lower bound on improvement, and because \(\pi\) is always feasible with objective value \(0\), the generated returns are guaranteed to be non-decreasing under exact expectations and a sufficiently conservative penalty. In the vocabulary of frozen policy iteration, this is best read as a conservative-update analogue: large deviations from the current policy are “frozen” by a trust-region-style penalty rather than by extended evaluation of a fixed policy [1602.09118]. ## 5. Modern reinterpretations: online RL and frozen-base-model alignment In online RL under linear \(Q^\pi\) realizability, Frozen Policy Iteration becomes a specific computationally efficient algorithm for episodic deterministic MDPs with stochastic initial states and stochastic rewards. The setting is a horizon-\(H\) MDP \((S,A,H,P,R,p)\) with deterministic transition maps \(f_h\), rewards in \([0,1]\), and approximate linear realizability \[ Q_h^\pi(s,a)=\phi(s,a)^\top w_h^\pi+\epsilon_h^\pi(s,a), \qquad |\epsilon_h^\pi(s,a)|\le \kappa, \] with \(\|\phi(s,a)\|_2\le 1\) and \(\|w_h^\pi\|_2\le \sqrt{H}\). The decisive mechanism is statewise freezing: once a state is well-explored, the algorithm freezes the policy there and computes \(\hat Q_h(s,\cdot)\) using only the first \(k_t(s)\) samples in the stage-\(h\) dataset. Coverage is defined through the self-normalized norm \[ \mathrm{Cover}(D_h,\epsilon)=\{(s,a):\|\phi(s,a)\|_{E_h^{-1}}\le \epsilon\}, \] where \[ E_{h,k}=\lambda I+\sum_{i=1}^k \phi(s_{h,i},a_{h,i})\phi(s_{h,i},a_{h,i})^\top, \qquad \hat Q_{h,k}(s,a)=\phi(s,a)^\top E_{h,k}^{-1}\sum_{i=1}^k \phi(s_{h,i},a_{h,i})q_{h,i}. \] Only the last non-covered step of each trajectory is added to the data. Determinism ensures that, after freezing, all returns used in regression remain effectively on-policy. The main guarantee is \[ R_T=\widetilde{O}\!\left(\sqrt{d^2H^6T}+\kappa\sqrt{dH^2T}\right), \] and in the exact case \(\kappa=0\), \[ R_T=\widetilde{O}\!\left(\sqrt{d^2H^6T}\right). \] For \(H=1\), this reduces to the optimal linear-bandit order \(\widetilde{O}(d\sqrt{T})\). The same framework yields a Uniform-PAC statement and extends to function classes with bounded eluder dimension [2603.00716]. A distinct reinterpretation arises in alignment of frozen large language models through Iterative Reweight-then-Optimize (IRO). Here the base model \(\pi_{\text{base}}\) remains frozen in its parameters, and policy iteration is carried out in the space of decoding distributions. Generation is modeled as a token-level MDP with state \(s_h=[x,y_{1:h-1}]\), action \(a_h=y_h\), deterministic transition \(s_{h+1}=[s_h,a_h]\), and terminal reward \(r(x,y)\). At iteration \(t\), IRO alternates: sampling trajectories using the current guided policy, fitting a lightweight value function by Monte Carlo regression, \[ \hat V^{\hat\pi_{t-1}} \in \arg\min_V \mathbb{E}_{\tau\sim\mathcal{D}_t} \left[ \sum_{h=1}^{H}(V(s_h)-r(\tau))^2 \right], \] and constructing an improved implicit decoding policy \[ \hat\pi_t(a_h|s_h) \propto \pi_{\text{base}}(a_h|s_h) \exp\!\left( \sum_{i=0}^{t-1}\frac{1}{\beta_i}\hat V^{\hat\pi_i}(s_{h+1}) \right). \] The update is derived from a TRPO-style KL-constrained policy-improvement step, \[ \pi_{\text{new}}(a_h|s_h) \propto \pi'(a_h|s_h)\exp\!\left(\frac{1}{\beta}Q^{\pi'}(s_h,a_h)\right) = \pi'(a_h|s_h)\exp\!\left(\frac{1}{\beta}V^{\pi'}(s_{h+1})\right). \] At test time, the learned values guide chunked beam-search-like decoding, and the final candidate is selected by the external reward model. The paper states that IRO is policy iteration “over the sampling distribution,” not policy-gradient training of the base model. Under bounded function class and concentrability assumptions, Theorem 1 gives convergence toward an optimal policy at \(\mathcal{O}(T^{-1/2})\) with adaptive \(\beta_t=\sqrt{t+1}/\omega\). Empirically, on AlpacaEval 2.0, Meta-Llama-3-8B-Instruct moves from a base length-controlled win rate of \(\sim 30.71\%\) to \(\sim 43\%\) by iteration 3, while Meta-Llama-3-70B-Instruct moves from \(\sim 43.11\%\) to \(\sim 49.77\%\) [2506.17828]. Taken together, these two works show that modern FPI-style algorithms need not freeze an entire policy uniformly. One may freeze it only on well-explored states, or freeze the underlying model weights while iterating over auxiliary value-guided policies. ## 6. Frozen branch selection beyond reinforcement learning: max-policy iteration and fixpoint computation In static analysis and invariant generation, frozen policy iteration no longer refers to an MDP policy over actions, but to branch selections inside monotone equation systems. The problem class is a system \[ X=F(X), \] with \(F:D^n\to D^n\) monotone and each component built from maxima and minima of affine forms, such as \[ X_i=\max_{p\in P_i} f_{i,p}(X), \qquad f_{i,p}(X)=a_{i,p}^\top X+b_{i,p}, \] or \[ X_i=\max_{j\in J_i}\min_{k\in K_{i,j}}(a_{i,j,k}^\top X+b_{i,j,k}). \] A policy selects one branch for each operator occurrence. Freezing that policy yields an induced affine system \(F_\pi\), and evaluation is performed by plain value iteration \[ X^{(k+1)}=F_\pi(X^{(k)}), \] starting from an appropriate lower bound [2606.10717]. The decisive observation is that mathematical optimization in max-policy iteration can be replaced by value iteration while preserving termination. If \(D\) is a finite-height poset and \(F_\pi\) is monotone, then the value-iteration sequence is monotone and stabilizes in finitely many steps at the least fixpoint of \(F_\pi\) above the start point. This applies directly to IEEE 754 floating-point domains, since the domain is finite, and to bounded integer systems. The overall frozen max-policy or max-min policy iteration terminates after finitely many improvements because each improvement step either strictly increases the current value componentwise or leaves the policy unchanged. For bounded systems, the algorithm returns the least fixpoint \(X^*=\mathrm{lfp}(F)\). At termination there exists a policy \(\pi^*\) such that \[ F(X^*)=F_{\pi^*}(X^*), \] and \(X^*\) is the least fixpoint of both \(F_{\pi^*}\) and \(F\). The framework also handles unbounded systems by propagating \(\pm\infty\) and detecting divergence. A further contribution is the construction of soundness and optimality certificates: a vector \(X\) together with a witness policy showing that the selected branch is tight at each component and that \(F(X)\le X\) or \(F(X)=X\), depending on the certificate type. In abstract-interpretation terms, this provides precise bound analysis for integer or floating-point variables while avoiding widening operators altogether. This non-RL usage is structurally faithful to the broader FPI idea. A policy is frozen, an induced monotone system is evaluated exactly by iterative refinement, and a subsequent improvement step changes only the branch choices that are suboptimal at the evaluated point. The result is a transfer of policy-iteration logic from dynamic programming to program analysis, with monotonicity and finite-height order replacing Bellman contraction as the underlying convergence mechanism [2606.10717]. Source: https://www.emergentmind.com/topics/frozen-policy-iteration