---
title: Two-Phase Bootstrap Policy Iteration (B-PI)
url: https://www.emergentmind.com/topics/two-phase-bootstrap-policy-iteration-b-pi
type: topic
---

# Two-Phase Bootstrap Policy Iteration (B-PI)

Searching arXiv for the cited B-PI-related papers to ground the article in current records.
arxiv_search(query="2508.20394", max_results=5, sort_by="relevance")
Two-Phase Bootstrap Policy Iteration (B-PI) denotes a policy-iteration architecture in which an initial policy that is trivial, non-stabilizing, or otherwise inadequate is first used to construct a usable improved policy class, after which ordinary policy-iteration refinement proceeds. In the arXiv literature represented here, the expression is used explicitly for continuous-time stochastic linear-quadratic tracking with multiplicative noise, where B-PI “bootstraps a stabilizing control gain from the trivially initialized zero-value start and proceeds with standard policy iteration” [2508.20394]. Closely related formulations appear in trajectory-driven online policy iteration, discounted deterministic policy iteration from non-stabilizing initializers, feasibility-preserving nonlinear PI, bounded-domain PI with invariant-region updates, and bootstrap-randomized adaptive LQ control, although those works do not always use the term “B-PI” itself [2604.15004], [2403.19007], [2210.14459], [2507.09994], [1903.05803].

## 1. Canonical meaning and terminological scope

In its explicit formulation, B-PI is a two-stage algorithm for continuous-time stochastic LQ tracking with multiplicative noise. The plant is
\[
\Sigma:\left\{
\begin{aligned}
\mathrm{d}x &= \left(Ax+Bu\right)\mathrm{d}t+(Cx+Du)\mathrm{d}w,\\
y &= Hx,\qquad x(0)=x_0\in\mathbb{R}^n,
\end{aligned}\right.
\]
and the reference is generated by
\[
\Sigma_d:\left\{
\begin{aligned}
\dot{x}_d &= A_d x_d,\qquad x_d(0)=x_{d0}\in\mathbb{R}^{n_d},\\
y_d &= H_d x_d.
\end{aligned}\right.
\]
The admissible controller is
\[
u(t)=-Kx(t)-F x_d(t),
\]
with feedback gain \(K\) and feedforward gain \(F\) [2508.20394].

The explicit “two-phase” designation refers to a separation between a bootstrap or stabilization-first stage and a subsequent standard policy-iteration stage. In the named B-PI paper, Phase I constructs a stabilizing gain from the trivial initialization \(K_0=0\), while Phase II applies ordinary stochastic PI to converge to the optimal feedback gain \(K^*\) [2508.20394]. This suggests a general editorial characterization of B-PI as a “bootstrap-then-refine” policy-iteration scheme.

The terminology is not uniform across the literature. Several closely related papers describe the same structural pattern without naming it B-PI. One line of work starts from a current policy, generates an improved trajectory, and then generates the next policy from that trajectory [2604.15004]. Another allows a non-stabilizing initial policy with finite discounted cost and proves that PI enters the stabilizing class after a finite and known number of iterations [2403.19007]. A further line assumes a stabilizing bootstrap policy and modifies PI into PI+ so that recursive feasibility is guaranteed a priori [2210.14459]. Accordingly, B-PI is best understood as a named algorithm in one paper and as a broader interpretive template in several others.

## 2. Optimal-control formulation underlying the explicit B-PI algorithm

The explicit B-PI paper studies optimal tracking for stochastic linear systems with multiplicative noise under the infinite-horizon average cost
\[
P_d(u)=\lim_{T\rightarrow\infty}\frac{1}{T}J_T(u),
\]
where
\[
J_T = \frac{1}{2}\mathbb{E}\left[\int_0^T \left(\|y-y_d\|_Q^2+\|u\|_R^2\right)\mathrm{d}t\right],
\]
with \(Q=Q^\top>0\) and \(R=R^\top>0\) [2508.20394].

The optimal controller has the structure
\[
u^*(t)=-K^*x^*(t)-F^*x_d(t),
\]
with
\[
K^*=(R+D^\top P^*D)^{-1}(B^\top P^*+D^\top P^*C),
\]
and
\[
F^*=(R+D^\top P^*D)^{-1}B^\top \Pi^*.
\]
The feedback matrix \(P^*\) solves the stochastic algebraic Riccati equation
\[
\begin{aligned}
&A^{\top}P^*+P^*A+Q-C^{\top}P^*C \\
&\quad -(P^*B+C^{\top}P^*D) (R+D^{\top}P^*D)^{-1} (B^{\top}P^*+D^{\top}P^*C)=0,
\end{aligned}
\]
while \(\Pi^*\) is obtained from the Sylvester equation
\[
\Pi^*A_d+(A-BK^*)^\top \Pi^*-H^\top QH_d=0.
\]
This decomposition separates the feedback-regulation problem from the feedforward-tracking problem [2508.20394].

The separation is central to the algorithm. B-PI is constructed for the feedback component \(K^*\), not for the feedforward term directly. The paper emphasizes that this avoids the usual state augmentation approach, which would solve one larger Riccati equation of size \((n+n_d)\times(n+n_d)\) [2508.20394]. A plausible implication is that the “bootstrap” burden is concentrated on producing a stabilizing regulation policy; only after that step does tracking feedforward become a linear recovery problem.

## 3. Two-phase algorithmic structure

Phase I introduces a parameterized auxiliary system
\[
\mathcal{S}(\alpha):=(A(\alpha),B\mid C,D), \qquad A(\alpha)=A-\frac{1}{2}(\gamma-\alpha)I_n,
\]
together with the stabilizing gain set
\[
\mathcal{Z}(\alpha):= \{K\in\mathbb{R}^{m\times n}\mid \sigma(\mathcal{L}_{[K;\mathcal{S}(\alpha)]})\subset\mathbb{C}^-\},
\]
where
\[
\mathcal{L}_{[K;\mathcal{S}]}(X) = (A-BK)^\top X+X(A-BK)+(C-DK)^\top X(C-DK).
\]
If \(\gamma>\alpha_0>0\) is chosen such that
\[
\gamma>\sigma(\mathcal{L}_{[0;\mathcal{S}]})+\alpha_0,
\]
then \(0\in \mathcal{Z}(\alpha_0)\), so the zero gain stabilizes the parameterized system [2508.20394].

Starting from \(K_0=0\), the bootstrap iteration solves
\[
\mathcal{L}_{[K_{i-1};\mathcal{S}(\alpha_{i-1})]}(P_i) +\|K_{i-1}\|_R^2+\Theta=0,
\]
updates the gain by
\[
K_i=(R+D^\top P_i D)^{-1}(B^\top P_i+D^\top P_i C),
\]
and updates the parameter via
\[
\alpha_i = \alpha_{i-1} +\eta \frac{\lambda_{\min}(\|K_i\|_R^2+\Theta)} {\lambda_{\max}(P_i)},
\qquad \eta\in(0,1).
\]
Phase I stops at
\[
\mathbb{I}:=\min\{i\in\mathbb{N}\mid \alpha_i\ge \gamma\}.
\]
At that point, \(K_{\mathbb{I}}\) stabilizes the original system [2508.20394].

Phase II then switches to standard policy iteration on the original system:
\[
\mathcal{L}_{[K_{i-1};\mathcal{S}]}(P_i) +\|K_{i-1}\|_R^2+\|H\|_Q^2=0,
\]
followed by
\[
K_i=(R+D^\top P_iD)^{-1}(B^\top P_i+D^\top P_iC),
\]
and termination when
\[
\|K_i-K_{i-1}\|_2\le \varepsilon.
\]
The role of the first phase is therefore not direct optimization of the original tracking criterion, but construction of a stabilizing entry point from which ordinary PI becomes valid [2508.20394].

A useful contrast is supplied by trajectory-driven online PI. There, the current policy \(\pi^\ell\) produces an improved trajectory through one-step lookahead minimizations against \(J_{k+1,\pi^\ell}\), and the next policy is generated from that improved trajectory:
\[
\pi^{\ell+1} = \mathcal{G}(x_0,u_0^{\ell+1},x_1^{\ell+1},\dots,u_{N-1}^{\ell+1},x_N^{\ell+1}).
\]
This is not the same algorithmic mechanism as the stochastic tracking B-PI, but it exhibits the same two-part organization: one phase generates improved decision data, and the next phase fits or generates the next policy [2604.15004].

## 4. Guarantees, data-driven realization, and empirical instantiations

Under mean-square stabilizability and exact detectability, the model-based B-PI theorem states that if \(\gamma>\alpha_0>0\) and
\[
\gamma>\sigma(\mathcal{L}_{[0;\mathcal{S}]})+\alpha_0,
\]
then the bootstrap index is finite,
\[
\mathbb{I}<\infty,
\]
with \(K_{\mathbb{I}}\in\mathcal{Z}(\gamma)\), and the refinement phase satisfies
\[
\lim_{i\to\infty}P_i=P^*,\qquad \lim_{i\to\infty}K_i=K^*.
\]
The sequence therefore first reaches the stabilizing class in finitely many steps and then converges to the optimal feedback gain [2508.20394].

The same paper develops an off-policy data-driven realization. With
\[
\tilde{\alpha}:=\frac12(\gamma-\alpha_0), \qquad
\chi(t)=e^{-\tilde{\alpha}t}x(t),\qquad
\zeta(t)=e^{-\tilde{\alpha}t}y(t),
\]
the transformed system corresponds to the parameterized system \(\mathcal{S}(\alpha_0)\). The data-driven method requires the interval excitation condition
\[
\mathrm{rank}(\mathcal{I}) = \frac12 n(n+1)+nm+\frac12 m(m+1), \qquad \forall l\ge l_1.
\]
Policy evaluation becomes a linear-algebraic recovery of
\[
\theta_i= \begin{bmatrix} \mathrm{vech}(P_i)^\top & \mathrm{vec}(M_i)^\top & \mathrm{vech}(\Lambda_i)^\top \end{bmatrix}^\top,
\]
using
\[
\theta_i = -\Psi_{i-1}^{\dagger}\mathcal{I}_{\chi}^{\top} \mathrm{vech}(\|K_{i-1}\|_R^2+\Theta)
\]
in Phase I and
\[
\theta_i = -\Phi_{i-1}^{\dagger} \left( \mathcal{I}_{\chi}^{\top}\mathrm{vech}(\|K_{i-1}\|_R^2) +\mathcal{I}_{\zeta}^{\top}\mathrm{vech}(Q) \right)
\]
in Phase II, followed by
\[
K_i=(R+\Lambda_i)^{-1}M_i.
\]
After feedback recovery, the feedforward parameters are computed from
\[
\vartheta^*=\Xi^\dagger \mathcal{I}_{y_d\bar{\zeta}}^\top \mathrm{vec}(Q).
\]
For \(D=0\), shadow systems are introduced so that probing noise on the physical plant can be avoided [2508.20394].

The numerical examples instantiate the two-phase pattern concretely. In the spring-mass-damper example with
\[
\gamma=1,\quad \alpha_0=0.1,\quad \eta=0.95,\quad \Theta=\mathrm{diag}([10,10]),
\]
the threshold \(\hat \alpha_i\ge \gamma\) is first reached at iteration \(i=2\), producing
\[
\hat K_2=[24.8937,\;42.5251],
\]
and the algorithm converges at iteration \(i=9\) with
\[
\hat K_9=[25.8381,\;7.5651].
\]
In the shadow-system example with \(D=0\) and \(u(t)=0\), the algorithm reaches \(\alpha_i\ge \gamma\) at \(i=4\), producing
\[
\hat K_4=[5.2039,4.4951,-1.8831,3.3963],
\]
and converges by iteration \(12\) [2508.20394].

## 5. Related formulations in policy iteration and reinforcement learning

Several arXiv papers fit the same structural idea even when they do not use the B-PI label.

| Paper | Relation to B-PI |
|---|---|
| "On-Line Policy Iteration with Trajectory-Driven Policy Generation" [2604.15004] | Improved trajectory first, policy generation second |
| "Policy iteration for discrete-time systems with discounted costs: stability and near-optimality guarantees" [2403.19007] | Recovery from non-stabilizing initializer, then stabilizing refinement |
| "Policy iteration: for want of recursive feasibility, all is not lost" [2210.14459] | Bootstrap stabilizer assumed; PI+ preserves feasibility and stability |
| "On the Convergence of the Policy Iteration for Infinite-Horizon Nonlinear Optimal Control Problems" [2507.09994] | Admissible bootstrap controller plus invariant-domain PI refinement |
| "On Applications of Bootstrap in Continuous Space Reinforcement Learning" [1903.05803] | Bootstrap-randomized model estimation followed by Riccati policy recomputation |
| "Sample-Efficient Model-Free Reinforcement Learning with Off-Policy Critics" [1903.04193] | Bootstrapped critics with conservative actor update; related but differently named |

The trajectory-driven formulation is the closest structural analogue on the finite-horizon side. Given a policy \(\pi^\ell\), it improves controls stagewise along a single trajectory by solving
\[
u_k^{\ell+1}\in \arg\min_{u_k\in U_k(x_k^{\ell+1})}
\Big[g_k(x_k^{\ell+1},u_k)+J_{k+1,\pi^\ell}(f_k(x_k^{\ell+1},u_k))\Big],
\]
then generates the next policy from the improved trajectory. Under a consistency condition requiring the generated policy to reproduce the trajectory controls on trajectory states,
\[
\mu_k(x_k)=u_k,\qquad k=0,\dots,N-1,
\]
the costs satisfy
\[
J_{\pi^{\ell+1}}(x_0)\le J_{\pi^\ell}(x_0).
\]
This suggests a trajectory-bootstrap interpretation of B-PI in which one phase improves local decision data and the next phase trains a policy representation that preserves those improvements [2604.15004].

The discounted deterministic PI paper gives a different two-phase reading. It assumes only that the initial policy \(h^0\) has finite discounted cost,
\[
V^0_\gamma(x)=J_\gamma(x,h^0)\le \overline\alpha_V(\sigma(x),\gamma),
\]
without requiring \(h^0\) to be stabilizing, and proves monotone improvement
\[
V^{i+1}_\gamma(x)\le V^i_\gamma(x),
\]
a near-optimality bound
\[
\big(V^{i}_{\gamma}-V^{\star}_{\gamma}\big)(x)
\leq \gamma^i\overline{\alpha}_{V}\big(\beta^{\star}_{\gamma}(\sigma(x),i),\gamma\big),
\]
and semiglobal practical stability after finitely many iterations once \(i\ge i^\star_\gamma\). This is an explicit recovery-phase/refinement-phase decomposition, even though it is not called B-PI [2403.19007].

The nonlinear undiscounted PI paper sharpens a different point: standard PI can fail to be recursively feasible, so a bootstrap policy alone is not enough. Its PI+ modification regularizes the improvement map and evaluates the best-cost regularized selection. Under stated continuity, level-boundedness, and outer-semicontinuity assumptions, \(H^i(x)\), \(H_r^i(x)\), and \(H_r^{\star,i}(x)\) remain non-empty for all \(i,x\), while recursive \(\mathcal{K}\mathcal{L}\)-stability and explicit near-optimality bounds are preserved [2210.14459]. This suggests that, in some settings, the second phase of a B-PI design must be feasibility-preserving rather than merely greedy.

The bounded-domain nonlinear PI paper contributes another closely related ingredient: a bootstrap policy must come with an admissible forward-invariant domain. For an admissible controller-domain pair \((u,\Omega)\), the value function \(v_u\) solves the GHJB equation
\[
\langle f(x)+g(x)u(x),\nabla v_u(x)\rangle +h(x)+u(x)^\top R\,u(x)=0,\qquad v_u(0)=0,
\]
and the improved policy
\[
u^+(x)=-\frac12 R^{-1}g(x)^\top \nabla v_u(x)
\]
is admissible on suitable proper sublevel subsets of \(v_u\). The resulting domain-aware PI sequence yields
\[
v^*(x)\le v_{s+1}(x)\le v_s(x),
\]
together with convergence on compact subsets under uniform derivative bounds [2507.09994]. In that setting, a plausible B-PI interpretation is “bootstrap controller plus safe-region construction, then monotone PI refinement.”

## 6. Conceptual boundaries, misconceptions, and structural significance

A first misconception is that B-PI names a single universally standardized method. The available record is narrower. The term “two-phase bootstrap policy iteration (B-PI)” is explicit in the stochastic LQ tracking paper [2508.20394], whereas several other works are best described as conceptual analogues rather than instances of a canonical shared algorithm [2604.15004], [2403.19007].

A second misconception is that “bootstrap” here necessarily refers to temporal-difference bootstrapping. In the cited literature, bootstrap has several meanings. It may denote bootstrapping a stabilizing gain from \(K_0=0\) by passing through an auxiliary parameterized system [2508.20394]; bootstrapping the next policy from an improved trajectory generated by the current policy [2604.15004]; bootstrapping from a non-stabilizing but finite-cost policy until PI enters the stabilizing class [2403.19007]; or bootstrap-randomized model estimation followed by Riccati recomputation in adaptive LQ control [1903.05803]. This suggests that B-PI is better viewed as a structural pattern than as a single estimator design.

A third misconception is that monotonic guarantees are automatically global. They are not. The trajectory-driven online PI theorem is only for the fixed initial state \(x_0\), not all states [2604.15004]. The discounted recovery result depends on detectability and optimal closed-loop stability assumptions [2403.19007]. The nonlinear PI+ results require recursive-feasibility machinery [2210.14459]. The bounded-domain convergence theory depends on admissibility, forward invariance, and sufficient GHJB regularity [2507.09994]. The explicit B-PI tracking guarantees rely on mean-square stabilizability, exact detectability, parameter choices \(\gamma,\alpha_0,\eta,\Theta\), and, in the data-driven case, interval excitation [2508.20394].

The structural significance of B-PI lies in this repeated decomposition. One stage repairs an initialization defect—lack of stability, lack of feasibility, lack of local training targets, or lack of informative model uncertainty. The next stage exploits ordinary PI-style monotone improvement, rollout-based enhancement, or conservative policy updating once the bootstrap stage has produced a viable operating point. Within the present literature, that is the most stable encyclopedia-level definition of Two-Phase Bootstrap Policy Iteration.

Source: https://www.emergentmind.com/topics/two-phase-bootstrap-policy-iteration-b-pi