Two-Phase Bootstrap Policy Iteration (B-PI)

• The paper demonstrates that B-PI bootstraps a stabilizing gain from a trivial initializer (K₀ = 0) before applying standard policy iteration to converge to the optimal feedback gain.
• Two-Phase Bootstrap Policy Iteration (B-PI) is a structural pattern for continuous-time stochastic LQ tracking that decouples the stabilization phase from the performance refinement phase.
• The method is supported by rigorous guarantees and a data-driven realization that ensure mean-square stability and convergence in a finite number of iterations.

Searching arXiv for the cited B-PI-related papers to ground the article in current records. arxiv_search(query="(Chen et al., 28 Aug 2025)", max_results=5, sort_by="relevance") Two-Phase Bootstrap Policy Iteration (B-PI) denotes a policy-iteration architecture in which an initial policy that is trivial, non-stabilizing, or otherwise inadequate is first used to construct a usable improved policy class, after which ordinary policy-iteration refinement proceeds. In the arXiv literature represented here, the expression is used explicitly for continuous-time stochastic linear-quadratic tracking with multiplicative noise, where B-PI “bootstraps a stabilizing control gain from the trivially initialized zero-value start and proceeds with standard policy iteration” (Chen et al., 28 Aug 2025). Closely related formulations appear in trajectory-driven online policy iteration, discounted deterministic policy iteration from non-stabilizing initializers, feasibility-preserving nonlinear PI, bounded-domain PI with invariant-region updates, and bootstrap-randomized adaptive LQ control, although those works do not always use the term “B-PI” itself (Li et al., 16 Apr 2026, Brusse et al., 2024, Granzotto et al., 2022, Ehring et al., 14 Jul 2025, Faradonbeh et al., 2019).

1. Canonical meaning and terminological scope

In its explicit formulation, B-PI is a two-stage algorithm for continuous-time stochastic LQ tracking with multiplicative noise. The plant is

$$\Sigma:\left\{ \begin{aligned} \mathrm{d}x &= \left(Ax+Bu\right)\mathrm{d}t+(Cx+Du)\mathrm{d}w,\ y &= Hx,\qquad x(0)=x_0\in\mathbb{R}^n, \end{aligned}\right.$$

and the reference is generated by

$$\Sigma_d:\left\{ \begin{aligned} \dot{x}_d &= A_d x_d,\qquad x_d(0)=x_{d0}\in\mathbb{R}^{n_d},\ y_d &= H_d x_d. \end{aligned}\right.$$

The admissible controller is

$u(t)=-Kx(t)-F x_d(t),$

with feedback gain $K$ and feedforward gain $F$ (Chen et al., 28 Aug 2025).

The explicit “two-phase” designation refers to a separation between a bootstrap or stabilization-first stage and a subsequent standard policy-iteration stage. In the named B-PI paper, Phase I constructs a stabilizing gain from the trivial initialization $K_0=0$, while Phase II applies ordinary stochastic PI to converge to the optimal feedback gain $K^*$ (Chen et al., 28 Aug 2025). This suggests a general editorial characterization of B-PI as a “bootstrap-then-refine” policy-iteration scheme.

The terminology is not uniform across the literature. Several closely related papers describe the same structural pattern without naming it B-PI. One line of work starts from a current policy, generates an improved trajectory, and then generates the next policy from that trajectory (Li et al., 16 Apr 2026). Another allows a non-stabilizing initial policy with finite discounted cost and proves that PI enters the stabilizing class after a finite and known number of iterations (Brusse et al., 2024). A further line assumes a stabilizing bootstrap policy and modifies PI into PI+ so that recursive feasibility is guaranteed a priori (Granzotto et al., 2022). Accordingly, B-PI is best understood as a named algorithm in one paper and as a broader interpretive template in several others.

2. Optimal-control formulation underlying the explicit B-PI algorithm

The explicit B-PI paper studies optimal tracking for stochastic linear systems with multiplicative noise under the infinite-horizon average cost

$P_d(u)=\lim_{T\rightarrow\infty}\frac{1}{T}J_T(u),$

where

$$J_T = \frac{1}{2}\mathbb{E}\left[\int_0^T \left(\|y-y_d\|_Q^2+\|u\|_R^2\right)\mathrm{d}t\right],$$

with $Q=Q^\top>0$ and $$\Sigma_d:\left\{ \begin{aligned} \dot{x}_d &= A_d x_d,\qquad x_d(0)=x_{d0}\in\mathbb{R}^{n_d},\ y_d &= H_d x_d. \end{aligned}\right.$$0 (Chen et al., 28 Aug 2025).

The optimal controller has the structure

$$\Sigma_d:\left\{ \begin{aligned} \dot{x}_d &= A_d x_d,\qquad x_d(0)=x_{d0}\in\mathbb{R}^{n_d},\ y_d &= H_d x_d. \end{aligned}\right.$$1

with

$$\Sigma_d:\left\{ \begin{aligned} \dot{x}_d &= A_d x_d,\qquad x_d(0)=x_{d0}\in\mathbb{R}^{n_d},\ y_d &= H_d x_d. \end{aligned}\right.$$2

and

$$\Sigma_d:\left\{ \begin{aligned} \dot{x}_d &= A_d x_d,\qquad x_d(0)=x_{d0}\in\mathbb{R}^{n_d},\ y_d &= H_d x_d. \end{aligned}\right.$$3

The feedback matrix $$\Sigma_d:\left\{ \begin{aligned} \dot{x}_d &= A_d x_d,\qquad x_d(0)=x_{d0}\in\mathbb{R}^{n_d},\ y_d &= H_d x_d. \end{aligned}\right.$$4 solves the stochastic algebraic Riccati equation

$$\Sigma_d:\left\{ \begin{aligned} \dot{x}_d &= A_d x_d,\qquad x_d(0)=x_{d0}\in\mathbb{R}^{n_d},\ y_d &= H_d x_d. \end{aligned}\right.$$5

while $$\Sigma_d:\left\{ \begin{aligned} \dot{x}_d &= A_d x_d,\qquad x_d(0)=x_{d0}\in\mathbb{R}^{n_d},\ y_d &= H_d x_d. \end{aligned}\right.$$6 is obtained from the Sylvester equation

$$\Sigma_d:\left\{ \begin{aligned} \dot{x}_d &= A_d x_d,\qquad x_d(0)=x_{d0}\in\mathbb{R}^{n_d},\ y_d &= H_d x_d. \end{aligned}\right.$$7

This decomposition separates the feedback-regulation problem from the feedforward-tracking problem (Chen et al., 28 Aug 2025).

The separation is central to the algorithm. B-PI is constructed for the feedback component $$\Sigma_d:\left\{ \begin{aligned} \dot{x}_d &= A_d x_d,\qquad x_d(0)=x_{d0}\in\mathbb{R}^{n_d},\ y_d &= H_d x_d. \end{aligned}\right.$$8, not for the feedforward term directly. The paper emphasizes that this avoids the usual state augmentation approach, which would solve one larger Riccati equation of size $$\Sigma_d:\left\{ \begin{aligned} \dot{x}_d &= A_d x_d,\qquad x_d(0)=x_{d0}\in\mathbb{R}^{n_d},\ y_d &= H_d x_d. \end{aligned}\right.$$9 (Chen et al., 28 Aug 2025). A plausible implication is that the “bootstrap” burden is concentrated on producing a stabilizing regulation policy; only after that step does tracking feedforward become a linear recovery problem.

3. Two-phase algorithmic structure

Phase I introduces a parameterized auxiliary system

$u(t)=-Kx(t)-F x_d(t),$0

together with the stabilizing gain set

$u(t)=-Kx(t)-F x_d(t),$1

where

$u(t)=-Kx(t)-F x_d(t),$2

If $u(t)=-Kx(t)-F x_d(t),$3 is chosen such that

$u(t)=-Kx(t)-F x_d(t),$4

then $u(t)=-Kx(t)-F x_d(t),$5, so the zero gain stabilizes the parameterized system (Chen et al., 28 Aug 2025).

Starting from $u(t)=-Kx(t)-F x_d(t),$6, the bootstrap iteration solves

$u(t)=-Kx(t)-F x_d(t),$7

updates the gain by

$u(t)=-Kx(t)-F x_d(t),$8

and updates the parameter via

$u(t)=-Kx(t)-F x_d(t),$9

Phase I stops at

$K$0

At that point, $K$1 stabilizes the original system (Chen et al., 28 Aug 2025).

Phase II then switches to standard policy iteration on the original system: $K$2 followed by

$K$3

and termination when

$K$4

The role of the first phase is therefore not direct optimization of the original tracking criterion, but construction of a stabilizing entry point from which ordinary PI becomes valid (Chen et al., 28 Aug 2025).

A useful contrast is supplied by trajectory-driven online PI. There, the current policy $K$5 produces an improved trajectory through one-step lookahead minimizations against $K$6, and the next policy is generated from that improved trajectory: $K$7 This is not the same algorithmic mechanism as the stochastic tracking B-PI, but it exhibits the same two-part organization: one phase generates improved decision data, and the next phase fits or generates the next policy (Li et al., 16 Apr 2026).

4. Guarantees, data-driven realization, and empirical instantiations

Under mean-square stabilizability and exact detectability, the model-based B-PI theorem states that if $K$8 and

$K$9

then the bootstrap index is finite,

$F$0

with $F$1, and the refinement phase satisfies

$F$2

The sequence therefore first reaches the stabilizing class in finitely many steps and then converges to the optimal feedback gain (Chen et al., 28 Aug 2025).

The same paper develops an off-policy data-driven realization. With

$F$3

the transformed system corresponds to the parameterized system $F$4. The data-driven method requires the interval excitation condition

$F$5

Policy evaluation becomes a linear-algebraic recovery of

$F$6

using

$F$7

in Phase I and

$F$8

in Phase II, followed by

$F$9

After feedback recovery, the feedforward parameters are computed from

$K_0=0$0

For $K_0=0$1, shadow systems are introduced so that probing noise on the physical plant can be avoided (Chen et al., 28 Aug 2025).

The numerical examples instantiate the two-phase pattern concretely. In the spring-mass-damper example with

$K_0=0$2

the threshold $K_0=0$3 is first reached at iteration $K_0=0$4, producing

$K_0=0$5

and the algorithm converges at iteration $K_0=0$6 with

$K_0=0$7

In the shadow-system example with $K_0=0$8 and $K_0=0$9, the algorithm reaches $K^*$0 at $K^*$1, producing

$K^*$2

and converges by iteration $K^*$3 (Chen et al., 28 Aug 2025).

5. Related formulations in policy iteration and reinforcement learning

Several arXiv papers fit the same structural idea even when they do not use the B-PI label.

Paper	Relation to B-PI
"On-Line Policy Iteration with Trajectory-Driven Policy Generation" (Li et al., 16 Apr 2026)	Improved trajectory first, policy generation second
"Policy iteration for discrete-time systems with discounted costs: stability and near-optimality guarantees" (Brusse et al., 2024)	Recovery from non-stabilizing initializer, then stabilizing refinement
"Policy iteration: for want of recursive feasibility, all is not lost" (Granzotto et al., 2022)	Bootstrap stabilizer assumed; PI+ preserves feasibility and stability
"On the Convergence of the Policy Iteration for Infinite-Horizon Nonlinear Optimal Control Problems" (Ehring et al., 14 Jul 2025)	Admissible bootstrap controller plus invariant-domain PI refinement
"On Applications of Bootstrap in Continuous Space Reinforcement Learning" (Faradonbeh et al., 2019)	Bootstrap-randomized model estimation followed by Riccati policy recomputation
"Sample-Efficient Model-Free Reinforcement Learning with Off-Policy Critics" (Steckelmacher et al., 2019)	Bootstrapped critics with conservative actor update; related but differently named

The trajectory-driven formulation is the closest structural analogue on the finite-horizon side. Given a policy $K^*$4, it improves controls stagewise along a single trajectory by solving

$K^*$5

then generates the next policy from the improved trajectory. Under a consistency condition requiring the generated policy to reproduce the trajectory controls on trajectory states,

$K^*$6

the costs satisfy

$K^*$7

This suggests a trajectory-bootstrap interpretation of B-PI in which one phase improves local decision data and the next phase trains a policy representation that preserves those improvements (Li et al., 16 Apr 2026).

The discounted deterministic PI paper gives a different two-phase reading. It assumes only that the initial policy $K^*$8 has finite discounted cost,

$K^*$9

without requiring $P_d(u)=\lim_{T\rightarrow\infty}\frac{1}{T}J_T(u),$0 to be stabilizing, and proves monotone improvement

$P_d(u)=\lim_{T\rightarrow\infty}\frac{1}{T}J_T(u),$1

a near-optimality bound

$P_d(u)=\lim_{T\rightarrow\infty}\frac{1}{T}J_T(u),$2

and semiglobal practical stability after finitely many iterations once $P_d(u)=\lim_{T\rightarrow\infty}\frac{1}{T}J_T(u),$3. This is an explicit recovery-phase/refinement-phase decomposition, even though it is not called B-PI (Brusse et al., 2024).

The nonlinear undiscounted PI paper sharpens a different point: standard PI can fail to be recursively feasible, so a bootstrap policy alone is not enough. Its PI+ modification regularizes the improvement map and evaluates the best-cost regularized selection. Under stated continuity, level-boundedness, and outer-semicontinuity assumptions, $P_d(u)=\lim_{T\rightarrow\infty}\frac{1}{T}J_T(u),$4, $P_d(u)=\lim_{T\rightarrow\infty}\frac{1}{T}J_T(u),$5, and $P_d(u)=\lim_{T\rightarrow\infty}\frac{1}{T}J_T(u),$6 remain non-empty for all $P_d(u)=\lim_{T\rightarrow\infty}\frac{1}{T}J_T(u),$7, while recursive $P_d(u)=\lim_{T\rightarrow\infty}\frac{1}{T}J_T(u),$8-stability and explicit near-optimality bounds are preserved (Granzotto et al., 2022). This suggests that, in some settings, the second phase of a B-PI design must be feasibility-preserving rather than merely greedy.

The bounded-domain nonlinear PI paper contributes another closely related ingredient: a bootstrap policy must come with an admissible forward-invariant domain. For an admissible controller-domain pair $P_d(u)=\lim_{T\rightarrow\infty}\frac{1}{T}J_T(u),$9, the value function $$J_T = \frac{1}{2}\mathbb{E}\left[\int_0^T \left(\|y-y_d\|_Q^2+\|u\|_R^2\right)\mathrm{d}t\right],$$0 solves the GHJB equation

$$J_T = \frac{1}{2}\mathbb{E}\left[\int_0^T \left(\|y-y_d\|_Q^2+\|u\|_R^2\right)\mathrm{d}t\right],$$1

and the improved policy

$$J_T = \frac{1}{2}\mathbb{E}\left[\int_0^T \left(\|y-y_d\|_Q^2+\|u\|_R^2\right)\mathrm{d}t\right],$$2

is admissible on suitable proper sublevel subsets of $$J_T = \frac{1}{2}\mathbb{E}\left[\int_0^T \left(\|y-y_d\|_Q^2+\|u\|_R^2\right)\mathrm{d}t\right],$$3. The resulting domain-aware PI sequence yields

$$J_T = \frac{1}{2}\mathbb{E}\left[\int_0^T \left(\|y-y_d\|_Q^2+\|u\|_R^2\right)\mathrm{d}t\right],$$4

together with convergence on compact subsets under uniform derivative bounds (Ehring et al., 14 Jul 2025). In that setting, a plausible B-PI interpretation is “bootstrap controller plus safe-region construction, then monotone PI refinement.”

6. Conceptual boundaries, misconceptions, and structural significance

A first misconception is that B-PI names a single universally standardized method. The available record is narrower. The term “two-phase bootstrap policy iteration (B-PI)” is explicit in the stochastic LQ tracking paper (Chen et al., 28 Aug 2025), whereas several other works are best described as conceptual analogues rather than instances of a canonical shared algorithm (Li et al., 16 Apr 2026, Brusse et al., 2024).

A second misconception is that “bootstrap” here necessarily refers to temporal-difference bootstrapping. In the cited literature, bootstrap has several meanings. It may denote bootstrapping a stabilizing gain from $$J_T = \frac{1}{2}\mathbb{E}\left[\int_0^T \left(\|y-y_d\|_Q^2+\|u\|_R^2\right)\mathrm{d}t\right],$$5 by passing through an auxiliary parameterized system (Chen et al., 28 Aug 2025); bootstrapping the next policy from an improved trajectory generated by the current policy (Li et al., 16 Apr 2026); bootstrapping from a non-stabilizing but finite-cost policy until PI enters the stabilizing class (Brusse et al., 2024); or bootstrap-randomized model estimation followed by Riccati recomputation in adaptive LQ control (Faradonbeh et al., 2019). This suggests that B-PI is better viewed as a structural pattern than as a single estimator design.

A third misconception is that monotonic guarantees are automatically global. They are not. The trajectory-driven online PI theorem is only for the fixed initial state $$J_T = \frac{1}{2}\mathbb{E}\left[\int_0^T \left(\|y-y_d\|_Q^2+\|u\|_R^2\right)\mathrm{d}t\right],$$6, not all states (Li et al., 16 Apr 2026). The discounted recovery result depends on detectability and optimal closed-loop stability assumptions (Brusse et al., 2024). The nonlinear PI+ results require recursive-feasibility machinery (Granzotto et al., 2022). The bounded-domain convergence theory depends on admissibility, forward invariance, and sufficient GHJB regularity (Ehring et al., 14 Jul 2025). The explicit B-PI tracking guarantees rely on mean-square stabilizability, exact detectability, parameter choices $$J_T = \frac{1}{2}\mathbb{E}\left[\int_0^T \left(\|y-y_d\|_Q^2+\|u\|_R^2\right)\mathrm{d}t\right],$$7, and, in the data-driven case, interval excitation (Chen et al., 28 Aug 2025).

The structural significance of B-PI lies in this repeated decomposition. One stage repairs an initialization defect—lack of stability, lack of feasibility, lack of local training targets, or lack of informative model uncertainty. The next stage exploits ordinary PI-style monotone improvement, rollout-based enhancement, or conservative policy updating once the bootstrap stage has produced a viable operating point. Within the present literature, that is the most stable encyclopedia-level definition of Two-Phase Bootstrap Policy Iteration.