PFDeePO: Perturbation-Free DeePO for LQR

• PFDeePO is a variant of data-enabled policy optimization for LQR that computes exact policy gradients from a single persistently exciting data batch without rollout perturbations.
• It leverages closed-loop data, Lyapunov equations, and projected gradient updates to ensure feasibility and achieve convergence to the certainty-equivalence solution.
• In adaptive formulations, PFDeePO eliminates unwanted state perturbations by pausing updates near equilibrium and using bounded multiplicative mean-1 excitation.

Searching arXiv for the cited DeePO and PFDeePO papers to ground the article in the latest available preprints. Perturbation-Free DeePO (PFDeePO) denotes perturbation-free variants of data-enabled policy optimization for the linear quadratic regulator (LQR). Across the DeePO literature, the designation has two closely related but distinct technical meanings. In “Data-enabled Policy Optimization for the Linear Quadratic Regulator” (Zhao et al., 2023), PFDeePO is the instantiation of DeePO that computes exact policy gradients directly from a finite batch of persistently exciting closed-loop data, without any on-policy perturbations, exploration noise, or trajectory rollouts during the policy-improvement loop. In “A Modified Adaptive Data-Enabled Policy Optimization Control to Resolve State Perturbations” (Kaheni et al., 28 Jul 2025), PFDeePO denotes a modified adaptive DeePO scheme that eliminates steady-state state perturbations by pausing gain updates near equilibrium and replacing additive probing noise with bounded multiplicative mean-1 excitation after controller convergence. A related adaptive DeePO formulation based on sample covariance is developed in “Data-Enabled Policy Optimization for Direct Adaptive Learning of the LQR” (Zhao et al., 2024), which explicitly notes that it does not define or use the term PFDeePO.

1. Terminology and conceptual scope

The DeePO framework is a direct data-driven method for solving the LQR without first identifying a parametric model. Its basic premise is that under controllability and a persistency of excitation (PE) condition, closed-loop policies can be parameterized directly from measured state-input data, so that policy optimization becomes an optimization problem over data-dependent variables rather than over unknown matrices $A$ and $B$ (Zhao et al., 2023).

Paper	Use of the term	Distinctive feature
(Zhao et al., 2023)	PFDeePO is an instantiation of DeePO	Exact policy gradients from one PE batch
(Zhao et al., 2024)	PFDeePO is not defined or used	Adaptive covariance-based DeePO
(Kaheni et al., 28 Jul 2025)	PFDeePO is a modified adaptive DeePO	No additive probing noise; multiplicative mean-1 excitation

This terminological split is significant. In the 2023 formulation, “perturbation-free” refers to the policy-improvement loop: once a single sufficiently exciting batch has been collected, all subsequent optimization is offline and algebraic. In the 2025 adaptive formulation, “perturbation-free” refers more specifically to the removal of additive probing noise that would otherwise induce undesirable state perturbations. A common misconception is that PFDeePO removes the PE requirement itself. The sources do not support that interpretation. In all versions, PE or an equivalent data-richness condition remains essential; what changes is how excitation is obtained and when it is applied.

2. Data-driven closed-loop parameterization

The foundational setting is the discrete-time linear system

$x_{t+1} = A x_t + B u_t,$

with $x_t \in \mathbb{R}^n$, $u_t \in \mathbb{R}^m$, controllable $(A,B)$, linear state-feedback $u_t = K x_t$, and infinite-horizon quadratic cost

$$J(K) = \mathbb{E}_{x(0)\sim D}\Big[\sum_{t=0}^{\infty} (x_t^\top Q x_t + u_t^\top R u_t)\Big],$$

where $Q \succ 0$ and $R \succ 0$ (Zhao et al., 2023). Classical LQR expresses the optimal gain through the algebraic Riccati equation, but DeePO replaces model knowledge with data matrices

$B$0

satisfying $B$1, together with

$B$2

Under this rank condition, any state-feedback gain $B$3 can be represented by a data-dependent matrix $B$4 satisfying

$B$5

The closed-loop matrix then becomes

$B$6

so both policy evaluation and policy improvement can be performed directly in terms of $B$7, $B$8, $B$9, and $x_{t+1} = A x_t + B u_t,$0, without explicit recovery of $x_{t+1} = A x_t + B u_t,$1 or $x_{t+1} = A x_t + B u_t,$2 (Zhao et al., 2023).

The adaptive DeePO formulation uses a covariance parameterization instead. Given batch data $x_{t+1} = A x_t + B u_t,$3, it defines

$x_{t+1} = A x_t + B u_t,$4

with $x_{t+1} = A x_t + B u_t,$5. This reparameterization has a fixed dimension depending only on $x_{t+1} = A x_t + B u_t,$6, not on the data length $x_{t+1} = A x_t + B u_t,$7, which is what enables recursive online adaptation in the 2024 adaptive DeePO paper (Zhao et al., 2024). The two parameterizations are different realizations of the same underlying idea: the policy is embedded in the column space generated by persistently exciting data.

3. Exact policy gradients without rollout perturbations

In the batch formulation, the LQR objective becomes

$x_{t+1} = A x_t + B u_t,$8

The value function admits a fully data-driven representation: $x_{t+1} = A x_t + B u_t,$9 where $x_t \in \mathbb{R}^n$0 and $x_t \in \mathbb{R}^n$1 solve the Lyapunov equations

$x_t \in \mathbb{R}^n$2

and

$x_t \in \mathbb{R}^n$3

The exact gradient is

$x_t \in \mathbb{R}^n$4

All quantities depend only on the pre-collected data and the current $x_t \in \mathbb{R}^n$5; no rollout-based gradient estimator, no exploration noise, and no on-policy perturbation are required (Zhao et al., 2023).

This perturbation-free property is the central distinction between PFDeePO and standard policy-gradient or actor–critic approaches for LQR. Standard model-free policy optimization typically estimates gradients from rollout costs under exploration noise and consequently requires many, potentially long trajectories for small estimation error. PFDeePO instead uses a single PE batch and exact algebraic evaluation through Lyapunov and covariance equations.

The adaptive covariance-based DeePO has an analogous gradient structure. With

$x_t \in \mathbb{R}^n$6

the gradient is

$x_t \in \mathbb{R}^n$7

The 2024 paper emphasizes that this gradient is computed from closed-loop data batches and can be used recursively online, performing one projected gradient step per sample (Zhao et al., 2024).

4. Projected updates, convex equivalence, and convergence

PFDeePO enforces the linear feasibility condition $x_t \in \mathbb{R}^n$8 through a projected gradient step,

$x_t \in \mathbb{R}^n$9

where $u_t \in \mathbb{R}^m$0 projects onto the nullspace of $u_t \in \mathbb{R}^m$1. By construction, the update preserves $u_t \in \mathbb{R}^m$2 at every iteration (Zhao et al., 2023).

A key theoretical result is an exact convex equivalence. Introducing variables $u_t \in \mathbb{R}^m$3 with $u_t \in \mathbb{R}^m$4, DeePO considers the convex objective

$u_t \in \mathbb{R}^m$5

subject to $u_t \in \mathbb{R}^m$6 and the LMI

$u_t \in \mathbb{R}^m$7

The paper proves that for any feasible $u_t \in \mathbb{R}^m$8, $u_t \in \mathbb{R}^m$9 is the minimum of $(A,B)$0 over all $(A,B)$1 satisfying $(A,B)$2. This equivalence underpins the projected gradient dominance property

$(A,B)$3

over sublevel sets, and yields global sublinear convergence of projected gradient descent. Specifically, for suitable $(A,B)$4, the iterates remain feasible and satisfy

$(A,B)$5

(Zhao et al., 2023).

The 2024 adaptive DeePO paper establishes an analogous projected gradient dominance result for the covariance parameterization $(A,B)$6, again with global convergence and a projected-gradient interpretation. It further derives an online regret bound of the form

$(A,B)$7

which is described informally as $(A,B)$8 and is independent of the noise statistics beyond boundedness (Zhao et al., 2024).

5. Adaptive PFDeePO for eliminating state perturbations

The 2025 PFDeePO paper starts from a different practical problem. In adaptive DeePO, PE is commonly maintained by additive probing noise. The paper identifies two failure modes when such probing noise is not added. First, as $(A,B)$9, both $u_t = K x_t$0 and $u_t = K x_t$1 accumulate near-zero columns, and the minimum singular value of the data covariance

$u_t = K x_t$2

tends to $u_t = K x_t$3, jeopardizing invertibility of $u_t = K x_t$4. Second, if the controller converges to a fixed gain $u_t = K x_t$5, then $u_t = K x_t$6 becomes a linear combination of $u_t = K x_t$7, so $u_t = K x_t$8, violating PE (Kaheni et al., 28 Jul 2025).

PFDeePO resolves these issues through two rules. First, it pauses gain updates near equilibrium: if

$u_t = K x_t$9

the algorithm keeps $$J(K) = \mathbb{E}_{x(0)\sim D}\Big[\sum_{t=0}^{\infty} (x_t^\top Q x_t + u_t^\top R u_t)\Big],$$0 and does not append the corresponding near-zero data columns. Second, once the controller is judged converged by

$$J(K) = \mathbb{E}_{x(0)\sim D}\Big[\sum_{t=0}^{\infty} (x_t^\top Q x_t + u_t^\top R u_t)\Big],$$1

and the state is not near equilibrium, PFDeePO applies multiplicative mean-1 excitation,

$$J(K) = \mathbb{E}_{x(0)\sim D}\Big[\sum_{t=0}^{\infty} (x_t^\top Q x_t + u_t^\top R u_t)\Big],$$2

with $$J(K) = \mathbb{E}_{x(0)\sim D}\Big[\sum_{t=0}^{\infty} (x_t^\top Q x_t + u_t^\top R u_t)\Big],$$3 and $$J(K) = \mathbb{E}_{x(0)\sim D}\Big[\sum_{t=0}^{\infty} (x_t^\top Q x_t + u_t^\top R u_t)\Big],$$4 bounded in an interval $$J(K) = \mathbb{E}_{x(0)\sim D}\Big[\sum_{t=0}^{\infty} (x_t^\top Q x_t + u_t^\top R u_t)\Big],$$5 that preserves closed-loop stability. The algorithmic description also uses $$J(K) = \mathbb{E}_{x(0)\sim D}\Big[\sum_{t=0}^{\infty} (x_t^\top Q x_t + u_t^\top R u_t)\Big],$$6 for the same role and gives a practical choice $$J(K) = \mathbb{E}_{x(0)\sim D}\Big[\sum_{t=0}^{\infty} (x_t^\top Q x_t + u_t^\top R u_t)\Big],$$7, with symmetric bounds around $$J(K) = \mathbb{E}_{x(0)\sim D}\Big[\sum_{t=0}^{\infty} (x_t^\top Q x_t + u_t^\top R u_t)\Big],$$8, such as $$J(K) = \mathbb{E}_{x(0)\sim D}\Big[\sum_{t=0}^{\infty} (x_t^\top Q x_t + u_t^\top R u_t)\Big],$$9 to $Q \succ 0$0.

This construction is called perturbation-free because the equilibrium is preserved: if $Q \succ 0$1, then $Q \succ 0$2 regardless of $Q \succ 0$3. It is also unbiased in the sense that

$Q \succ 0$4

A plausible implication is that the term “perturbation-free” is used here in a control-theoretic rather than stochastic-estimation sense: the injected variability does not create an additive offset at equilibrium and its magnitude decays with $Q \succ 0$5.

6. Guarantees, empirical behavior, and limitations

For the adaptive 2025 PFDeePO scheme, the paper proves a rank-preservation theorem: under PFDeePO,

$Q \succ 0$6

It also gives a closed-loop stability result for multiplicative excitation. If the converged DeePO gain $Q \succ 0$7 is the certainty-equivalence LQR gain for $Q \succ 0$8, computed from a $Q \succ 0$9-shifted DARE, and if the interval $R \succ 0$0 satisfies

$R \succ 0$1

for all $R \succ 0$2 in the interval, then the time-varying feedback $R \succ 0$3 renders the origin exponentially stable for $R \succ 0$4, with Lyapunov function $R \succ 0$5 (Kaheni et al., 28 Jul 2025).

The empirical evidence in the sources reflects the differing emphases of the two PFDeePO interpretations. In the 2023 batch setting, simulations with $R \succ 0$6, $R \succ 0$7, $R \succ 0$8, $R \succ 0$9, $B$00, Gaussian inputs, and $B$01 show that all methods exhibit linear empirical convergence in relative error; robustness regularization accelerates convergence, and the unregularized PFDeePO converges to the certainty-equivalence solution when $B$02 (Zhao et al., 2023). In the 2025 adaptive setting, simulations with $B$03, $B$04, $B$05, $B$06, $B$07, $B$08, $B$09, $B$10, $B$11, and a state perturbation at $B$12 show that PFDeePO avoids oscillations once the states reach equilibrium, while DeePO with additive probing noise induces continuous oscillations and increased control effort (Kaheni et al., 28 Jul 2025).

The limitations are equally consistent across the papers. PFDeePO relies on sufficiently strong PE or quantitative PE; poor excitation causes rank deficiency and invalidates the data-driven parameterization. The proven convergence rate is sublinear, although empirical behavior is often linear. In the adaptive perturbation-elimination variant, the stability interval depends on $B$13 and $B$14, so conservative choices may be required when identification is uncertain. Current guarantees are stated for LTI systems with controllability and positive definite $B$15 and $B$16; extensions to time-varying or nonlinear systems are presented as future work (Zhao et al., 2023, Zhao et al., 2024, Kaheni et al., 28 Jul 2025).

Taken together, the literature presents PFDeePO not as a single universally standardized algorithm, but as a perturbation-free design philosophy within DeePO. In one form, it means exact policy optimization from a finite PE batch without online perturbations. In another, it means adaptive DeePO without additive probing noise, achieved through update pausing near equilibrium and bounded multiplicative mean-1 excitation. Both forms preserve the central DeePO objective: direct, data-driven recovery of the certainty-equivalence LQR solution through projected policy optimization on informative closed-loop data.