---
title: 'PFDeePO: Perturbation-Free DeePO for LQR'
url: https://www.emergentmind.com/topics/perturbation-free-deepo-pfdeepo
type: topic
---

# PFDeePO: Perturbation-Free DeePO for LQR

Searching arXiv for the cited DeePO and PFDeePO papers to ground the article in the latest available preprints.
Perturbation-Free DeePO (PFDeePO) denotes perturbation-free variants of data-enabled policy optimization for the linear quadratic regulator (LQR). Across the DeePO literature, the designation has two closely related but distinct technical meanings. In “Data-enabled Policy Optimization for the Linear Quadratic Regulator” [2303.17958], PFDeePO is the instantiation of DeePO that computes exact policy gradients directly from a finite batch of persistently exciting closed-loop data, without any on-policy perturbations, exploration noise, or trajectory rollouts during the policy-improvement loop. In “A Modified Adaptive Data-Enabled Policy Optimization Control to Resolve State Perturbations” [2507.20580], PFDeePO denotes a modified adaptive DeePO scheme that eliminates steady-state state perturbations by pausing gain updates near equilibrium and replacing additive probing noise with bounded multiplicative mean-1 excitation after controller convergence. A related adaptive DeePO formulation based on sample covariance is developed in “Data-Enabled Policy Optimization for Direct Adaptive Learning of the LQR” [2401.14871], which explicitly notes that it does not define or use the term PFDeePO.

## 1. Terminology and conceptual scope

The DeePO framework is a direct data-driven method for solving the LQR without first identifying a parametric model. Its basic premise is that under controllability and a persistency of excitation (PE) condition, closed-loop policies can be parameterized directly from measured state-input data, so that policy optimization becomes an optimization problem over data-dependent variables rather than over unknown matrices \(A\) and \(B\) [2303.17958].

| Paper | Use of the term | Distinctive feature |
|---|---|---|
| [2303.17958] | PFDeePO is an instantiation of DeePO | Exact policy gradients from one PE batch |
| [2401.14871] | PFDeePO is not defined or used | Adaptive covariance-based DeePO |
| [2507.20580] | PFDeePO is a modified adaptive DeePO | No additive probing noise; multiplicative mean-1 excitation |

This terminological split is significant. In the 2023 formulation, “perturbation-free” refers to the policy-improvement loop: once a single sufficiently exciting batch has been collected, all subsequent optimization is offline and algebraic. In the 2025 adaptive formulation, “perturbation-free” refers more specifically to the removal of additive probing noise that would otherwise induce undesirable state perturbations. A common misconception is that PFDeePO removes the PE requirement itself. The sources do not support that interpretation. In all versions, PE or an equivalent data-richness condition remains essential; what changes is how excitation is obtained and when it is applied.

## 2. Data-driven closed-loop parameterization

The foundational setting is the discrete-time linear system
\[
x_{t+1} = A x_t + B u_t,
\]
with \(x_t \in \mathbb{R}^n\), \(u_t \in \mathbb{R}^m\), controllable \((A,B)\), linear state-feedback \(u_t = K x_t\), and infinite-horizon quadratic cost
\[
J(K) = \mathbb{E}_{x(0)\sim D}\Big[\sum_{t=0}^{\infty} (x_t^\top Q x_t + u_t^\top R u_t)\Big],
\]
where \(Q \succ 0\) and \(R \succ 0\) [2303.17958]. Classical LQR expresses the optimal gain through the algebraic Riccati equation, but DeePO replaces model knowledge with data matrices
\[
X_- = [x(0)\ x(1)\ \cdots\ x(T-1)],\quad
U_- = [u(0)\ u(1)\ \cdots\ u(T-1)],\quad
X_+ = [x(1)\ x(2)\ \cdots\ x(T)],
\]
satisfying \(X_+ = A X_- + B U_-\), together with
\[
D_- = \begin{bmatrix} U_- \\ X_- \end{bmatrix},\qquad \operatorname{rank}(D_-) = m+n.
\]

Under this rank condition, any state-feedback gain \(K\) can be represented by a data-dependent matrix \(G \in \mathbb{R}^{T\times n}\) satisfying
\[
[K\ I_n] = D_- G,\qquad K = U_- G,\qquad X_- G = I_n.
\]
The closed-loop matrix then becomes
\[
A + BK = X_+ G,
\]
so both policy evaluation and policy improvement can be performed directly in terms of \(G\), \(U_-\), \(X_-\), and \(X_+\), without explicit recovery of \(A\) or \(B\) [2303.17958].

The adaptive DeePO formulation uses a covariance parameterization instead. Given batch data \(D_0 = [U_0; X_0]\), it defines
\[
\Lambda = \frac{1}{t} D_0 D_0^\top,
\qquad
\begin{bmatrix}K & I_n\end{bmatrix} = \Lambda V,
\]
with \(V \in \mathbb{R}^{(n+m)\times n}\). This reparameterization has a fixed dimension depending only on \(m+n\), not on the data length \(t\), which is what enables recursive online adaptation in the 2024 adaptive DeePO paper [2401.14871]. The two parameterizations are different realizations of the same underlying idea: the policy is embedded in the column space generated by persistently exciting data.

## 3. Exact policy gradients without rollout perturbations

In the batch formulation, the LQR objective becomes
\[
\min J(G)\quad \text{subject to}\quad G \in \mathcal{S}_G := \{G \mid X_- G = I_n,\ \rho(X_+ G) < 1\}.
\]
The value function admits a fully data-driven representation:
\[
J(G) = \operatorname{Tr}\{P_G\}
      = \operatorname{Tr}\{(Q + G^\top U_-^\top R U_- G)\Sigma_G\},
\]
where \(P_G\) and \(\Sigma_G\) solve the Lyapunov equations
\[
P_G = Q + G^\top U_-^\top R U_- G + G^\top X_+^\top P_G X_+ G,
\]
and
\[
\Sigma_G = I_n + X_+ G \Sigma_G G^\top X_+^\top.
\]
The exact gradient is
\[
\nabla J(G) = 2 E_G \Sigma_G,
\qquad
E_G := (U_-^\top R U_- + X_+^\top P_G X_+)G.
\]
All quantities depend only on the pre-collected data and the current \(G\); no rollout-based gradient estimator, no exploration noise, and no on-policy perturbation are required [2303.17958].

This perturbation-free property is the central distinction between PFDeePO and standard policy-gradient or actor–critic approaches for LQR. Standard model-free policy optimization typically estimates gradients from rollout costs under exploration noise and consequently requires many, potentially long trajectories for small estimation error. PFDeePO instead uses a single PE batch and exact algebraic evaluation through Lyapunov and covariance equations.

The adaptive covariance-based DeePO has an analogous gradient structure. With
\[
P_V = Q + V^\top \overline{U}_0^\top R\, \overline{U}_0 V + V^\top \overline{X}_1^\top P_V\, \overline{X}_1 V,
\]
the gradient is
\[
\nabla J(V) = 2 \left(\overline{U}_0^\top R\,\overline{U}_0 + \overline{X}_1^\top P_V\,\overline{X}_1\right)V\Sigma_V.
\]
The 2024 paper emphasizes that this gradient is computed from closed-loop data batches and can be used recursively online, performing one projected gradient step per sample [2401.14871].

## 4. Projected updates, convex equivalence, and convergence

PFDeePO enforces the linear feasibility condition \(X_- G = I_n\) through a projected gradient step,
\[
G^+ = G - \eta \Pi_{X_-}\nabla J(G),
\qquad
\Pi_{X_-} := I_T - X_-^\dagger X_-,
\]
where \(\Pi_{X_-}\) projects onto the nullspace of \(X_-\). By construction, the update preserves \(X_- G = I_n\) at every iteration [2303.17958].

A key theoretical result is an exact convex equivalence. Introducing variables \((L,\Sigma)\) with \(G = L\Sigma^{-1}\), DeePO considers the convex objective
\[
f(L,\Sigma) := \operatorname{Tr}\{Q\Sigma\}
+ \operatorname{Tr}\{L\Sigma^{-1}L^\top U_-^\top R U_- \},
\]
subject to \(\Sigma = X_- L\) and the LMI
\[
\begin{bmatrix}
\Sigma - I_n & X_+ L \\
L^\top X_+^\top & \Sigma
\end{bmatrix} \succeq 0.
\]
The paper proves that for any feasible \(G\), \(J(G)\) is the minimum of \(f(L,\Sigma)\) over all \((L,\Sigma)\) satisfying \(L\Sigma^{-1}=G\). This equivalence underpins the projected gradient dominance property
\[
J(G) - J^* \le \mu(a)\|\Pi_{X_-}\nabla J(G)\|,
\]
over sublevel sets, and yields global sublinear convergence of projected gradient descent. Specifically, for suitable \(\eta \in (0,1/l_0]\), the iterates remain feasible and satisfy
\[
k \ge \frac{2\mu_0^2}{\epsilon(2\eta - l_0\eta^2)}
\quad \Rightarrow \quad
J(G^k) - J^* \le \epsilon
\]
[2303.17958].

The 2024 adaptive DeePO paper establishes an analogous projected gradient dominance result for the covariance parameterization \(V\), again with global convergence and a projected-gradient interpretation. It further derives an online regret bound of the form
\[
\mathrm{Regret}_T \le \frac{b_1}{\sqrt{T}} + \sqrt{\frac{b_2\delta}{\gamma}},
\]
which is described informally as \(\mathcal{O}(1/\sqrt{T}) + \mathcal{O}(\mathrm{SNR}^{-1/2})\) and is independent of the noise statistics beyond boundedness [2401.14871].

## 5. Adaptive PFDeePO for eliminating state perturbations

The 2025 PFDeePO paper starts from a different practical problem. In adaptive DeePO, PE is commonly maintained by additive probing noise. The paper identifies two failure modes when such probing noise is not added. First, as \(x_k \to 0\), both \(U_0\) and \(X_0\) accumulate near-zero columns, and the minimum singular value of the data covariance
\[
\Phi = \frac{1}{t} D D^\top,\qquad D = \begin{bmatrix}U_0 \\ X_0\end{bmatrix},
\]
tends to \(0\), jeopardizing invertibility of \(\Phi\). Second, if the controller converges to a fixed gain \(K\), then \(U_0\) becomes a linear combination of \(X_0\), so \(\operatorname{rank}(D)=n\), violating PE [2507.20580].

PFDeePO resolves these issues through two rules. First, it pauses gain updates near equilibrium: if
\[
\|x_i\| \le \gamma,
\]
the algorithm keeps \(K_{i+1}=K_i\) and does not append the corresponding near-zero data columns. Second, once the controller is judged converged by
\[
\|\Delta K\| = \|K_i - K_{i-1}\| \le \delta,
\]
and the state is not near equilibrium, PFDeePO applies multiplicative mean-1 excitation,
\[
u_i = \gamma_i K_i x_i,
\]
with \(\mathbb{E}[\gamma_i]=1\) and \(\gamma_i\) bounded in an interval \([\underline{v},\overline{v}]\) that preserves closed-loop stability. The algorithmic description also uses \(v_i\) for the same role and gives a practical choice \(v_i \sim \mathcal{U}(\underline{v},\overline{v})\), with symmetric bounds around \(1\), such as \(0.5\) to \(1.5\).

This construction is called perturbation-free because the equilibrium is preserved: if \(x_k=0\), then \(u_k=\gamma_k K x_k=0\) regardless of \(\gamma_k\). It is also unbiased in the sense that
\[
\mathbb{E}[u_k\mid x_k] = K x_k.
\]
A plausible implication is that the term “perturbation-free” is used here in a control-theoretic rather than stochastic-estimation sense: the injected variability does not create an additive offset at equilibrium and its magnitude decays with \(\|x_k\|\).

## 6. Guarantees, empirical behavior, and limitations

For the adaptive 2025 PFDeePO scheme, the paper proves a rank-preservation theorem: under PFDeePO,
\[
\operatorname{rank}(\Phi_i)=n+m,
\qquad
\underline{\sigma}(\Phi_i)>0.
\]
It also gives a closed-loop stability result for multiplicative excitation. If the converged DeePO gain \(K\) is the certainty-equivalence LQR gain for \((\hat A,\hat B)\), computed from a \(\beta\)-shifted DARE, and if the interval \([\underline{v},\overline{v}]\) satisfies
\[
Q - K^\top\Big((v-1)^2\,\hat{B}^\top H \hat{B} + (1 - 2v)\,R\Big)K \succeq 0
\]
for all \(v\) in the interval, then the time-varying feedback \(u_k=-v_k K x_k\) renders the origin exponentially stable for \(x_{k+1}=(\hat A-v_k\hat B K)x_k\), with Lyapunov function \(V(x)=x^\top H x\) [2507.20580].

The empirical evidence in the sources reflects the differing emphases of the two PFDeePO interpretations. In the 2023 batch setting, simulations with \(n=4\), \(m=2\), \(T=10\), \(Q=I_4\), \(R=I_2\), Gaussian inputs, and \(\eta = 2\times 10^{-3}\) show that all methods exhibit linear empirical convergence in relative error; robustness regularization accelerates convergence, and the unregularized PFDeePO converges to the certainty-equivalence solution when \(\Pi_{D_-}G^0=0\) [2303.17958]. In the 2025 adaptive setting, simulations with \(n=4\), \(m=2\), \(t=8\), \(\sigma_u=0.01\), \(\sigma_\omega=0.01\), \(\eta=10^{-4}\), \(\gamma=0.1\), \(\delta=0.1\), \([\underline{v},\overline{v}] = [0.5,1.5]\), and a state perturbation at \(k=15\) show that PFDeePO avoids oscillations once the states reach equilibrium, while DeePO with additive probing noise induces continuous oscillations and increased control effort [2507.20580].

The limitations are equally consistent across the papers. PFDeePO relies on sufficiently strong PE or quantitative PE; poor excitation causes rank deficiency and invalidates the data-driven parameterization. The proven convergence rate is sublinear, although empirical behavior is often linear. In the adaptive perturbation-elimination variant, the stability interval depends on \(\hat B\) and \(H\), so conservative choices may be required when identification is uncertain. Current guarantees are stated for LTI systems with controllability and positive definite \(Q\) and \(R\); extensions to time-varying or nonlinear systems are presented as future work [2303.17958; 2401.14871; 2507.20580].

Taken together, the literature presents PFDeePO not as a single universally standardized algorithm, but as a perturbation-free design philosophy within DeePO. In one form, it means exact policy optimization from a finite PE batch without online perturbations. In another, it means adaptive DeePO without additive probing noise, achieved through update pausing near equilibrium and bounded multiplicative mean-1 excitation. Both forms preserve the central DeePO objective: direct, data-driven recovery of the certainty-equivalence LQR solution through projected policy optimization on informative closed-loop data.

Source: https://www.emergentmind.com/topics/perturbation-free-deepo-pfdeepo