--- title: Adaptive Indirect Estimator in LQR Control url: https://www.emergentmind.com/topics/adaptive-indirect-estimator type: topic --- # Adaptive Indirect Estimator in LQR Control An Adaptive Indirect Estimator, in the sense developed for the linear quadratic regulator (LQR), is the indirect Policy Gradient Adaptive Control (PGAC) scheme in which the feedback gain is updated from online closed-loop data through model-based gradient estimation rather than direct covariance-based differentiation. The estimator identifies the unknown dynamics online, computes Lyapunov-based policy gradients from the estimated model, and updates the controller while preserving closed-loop stability. In the 2025 formulation, the method includes vanilla gradient descent, natural gradient, and Gauss–Newton updates, together with a regularization mechanism that compensates for noise-induced model uncertainty, and it is shown to converge to the optimal LQR gain under persistency of excitation and signal-to-noise conditions [2505.03706]. ## 1. Problem formulation and analytical objects The setting is a discrete-time linear time-invariant system $$ x_{t+1} = A x_t + B u_t + w_t, $$ with $x_t \in \mathbb{R}^n$, $u_t \in \mathbb{R}^m$, and zero-mean process noise of covariance $W$. The performance output is $$ z_t= \begin{bmatrix} Q^{1/2} & 0\\ 0 & R^{1/2} \end{bmatrix} \begin{bmatrix} x_t\\ u_t \end{bmatrix}, \qquad Q\succ 0,\ R\succ 0. $$ Under static state feedback $u_t=Kx_t$, the closed-loop matrix is $F(K)=A+BK$. When $F(K)$ is Schur, the stationary covariance $\Sigma_K$ solves $$ \Sigma_K = F(K)\Sigma_K F(K)^\top + W, $$ and in the normalized presentation used for the performance analysis, $W=I_n$, so $$ \Sigma = I_n + F(K)\Sigma F(K)^\top. $$ The stationary LQR average cost is $$ J(K)=\operatorname{tr}\!\big((Q+K^\top R K)\Sigma_K\big), $$ which equals the squared $\mathcal{H}_2$ norm from $w$ to $z$ under stationarity. A Riccati-like costate $P_K$ is defined by $$ P_K = Q + K^\top R K + F(K)^\top P_K F(K), $$ and the policy gradient has the exact form $$ \nabla_K J(K)=2\Big((R+B^\top P_K B)K + B^\top P_K A\Big)\Sigma_K. $$ The adaptation theory assumes an initial stabilizing controller $$ K_{t_0}\in \mathcal{S}:=\{K:\rho(A+BK)<1\}, $$ together with persistency of excitation for the block data matrix $D_{0,t}=[U_{0,t};X_{0,t}]$. In covariance form, with $$ \Phi_t=\frac{1}{t}D_{0,t}D_{0,t}^\top, $$ the excitation level is quantified by $\underline{\sigma}(\Phi_t)\ge \gamma_t^2$, while the noise energy satisfies $$ \left\|\frac{1}{t}W_{0,t}W_{0,t}^\top\right\|\le \delta_t^2. $$ The signal-to-noise ratio is $$ \mathrm{SNR}_t:=\frac{\gamma_t}{\delta_t}, $$ and the least-squares identification error obeys $$ \big\|[\hat B_t,\hat A_t]-[B,A]\big\|\le \frac{1}{\mathrm{SNR}_t}. $$ For i.i.d. Gaussian noise, $\delta_t=\mathcal{O}(1/\sqrt{t})$, hence $\mathrm{SNR}_t=\mathcal{O}(\sqrt{t})$ under constant excitation [2505.03706]. ## 2. Indirect estimator construction The indirect estimator is model-based. It first identifies $(A,B)$ online by least squares and then evaluates the policy gradient using the estimated dynamics. In batch form, $$ [\hat B,\hat A]=X_1 D_0^\dagger,\qquad D_0=[U_0;X_0], $$ and in recursive form, $$ [\hat B_{t+1},\hat A_{t+1}] = [\hat B_t,\hat A_t] + \big(x_{t+1}-[\hat B_t,\hat A_t]\phi_t\big)\, \frac{\phi_t^\top \Phi_t^{-1}}{t+\phi_t^\top \Phi_t^{-1}\phi_t}, \qquad \phi_t=\begin{bmatrix}u_t\\x_t\end{bmatrix}. $$ Persistency of excitation guarantees identifiability. Given $(\hat A_{t+1},\hat B_{t+1})$, the estimator solves the certainty-equivalence Lyapunov equations $$ \hat\Sigma_{t+1} = I_n+(\hat A_{t+1}+\hat B_{t+1}K)\hat\Sigma_{t+1}(\hat A_{t+1}+\hat B_{t+1}K)^\top, $$ $$ \hat P_{t+1} = Q+K^\top R K + (\hat A_{t+1}+\hat B_{t+1}K)^\top \hat P_{t+1} (\hat A_{t+1}+\hat B_{t+1}K). $$ The indirect gradient estimate is then $$ \widehat{\nabla_K J}(K) = 2\Big((R+\hat B^\top \hat P\hat B)K+\hat B^\top \hat P\hat A\Big)\hat\Sigma. $$ The corresponding vanilla indirect PGAC update is $$ K_{t+1}=K_t-\eta\,\widehat{\nabla_K J}(K_t), $$ with constant stepsize $\eta>0$ selected so that policy changes remain small enough for sequential stability. In the pseudocode-level implementation, the controller applies $$ u_t=K_t x_t + e_t, $$ where the probing signal $e_t$ is used to ensure persistency of excitation, then updates the least-squares model, solves the Lyapunov equations for $(\hat\Sigma_{t+1},\hat P_{t+1})$, computes the gradient estimate, and performs the gain update [2505.03706]. ## 3. Update geometries: vanilla gradient, natural gradient, and Gauss–Newton The estimator admits three policy-update geometries. | Variant | Update | Distinctive property | |---|---|---| | Vanilla gradient | $K_{t+1}=K_t-\eta\,\widehat{\nabla_K J}(K_t)$ | Uses estimated Lyapunov factors directly | | Natural gradient | $K_{t+1}=K_t-2\eta((R+\hat B^\top\hat P\hat B)K_t-\hat B^\top\hat P\hat A)$ | Bridges indirect and direct PGAC | | Gauss–Newton | $K_{t+1}=K_t-2\eta(R+\hat B^\top\hat P\hat B)^{-1}((R+\hat B^\top\hat P\hat B)K_t-\hat B^\top\hat P\hat A)$ | Yields adaptive Hewer iteration at $\eta=\tfrac12$ | The natural gradient uses the Fisher-like metric of the LQR parameterization and yields $$ K' = K-\eta\,\nabla C(K)\Sigma^{-1} =K-2\eta\Big((R+B^\top P B)K-B^\top P A\Big). $$ In the indirect variant, $(A,B,P)$ are replaced with $(\hat A,\hat B,\hat P)$. A central structural result is that, under the covariance parameterization and projection used for direct PGAC, the direct natural-gradient step in $V$-space maps exactly to the indirect natural-gradient step in $K$-space. A common misconception is therefore that indirect and direct PGAC are intrinsically separate algorithms; the natural-gradient result shows that they are equivalent up to parameterization in this case. The Gauss–Newton step uses the local curvature metric $$ K' = K-\eta\,(R+B^\top P B)^{-1}\nabla C(K)\Sigma^{-1} =K-2\eta\,(R+B^\top P B)^{-1}\Big((R+B^\top P B)K-B^\top P A\Big). $$ At $\eta=\tfrac12$, $$ K'=(R+B^\top P B)^{-1}B^\top P A, $$ which recovers Hewer’s policy-improvement step. The indirect form therefore yields an adaptive Hewer or adaptive Kleinman iteration: $$ \hat P_{t+1} = Q+K_t^\top R K_t +(\hat A_{t+1}+\hat B_{t+1}K_t)^\top \hat P_{t+1} (\hat A_{t+1}+\hat B_{t+1}K_t), $$ followed by $$ K_{t+1} = \big(R+\hat B_{t+1}^\top\hat P_{t+1}\hat B_{t+1}\big)^{-1} \hat B_{t+1}^\top\hat P_{t+1}\hat A_{t+1}. $$ For $\eta=\tfrac12$, sequential stability is guaranteed only locally near $K_\star$ or under sufficiently small identification error [2505.03706]. ## 4. Regularization under noise and uncertainty To account for model mismatch induced by noisy data, the indirect estimator can be regularized through a variance-based penalty built from the sample covariance matrix $\Phi_t$. Writing $$ \Phi^{-1}= \begin{bmatrix} (\Phi^{-1})_{uu} & (\Phi^{-1})_{ux}\\ (\Phi^{-1})_{xu} & (\Phi^{-1})_{xx} \end{bmatrix}, $$ the regularized weights are $$ Q_\lambda=Q+\lambda(\Phi^{-1})_{xx}, \qquad R_\lambda=R+\lambda(\Phi^{-1})_{uu}. $$ The regularized indirect cost is $$ \hat C(K;\lambda) = \operatorname{tr}\!\left( \left( \begin{bmatrix} R & 0\\ 0 & Q \end{bmatrix} +\lambda \Phi^{-1} \right) \begin{bmatrix} K & I_n \end{bmatrix} \Sigma \begin{bmatrix} K & I_n \end{bmatrix}^\top \right), $$ and its gradient is $$ \nabla \hat C(K;\lambda) = 2\Big(R_\lambda K+\hat B^\top P(\hat A+\hat B K)+\lambda(\Phi^{-1})_{ux}\Big)\Sigma, $$ where $P$ solves the corresponding regularized Lyapunov equation. The regularization coefficient is chosen so that $$ \lambda_t\le c\,\delta_t\,\gamma_t, $$ hence the regularizer magnitude scales like $\mathcal{O}(1/\mathrm{SNR}_t)$. This preserves vanishing bias while improving robustness against noise-induced model uncertainty and reducing the variance of gradient estimates. The regularized variants retain the same qualitative stability and convergence guarantees as the unregularized schemes when $\lambda_t=\mathcal{O}(\delta_t\gamma_t)$. A plausible implication is that regularization is not an ancillary numerical device but part of the estimator’s asymptotic design, because its scale is tied explicitly to excitation and noise levels rather than tuned independently of them [2505.03706]. ## 5. Stability, convergence, and empirical behavior The indirect PGAC iterates are sequentially strongly stable if the signal-to-noise ratio is sufficiently large and the stepsize is sufficiently small. For vanilla gradient descent, the stated conditions are $$ \mathrm{SNR}_t\ge \nu_1,\qquad \eta\le \min\{\nu_2,\,2\mu\}, $$ with analogous bounds for natural gradient and Gauss–Newton. Under these conditions, $$ \|x_t\| \le \nu_3\Big(1-\frac{\nu_4}{2}\Big)^{t-t_0}\|x_{t_0}\| + \frac{2\nu_3}{\nu_4}\max_{t_0\le i