Adaptive Indirect Estimator in LQR Control

• Adaptive Indirect Estimator is a model-based scheme that identifies system dynamics online to inform LQR policy gradient updates.
• It employs recursive least-squares estimation and Lyapunov equations to compute gradients and update feedback gains, ensuring closed-loop stability.
• Its variants include vanilla gradient, natural gradient, and Gauss–Newton methods, with regularization enhancing robustness against noise.

An Adaptive Indirect Estimator, in the sense developed for the linear quadratic regulator (LQR), is the indirect Policy Gradient Adaptive Control (PGAC) scheme in which the feedback gain is updated from online closed-loop data through model-based gradient estimation rather than direct covariance-based differentiation. The estimator identifies the unknown dynamics online, computes Lyapunov-based policy gradients from the estimated model, and updates the controller while preserving closed-loop stability. In the 2025 formulation, the method includes vanilla gradient descent, natural gradient, and Gauss–Newton updates, together with a regularization mechanism that compensates for noise-induced model uncertainty, and it is shown to converge to the optimal LQR gain under persistency of excitation and signal-to-noise conditions (Zhao et al., 6 May 2025).

1. Problem formulation and analytical objects

The setting is a discrete-time linear time-invariant system

$x_{t+1} = A x_t + B u_t + w_t,$

with $x_t \in \mathbb{R}^n$, $u_t \in \mathbb{R}^m$, and zero-mean process noise of covariance $W$. The performance output is

$$z_t= \begin{bmatrix} Q^{1/2} & 0\ 0 & R^{1/2} \end{bmatrix} \begin{bmatrix} x_t\ u_t \end{bmatrix}, \qquad Q\succ 0,\ R\succ 0.$$

Under static state feedback $u_t=Kx_t$, the closed-loop matrix is $F(K)=A+BK$. When $F(K)$ is Schur, the stationary covariance $\Sigma_K$ solves

$\Sigma_K = F(K)\Sigma_K F(K)^\top + W,$

and in the normalized presentation used for the performance analysis, $x_t \in \mathbb{R}^n$0, so

$x_t \in \mathbb{R}^n$1

The stationary LQR average cost is

$x_t \in \mathbb{R}^n$2

which equals the squared $x_t \in \mathbb{R}^n$3 norm from $x_t \in \mathbb{R}^n$4 to $x_t \in \mathbb{R}^n$5 under stationarity. A Riccati-like costate $x_t \in \mathbb{R}^n$6 is defined by

$x_t \in \mathbb{R}^n$7

and the policy gradient has the exact form

$x_t \in \mathbb{R}^n$8

The adaptation theory assumes an initial stabilizing controller

$x_t \in \mathbb{R}^n$9

together with persistency of excitation for the block data matrix $u_t \in \mathbb{R}^m$0. In covariance form, with

$u_t \in \mathbb{R}^m$1

the excitation level is quantified by $u_t \in \mathbb{R}^m$2, while the noise energy satisfies

$u_t \in \mathbb{R}^m$3

The signal-to-noise ratio is

$u_t \in \mathbb{R}^m$4

and the least-squares identification error obeys

$u_t \in \mathbb{R}^m$5

For i.i.d. Gaussian noise, $u_t \in \mathbb{R}^m$6, hence $u_t \in \mathbb{R}^m$7 under constant excitation (Zhao et al., 6 May 2025).

2. Indirect estimator construction

The indirect estimator is model-based. It first identifies $u_t \in \mathbb{R}^m$8 online by least squares and then evaluates the policy gradient using the estimated dynamics. In batch form,

$u_t \in \mathbb{R}^m$9

and in recursive form,

$W$0

Persistency of excitation guarantees identifiability.

Given $W$1, the estimator solves the certainty-equivalence Lyapunov equations

$W$2

$W$3

The indirect gradient estimate is then

$W$4

The corresponding vanilla indirect PGAC update is

$W$5

with constant stepsize $W$6 selected so that policy changes remain small enough for sequential stability. In the pseudocode-level implementation, the controller applies

$W$7

where the probing signal $W$8 is used to ensure persistency of excitation, then updates the least-squares model, solves the Lyapunov equations for $W$9, computes the gradient estimate, and performs the gain update (Zhao et al., 6 May 2025).

3. Update geometries: vanilla gradient, natural gradient, and Gauss–Newton

The estimator admits three policy-update geometries.

Variant	Update	Distinctive property
Vanilla gradient	$$z_t= \begin{bmatrix} Q^{1/2} & 0\ 0 & R^{1/2} \end{bmatrix} \begin{bmatrix} x_t\ u_t \end{bmatrix}, \qquad Q\succ 0,\ R\succ 0.$$0	Uses estimated Lyapunov factors directly
Natural gradient	$$z_t= \begin{bmatrix} Q^{1/2} & 0\ 0 & R^{1/2} \end{bmatrix} \begin{bmatrix} x_t\ u_t \end{bmatrix}, \qquad Q\succ 0,\ R\succ 0.$$1	Bridges indirect and direct PGAC
Gauss–Newton	$$z_t= \begin{bmatrix} Q^{1/2} & 0\ 0 & R^{1/2} \end{bmatrix} \begin{bmatrix} x_t\ u_t \end{bmatrix}, \qquad Q\succ 0,\ R\succ 0.$$2	Yields adaptive Hewer iteration at $$z_t= \begin{bmatrix} Q^{1/2} & 0\ 0 & R^{1/2} \end{bmatrix} \begin{bmatrix} x_t\ u_t \end{bmatrix}, \qquad Q\succ 0,\ R\succ 0.$$3

The natural gradient uses the Fisher-like metric of the LQR parameterization and yields

$$z_t= \begin{bmatrix} Q^{1/2} & 0\ 0 & R^{1/2} \end{bmatrix} \begin{bmatrix} x_t\ u_t \end{bmatrix}, \qquad Q\succ 0,\ R\succ 0.$$4

In the indirect variant, $$z_t= \begin{bmatrix} Q^{1/2} & 0\ 0 & R^{1/2} \end{bmatrix} \begin{bmatrix} x_t\ u_t \end{bmatrix}, \qquad Q\succ 0,\ R\succ 0.$$5 are replaced with $$z_t= \begin{bmatrix} Q^{1/2} & 0\ 0 & R^{1/2} \end{bmatrix} \begin{bmatrix} x_t\ u_t \end{bmatrix}, \qquad Q\succ 0,\ R\succ 0.$$6. A central structural result is that, under the covariance parameterization and projection used for direct PGAC, the direct natural-gradient step in $$z_t= \begin{bmatrix} Q^{1/2} & 0\ 0 & R^{1/2} \end{bmatrix} \begin{bmatrix} x_t\ u_t \end{bmatrix}, \qquad Q\succ 0,\ R\succ 0.$$7-space maps exactly to the indirect natural-gradient step in $$z_t= \begin{bmatrix} Q^{1/2} & 0\ 0 & R^{1/2} \end{bmatrix} \begin{bmatrix} x_t\ u_t \end{bmatrix}, \qquad Q\succ 0,\ R\succ 0.$$8-space. A common misconception is therefore that indirect and direct PGAC are intrinsically separate algorithms; the natural-gradient result shows that they are equivalent up to parameterization in this case.

The Gauss–Newton step uses the local curvature metric

$$z_t= \begin{bmatrix} Q^{1/2} & 0\ 0 & R^{1/2} \end{bmatrix} \begin{bmatrix} x_t\ u_t \end{bmatrix}, \qquad Q\succ 0,\ R\succ 0.$$9

At $u_t=Kx_t$0,

$u_t=Kx_t$1

which recovers Hewer’s policy-improvement step. The indirect form therefore yields an adaptive Hewer or adaptive Kleinman iteration:

$u_t=Kx_t$2

followed by

$u_t=Kx_t$3

For $u_t=Kx_t$4, sequential stability is guaranteed only locally near $u_t=Kx_t$5 or under sufficiently small identification error (Zhao et al., 6 May 2025).

4. Regularization under noise and uncertainty

To account for model mismatch induced by noisy data, the indirect estimator can be regularized through a variance-based penalty built from the sample covariance matrix $u_t=Kx_t$6. Writing

$u_t=Kx_t$7

the regularized weights are

$u_t=Kx_t$8

The regularized indirect cost is

$u_t=Kx_t$9

and its gradient is

$F(K)=A+BK$0

where $F(K)=A+BK$1 solves the corresponding regularized Lyapunov equation.

The regularization coefficient is chosen so that

$F(K)=A+BK$2

hence the regularizer magnitude scales like $F(K)=A+BK$3. This preserves vanishing bias while improving robustness against noise-induced model uncertainty and reducing the variance of gradient estimates. The regularized variants retain the same qualitative stability and convergence guarantees as the unregularized schemes when $F(K)=A+BK$4. A plausible implication is that regularization is not an ancillary numerical device but part of the estimator’s asymptotic design, because its scale is tied explicitly to excitation and noise levels rather than tuned independently of them (Zhao et al., 6 May 2025).

5. Stability, convergence, and empirical behavior

The indirect PGAC iterates are sequentially strongly stable if the signal-to-noise ratio is sufficiently large and the stepsize is sufficiently small. For vanilla gradient descent, the stated conditions are

$F(K)=A+BK$5

with analogous bounds for natural gradient and Gauss–Newton. Under these conditions,

$F(K)=A+BK$6

The cost convergence bound for vanilla indirect PGAC is

$F(K)=A+BK$7

while for Gauss–Newton it becomes

$F(K)=A+BK$8

At $F(K)=A+BK$9, the adaptive Hewer iteration enjoys a local quadratic bound near the optimum,

$F(K)$0

The simulations use a marginally unstable Laplacian $F(K)$1, $F(K)$2, $F(K)$3, $F(K)$4, offline sample size $F(K)$5, online probing noise $F(K)$6, and process noise $F(K)$7. The optimality gap decreases empirically at $F(K)$8, whereas the SNR-based theory yields $F(K)$9. One-shot certainty-equivalence can diverge early under noise, whereas the PGAC variants converge smoothly and stably. Over 500 steps, mean runtime over 20 trials is approximately $\Sigma_K$0 for indirect PGAC, $\Sigma_K$1 for natural gradient, $\Sigma_K$2 for Gauss–Newton, $\Sigma_K$3 for direct PGAC, and $\Sigma_K$4 for one-shot certainty-equivalence. Regularization with $\Sigma_K$5 raises convergence percentages from about $\Sigma_K$6 to about $\Sigma_K$7 for the indirect method and about $\Sigma_K$8 for the direct method, while improving the median optimality gap (Zhao et al., 6 May 2025).

6. Scope, limitations, and related uses of the term

The estimator relies on standard LQR assumptions: controllability of $\Sigma_K$9, detectability of $\Sigma_K = F(K)\Sigma_K F(K)^\top + W,$0, an initial stabilizing gain, and sufficient excitation. If excitation is weak, so that $\Sigma_K = F(K)\Sigma_K F(K)^\top + W,$1 is small, or the noise is heavy-tailed or adversarial, so that $\Sigma_K = F(K)\Sigma_K F(K)^\top + W,$2 is large, then $\Sigma_K = F(K)\Sigma_K F(K)^\top + W,$3 degrades, the least-squares model becomes inaccurate, and the policy gradient becomes biased. The prescribed mitigations are probing noise $\Sigma_K = F(K)\Sigma_K F(K)^\top + W,$4, persistency of excitation, regularization with $\Sigma_K = F(K)\Sigma_K F(K)^\top + W,$5, and smaller stepsizes. Gauss–Newton with $\Sigma_K = F(K)\Sigma_K F(K)^\top + W,$6 can produce larger policy jumps and is therefore proved sequentially stable only near the optimum or when the model error is already small (Zhao et al., 6 May 2025).

The phrase “adaptive indirect estimator” is not unique to adaptive control. In time-series econometrics, it refers to local indirect inference for locally stationary models, where the parameter function $\Sigma_K = F(K)\Sigma_K F(K)^\top + W,$7 is estimated by kernel-weighted auxiliary matching at each rescaled time point $\Sigma_K = F(K)\Sigma_K F(K)^\top + W,$8 (Frazier et al., 2019). In continuous-time stochastic processes, robust CARMA estimation uses indirect inference through an auxiliary AR$\Sigma_K = F(K)\Sigma_K F(K)^\top + W,$9 representation and a GM-estimator (Fasen-Hartmann et al., 2018). In Bayesian inverse problems, the term appears in indirect Gaussian sequence models with hierarchical sieve priors and adaptive posterior contraction (Johannes et al., 2015). In differential privacy, adaptive indirect estimation is used to invert clamped-and-noised summary statistics and obtain asymptotically optimal debiased inference (Wang et al., 14 Jul 2025). Related terminology also appears in adaptive filtering for nonlinear stochastic systems, where process and measurement covariances are inferred from data and then inserted into a Kalman-type update (Busch et al., 2014).

This diversity suggests that “adaptive indirect estimator” is best understood as a methodological pattern rather than a single standardized algorithm: an intermediate model, auxiliary statistic, or surrogate inverse map is estimated adaptively from data and then used to perform inference or control indirectly. In the LQR setting, that pattern takes the specific form of recursive model identification followed by policy-gradient control updates on the estimated model (Zhao et al., 6 May 2025).