Adaptive Linear Quadratic Control of Unknown Linear Time-Varying Systems via Policy Gradient Methods

Abstract: Unknown linear time-varying (LTV) systems require the control policy to adapt from online closed-loop data as dynamics evolve. Existing methods usually update the policy by solving a one-shot optimization problem, which can be computationally demanding and sensitive to noisy model estimates. In this paper, we propose a policy gradient adaptive control (PGAC) method for LTV system control with unknown model parameters. Specifically, PGAC integrates online policy optimization into feedback by updating the state-feedback policy with one-step gradient descent of the linear quadratic regulator cost at each time instant. This incremental update is computationally light and naturally limits policy variations caused by noisy data. To explicitly compute the policy gradient online, we estimate local models from recent closed-loop trajectories using normalized sliding-window least-squares. We provide stability and convergence certificates of PGAC for two classes of LTV systems. For slowly time-varying systems, we prove that the closed-loop system achieves practical exponential stability without a dwell-time condition. For piecewise-constant LTV systems, we establish practical stability through a dwell-time contraction argument. We also provide average frozen-time optimality-gap bounds of the policy sequence for both classes. Finally, we validate the effectiveness of our method via numerical case studies of both LTV and nonlinear systems.

• The paper presents a first-order policy gradient adaptive control method that incrementally refines state feedback gains using online model estimation.
• The method achieves practically exponential stability in slowly time-varying systems and guarantees interval-wise performance under abrupt mode switches.
• Empirical evaluations, including tests on a nonlinear quadrotor, confirm the approach’s computational efficiency and robustness to noise and parameter variations.

Adaptive Linear Quadratic Control of Unknown Linear Time-Varying Systems via Policy Gradient Methods

Introduction

This paper addresses the adaptive regulation of linear time-varying (LTV) systems with unknown and evolving parameters, a challenging and practically relevant problem in modern control. Standard data-driven approaches for LTI systems, both direct and indirect, do not transfer naturally to such settings due to the non-stationarity of the dynamics and the requirement for continuous adaptation from online, closed-loop data. Recent LTV control methods often utilize one-shot optimization-based updates per time step (e.g., solving SDPs or Riccati equations), which are computationally demanding and can induce instability under noisy or inaccurate model estimates.

This work contributes a first-order, lightweight Policy Gradient Adaptive Control (PGAC) method for online synthesis of LQR policies in unknown LTV systems. The methodology incrementally refines the state feedback gain via online policy gradient steps computed using normalized, sliding-window least-squares model identification from recent closed-loop trajectories. This architecture directly addresses the dual challenge of computational tractability and robustness to temporal model variation and measurement noise. The approach is analyzed for two canonical classes of LTV dynamics: slowly time-varying systems and piecewise-constant (mode-switching) systems. Non-asymptotic stability guarantees and bounds on the frozen-time optimality gap are established for both.

Methodology

Policy Gradient Adaptive Control Algorithm

The PGAC algorithm interleaves online model identification with incremental policy optimization:

• Model Estimation: At each time step, the current input/state-to-next-state map is estimated via normalized least-squares on recent closed-loop trajectories within a sliding window. Normalization ensures identification error bounds that are independent of the unboundedness of state trajectories, overcoming key analysis obstacles of ordinary least squares in adaptive scenarios.
• Policy Update: The feedback gain is refined via one-step descent of the surrogate (certainty-equivalent) LQR cost associated with the estimated local model. The policy gradient has a closed-form expression derived from Lyapunov equations for the estimated system. Policy variation is tightly regulated by the gradient stepsize, mitigating instability from noisy estimates.
• Excitation and Robustness: Persistency of excitation is ensured through bounded, noisy probing signals superposed on the control input. This design secures identifiability of the local models throughout the adaptive process.

The algorithm does not require system identification with an explicit state bound—a major advantage in time-varying settings—and all computational operations are efficiently scalable.

Theoretical Results

Guarantees for Slowly Time-Varying LTV Systems

The stability and performance analysis leverages sequential stability notions adapted from [cohen2019learning]. Under uniform bounds on system matrix rate-of-change, excitation, noise, and a suitably small stepsize, PGAC produces a sequence of policies yielding practically exponential stability: the closed-loop state norm decays exponentially to a ball whose size depends on the probing and process noise.

After each paragraph referencing figures, insert the first relevant figure:

Figure 1: Spectral radius $\rho(A_t)$ of the slowly time-varying system and the state norm $\|x_t\|$ under the PGAC controller with bounded uniform noise.

A quantitative bound on the average frozen-time optimality gap is also provided. It exhibits explicit decomposition into a vanishing transient term, a drift-induced term scaling with model variation, and a noise/amalgamation-induced bias, all controlled through analytic functions of the excitation and adaptation parameters.

Figure 2: Frozen-time optimality gap of PGAC and the fixed nominal LQR controller for the slowly time-varying system with bounded uniform noise.

Importantly, these results do not require a dwell-time assumption (i.e., a lower bound on the interval between changes), differentiating them from most prior works.

Guarantees for Piecewise-Constant LTV Systems

When the system switches abruptly among a finite (or countably infinite) set of modes according to a piecewise-constant law, generally with large parameter jumps, global sequential stability fails. The analysis proceeds via a two-layer approach:

• Within-interval stability: On each dwell interval of the mode, the closed-loop system is shown to be sequentially stable and thus exhibits time-step-wise exponential decay.
• Across-switch contraction: By imposing a dwell-time lower bound (each mode is active for a minimum time), it is shown that the maximum state norm contracts across intervals, yielding interval-wise practical exponential stability.

The average frozen-time optimality gap is controlled via the number/frequency of switches and the estimation error, extending non-asymptotic optimality tracking to switched LTV systems.

Practical Evaluation

Experiments on LTV Benchmarks

The methodology is evaluated on high-dimensional LTV testbeds, including both slowly varying and mode-switching linear systems, with structured process/measurement noise and persistently excited control inputs. In each regime, the PGAC policy maintains closed-loop stability and tracks time-varying optimal policies:

• In the slowly-varying setting, the state norm rapidly converges to a small noise-determined neighborhood of the origin even when the open-loop system is intermittently unstable (Figure 1). The performance gap to the frozen-time optimum remains uniformly small and tracks the underlying variation (Figure 2).
• In the piecewise-constant scenario, the state remains bounded after abrupt mode switches, and the method adapts in each new frozen mode, maintaining near-optimal cost.

Adaptive Control for Nonlinear Quadrotor

The framework is further validated on a nonlinear planar quadrotor subject to time-varying wind and inertia disturbances, where the linear time-varying model is only locally valid. The PGAC approach, applied with online local linearization and identification, successfully stabilizes the vehicle state and adapts to exogenous environmental changes that destabilize fixed-gain designs.

Figure 3: State norm of the nonlinear planar quadrotor under persistent square-wave disturbance and bounded uniform process noise. PGAC adapts from closed-loop data using normalized least-squares, while the baseline is a fixed nominal LQR controller.

Figure 4: Wind and inertia profiles used in the nonlinear planar quadrotor simulation.

Discussion and Implications

This work establishes the viability of first-order policy gradient methods, with online local identification, as a practical and theoretically analyzable approach for the adaptive control of unknown LTV systems. The structure of PGAC offers several advantages:

• Computational efficiency: Updates require only model estimation (with recursive least-squares) and Lyapunov equation solutions for gradient calculation, offering major reductions compared to online Riccati/SDP solvers.
• Robustness to noise/model variation: Continuous, small-step adaptation prevents destabilizing policy changes, a known weakness of one-shot, optimization-based adaptive control methods.
• Unified analysis: The method provides both high-probability bounds and deterministic guarantees applicable to both continuous and switching time-variation regimes.

Directions for Further Research

The proposed framework motivates extensions in several directions:

• Stochastic variants: High-probability stability and optimality guarantees under stochastic (rather than bounded, deterministic) process/noise models.
• Alternative objective functions: Incorporation and analysis of discounted or finite-horizon cost surrogates for rapidly varying or high-uncertainty environments.
• Direct data-driven policy optimization: Extending similar guarantees to direct policy optimization frameworks (e.g., DeePO) that avoid local model estimation altogether.
• Nonlinear adaptive control: Rigorous extension to systems where the LTV model is only ever a local approximation, with theoretical guarantees.

Conclusion

This paper presents a first-order, online-adaptive LQR control strategy for unknown linear time-varying systems, equipped with practical stability and optimality certificates for both slowly drifting and abruptly switching dynamics. Empirical results, including on nonlinear platforms, substantiate the effectiveness and robustness of the proposed approach, opening new possibilities for truly adaptive, data-driven control in time-varying and uncertain environments (2607.03251).