Data-Driven Continuous-Time Linear Quadratic Regulator via Closed-Loop and Reinforcement Learning Parameterizations

Abstract: This paper studies data-driven approaches to the continuous-time linear quadratic regulator (LQR) problem based on two existing parameterizations, namely a closed-loop (CL) parameterization from behavioral system theory and an integral reinforcement learning (IRL) parameterization. The CL parameterization characterizes the closed-loop system via a matrix that satisfies equality constraints. While this parameterization has been extensively studied for discrete-time systems, we adapt key results to the continuous-time setting and develop a policy iteration (PI) scheme, derive a data-driven continuous-time algebraic Riccati equation (CARE), and introduce an alternative convex problem formulation. The IRL parameterization utilizes off-policy data to perform policy evaluation, which is then used for PI or value iteration. Within the IRL framework, we derive a policy gradient flow and propose convex reformulations of the LQR problem. Finally, we provide a unified treatment of these parameterizations that enables a systematic understanding of existing approaches and clarifies their structural relationships.

• The paper unifies closed-loop and integral reinforcement learning parameterizations for continuous-time LQR synthesis directly from data, bypassing explicit system identification.
• The paper introduces efficient policy iteration and gradient flow methods that guarantee convergence to the optimal feedback under mild data informativity conditions.
• The paper demonstrates that closed-loop methods have lower sample complexity while IRL scales robustly with larger datasets, highlighting practical trade-offs in controller synthesis.

Data-Driven Continuous-Time LQR via Closed-Loop and Reinforcement Learning Parameterizations

Introduction and Context

The paper "Data-Driven Continuous-Time Linear Quadratic Regulator via Closed-Loop and Reinforcement Learning Parameterizations" (2604.27922) presents a comprehensive theoretical and methodological analysis of data-driven controller synthesis for the continuous-time LQR problem. It focuses on two central parameterizations: the closed-loop (CL) parameterization rooted in behavioral systems theory, and the integral reinforcement learning (IRL) parameterization from the data-driven RL perspective. Both frameworks leverage input-state or input-output data directly, thereby bypassing explicit system identification.

The authors provide a detailed comparison, extension, and unification of these frameworks in the continuous-time regime, offering algorithms, convex formulations, policy gradient flows, and systematic connections between the respective approaches.

Problem Statement and Parameterizations

The standard continuous-time LQR problem aims to minimize an infinite-horizon quadratic cost functional for the linear system $\dot{x}(t) = A x(t) + B u(t)$, with cost:

$$J(u) = \int_0^\infty x(t)^T Q x(t) + u(t)^T R u(t) \, dt$$

where $Q \succeq 0$ and $R \succ 0$. Traditionally, the optimal state feedback $u^* = -K^*x$ is computed via the Algebraic Riccati Equation (CARE):

$$A^\top P^* + P^* A - P^* B R^{-1} B^\top P^* + Q = 0$$

yielding $K^* = R^{-1} B^\top P^*$. In data-driven control, $A$, $B$ need not be known—the aim is direct controller synthesis from data.

Closed-Loop (CL) Parameterization

The CL parameterization characterizes the closed-loop system by expressing the dynamics in terms of data-driven quantities, specifically as $\dot{x} = \bar{X} G x$ with equality constraints coupling $$J(u) = \int_0^\infty x(t)^T Q x(t) + u(t)^T R u(t) \, dt$$0, $$J(u) = \int_0^\infty x(t)^T Q x(t) + u(t)^T R u(t) \, dt$$1, and the aggregated input/state integrals. Notably, the paper adapts these techniques to remove reliance on state-derivative information using time integrals, addressing a major practical bottleneck.

Integral Reinforcement Learning (IRL) Parameterization

The IRL parameterization employs off-policy data to estimate value functions, leading to least-squares policy evaluation without knowledge of either $$J(u) = \int_0^\infty x(t)^T Q x(t) + u(t)^T R u(t) \, dt$$2 or $$J(u) = \int_0^\infty x(t)^T Q x(t) + u(t)^T R u(t) \, dt$$3. The key step is reformulating the continuous-time Bellman equation as an integral identity, facilitating model-free learning and controller improvement.

Unification and Analytical Properties

The authors provide a unified mathematical framework showing that both CL and IRL parameterizations can recover the optimal LQR policy under mild data informativity assumptions, though with different sample complexity, computational overhead, and robustness characteristics.

• Sample efficiency: CL requires fewer data samples ($$J(u) = \int_0^\infty x(t)^T Q x(t) + u(t)^T R u(t) \, dt$$4) compared to IRL ($$J(u) = \int_0^\infty x(t)^T Q x(t) + u(t)^T R u(t) \, dt$$5), where $$J(u) = \int_0^\infty x(t)^T Q x(t) + u(t)^T R u(t) \, dt$$6 and $$J(u) = \int_0^\infty x(t)^T Q x(t) + u(t)^T R u(t) \, dt$$7 are state and input dimensions.
• Decision variable geometry: In CL, the feasible set $$J(u) = \int_0^\infty x(t)^T Q x(t) + u(t)^T R u(t) \, dt$$8 is relatively open and path-connected in an affine subspace, while IRL's feasible set coincides with the set of stabilizing gains $$J(u) = \int_0^\infty x(t)^T Q x(t) + u(t)^T R u(t) \, dt$$9.
• Non-uniqueness: The CL approach has a non-unique solution in its variable $Q \succeq 0$0 (distinct $Q \succeq 0$1 yield the same $Q \succeq 0$2), while IRL's solution in $Q \succeq 0$3 is unique.

Policy Iteration and Policy Evaluation

Both parameterizations admit data-driven forms of policy iteration (PI), inspired by Kleinman's algorithm. The CL algorithm iteratively alternates between Lyapunov-based evaluation and constrained minimization for policy improvement in the $Q \succeq 0$4 variable. In IRL, PI consists of repeated least-squares identification of the value matrix and direct gain update.

Strong convergence results show that, in both cases, the iterates converge to the optimal feedback $Q \succeq 0$5 and value matrix $Q \succeq 0$6, provided a stabilizing initial controller is available. The computational complexity per PI iteration is, however, lower for CL (Lyapunov equation) than for IRL (large least-squares problem).

Riccati Flows and Value Iteration

The paper formulates data-driven continuous-time Riccati flows for both parameterizations:

• In CL, substituting optimal policy expressions into policy evaluation leads to a data-driven CARE (with coefficients expressed in data integrals), and thus enables Riccati flow or VI.
• In IRL, the Riccati flow and value iteration are constructed from lifted equations involving value function increments observed through trajectory data segments.

Both flows are shown to converge to $Q \succeq 0$7 and allow recovery of $Q \succeq 0$8. Practical considerations include sample complexity, computational scalability, and numerical accuracy.

Convex Optimization Formulations

The authors present several convex SDP relaxations for data-driven LQR synthesis:

• CL-based SDPs: Convexification via variable changes (e.g., $Q \succeq 0$9) and Schur complements, resulting in tractable SDPs for computing $R \succ 0$0 and $R \succ 0$1 directly from data.
• IRL-based SDPs: Relaxing the fundamental (generally nonconvex) value-matching equations by introducing auxiliary variables and Schur complements, yielding SDPs whose constraints scale linearly with sample size, while LMIs remain of manageable size.

A succinct comparison of computational costs, variable counts, constraint sizes, and practical implementation trade-offs is provided.

Figure 1: Normalized residuals in $R \succ 0$2 for PI and gradient flow under CL and IRL parameterizations across randomized trials, demonstrating similar average convergence but minor accuracy differences.

Figure 2: Convergence of $R \succ 0$3-residuals in VI and Riccati flow for both parameterizations, showing near-identical convergence rates but higher numerical accuracy in the CL case.

Figure 3: Residuals in recovered $R \succ 0$4 for convex programs (varied formulations, CL and IRL) across all runs, highlighting variability and consistency between methods.

Policy Gradient Flows

Both parameterizations permit policy search via gradient flows:

• CL: Gradient descent on the data-driven cost $R \succ 0$5, with enforcement of constraints via projection onto the feasible affine subspace, yielding unique equilibria corresponding to optimal $R \succ 0$6. Non-coercivity in $R \succ 0$7 necessitates careful regularization for guaranteeing bounded trajectories.
• IRL: Standard policy gradient, where the gradient expression is constructed through data-driven least-squares quantities; trajectories initialized in $R \succ 0$8 remain stable and converge globally to $R \succ 0$9 without the need for explicit projection.

The projected gradient flows theoretically guarantee convergence to the LQR optimum under natural conditions.

Numerical Results

Extensive experiments corroborate the theoretical claims:

• PI and gradient flow methods show comparable convergence under both parameterizations, with minor differences in numerical accuracy.
• The Riccati flow and VI methods produce matching trajectories in both CL and IRL, with computational timing comparable except for certain SDPs whose constraint size can grow with the data.
• Convex programs exhibit small residuals for all tested systems, with their practical performance depending on the parameterization and specific data realization.
• Table-based summary indicates that CL-based methods offer better computational efficiency at lower sample complexity, while IRL-based methods can scale more robustly with large datasets due to fixed LMI size.

Implications and Future Directions

The rigorous unification and comparative analysis provided clarify when to prefer CL or IRL parameterizations based on available data, computational resources, and problem context. For high-dimensional systems or sample-constrained scenarios, CL enjoys clear advantages, while IRL-based approaches are more modular for extending to broader RL frameworks.

Practically, these results imply that high-quality LQR controllers can be synthesized with statistical guarantees using only data, without ever explicitly estimating system matrices. The methods promote sample efficiency, numerical reliability, and algorithmic flexibility, opening up new prospects for real-time adaptive control and reinforcement learning in continuous-time settings.

Potential future developments include extensions for robust, noise-resilient control synthesis (stochastic formulations, regularization), online adaptive implementations without stabilizing initialization, and application to broader classes of optimal control problems beyond LQR, including nonlinear and constrained systems.

Conclusion

This work systematically develops and unifies data-driven controller synthesis frameworks for continuous-time LQR using both closed-loop and reinforcement learning parameterizations. The theoretical insights, algorithmic contributions, and extensive numerical validation provide a comprehensive toolkit for implementing efficient and reliable LQR synthesis directly from measured trajectories. It sets the stage for further advances in model-free and data-driven optimal control for complex dynamical systems.