Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
149 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

The Role of Identification in Data-driven Policy Iteration: A System Theoretic Study (2401.06721v2)

Published 12 Jan 2024 in eess.SY and cs.SY

Abstract: The goal of this article is to study fundamental mechanisms behind so-called indirect and direct data-driven control for unknown systems. Specifically, we consider policy iteration applied to the linear quadratic regulator problem. Two iterative procedures, where data collected from the system are repeatedly used to compute new estimates of the desired optimal controller, are considered. In indirect policy iteration, data are used to obtain an updated model estimate through a recursive identification scheme, which is used in a certainty-equivalent fashion to perform the classic policy iteration update. By casting the concurrent model identification and control design as a feedback interconnection between two algorithmic systems, we provide a closed-loop analysis that shows convergence and robustness properties for arbitrary levels of excitation in the data. In direct policy iteration, data are used to approximate the value function and design the associated controller without requiring the intermediate identification step. After proposing an extension to a recently proposed scheme that overcomes potential identifiability issues, we establish under which conditions this procedure is guaranteed to deliver the optimal controller. Based on these analyses we are able to compare the strengths and limitations of the two approaches, highlighting aspects such as the required samples, convergence properties, and excitation requirement. Simulations are also provided to illustrate the results.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (36)
  1. From model-based control to data-driven control: Survey, classification and perspective. Information Sciences, 235:3–35, 2013. Data-based Control, Decision, Scheduling and Fault Diagnostics.
  2. A. Khaki-Sedigh. An Introduction to Data-Driven Control Systems. Wiley, 2023.
  3. Data-driven control: Theory and applications. In 2023 American Control Conference (ACC), pages 1922–1939. 2023 American Control Conference (ACC), 2023.
  4. Florian Dörfler. Data-driven control: Part two of two: Hot take: Why not go with models? IEEE Control Systems Magazine, 43(6):27–31, 2023.
  5. L. Ljung. System Identification: Theory for the User. Prentice Hall information and system sciences series. Prentice Hall PTR, 1999.
  6. K.J. Åström and B. Wittenmark. Adaptive Control. Dover Books on Electrical Engineering. Dover Publications, 2008.
  7. Benjamin Recht. A tour of reinforcement learning: The view from continuous control. Annual Review of Control, Robotics, and Autonomous Systems, 2(1):253–279, 2019.
  8. Anuradha M. Annaswamy. Adaptive control and intersections with reinforcement learning. Annual Review of Control, Robotics, and Autonomous Systems, 6(1):65–93, 2023.
  9. Bridging direct and indirect data-driven control formulations via regularizations and relaxations. IEEE Transactions on Automatic Control, 68(2):883–897, 2023.
  10. The gap between model-based and model-free methods on the linear quadratic regulator: An asymptotic viewpoint. In Alina Beygelzimer and Daniel Hsu, editors, Proceedings of the Thirty-Second Conference on Learning Theory, volume 99, pages 3036–3083. PMLR, 25–28 Jun 2019.
  11. A. Bhaya and E. Kaszkurewicz. Control Perspectives on Numerical Algorithms and Matrix Problems. Advances in Design and Control. Society for Industrial and Applied Mathematics, 2006.
  12. Convex synthesis of accelerated gradient algorithms. SIAM Journal on Control and Optimization, 59(6):4615–4645, 2021.
  13. Laurent Lessard. The analysis of optimization algorithms: A dissipativity approach. IEEE Control Systems Magazine, 42(3):58–72, 2022.
  14. An analysis of closed-loop stability for linear model predictive control based on time-distributed optimization. IEEE Transactions on Automatic Control, 67(5):2618–2625, 2022.
  15. A lyapunov function for the combined system-optimizer dynamics in inexact model predictive control. Automatica, 134:109901, 2021.
  16. Optimal control. John Wiley & Sons, 2012.
  17. D. Bertsekas. Abstract Dynamic Programming: 3rd Edition. Athena Scientific., 2022.
  18. Structured policy iteration for linear quadratic regulator. In Proceedings of the 37th International Conference on Machine Learning, ICML’20. Proceedings of the 37th International Conference on Machine Learning, JMLR.org, 2020.
  19. Donghwan Lee. Convergence of dynamic programming on the semidefinite cone for discrete-time infinite-horizon lqr. IEEE Transactions on Automatic Control, 67(10):5661–5668, 2022.
  20. Robust policy iteration for continuous-time linear quadratic regulation. IEEE Transactions on Automatic Control, 67(1):504–511, 2022.
  21. Dimitri Bertsekas. Newton’s method for reinforcement learning and model predictive control. Results in Control and Optimization, 7:100121, 2022.
  22. Stability and monotone convergence of generalised policy iteration for discrete-time linear quadratic regulations. International Journal of Control, 89(3):437–450, 2016.
  23. Convergence and sample complexity of gradient methods for the model-free linear–quadratic regulator problem. IEEE Transactions on Automatic Control, 67(5):2435–2450, 2022.
  24. Linear quadratic control using model-free reinforcement learning. IEEE Transactions on Automatic Control, 68(2):737–752, Feb 2023.
  25. On the sample complexity of the linear quadratic regulator. Foundations of Computational Mathematics, 20, 10 2017.
  26. Learning robust lq-controllers using application oriented exploration. IEEE Control Systems Letters, 4(1):19–24, 2020.
  27. The generalised discrete algebraic riccati equation in linear-quadratic optimal control. Automatica, 49(2):471–478, 2013.
  28. G. Hewer. An iterative technique for the computation of the steady state gains for the discrete optimal regulator. IEEE Transactions on Automatic Control, 16(4):382–384, 1971.
  29. G.A Hewer. Analysis of a discrete matrix riccati equation of linear control and kalman filtering. Journal of Mathematical Analysis and Applications, 42(1):226–236, 1973.
  30. The matrix cookbook. Technical University of Denmark, October 2008. Version 20081110.
  31. Convergence and consistency of recursive least squares with variable-rate forgetting. Automatica, 119:109052, 2020.
  32. Eduardo D. Sontag. Input to State Stability: Basic Concepts and Results, pages 163–220. Springer Berlin Heidelberg, Berlin, Heidelberg, 2008.
  33. Input-to-state stability for discrete-time nonlinear systems. Automatica, 37(6):857–869, 2001.
  34. Model-free linear quadratic control via reduction to expert prediction. In International Conference on Artificial Intelligence and Statistics. International Conference on Artificial Intelligence and Statistics, 2018.
  35. The lidskii-mirsky-wielandt theorem – additive and multiplicative versions. Numerische Mathematik, 81, 06 2001.
  36. C.D. Meyer. Matrix Analysis and Applied Linear Algebra. Society for Industrial and Applied Mathematics, 2023.
Citations (3)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets