2000 character limit reached
Actor-Critic Physics-informed Neural Lyapunov Control (2403.08448v2)
Published 13 Mar 2024 in cs.LG, cs.RO, cs.SY, and eess.SY
Abstract: Designing control policies for stabilization tasks with provable guarantees is a long-standing problem in nonlinear control. A crucial performance metric is the size of the resulting region of attraction, which essentially serves as a robustness "margin" of the closed-loop system against uncertainties. In this paper, we propose a new method to train a stabilizing neural network controller along with its corresponding Lyapunov certificate, aiming to maximize the resulting region of attraction while respecting the actuation constraints. Crucial to our approach is the use of Zubov's Partial Differential Equation (PDE), which precisely characterizes the true region of attraction of a given control policy. Our framework follows an actor-critic pattern where we alternate between improving the control policy (actor) and learning a Zubov function (critic). Finally, we compute the largest certifiable region of attraction by invoking an SMT solver after the training procedure. Our numerical experiments on several design problems show consistent and significant improvements in the size of the resulting region of attraction.
- S. Li, E. Öztürk, C. De Wagter, G. C. De Croon, and D. Izzo, “Aggressive online control of a quadrotor via deep network representations of optimality principles,” in 2020 IEEE International Conference on Robotics and Automation (ICRA), pp. 6282–6287, IEEE, 2020.
- X. Yang, D. Liu, and Y. Huang, “Neural-network-based online optimal control for uncertain non-linear continuous-time systems with control constraints,” IET Control Theory & Applications, vol. 7, no. 17, pp. 2037–2047, 2013.
- J. Zhang, Q. Zhu, and W. Lin, “Neural stochastic control,” Advances in Neural Information Processing Systems, vol. 35, pp. 9098–9110, 2022.
- E. Kaufmann, L. Bauersfeld, A. Loquercio, M. Müller, V. Koltun, and D. Scaramuzza, “Champion-level drone racing using deep reinforcement learning,” Nature, vol. 620, no. 7976, pp. 982–987, 2023.
- Wiley-interscience New York, 1972.
- A. A. Ahmadi and A. Majumdar, “Some applications of polynomial optimization in operations research and real-time decision making,” Optimization Letters, vol. 10, pp. 709–729, 2016.
- A. Abate, D. Ahmed, M. Giacobbe, and A. Peruffo, “Formal synthesis of lyapunov neural networks,” IEEE Control Systems Letters, vol. 5, no. 3, pp. 773–778, 2020.
- N. Gaby, F. Zhang, and X. Ye, “Lyapunov-net: A deep neural network architecture for lyapunov function approximation,” in 2022 IEEE 61st Conference on Decision and Control (CDC), pp. 2091–2096, IEEE, 2022.
- L. Grüne, “Computing lyapunov functions using deep neural networks,” arXiv preprint arXiv:2005.08965, 2020.
- Y.-C. Chang, N. Roohi, and S. Gao, “Neural lyapunov control,” Advances in neural information processing systems, vol. 32, 2019.
- W. Jin, Z. Wang, Z. Yang, and S. Mou, “Neural certificates for safe control policies,” arXiv preprint arXiv:2006.08465, 2020.
- R. Zhou, T. Quartz, H. D. Sterck, and J. Liu, “Neural lyapunov control of unknown nonlinear systems with stability guarantees,” 2022.
- C. Barrett and C. Tinelli, Satisfiability modulo theories. Springer, 2018.
- N. Boffi, S. Tu, N. Matni, J.-J. Slotine, and V. Sindhwani, “Learning stability certificates from data,” in Conference on Robot Learning, pp. 1341–1350, PMLR, 2021.
- M. Raissi, P. Perdikaris, and G. E. Karniadakis, “Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations,” Journal of Computational physics, vol. 378, pp. 686–707, 2019.
- J. Wu, A. Clark, Y. Kantaros, and Y. Vorobeychik, “Neural lyapunov control for discrete-time systems,” Advances in Neural Information Processing Systems, vol. 36, 2024.
- V. Konda and J. Tsitsiklis, “Actor-critic algorithms,” Advances in neural information processing systems, vol. 12, 1999.
- R. S. Sutton, A. G. Barto, et al., “Introduction to reinforcement learning. vol. 135,” 1998.
- Y. Meng, R. Zhou, A. Mukherjee, M. Fitzsimmons, C. Song, and J. Liu, “Physics-informed neural network policy iteration: Algorithms, convergence, and verification,” arXiv preprint arXiv:2402.10119, 2024.
- R. Leake and R.-W. Liu, “Construction of suboptimal control sequences,” SIAM Journal on Control, vol. 5, no. 1, pp. 54–63, 1967.
- W. M. Haddad and V. Chellaboina, Nonlinear Dynamical Systems and control: A Lyapunov-based approach. Princeton University, 2008.
- A. Vannelli and M. Vidyasagar, “Maximal lyapunov functions and domains of attraction for autonomous nonlinear systems,” Automatica, vol. 21, no. 1, pp. 69–80, 1985.
- US Atomic Energy Commission, 1961.
- W. Kang, K. Sun, and L. Xu, “Data-driven computational methods for the domain of attraction and zubov’s equation,” IEEE Transactions on Automatic Control, 2023.
- J. Liu, Y. Meng, M. Fitzsimmons, and R. Zhou, “Physics-informed neural network lyapunov functions: Pde characterization, learning, and verification,” arXiv preprint arXiv:2312.09131, 2023.
- A. D. Ames, X. Xu, J. W. Grizzle, and P. Tabuada, “Control barrier function based quadratic programs for safety critical systems,” IEEE Transactions on Automatic Control, vol. 62, no. 8, pp. 3861–3876, 2016.
- A. Agrawal, B. Amos, S. Barratt, S. Boyd, S. Diamond, and J. Z. Kolter, “Differentiable convex optimization layers,” Advances in neural information processing systems, vol. 32, 2019.
- S. Gao, S. Kong, and E. M. Clarke, “dreal: An smt solver for nonlinear theories over the reals,” in Automated Deduction–CADE-24: 24th International Conference on Automated Deduction, Lake Placid, NY, USA, June 9-14, 2013. Proceedings 24, pp. 208–214, Springer, 2013.