Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
97 tokens/sec
GPT-4o
53 tokens/sec
Gemini 2.5 Pro Pro
44 tokens/sec
o3 Pro
5 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Accelerated Optimization Landscape of Linear-Quadratic Regulator (2307.03590v3)

Published 7 Jul 2023 in math.OC and cs.LG

Abstract: Linear-quadratic regulator (LQR) is a landmark problem in the field of optimal control, which is the concern of this paper. Generally, LQR is classified into state-feedback LQR (SLQR) and output-feedback LQR (OLQR) based on whether the full state is obtained. It has been suggested in existing literature that both SLQR and OLQR could be viewed as \textit{constrained nonconvex matrix optimization} problems in which the only variable to be optimized is the feedback gain matrix. In this paper, we introduce a first-order accelerated optimization framework of handling the LQR problem, and give its convergence analysis for the cases of SLQR and OLQR, respectively. Specifically, a Lipschiz Hessian property of LQR performance criterion is presented, which turns out to be a crucial property for the application of modern optimization techniques. For the SLQR problem, a continuous-time hybrid dynamic system is introduced, whose solution trajectory is shown to converge exponentially to the optimal feedback gain with Nesterov-optimal order $1-\frac{1}{\sqrt{\kappa}}$ ($\kappa$ the condition number). Then, the symplectic Euler scheme is utilized to discretize the hybrid dynamic system, and a Nesterov-type method with a restarting rule is proposed that preserves the continuous-time convergence rate, i.e., the discretized algorithm admits the Nesterov-optimal convergence order. For the OLQR problem, a Hessian-free accelerated framework is proposed, which is a two-procedure method consisting of semiconvex function optimization and negative curvature exploitation. In a time $\mathcal{O}(\epsilon{-7/4}\log(1/\epsilon))$, the method can find an $\epsilon$-stationary point of the performance criterion; this entails that the method improves upon the $\mathcal{O}(\epsilon{-2})$ complexity of vanilla gradient descent. Moreover, our method provides the second-order guarantee of stationary point.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (45)
  1. T. Polyak, “Some methods of speeding up the convergence of iteration methods,” Ussr Computational Mathematics and Mathematical Physics, vol. 4, no. 5, pp. 1-17, 1964.
  2. Y. Nesterov, “A method of solving a convex programming problem with convergence rate O⁢(1k2)𝑂1superscript𝑘2O\bigl{(}\frac{1}{k^{2}}\bigr{)}italic_O ( divide start_ARG 1 end_ARG start_ARG italic_k start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG ),” Akademii Nauk. Russian Academy of Sciences, vol. 269, no. 3, pp. 543-547, 1983.
  3. W. Su, S. Boyd, and E. Candes, “A differential equation for modeling Nesterov’s accelerated gradient method: theory and insights,” The Journal of Machine Learning Research, vol. 17, no. 6, pp. 5312-5354, 2016.
  4. A. Wibisono, A. Wilson, and M. Jordan, “A variational perspective on accelerated methods in optimization,” Proceedings of the National Academy of Sciences, vol. 113, no. 47, pp. 7351-7358, 2016.
  5. A. Wilson, B. Recht, and M. Jordan, “A Lyapunov analysis of accelerated methods in optimization,” The Journal of Machine Learning Research, vol. 22, no. 1, pp. 5040-5073, 2021.
  6. H. Luo, and L. Chen, “From differential equation solvers to accelerated first-order methods for convex optimization,” Mathematical Programming, vol. 195, no. 1-2, pp. 735-781, 2022.
  7. L. Chen, and H. Luo, “A unified convergence analysis of first order convex optimization methods via strong Lyapunov functions,” arXiv: 2108.00132.
  8. H. Attouch, Z. Chbani, J. Fadili and H. Riahi, “First-order optimization algorithms via inertial systems with Hessian driven damping,” Mathematical Programming, vol. 193, pp. 113-155, 2022.
  9. M. Fazel, R. Ge, S. Kakade, and M. Mesbahi, “Global convergence of policy gradient methods for the linear quadratic regulator,” in International Conference on Machine Learning, July 2018, pp. 1467-1476.
  10. H. Mohammadi, A. Zare, M. Soltanolkotabi, and M. Jovanović, “Convergence and sample complexity of gradient methods for the model-free linear–quadratic regulator problem,” IEEE Transactions on Automatic Control, vol. 67, no. 5, pp. 2435-2450, 2021.
  11. L. Furieri, Y. Zheng and M. Kamgarpour, “Learning the globally optimal distributed LQ regulator,” in Learning for Dynamics and Control, June 2020, pp. 287-297.
  12. H. Feng and J. Lavaei, “Connectivity properties of the set of stabilizing static decentralized controllers,” SIAM Journal on Control and Optimization, vol. 58, no. 5 pp. 2790-2820, 2020.
  13. J. Duan, J. Li, S. Li and L. Zhao, “Optimization landscape of gradient descent for discrete-time static output feedback,” in American Control Conference June 2022, pp. 2932-2937.
  14. J. Duan, W. Cao, Y. Zheng and L. Zhao, “On the optimization landscape of dynamical output feedback linear quadratic control,” IEEE Transactions on Automatic Control, 2023.
  15. J. Perdomo, J. Umenberger, and M. Simchowitz, “Stabilizing dynamical systems via policy gradient methods,” in Advances in Neural Information Processing Systems, Dec. 2021, pp. 29274-29286.
  16. I. Fatkhullin, and B. Polyak, “Optimizing static linear feedback: Gradient method,” SIAM Journal on Control and Optimization, vol. 59, no. 5, pp. 3887-3911, 2021.
  17. H. Feng, and J. Lavaei, “Connectivity properties of the set of stabilizing static decentralized controllers,” SIAM Journal on Control and Optimization, vol. 58, no. 5, 2790-2820, 2020.
  18. Y. Zheng, Y. Tang, and N. Li, “Analysis of the optimization landscape of linear quadratic gaussian (lqg) control,” Mathematical Programming, 2023, DOI: 10.1007/s10107-023-01938-4.
  19. J. Wang, C. Lin, A. Wibisono, and B. Hu, “Provable acceleration of heavy ball beyond quadratics for a class of Polyak-Lojasiewicz functions when the non-convexity is averaged-out,” in International Conference on Machine Learning, June 2022, pp. 22839-22864.
  20. H. Karimi, J. Nutini, and M. Schmidt, “Linear convergence of gradient and proximal-gradient methods under the polyak-łojasiewicz condition,” in Joint European Conference on Machine Learning and Knowledge Discovery in Databases, September 2016, pp. 795-811.
  21. M. Muehlebach, and M. Jordan, “Optimization with momentum: dynamical, control-theoretic, and symplectic perspectives,” The Journal of Machine Learning Research, vol. 22, no. 73, pp. 1-50, 2021.
  22. M. Muehlebach, and M. Jordan, “A dynamical systems perspective on Nesterov acceleration,” In International Conference on Machine Learning, May 2019, pp. 4656-4662.
  23. J. Diakonikolas and M. Jordan, “Generalized momentum-based methods: A Hamiltonian perspective,” SIAM Journal on Optimization, vol. 31, no. 1, pp. 915-944, 2021.
  24. Y. Carmon, J. Duchi, O. Hinder and A. Sidford, “Accelerated methods for nonconvex optimization,” SIAM Journal on Optimization, vol. 28, no. 2, pp. 1751-1772, 2018.
  25. K. Garg, and D. Panagou, “Fixed-time stable gradient flows: Applications to continuous-time optimization,” IEEE Transactions on Automatic Control, vol. 66, no. 5, pp. 2002-2015, 2020.
  26. M. Vaquero, P. Mestres and J. Cortes, “Resource-aware discretization of accelerated optimization flows: the heavy-ball dynamics case,” IEEE Transactions on Automatic Control, vol. 68, no. 4, pp. 2109-2124, 2022.
  27. F. Alvarez, “On the minimizing property of a second order dissipative system in Hilbert spaces,” SIAM Journal on Control and Optimization, vol. 38, no. 4, pp. 1102-1119, 2000.
  28. S. T. Smith, “Geometric optimization methods for adaptive filtering,” Harvard University, 1993.
  29. W. Krichene, A. Bayen and P. L. Bartlett, “Accelerated mirror descent in continuous and discrete time,” in Advances in neural information processing systems, 2015, vol. 28.
  30. H. Attouch and P. Redont, “The second-order in time continuous Newton method,” In Approximation, optimization and mathematical economics, 2001, pp. 25-36.
  31. M. Betancourt, M. I. Jordan and A. C. Wilson, “On symplectic optimization,” arXiv: 1802.03653.
  32. G. Franca, J. Sulam, D. Robinson and R. Vidal, “Conformal symplectic and relativistic optimization,” in Advances in Neural Information Processing Systems, 2020, vol. 33, pp. 16916-16926.
  33. B. Shi, S. S. Du, W. Su and M. I. Jordan, “Acceleration via symplectic discretization of high-resolution differential equations,” in Advances in Neural Information Processing Systems, 2019, vol. 32.
  34. J. Bu, A. Mesbahi, M. Fazel and M. Mesbahi, “LQR through the lens of first order methods: Discrete-time case,” arXiv: 1907.08921.
  35. A. Ilka and N. Murgovski, “Novel results on output-feedback LQR design,” IEEE Transactions on Automatic Control, vol. 68, no. 9, pp. 5187-5200, 2023.
  36. Y. Zheng, M. Kamgarpour, A. Sootla and A. Papachristodoulou, “Distributed design for decentralized control using chordal decomposition and ADMM,” IEEE Transactions on Control of Network Systems, vol. 7, no. 2, pp. 614-626, 2019.
  37. B. Yang, X. Zhao, X. Li and D. Sun, “An accelerated proximal alternating direction method of multipliers for optimal decentralized control of uncertain systems,” arXiv: 2304.11037.
  38. F. Zhao, K. You and T. Basar, “Global convergence of policy gradient primal-dual methods for risk-constrained LQRs,” IEEE Transactions on Automatic Control, vol. 68, no. 5, pp. 2934-2949, 2023.
  39. K. Zhang, B. Hu and T. Basar, “Policy optimization for H2subscript𝐻2H_{2}italic_H start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT linear control with H∞subscript𝐻H_{\infty}italic_H start_POSTSUBSCRIPT ∞ end_POSTSUBSCRIPT robustness guarantee: Implicit regularization and global convergence,” SIAM Journal on Control and Optimization, vol. 59, no. 6, pp. 4081-4109, 2021.
  40. R. Mifflin, “Semismooth and semiconvex functions in constrained optimization,” SIAM Journal on Control and Optimization, vol. 15, no. 6, pp. 959-972, 1977.
  41. V. H. Ngai, and J. P. Penot, “The semiconvex regularization of functions,” SIAM Journal on Optimization, vol. 33, no. 3, pp. 2457-2483, 2023.
  42. D. Goldfarb, “Curvilinear path steplength algorithms for minimization which use directions of negative curvature,” Mathematical Programming, vol. 18, no. 1, pp. 31-40, 1980.
  43. C. Jin, P. Netrapalli and M. I. Jordan, “Accelerated gradient descent escapes saddle points faster than gradient descent,” in Conference On Learning Theory, 2018, pp. 1042-1085.
  44. M. Liu, Z. Li, X. Wang, J. Yi and T. Yang, “Adaptive negative curvature descent with applications in non-convex optimization,” in Advances in Neural Information Processing Systems, 2018, vol. 31.
  45. Y. Zheng, Y. Sun, M. Fazel and N. Li, “Escaping high-order saddles in policy optimization for Linear Quadratic Gaussian (LQG) control,” in IEEE Conference on Decision and Control, 2022, pp. 5329-5334.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (2)
  1. Lechen Feng (2 papers)
  2. Yuan-Hua Ni (20 papers)

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now