2000 character limit reached
A new framework for constrained optimization via feedback control of Lagrange multipliers (2403.12738v2)
Published 19 Mar 2024 in math.OC, cs.SY, and eess.SY
Abstract: The continuous-time analysis of existing iterative algorithms for optimization has a long history. This work proposes a novel continuous-time control-theoretic framework for equality-constrained optimization. The key idea is to design a feedback control system where the Lagrange multipliers are the control input, and the output represents the constraints. The system converges to a stationary point of the constrained optimization problem through suitable regulation. Regarding the Lagrange multipliers, we consider two control laws: proportional-integral control and feedback linearization. These choices give rise to a family of different methods. We rigorously develop the related algorithms, theoretically analyze their convergence and present several numerical experiments to support their effectiveness concerning the state-of-the-art approaches.
- A. Aggarwal and C.A. Floudas. Synthesis of general distillation sequences-nonsharp separations. Comput. Chemical Eng., 14(6):631–653, 1990.
- Studies in linear and non-linear programming. Stanford University Press, 1958.
- D.P. Bertsekas. Nonlinear Programming. Athena Scientific, Belmont, Massachusetts, 2nd edition, 1999.
- A control-theoretic approach to the design of zero finding numerical methods. IEEE Trans. Autom. Control, 52(6):1014–1026, 2007.
- Distributed optimization and statistical learning via the alternating direction method of multipliers. Found. Trends Mach. Learn., 3(1):1 – 122, 2010.
- A frequency domain philosophy for nonlinear systems, with applications to stabilization and to adaptive control. In IEEE Conf. Decis. Control (CDC), pages 1569–1573, 1984.
- On constrained nonconvex stochastic optimization: A case study for generalized eigenvalue decomposition. In Proc. Int. Conf. Arti. Intell. Stat. (AISTATS), volume 89, pages 916–925, 2019.
- The proximal augmented lagrangian method for nonsmooth composite optimization. IEEE Trans. Autom. Control, 64(7):2861–2868, 2019.
- Global exponential stability of primal-dual gradient flow dynamics based on the proximal augmented lagrangian. In Proc. Amer. Control Conf. (ACC), pages 3414–3419, 2019.
- Continuous vs. discrete optimization of deep neural networks. In Proc. Adv. Neural Infor. Process. Syst. (NeurIPS), volume 34, pages 4947–4960, 2021.
- ADMM and accelerated ADMM as continuous dynamical systems. In Proc. Int. Conf. Mach. Learn. (ICML), pages 1559–1567, 2018.
- Deep Learning. MIT Press, 2016.
- Control interpretations for first-order optimization methods. In Amer. Control Conf. (ACC), pages 3114–3119, 2017.
- Alberto Isidori. Nonlinear Control Systems. Springer London, 1995.
- Alberto Isidori. The zero dynamics of a nonlinear system: From the origin to the latest progresses of a long successful story. Europ. J. Control, 19(5):369–378, 2013.
- H. K. Khalil. Nonlinear Systems. Pearson Education. Prentice Hall, 3rd edition, 2002.
- T. Kose. Solutions of saddle value problems by differential equations. Econometrica, 24(1):59–70, 1956.
- A test-suite of non-convex constrained optimization problems from the real-world and some baseline results. Swarm and Evolutionary Computation, 56:100693, 2020.
- Analysis and design of optimization algorithms via integral quadratic constraints. SIAM J. Optim., 26(1):57–95, 2016.
- Linear and Nonlinear Programming. Springer Publishing Company, Inc., 4th edition, 2015.
- Deyuan Meng. Feedback of control on mathematics: Bettering iterative methods by observer system design. IEEE Trans. Autom. Control, 68(4):2498–2505, 2023.
- Control design for iterative methods in solving linear algebraic equations. IEEE Trans. Autom. Control, 67(10):5039–5054, 2022.
- A dynamical systems perspective on nesterov acceleration. In Proc. Int. Conf. Mach. Learn. (ICML), pages 4656–4662. PMLR, 2019.
- Numerical optimization. Springer, 2006.
- Proximal algorithms. Foundations and Trends in Optimization, 1(3):127–239, 2014.
- Guannan Qu and Na Li. On the exponential stability of primal-dual gradient dynamics. IEEE Control Syst. Lett., 3(1):43–48, 2019.
- Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. In Proc. of the International Conference on Learning Represenatations 2014. International Conference on Learning Represenatations 2014, 2014.
- A dynamical systems approach to constrained minimization. Numer. Funct. Anal. Optim., 21:537–551, 2000.
- A differential equation for modeling Nesterov’s accelerated gradient method: Theory and insights. J. Mach. Learn. Res., 17(153):1–43, 2016.
- Hiroshi Yamashita. A differential equation approach to nonlinear programming. Mathematical Programming, 18(1):155–168, 1980.
- Convergence analysis of a differential equation approach for solving nonlinear programming problems. Applied Mathematics and Computation, 184(2):789–797, 2007.