2000 character limit reached
An L-BFGS-B approach for linear and nonlinear system identification under $\ell_1$ and group-Lasso regularization (2403.03827v3)
Published 6 Mar 2024 in eess.SY, cs.LG, cs.SY, and math.OC
Abstract: In this paper, we propose a very efficient numerical method based on the L-BFGS-B algorithm for identifying linear and nonlinear discrete-time state-space models, possibly under $\ell_1$ and group-Lasso regularization for reducing model complexity. For the identification of linear models, we show that, compared to classical linear subspace methods, the approach often provides better results, is much more general in terms of the loss and regularization terms used (such as penalties for enforcing system stability), and is also more stable from a numerical point of view. The proposed method not only enriches the existing set of linear system identification tools but can also be applied to identifying a very broad class of parametric nonlinear state-space models, including recurrent neural networks. We illustrate the approach on synthetic and experimental datasets and apply it to solve a challenging industrial robot benchmark for nonlinear multi-input/multi-output system identification. A Python implementation of the proposed identification method is available in the package jax-sysid, available at https://github.com/bemporad/jax-sysid.
- Input convex neural networks. In Proc. 34th Int. Conf. on Machine Learning. Proceedings of Machine Learning Research, volume 70, pages 146–155, Sydney, Australia, 2017.
- G. Andrew and J. Gao. Scalable training of ℓ1subscriptℓ1\ell_{1}roman_ℓ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT-regularized log-linear models. In Proc. 24th Int. Conf. on Machine Learning, pages 33–40, 2007.
- A Levenberg-Marquardt method for nonsmooth regularized least squares. 2023. https://arxiv.org/abs/2301.02347.
- An open-source system identification package for multivariable processes. In UKACC 12th Int. Conf. Control, pages 152–157, Sheffield, UK, 2018.
- Nonlinear state-space identification using deep encoder networks. In Proc. Machine Learning Research, volume 144, pages 241–250, 2021.
- A. Bemporad. Training recurrent neural networks by sequential least squares and the alternating direction method of multipliers. 2021. Submitted for publication. Available on http://arxiv.org/abs/2112.15348.
- A. Bemporad. Training recurrent neural networks by sequential least squares and the alternating direction method of multipliers. 2022. Uploaded on http://arxiv.org/abs/2112.15348 [v3].
- A. Bemporad. Recurrent neural network training with convex loss and regularization functions by extended Kalman filtering. IEEE Transactions on Automatic Control, 68(9):5661–5668, 2023.
- A. Bemporad. Training recurrent neural-network models on the industrial robot dataset under ℓ1subscriptℓ1\ell_{1}roman_ℓ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT-regularization. In 7th Workshop on Nonlinear System Identification Benchmarks, Eindhoven, The Netherlands, April 2023.
- A. Bemporad. Training recurrent neural networks by sequential least squares and the alternating direction method of multipliers. Automatica, 156:111183, October 2023.
- Efficient and modular implicit differentiation. arXiv preprint arXiv:2105.15183, 2021.
- Predictive control for linear and hybrid systems. Cambridge University Press, 2017.
- JAX: composable transformations of Python+NumPy programs, 2018.
- Gradient sampling methods for nonsmooth optimization. In A. Bagirov et al., editor, Numerical Nonsmooth Optimization, pages 201–225. 2020. http://www.cs.nyu.edu/overton/software/hanso/.
- A limited memory algorithm for bound constrained optimization. SIAM Journal on Scientific Computing, 16(5):1190–1208, 1995.
- A BFGS-SQP method for nonsmooth, nonconvex, constrained optimization and its evaluation using relative minimization profiles. Optimization Methods and Software, 32(1):148–181, 2017. http://www.timmitchell.com/software/GRANSO/.
- D.P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
- D.C. Liu and J. Nocedal. On the limited memory BFGS method for large scale optimization. Mathematical programming, 45(1-3):503–528, 1989.
- L. Ljung. System Identification : Theory for the User. Prentice Hall, 2 edition, 1999.
- L. Ljung. System Identification Toolbox for MATLAB. The Mathworks, Inc., 2001. https://www.mathworks.com/help/ident.
- Deep learning and system identification. IFAC-PapersOnLine, 53(2):1175–1181, 2020.
- D. Masti and A. Bemporad. Learning nonlinear state-space models using autoencoders. Automatica, 129:109666, 2021.
- Model Predictive Control: Theory and Design. Nob Hill Publishing, LCC, Madison,WI, 2 edition, 2018.
- A unifying theorem for three subspace system identification algorithms. Automatica, 31(12):1853–1864, 1995.
- N4SID: Subspace algorithms for the identification of combined deterministic-stochastic systems. Automatica, 30(1):75–93, 1994.
- Offset-free MPC explained: novelties, subtleties, and applications. IFAC-PapersOnLine, 48(23):342–351, 2015.
- Deep networks for system identification: a survey. arXiv preprint 2301.12832, 2023.
- Neurocontrol of nonlinear dynamical systems with Kalman filter trained recurrent networks. IEEE Transactions on Neural Networks, 5(2):279–297, 1994.
- Recent advances in recurrent neural networks. 2017. https://arxiv.org/abs/1801.01078.
- S. Särkkä and L. Svensson. Bayesian filtering and smoothing, volume 17. Cambridge University Press, 2023.
- Projected Newton-type methods in machine learning. In S. Sra, S. Nowozin, and S.J. Wright, editors, Optimization for Machine Learning, pages 305–329. MIT Press, 2012.
- J. Schoukens and L. Ljung. Nonlinear system identification: A user-oriented road map. IEEE Control Systems Magazine, 39(6):28–99, 2019.
- Cascaded tanks benchmark combining soft and hard nonlinearities. Technical report, Eindhoven University of Technology, 2016.
- Forward–backward quasi-Newton methods for nonsmooth optimization problems. Computational Optimization and Applications, 67(3):443–487, 2017. https://github.com/kul-optec/ForBES.
- Dataset and baseline for an industrial robot identification benchmark. 2022. https://www.nonlinearbenchmark.org/benchmarks/industrial-robot.
- M. Yuan and Y. Lin. Model selection and estimation in regression with grouped variables. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 68(1):49–67, 2006.
Sponsor
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.
