A Class of Convex Optimization-Based Recursive Algorithms for Identification of Stochastic Systems (2405.07483v1)
Abstract: Focusing on identification, this paper develops a class of convex optimization-based criteria and correspondingly the recursive algorithms to estimate the parameter vector $\theta{*}$ of a stochastic dynamic system. Not only do the criteria include the classical least-squares estimator but also the $L_l=|\cdot|l, l\geq 1$, the Huber, the Log-cosh, and the Quantile costs as special cases. First, we prove that the minimizers of the convex optimization-based criteria converge to $\theta{*}$ with probability one. Second, the recursive algorithms are proposed to find the estimates, which minimize the convex optimization-based criteria, and it is shown that these estimates also converge to the true parameter vector with probability one. Numerical examples are given, justifying the performance of the proposed algorithms including the strong consistency of the estimates, the robustness against outliers in the observations, and higher efficiency in online computation compared with the kernel-based regularization method due to the recursive nature.
- K. J. Åström and P. Eykhoff, “System identification a survey,” Automatica, vol. 7, no. 2, pp. 123–162, 1971.
- L. Ljung, “System Identification,” in Signal Analysis and Prediction. Springer, 1998, pp. 163–173.
- G. Pillonetto and G. De Nicolao, “A new kernel-based approach for linear system identification,” Automatica, vol. 46, no. 1, pp. 81–93, 2010.
- T. Chen, H. Ohlsson, and L. Ljung, “On the estimation of transfer functions, regularizations and Gaussian processes-Revisited,” Automatica, vol. 48, no. 8, pp. 1525–1535, 2012.
- G. Pillonetto, F. Dinuzzo, T. Chen, G. De Nicolao, and L. Ljung, “Kernel methods in system identification, machine learning and function estimation: a survey,” Automatica, vol. 50, no. 3, pp. 657–682, 2014.
- K. Chaudhuri, C. Monteleoni, and A. D. Sarwate, “Differentially private empirical risk minimization.” Journal of Machine Learning Research, vol. 12, no. 3, 2011.
- P. Jain and A. G. Thakurta, “(Near) dimension independent risk bounds for differentially private learning,” in International Conference on Machine Learning. PMLR, 2014, pp. 476–484.
- P. Jain and A. Thakurta, “Differentially private learning with kernels,” in International Conference on Machine Learning. PMLR, 2013, pp. 118–126.
- L. Ljung, “Convergence analysis of parametric identification methods,” IEEE Transactions on Automatic Control, vol. 23, no. 5, pp. 770–783, 1978.
- B. Mu, T. Chen, and L. Ljung, “On asymptotic properties of hyperparameter estimators for kernel-based regularization methods,” Automatica, vol. 94, pp. 381–395, 2018.
- B. Vau and T.-B. Airimitoaie, “Recursive identification with regularization and on-line hyperparameters estimation,” arXiv preprint arXiv:2401.00097, 2023.
- T. Murata and T. Suzuki, “Doubly accelerated stochastic variance reduced dual averaging method for regularized empirical risk minimization,” Advances in Neural Information Processing Systems, vol. 30, 2017.
- M. Schmidt, N. Le Roux, and F. Bach, “Minimizing finite sums with the stochastic average gradient,” Mathematical Programming, vol. 162, no. 1, pp. 83–112, 2017.
- Q. Wang, Y. Ma, K. Zhao, and Y. Tian, “A comprehensive survey of loss functions in machine learning,” Annals of Data Science, vol. 9, no. 2, pp. 187–212, 2022.
- W.-X. Zhao, H.-F. Chen, and W. X. Zheng, “Recursive identification for nonlinear ARX systems based on stochastic approximation algorithm,” IEEE Transactions on Automatic Control, vol. 55, no. 6, pp. 1287–1299, 2010.
- W. Zhao, E. Weyer, and G. Yin, “A general framework for nonparametric identification of nonlinear stochastic systems,” IEEE Transactions on Automatic Control, vol. 66, no. 6, pp. 2449–2464, 2020.
- T. Chen, “On kernel design for regularized LTI system identification,” Automatica, vol. 90, pp. 109–122, 2018.
- T. Chen, H. Ohlsson, G. C. Goodwin, and L. Ljung, “Kernel selection in linear system identification part ii: A classical perspective,” in 2011 50th IEEE conference on Decision and Control and European Control Conference. IEEE, 2011, pp. 4326–4331.
- I. Diakonikolas, G. Kamath, D. Kane, J. Li, J. Steinhardt, and A. Stewart, “Sever: A robust meta-algorithm for stochastic optimization,” in International Conference on Machine Learning. PMLR, 2019, pp. 1596–1606.
- Y. A. Davydov, “Mixing conditions for markov chains,” Teoriya Veroyatnostei i ee Primeneniya, vol. 18, no. 2, pp. 321–338, 1973.
- E. Masry and L. Györfi, “Strong consistency and rates for recursive probability density estimators of stationary processes,” Journal of Multivariate Analysis, vol. 22, no. 1, pp. 79–93, 1987.