Papers
Topics
Authors
Recent
Assistant
AI Research Assistant
Well-researched responses based on relevant abstracts and paper content.
Custom Instructions Pro
Preferences or requirements that you'd like Emergent Mind to consider when generating responses.
Gemini 2.5 Flash
Gemini 2.5 Flash 134 tok/s
Gemini 2.5 Pro 41 tok/s Pro
GPT-5 Medium 24 tok/s Pro
GPT-5 High 25 tok/s Pro
GPT-4o 113 tok/s Pro
Kimi K2 216 tok/s Pro
GPT OSS 120B 428 tok/s Pro
Claude Sonnet 4.5 37 tok/s Pro
2000 character limit reached

Variable Projection Algorithms: Theoretical Insights and A Novel Approach for Problems with Large Residual (2402.13865v2)

Published 21 Feb 2024 in math.OC

Abstract: This paper delves into an in-depth exploration of the Variable Projection (VP) algorithm, a powerful tool for solving separable nonlinear optimization problems across multiple domains, including system identification, image processing, and machine learning. We first establish a theoretical framework to examine the effect of the approximate treatment of the coupling relationship among parameters on the local convergence of the VP algorithm and theoretically prove that the Kaufman's VP algorithm can achieve a similar convergence rate as the Golub & Pereyra's form. These studies fill the gap in the existing convergence theory analysis, and provide a solid foundation for understanding the mechanism of VP algorithm and broadening its application horizons. Furthermore, drawing inspiration from these theoretical revelations, we design a refined VP algorithm for handling separable nonlinear optimization problems characterized by large residual, called VPLR, which boosts the convergence performance by addressing the interdependence of parameters within the separable model and by continually correcting the approximated Hessian matrix to counteract the influence of large residual during the iterative process. The effectiveness of this refined algorithm is corroborated through numerical experimentation.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (61)
  1. An efficient bounded-variable nonlinear least-squares algorithm for embedded mpc. Automatica, 141:110293, 2022.
  2. Robust model predictive control algorithm with variable feedback gains for output tracking. IEEE Transactions on Industrial Electronics, 68(5):4228–4237, 2020.
  3. Modified multi-direction iterative algorithm for separable nonlinear models with missing data. IEEE Signal Processing Letters, 29:1968–1972, 2022.
  4. Sequential stabilizing spline algorithm for linear systems: Eigenvalue approximation and polishing. Automatica, 159:111313, 2024.
  5. Variable projection methods for separable nonlinear inverse problems with general-form tikhonov regularization. Inverse Problems, 2023.
  6. Robust matrix factorization by majorization minimization. IEEE transactions on Pattern Analysis and Machine Intelligence, 40(1):208–220, 2017.
  7. A constrained variable projection reconstruction method for photoacoustic computed tomography without accurate knowledge of transducer responses. IEEE transactions on Medical Imaging, 34(12):2443–2458, 2015.
  8. Hybrid projection methods for solution decomposition in large-scale bayesian inverse problems. SIAM Journal on Scientific Computing, pages S97–S119, 2023.
  9. Generalized rational variable projection with application in ecg compression. IEEE Transactions on Signal Processing, 68:478–492, 2019.
  10. A two-dimensional prony’s method for spectral estimation. IEEE Transactions on Signal Processing, 40(11):2747–2756, 1992.
  11. Vpnet: variable projection networks. International Journal of Neural Systems, 32(01):2150054, 2022.
  12. Cooperative control of uncertain multi-agent systems via distributed gaussian processes. IEEE Transactions on Automatic Control, 68(5):3091–3098, 2023.
  13. Train like a (var) pro: Efficient training of neural networks with variable projection. SIAM Journal on Mathematics of Data Science, 3(4):1041–1066, 2021.
  14. Separable nonlinear least squares: the variable projection method and its applications. Inverse Problems, 19(2):R1, 2003.
  15. Shape and motion from image streams under orthography: a factorization method. International Journal of Computer Vision, 9:137–154, 1992.
  16. Secrets of matrix factorization: Approximations, numerics, manifold optimization and random restarts. In Proceedings of the IEEE International Conference on Computer Vision, pages 4130–4138, 2015.
  17. Sar structure-from-motion via matrix factorization. In IGARSS 2023-2023 IEEE International Geoscience and Remote Sensing Symposium, pages 6967–6970. IEEE, 2023.
  18. Sparse principal component analysis via variable projection. SIAM Journal on Applied Mathematics, 80(2):977–1002, 2020.
  19. Ensemble principal component analysis. arXiv preprint arXiv:2311.01826, 2023.
  20. A variable projection-based algorithm for fault detection and diagnosis. IEEE Transactions on Instrumentation and Measurement, 2023.
  21. Multidirection gradient iterative algorithm: A unified framework for gradient iterative and least squares algorithms. IEEE Transactions on Automatic Control, 67(12):6770–6777, 2022.
  22. System identification of miso fractional systems: parameter and differentiation order estimation. Automatica, 141:110268, 2022.
  23. A novel reduced-order algorithm for rational models based on arnoldi process and krylov subspace. Automatica, 129:109663, 2021.
  24. Dc-distadmm: Admm algorithm for constrained optimization over directed graphs. IEEE Transactions on Automatic Control, 68(9):5365–5380, 2023.
  25. Moritz Hardt. Understanding alternating minimization for matrix completion. In 2014 IEEE 55th Annual Symposium on Foundations of Computer Science, pages 651–660. IEEE, 2014.
  26. Proximal admm for nonconvex and nonsmooth optimization. Automatica, 146:110551, 2022.
  27. Kalman filtering under unknown inputs and norm constraints. Automatica, 133:109871, 2021.
  28. Damped newton algorithms for matrix factorization with missing data. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), volume 2, pages 316–322 vol. 2, 2005.
  29. T Wiberg. Computation of principal components when data are missing. In Proc. of Second Symp. Computational Statistics, pages 229–236, 1976.
  30. On the wiberg algorithm for matrix factorization in the presence of missing components. International Journal of Computer Vision, 72(3):329–337, 2007.
  31. Mm optimization: Proximal distance algorithms, path following, and trust regions. Proceedings of the National Academy of Sciences, 120(27):e2303168120, 2023.
  32. Variable projection for nonsmooth problems. SIAM journal on scientific computing, 43(5):S249–S268, 2021.
  33. The differentiation of pseudo-inverses and nonlinear least squares problems whose variables separate. SIAM Journal on Numerical Analysis, 10(2):413–432, 1973.
  34. Embedded point iteration based recursive algorithm for online identification of nonlinear regression models. IEEE Transactions on Automatic Control, 68:4257–4264, 2022.
  35. A regularized snpom for stable parameter estimation of rbf-ar (x) model. IEEE Transactions on Neural Networks and Learning Systems, 29(4):779–791, 2017.
  36. Online estimation of aluminum electrolytic-capacitor parameters using a modified prony’s method. IEEE Transactions on Industry Applications, 54(5):4764–4774, 2018.
  37. Ecg beat representation and delineation by means of variable projection. IEEE Transactions on Biomedical engineering, 68(10):2997–3008, 2021.
  38. Training two-layered feedforward networks with variable projection method. IEEE Transactions on Neural Networks, 19(2):371–375, 2008.
  39. Nonmonotone variable projection algorithms for matrix decomposition with missing data. Pattern Recognition, page 110150, 2023.
  40. Linda Kaufman. A variable projection method for solving separable nonlinear least squares problems. BIT Numerical Mathematics, 15:49–57, 1975.
  41. A new formulation of the learning problem of a neural network controller. In [1991] Proceedings of the 30th IEEE Conference on Decision and Control, pages 865–866. IEEE, 1991.
  42. Secant variable projection method for solving nonnegative separable least squares problems. Numerical Algorithms, 85:737–761, 2020.
  43. A generalization of variable elimination for separable inverse problems beyond least squares. Inverse Problems, 29(4):045003, 2013.
  44. Insights into algorithms for separable nonlinear least squares problems. IEEE transactions on Image Processing, 30(2):1207–1218, 2021.
  45. Algorithms for separable nonlinear least squares problems. SIAM review, 22(3):318–337, 1980.
  46. Offline state estimation for hybrid systems via nonsmooth variable projection. Automatica, 115:108871, 2020.
  47. Separable non-linear least-squares minimization-possible improvements for neural net fitting. In Neural Networks for Signal Processing VII. Proceedings of the 1997 IEEE Signal Processing Society Workshop, pages 345–354. IEEE, 1997.
  48. Insights into algorithms for separable nonlinear least squares problems. IEEE Transactions on Image Processing, 2020.
  49. Carl T Kelley. Iterative methods for linear and nonlinear equations. SIAM, 1995.
  50. On some separated algorithms for separable nonlinear least squares problems. IEEE Transactions on Cybernetics, 48(10):2866–2874, 2018.
  51. Greedy search method for separable nonlinear models using stage aitken gradient descent and least squares algorithms. IEEE Transactions on Automatic Control, 68(8):5044–5051, 2023.
  52. Efficient quadratic penalization through the partial minimization technique. IEEE Transactions on Automatic Control, 63(7):2131–2138, 2017.
  53. Numerical methods for unconstrained optimization and nonlinear equations. SIAM, 1996.
  54. On a combination of alternating minimization and nesterov’s momentum. In International Conference on Machine Learning, pages 3886–3898. PMLR, 2021.
  55. Let’s make block coordinate descent converge faster: faster greedy rules, message-passing, active-set complexity, and superlinear convergence. Journal of Machine Learning Research, 23(131):1–74, 2022.
  56. Stephen J Wright. Coordinate descent algorithms. Mathematical Programming, 151(1):3–34, 2015.
  57. Exploiting the interpretability and forecasting ability of the rbf-ar model for nonlinear time series. International Journal of Systems Science, 47(8):1868–1876, 2016.
  58. Nonlinear system modeling and predictive control using the rbf nets-based quasi-linear arx model. Control Engineering Practice, 17(1):59–66, 2009.
  59. Rbf-arx model-based nonlinear system modeling and predictive control with application to a nox decomposition process. Control Engineering Practice, 12(2):191–203, 2004.
  60. Generalized exponential autoregressive models for nonlinear time series: Stationarity, estimation and applications. Information Sciences, 438:46–57, 2018.
  61. I-C Yeh. Modeling of strength of high-performance concrete using artificial neural networks. Cement and Concrete Research, 28(12):1797–1808, 1998.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Dice Question Streamline Icon: https://streamlinehq.com

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now
Lightbulb Streamline Icon: https://streamlinehq.com

Continue Learning

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now
List To Do Tasks Checklist Streamline Icon: https://streamlinehq.com

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up