Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
129 tokens/sec
GPT-4o
28 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

On adaptive stochastic extended iterative methods for solving least squares (2405.19044v1)

Published 29 May 2024 in math.NA and cs.NA

Abstract: In this paper, we propose a novel adaptive stochastic extended iterative method, which can be viewed as an improved extension of the randomized extended Kaczmarz (REK) method, for finding the unique minimum Euclidean norm least-squares solution of a given linear system. In particular, we introduce three equivalent stochastic reformulations of the linear least-squares problem: stochastic unconstrained and constrained optimization problems, and the stochastic multiobjective optimization problem. We then alternately employ the adaptive variants of the stochastic heavy ball momentum (SHBM) method, which utilize iterative information to update the parameters, to solve the stochastic reformulations. We prove that our method converges linearly in expectation, addressing an open problem in the literature related to designing theoretically supported adaptive SHBM methods. Numerical experiments show that our adaptive stochastic extended iterative method has strong advantages over the non-adaptive one.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (62)
  1. On the augmented system approach to sparse least-squares problems. Numerische Mathematik, 55(6):667–684, 1989.
  2. On greedy randomized Kaczmarz method for solving large sparse linear systems. SIAM J. Sci. Comput., 40(1):A592–A606, 2018.
  3. On partially randomized extended Kaczmarz method for solving large sparse overdetermined inconsistent linear systems. Linear Algebra and Its Applications, 578:225–250, 2019.
  4. On greedy randomized augmented Kaczmarz method for solving large sparse inconsistent linear systems. SIAM J. Sci. Comput., 43(6):A3892–A3911, 2021.
  5. Complexity guarantees for Polyak steps with momentum. In Conference on Learning Theory, pages 452–478. PMLR, 2020.
  6. Generalized inverses: theory and applications, volume 15. Springer Science & Business Media, 2003.
  7. On the fast convergence of minibatch heavy ball momentum. arXiv preprint arXiv:2206.07553, 2022.
  8. Charles Byrne. A unified treatment of some iterative algorithms in signal processing and image reconstruction. Inverse Problems, 20(1):103–120, 2003.
  9. LIBSVM: a library for support vector machines. ACM transactions on intelligent systems and technology (TIST), 2(3):1–27, 2011.
  10. Coordinate descent method for large-scale l2-loss linear support vector machines. J. Mach. Learn. Res., 9(7):1369—1398, 2008.
  11. Kui Du. Tight upper bounds for the convergence of the randomized extended Kaczmarz and Gauss-Seidel algorithms. Numer. Linear Algebra Appl., 26(3):e2233, 2019.
  12. Randomized extended average block Kaczmarz for solving least squares. SIAM J. Sci. Comput., 42(6):A3541–A3559, 2020.
  13. Pseudoinverse-free randomized block iterative algorithms for consistent and inconsistent linear systems. arXiv preprint arXiv:2011.10353, 2020.
  14. Ellen H Fukuda and Luis Mauricio Graña Drummond. A survey on multiobjective descent methods. Pesquisa Operacional, 34:585–620, 2014.
  15. Global convergence of the heavy-ball method for convex optimization. In 2015 European control conference (ECC), pages 310–315. IEEE, 2015.
  16. Matrix computations. JHU press, 2013.
  17. Algebraic reconstruction techniques (ART) for three-dimensional electron microscopy and X-ray photography. J. Theor. Biol., 29(3):471–481, 1970.
  18. On adaptive sketch-and-project for solving linear systems. SIAM J. Matrix Anal. Appl., 42(2):954–989, 2021.
  19. Randomized iterative methods for linear systems. SIAM J. Matrix Anal. Appl., 36(4):1660–1690, 2015.
  20. Randomized Douglas-Rachford methods for linear systems: Improved accuracy and efficiency. SIAM J. Optim., 34(1):1045–1070, 2024.
  21. On pseudoinverse-free randomized methods for linear systems: Unified framework and acceleration. arXiv preprint arXiv:2208.05437, 2022.
  22. On the acceleration of Kaczmarz’s method for inconsistent linear systems. Linear Algebra and its Applications, 130:83–98, 1990.
  23. Algebraic reconstruction techniques can be made computationally efficient (positron emission tomography application). IEEE Trans. Medical Imaging, 12(3):600–609, 1993.
  24. An adaptive heavy ball method for ill-posed inverse problems. arXiv preprint arXiv:2404.03218, 2024.
  25. S Karczmarz. Angenäherte auflösung von systemen linearer glei-chungen. Bull. Int. Acad. Pol. Sic. Let., Cl. Sci. Math. Nat., pages 355–357, 1937.
  26. The suitesparse matrix collection website interface. J. Open Source Softw., 4(35):1244, 2019.
  27. Guanghui Lan. First-order and stochastic optimization methods for machine learning. Springer, 2020.
  28. Ji Liu and Stephen Wright. An accelerated randomized Kaczmarz algorithm. Math. Comp., 85(297):153–178, 2016.
  29. Momentum and stochastic momentum for stochastic gradient, newton, proximal point and subspace descent methods. Comput. Optim. Appl., 77(3):653–710, 2020.
  30. Revisiting randomized gossip algorithms: General framework, convergence rates and novel block and accelerated protocols. IEEE Trans. Inform. Theory, 67(12):8300–8324, 2021.
  31. Minimal error momentum Bregman-Kaczmarz. arXiv preprint arXiv:2307.15435, 2023.
  32. Stochastic gradient descent for linear systems with missing data. Numerical Mathematics: Theory, Methods and Applications, 12(1):1–20, 2019.
  33. Convergence properties of the randomized extended Gauss–Seidel and Kaczmarz methods. SIAM Journal on Matrix Analysis and Applications, 36(4):1590–1604, 2015.
  34. Randomized Kaczmarz with averaging. BIT., 61(1):337–359, 2021.
  35. Stochastic steepest descent methods for linear systems: Greedy sampling & momentum. arXiv preprint arXiv:2012.13087, 2020.
  36. Frank Natterer. The mathematics of computerized tomography. SIAM, 2001.
  37. I Necoara. Stochastic block projection algorithms with extrapolation for convex feasibility problems. Optimization Methods and Software, 37(5):1845–1875, 2022.
  38. Ion Necoara. Faster randomized block Kaczmarz algorithms. SIAM J. Matrix Anal. Appl., 40(4):1425–1452, 2019.
  39. Deanna Needell. Randomized Kaczmarz solver for noisy linear systems. BIT Numerical Mathematics, 50(2):395–403, 2010.
  40. Stochastic gradient descent, weighted sampling, and the randomized Kaczmarz algorithm. Mathematical Programming, 155:549–573, 2016.
  41. Paved with good intentions: analysis of a randomized block Kaczmarz method. Linear Algebra and its Applications, 441:199–221, 2014.
  42. Two-subspace projection method for coherent overdetermined systems. Journal of Fourier Analysis and Applications, 19(2):256–269, 2013.
  43. Yurii Nesterov. Introductory lectures on convex optimization: A basic course, volume 87. Springer Science & Business Media, 2003.
  44. Yurii E Nesterov. A method for solving the convex programming problem with convergence rate O(1/k2)1superscript𝑘2(1/k^{2})( 1 / italic_k start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT ). In Dokl. akad. nauk Sssr, volume 269, pages 543–547, 1983.
  45. Boris T Polyak. Some methods of speeding up the convergence of iteration methods. Comput. Math. Math. Phys., 4(5):1–17, 1964.
  46. Constantin Popa. Extensions of block-projections methods with relaxation parameters to inconsistent and rank-deficient least-squares problems. BIT Numerical Mathematics, 38(1):151–176, 1998.
  47. Constantin Popa. Characterization of the solutions set of inconsistent least-squares problems by an extended Kaczmarz algorithm. Korean Journal of Computational and Applied Mathematics, 6(1):51–64, 1999.
  48. Stochastic reformulations of linear systems: Algorithms and convergence theory. SIAM J. Matrix Anal. Appl., 41(2):487–524, 2020.
  49. Stochastic reformulations of linear systems: algorithms and convergence theory. SIAM Journal on Matrix Analysis and Applications, 41(2):487–524, 2020.
  50. An adaptive Polyak heavy-ball method. Machine Learning, 111(9):3245–3277, 2022.
  51. Theory of multiobjective optimization. Elsevier, 1985.
  52. Linear convergence of the randomized sparse Kaczmarz method. Math. Program., 173(1):509–536, 2019.
  53. Almost sure convergence rates for stochastic gradient descent and stochastic heavy ball. In Conference on Learning Theory, pages 3935–3971. PMLR, 2021.
  54. A randomized Kaczmarz algorithm with exponential convergence. J. Fourier Anal. Appl., 15(2):262–278, 2009.
  55. Proximal gradient methods for multiobjective optimization and their applications. Computational Optimization and Applications, 72:339–361, 2019.
  56. On the extended randomized multiple row method for solving linear least-squares problems. arXiv preprint arXiv:2210.03478, 2022.
  57. Semiconvergence analysis of the randomized row iterative method and its extended variants. Numerical Linear Algebra with Applications, 28(1):e2334, 2021.
  58. Wen-Ting Wu. On two-subspace randomized extended Kaczmarz method for solving large linear least-squares problems. Numerical Algorithms, 89(1):1–31, 2022.
  59. Fast stochastic dual coordinate descent algorithms for linearly constrained convex optimization. arXiv preprint arXiv:2307.16702, 2023.
  60. On adaptive stochastic heavy ball momentum for solving linear systems. arXiv preprint arXiv:2305.05482, to appear in SIAM Journal on Matrix Analysis and Applications, 2023.
  61. Randomized kaczmarz method with adaptive stepsizes for inconsistent linear systems. Numerical Algorithms, 94(3):1403–1420, 2023.
  62. Randomized extended Kaczmarz for solving least squares. SIAM J. Matrix Anal. Appl., 34(2):773–793, 2013.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets