Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
184 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

On Quantile Randomized Kaczmarz for Linear Systems with Time-Varying Noise and Corruption (2403.19874v2)

Published 28 Mar 2024 in math.NA and cs.NA

Abstract: Large-scale systems of linear equations arise in machine learning, medical imaging, sensor networks, and in many areas of data science. When the scale of the systems are extreme, it is common for a fraction of the data or measurements to be corrupted. The Quantile Randomized Kaczmarz (QRK) method is known to converge on large-scale systems of linear equations $A\mathbf{x}=\mathbf{b}$ that are inconsistent due to static corruptions in the measurement vector $\mathbf{b}$. We prove that QRK converges even for systems corrupted by time-varying perturbations. Additionally, we prove that QRK converges up to a convergence horizon on systems affected by time-varying noise and corruption. Finally, we utilize Markov's inequality to prove a lower bound on the probability that the largest entries of the QRK residual reveal the time-varying corruption in each iteration. We present numerical experiments which illustrate our theoretical results.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (45)
  1. Byzantine stochastic gradient descent. Advances in Neural Information Processing Systems, 31, 2018.
  2. The power of localization for efficiently learning linear separators with noise. In Proceedings of the forty-sixth annual ACM symposium on Theory of computing, pages 449–458, 2014.
  3. Consistent robust regression. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
  4. Machine learning with adversaries: Byzantine tolerant gradient descent. Advances in neural information processing systems, 30, 2017.
  5. S. Boyd and L. Vandenberghe. Convex optimization. Cambridge university press, 2004.
  6. Learning from untrusted data. In Proceedings of the 49th Annual ACM SIGACT Symposium on Theory of Computing, pages 47–60, 2017.
  7. X. Chen and A. Powell. Almost sure convergence of the Kaczmarz algorithm with random measurements. J. Fourier Anal. Appl., pages 1–20, 2012. 10.1007/s00041-012-9237-2.
  8. On block accelerations of quantile randomized kaczmarz for corrupted systems of linear equations. Inverse Problems, 2022. To appear.
  9. Robust estimators in high-dimensions without the computational intractability. SIAM Journal on Computing, 48(2):742–864, 2019.
  10. Sever: A robust meta-algorithm for stochastic optimization. In International Conference on Machine Learning, pages 1596–1606. PMLR, 2019.
  11. B. Dumitrescu. On the relation between the randomized extended Kaczmarz algorithm and coordinate descent. BIT, pages 1–11, 2014.
  12. Y. C. Eldar and D. Needell. Acceleration of randomized Kaczmarz method via the Johnson-Lindenstrauss lemma. Numer. Algorithms, 58(2):163–177, 2011.
  13. Algebraic reconstruction techniques (ART) for three-dimensional electron microscopy and X-ray photography. J. Theoret. Biol., 29:471–481, 1970.
  14. On subsampled quantile randomized Kaczmarz. In Proc. Allerton Conf. on Communication, Control, and Computing, 2023.
  15. Quantile-based iterative methods for corrupted systems of linear equations. SIAM Journal on Matrix Analysis and Applications, 43(2):605–637, 2022.
  16. G. Herman and L. Meyer. Algebraic reconstruction techniques can be made computationally efficient. IEEE T. Med. Imaging, 12(3):600–609, 1993.
  17. B. Jarman and D. Needell. QuantileRK: Solving large-scale linear systems with corrupted, noisy data. 2021.
  18. S. Kaczmarz. Angenäherte auflösung von systemen linearer gleichungen. Bull. Int. Acad. Polon. Sci. Lett. Ser. A, pages 335–357, 1937.
  19. Agnostic estimation of mean and covariance. In 2016 IEEE 57th Annual Symposium on Foundations of Computer Science (FOCS), pages 665–674. IEEE, 2016.
  20. The folded normal distribution. Technometrics, 3(4):543–550, 1961.
  21. Distributed randomized kaczmarz for the adversarial workers. arXiv preprint arXiv:2203.00095, 2022.
  22. An asynchronous parallel randomized Kaczmarz algorithm. arXiv preprint arXiv:1401.4780, 2014.
  23. Convergence properties of the randomized extended Gauss–Seidel and Kaczmarz methods. SIAM J. Matrix Anal. A., 36(4):1590–1604, 2015.
  24. N. F. Marshall and O. Mickelin. An optimal scheduled learning rate for a randomized kaczmarz algorithm. SIAM Journal on Matrix Analysis and Applications, 44(1):312–330, 2023.
  25. Randomized Kaczmarz with averaging. BIT, pages 1–23, 2020.
  26. Accelerated sampling Kaczmarz Motzkin algorithm for the linear feasibility problem. J. Global Optim., pages 1–22, 2019.
  27. D. Needell. Randomized Kaczmarz solver for noisy linear systems. BIT, 50(2):395–403, 2010.
  28. Stochastic gradient descent and the randomized Kaczmarz algorithm. Math. Program. A, 155(1):549–573, 2016.
  29. D. Needell and J. A. Tropp. Paved with good intentions: Analysis of a randomized block Kaczmarz method. Linear Algebra Appl., 2013.
  30. C. Popa. A fast Kaczmarz-Kovarik algorithm for consistent least-squares problems. Korean J. Comput. Appl. Math., 8(1):9–26, 2001.
  31. C. Popa. A Kaczmarz-Kovarik algorithm for symmetric ill-conditioned matrices. An. Ştiinţ. Univ. Ovidius Constanţa Ser. Mat., 12(2):135–146, 2004.
  32. On Kaczmarz’s projection iteration as a direct solver for linear least squares problems. Linear Algebra Appl., 436(2):389–404, 2012.
  33. Robust estimation via robust gradient estimation. Journal of the Royal Statistical Society Series B, 82(3):601–627, 2020.
  34. H. Robbins and S. Monro. A stochastic approximation method. Ann. Math. Statist., 22:400–407, 1951.
  35. P. J. Rousseeuw. Least median of squares regression. Journal of the American statistical association, 79(388):871–880, 1984.
  36. Y. Saad. Iterative methods for sparse linear systems. SIAM, 2003.
  37. Dynamic fine-grained localization in ad-hoc networks of sensors. In Proceedings of the 7th annual international conference on Mobile computing and networking, pages 166–179, 2001.
  38. Choosing the sample with lowest loss makes SGD robust. In S. Chiappa and R. Calandra, editors, Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, volume 108 of Proceedings of Machine Learning Research, pages 2120–2130. PMLR, 26–28 Aug 2020.
  39. S. Steinerberger. Quantile-based random Kaczmarz for corrupted linear systems of equations. Information and Inference: A Journal of the IMA, 12(1):448–465, 2023.
  40. T. Strohmer and R. Vershynin. A randomized Kaczmarz algorithm with exponential convergence. J. Fourier Anal. Appl., 15(2):262–278, 2009.
  41. T. Strohmer and R. Vershynin. A randomized Kaczmarz algorithm with exponential convergence. Journal of Fourier Analysis and Applications, 15(2):262, 2009.
  42. Adaptive bregman-kaczmarz: An approach to solve linear inverse problems with independent noise exactly. arXiv preprint arXiv:2309.06186, 2023.
  43. J. Á. Víšek. The least trimmed squares. Part I: Consistency. Kybernetika, 42(1):1–36, 2006.
  44. A. Zouzias and N. M. Freris. Randomized extended Kaczmarz for solving least squares. SIAM J. Matrix Anal. A., 34(2):773–793, 2013.
  45. A. Zouzias and N. M. Freris. Randomized gossip algorithms for solving Laplacian systems. In 2015 European Control Conference (ECC), pages 1920–1925. IEEE, 2015.
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now