Papers
Topics
Authors
Recent
Search
2000 character limit reached

Matrices with Gaussian noise: optimal estimates for singular subspace perturbation

Published 2 Mar 2018 in stat.ML, cs.IT, cs.LG, math.IT, and math.PR | (1803.00679v3)

Abstract: The Davis-Kahan-Wedin $\sin \Theta$ theorem describes how the singular subspaces of a matrix change when subjected to a small perturbation. This classic result is sharp in the worst case scenario. In this paper, we prove a stochastic version of the Davis-Kahan-Wedin $\sin \Theta$ theorem when the perturbation is a Gaussian random matrix. Under certain structural assumptions, we obtain an optimal bound that significantly improves upon the classic Davis-Kahan-Wedin $\sin \Theta$ theorem. One of our key tools is a new perturbation bound for the singular values, which may be of independent interest.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (77)
  1. Entrywise eigenvector analysis of random matrices with low expected rank. Ann. Statist., 48(3):1452ā€“1474, 2020.
  2. R.Ā Allez and J.-P. Bouchaud. Eigenvector dynamics under free addition. Random Matrices Theory Appl., 3(3):1450010, 17, 2014.
  3. Z.Ā Bai and J.Ā W. Silverstein. Spectral analysis of large dimensional random matrices. Springer Series in Statistics. Springer, New York, second edition, 2010.
  4. Z.Ā Bai and J.-f. Yao. Central limit theorems for eigenvalues in a spiked population model. Ann. Inst. Henri PoincarĆ© Probab. Stat., 44(3):447ā€“474, 2008.
  5. Phase transition of the largest eigenvalue for nonnull complex sample covariance matrices. Ann. Probab., 33(5):1643ā€“1697, 2005.
  6. Statistical inference for principal components of spiked covariance matrices. Ann. Statist., 50(2):1144ā€“1169, 2022.
  7. Singular vector and singular subspace distribution for the matrix denoising model. The Annals of Statistics, 49(1):370ā€“392, 2021.
  8. Z.Ā Bao and D.Ā Wang. Eigenvector distribution in the critical regime of BBP transition. Probab. Theory Related Fields, 182(1-2):399ā€“479, 2022.
  9. Eigenvectors of a matrix under random perturbation. Random Matrices Theory Appl., 10(2):Paper No. 2150023, 17, 2021.
  10. Fluctuations of the extreme eigenvalues of finite rank deformations of random matrices. Electron. J. Probab., 16:no. 60, 1621ā€“1662, 2011.
  11. F.Ā Benaych-Georges and R.Ā R. Nadakuditi. The eigenvalues and eigenvectors of finite, low rank perturbations of large random matrices. Adv. Math., 227(1):494ā€“521, 2011.
  12. F.Ā Benaych-Georges and R.Ā R. Nadakuditi. The singular values and vectors of low rank perturbations of large rectangular random matrices. J. Multivariate Anal., 111:120ā€“135, 2012.
  13. R.Ā Bhatia. Matrix analysis, volume 169 of Graduate Texts in Mathematics. Springer-Verlag, New York, 1997.
  14. A.Ā Bjƶrck and G.Ā H. Golub. Numerical methods for computing angles between linear subspaces. Math. Comp., 27:579ā€“594, 1973.
  15. Isotropic local laws for sample covariance and generalized Wigner matrices. Electron. J. Probab., 19:no. 33, 53, 2014.
  16. On the principal components of sample covariance matrices. Probability theory and related fields, 164(1):459ā€“552, 2016.
  17. Optimal estimation and rank detection for sparse spiked covariance matrices. Probab. Theory Related Fields, 161(3-4):781ā€“815, 2015.
  18. E.Ā J. CandĆØs and Y.Ā Plan. Matrix completion with noise. Proceedings of the IEEE, 98:925ā€“936, 2010.
  19. E.Ā J. CandĆØs and B.Ā Recht. Exact matrix completion via convex optimization. Found. Comput. Math., 9(6):717ā€“772, 2009.
  20. Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information. IEEE Trans. Inform. Theory, 52(2):489ā€“509, 2006.
  21. Signal-plus-noise matrix models: eigenvector deviations and fluctuations. Biometrika, 106(1):243ā€“250, 2019.
  22. The two-to-infinity norm and singular subspace geometry with applications to high-dimensional statistics. Ann. Statist., 47(5):2405ā€“2439, 2019.
  23. Central limit theorems for eigenvalues of deformations of Wigner matrices. Ann. Inst. Henri PoincarĆ© Probab. Stat., 48(1):107ā€“133, 2012.
  24. Stochastic block model and community detection in sparse graphs: A spectral algorithm with optimal rate of recovery. In P.Ā GrĆ¼nwald, E.Ā Hazan, and S.Ā Kale, editors, Proceedings of The 28th Conference on Learning Theory, volumeĀ 40 of Proceedings of Machine Learning Research, pages 391ā€“423, Paris, France, 03ā€“06 Jul 2015. PMLR.
  25. C.Ā Davis and W.Ā M. Kahan. The rotation of eigenvectors by a perturbation. III. SIAM J. Numer. Anal., 7:1ā€“46, 1970.
  26. Unperturbed: spectral analysis beyond Davis-Kahan. In Algorithmic learning theory 2018, volumeĀ 83 of Proc. Mach. Learn. Res. (PMLR), pageĀ 38. Proceedings of Machine Learning Research PMLR, [place of publication not identified], 2018.
  27. Bulk universality for generalized Wigner matrices. Probab. Theory Related Fields, 154(1-2):341ā€“407, 2012.
  28. Rigidity of eigenvalues of generalized Wigner matrices. Adv. Math., 229(3):1435ā€“1515, 2012.
  29. Spectral statistics of erdős-rĆ©nyi graphs ii: Eigenvalue spacing and the extreme eigenvalues. Communications in Mathematical Physics, 314(3):587ā€“640, 2012.
  30. Spectral statistics of erdős-rĆ©nyi graphs I: Local semicircle law. Ann. Probab., 41(3B):2279ā€“2375, 2013.
  31. J.Ā Fan and X.Ā Han. Estimation of the false discovery proportion with unknown dependence. J. R. Stat. Soc. Ser. B. Stat. Methodol., 79(4):1143ā€“1164, 2017.
  32. Large covariance estimation by thresholding principal orthogonal complements. J. R. Stat. Soc. Ser. B. Stat. Methodol., 75(4):603ā€“680, 2013. With 33 discussions by 57 authors and a reply by Fan, Liao and Mincheva.
  33. An ā„“āˆžsubscriptā„“\ell_{\infty}roman_ā„“ start_POSTSUBSCRIPT āˆž end_POSTSUBSCRIPT eigenvector perturbation bound and its application to robust covariance estimation. J. Mach. Learn. Res., 18:Paper No. 207, 42, 2017.
  34. Matrix computations. JHU press, 2013.
  35. O.Ā GuĆ©don and R.Ā Vershynin. Community detection in sparse networks via Grothendieckā€™s inequality. Probab. Theory Related Fields, 165(3-4):1025ā€“1049, 2016.
  36. Isotropic self-consistent equations for mean-field random matrices. Probab. Theory Related Fields, 171(1-2):203ā€“249, 2018.
  37. N.Ā J. Higham. A survey of componentwise perturbation theory in numerical linear algebra. In Mathematics of Computation 1943ā€“1993: a half-century of computational mathematics (Vancouver, BC, 1993), volumeĀ 48 of Proc. Sympos. Appl. Math., pages 49ā€“77. Amer. Math. Soc., Providence, RI, 1994.
  38. Matrix analysis. Cambridge University Press, Cambridge, second edition, 2013.
  39. I.Ā C.Ā F. Ipsen and R.Ā Rehman. Perturbation bounds for determinants and characteristic polynomials. SIAM J. Matrix Anal. Appl., 30(2):762ā€“776, 2008.
  40. I.Ā T. Jolliffe. Principal component analysis. Springer Series in Statistics. Springer-Verlag, New York, 1986.
  41. R.Ā Kannan and S.Ā Vempala. Spectral algorithms. Found. Trends Theor. Comput. Sci., 4(3-4):front matter, 157ā€“288 (2009), 2008.
  42. A.Ā Knowles and J.Ā Yin. The isotropic semicircle law and deformation of Wigner matrices. Comm. Pure Appl. Math., 66(11):1663ā€“1750, 2013.
  43. A.Ā Knowles and J.Ā Yin. Anisotropic local laws for random matrices. Probab. Theory Related Fields, 169(1-2):257ā€“352, 2017.
  44. V.Ā Koltchinskii and D.Ā Xia. Perturbation of linear forms of singular vectors under Gaussian noise. In High dimensional probability VII, volumeĀ 71 of Progr. Probab., pages 397ā€“423. Springer, [Cham], 2016.
  45. J.Ā O. Lee and K.Ā Schnelli. Local law and Tracy-Widom limit for sparse random matrices. Probab. Theory Related Fields, 171(1-2):543ā€“616, 2018.
  46. Bulk universality for deformed Wigner matrices. Ann. Probab., 44(3):2349ā€“2425, 2016.
  47. Reconstruction in the labelled stochastic block model. IEEE Trans. Network Sci. Eng., 2(4):152ā€“163, 2015.
  48. F.Ā McSherry. Spectral partitioning of random graphs. In 42nd IEEE Symposium on Foundations of Computer Science (Las Vegas, NV, 2001), pages 529ā€“537. IEEE Computer Soc., Los Alamitos, CA, 2001.
  49. B.Ā Nadler. Finite sample approximation results for principal component analysis: a matrix perturbation approach. Ann. Statist., 36(6):2791ā€“2817, 2008.
  50. Eigenvectors of random matrices: A survey. Journal of Combinatorial Theory, Series A, 144:361 ā€“ 442, 2016. Fifty Years of the Journal of Combinatorial Theory.
  51. Random perturbation of low rank matrices: improving classical bounds. Linear Algebra Appl., 540:26ā€“59, 2018.
  52. S.Ā Oā€™Rourke and N.Ā Williams. Pairing between zeros and critical points of random polynomials with independent roots. Trans. Amer. Math. Soc., 371(4):2343ā€“2381, 2019.
  53. C.Ā Pozrikidis. An introduction to grids, graphs, and networks. Oxford University Press, Oxford, 2014.
  54. J.Ā W.Ā S. Rayleigh, Baron. The Theory of Sound. Dover Publications, New York, N. Y., 1945. 2d ed.
  55. Spectral clustering and the high-dimensional stochastic blockmodel. Ann. Statist., 39(4):1878ā€“1915, 2011.
  56. M.Ā Rudelson and R.Ā Vershynin. Non-asymptotic theory of random matrices: extreme singular values. In Proceedings of the International Congress of Mathematicians. Volume III, pages 1576ā€“1602. Hindustan Book Agency, New Delhi, 2010.
  57. P.Ā Sarkar and P.Ā J. Bickel. Role of normalization in spectral clustering for stochastic blockmodels. Ann. Statist., 43(3):962ā€“990, 2015.
  58. E.Ā Schrƶdinger. Quantisierung als eigenwertproblem. Annalen der Physik, 384(4):361ā€“376, 1926.
  59. Matrix perturbation theory. Computer Science and Scientific Computing. Academic Press, Inc., Boston, MA, 1990.
  60. A semiparametric two-sample hypothesis testing problem for random graphs. J. Comput. Graph. Statist., 26(2):344ā€“354, 2017.
  61. Asymptotically efficient estimators for stochastic blockmodels: the naive MLE, the rank-constrained MLE, and the spectral estimator. Bernoulli, 28(2):1049ā€“1073, 2022.
  62. T.Ā Tao. Outliers in the spectrum of iid matrices with bounded rank perturbations. Probab. Theory Related Fields, 155(1-2):231ā€“263, 2013.
  63. C.Ā Tomasi and T.Ā Kanade. Shape and motion from image streams: a factorization method. Proceedings of the National Academy of Sciences, 90(21):9795ā€“9802, 1993.
  64. R.Ā Vershynin. High-dimensional probability: An introduction with applications in data science, volumeĀ 47. Cambridge university press, 2018.
  65. U.Ā von Luxburg. A tutorial on spectral clustering. Stat. Comput., 17(4):395ā€“416, 2007.
  66. V.Ā Vu. Singular vectors under random perturbation. Random Structures Algorithms, 39(4):526ā€“538, 2011.
  67. V.Ā Vu. A simple SVD algorithm for finding hidden partitions. Combin. Probab. Comput., 27(1):124ā€“140, 2018.
  68. M.Ā J. Wainwright. High-dimensional statistics: A non-asymptotic viewpoint, volumeĀ 48. Cambridge University Press, 2019.
  69. R.Ā Wang. Singular vector perturbation under Gaussian noise. SIAM J. Matrix Anal. Appl., 36(1):158ā€“177, 2015.
  70. P.-A. Wedin. Perturbation bounds in connection with singular value decomposition. Nordisk Tidskr. Informationsbehandling (BIT), 12:99ā€“111, 1972.
  71. D.Ā Xia and F.Ā Zhou. The sup-norm perturbation of HOSVD and low rank tensor denoising. J. Mach. Learn. Res., 20:Paper No. 61, 42, 2019.
  72. A useful variant of the Davis-Kahan theorem for statisticians. Biometrika, 102(2):315ā€“323, 2015.
  73. S.-Y. Yun and A.Ā Proutiere. Accurate community detection in the stochastic block model via spectral algorithms. Available at arXiv:1412.7335, 2014.
  74. S.-Y. Yun and A.Ā Proutiere. Optimal cluster recovery in the labeled stochastic block model. In D.Ā Lee, M.Ā Sugiyama, U.Ā Luxburg, I.Ā Guyon, and R.Ā Garnett, editors, Advances in Neural Information Processing Systems, volumeĀ 29. Curran Associates, Inc., 2016.
  75. Empirical Bayes PCA in high dimensions. J. R. Stat. Soc. Ser. B. Stat. Methodol., 84(3):853ā€“878, 2022.
  76. Y.Ā Zhong. Eigenvector under random perturbation: A nonasymptotic rayleigh-schrƶdinger theory. Available at arXiv:1702.00139, 2017.
  77. Y.Ā Zhong and N.Ā Boumal. Near-optimal bounds for phase synchronization. SIAM J. Optim., 28(2):989ā€“1016, 2018.
Citations (5)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Sign Up to Summarize

Paper to Video (Beta)

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Authors (3)

  1. Sean O'Rourke 
  2. Van Vu 
  3. Ke Wang 

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up