Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
169 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Sharp Analysis of Power Iteration for Tensor PCA (2401.01047v1)

Published 2 Jan 2024 in cs.LG, cs.NA, math.NA, and stat.ML

Abstract: We investigate the power iteration algorithm for the tensor PCA model introduced in Richard and Montanari (2014). Previous work studying the properties of tensor power iteration is either limited to a constant number of iterations, or requires a non-trivial data-independent initialization. In this paper, we move beyond these limitations and analyze the dynamics of randomly initialized tensor power iteration up to polynomially many steps. Our contributions are threefold: First, we establish sharp bounds on the number of iterations required for power method to converge to the planted signal, for a broad range of the signal-to-noise ratios. Second, our analysis reveals that the actual algorithmic threshold for power iteration is smaller than the one conjectured in literature by a polylog(n) factor, where n is the ambient dimension. Finally, we propose a simple and effective stopping criterion for power iteration, which provably outputs a solution that is highly correlated with the true signal. Extensive numerical experiments verify our theoretical results.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (63)
  1. Homotopy analysis for tensor pca. In Conference on Learning Theory, pages 79–104. PMLR, 2017.
  2. A tensor spectral approach to learning mixed membership community models. In Conference on Learning Theory, pages 867–881. PMLR, 2013.
  3. Analyzing tensor power method dynamics in overcomplete regime. Journal of Machine Learning Research, 18(22):1–40, 2017.
  4. The landscape of the spiked tensor model. Communications on Pure and Applied Mathematics, 72(11):2282–2330, 2019.
  5. Algorithmic thresholds for tensor pca. 2020.
  6. Phase transition of the largest eigenvalue for nonnull complex sample covariance matrices. 2005.
  7. How to iron out rough landscapes and get optimal performances: averaged gradient descent and its application to tensor pca. Journal of Physics A: Mathematical and Theoretical, 53(17):174003, 2020.
  8. The singular values and vectors of low rank perturbations of large rectangular random matrices. Journal of Multivariate Analysis, 111:120–135, 2012.
  9. The dynamics of message passing on dense graphs, with applications to compressed sensing. IEEE Transactions on Information Theory, 57(2):764–785, 2011.
  10. Efficient tensor completion for color image and video recovery: Low-rank tensor train. IEEE Transactions on Image Processing, 26(5):2466–2479, 2017.
  11. Multilayer tensor factorization with applications to recommender systems. 2018.
  12. Optimal detection of sparse principal components in high dimension. 2013.
  13. Wei-Kuo Chen. Phase transition in the spiked random tensor with rademacher prior. The Annals of Statistics, 47(5):2734–2756, 2019.
  14. Nonconvex low-rank tensor completion from noisy data. Advances in neural information processing systems, 32, 2019.
  15. Statistical query lower bounds for tensor pca. The Journal of Machine Learning Research, 22(1):3729–3779, 2021.
  16. Sparse pca via covariance thresholding. Journal of Machine Learning Research, 17(141):1–41, 2016.
  17. Fundamental limits of detection in the spiked wigner model. 2020.
  18. Tensor methods and recommender systems. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 7(3):e1201, 2017.
  19. David Gamarnik. The overlap gap property: A topological barrier to optimizing over random structures. Proceedings of the National Academy of Sciences, 118(41):e2108492118, 2021.
  20. A literature survey of low-rank tensor approximation techniques. GAMM-Mitteilungen, 36(1):53–78, 2013.
  21. Power iteration for tensor pca. Journal of Machine Learning Research, 23(128):1–47, 2022.
  22. Most tensor problems are np-hard. Journal of the ACM (JACM), 60(6):1–39, 2013.
  23. Tensor principal component analysis via sum-of-square proofs. In Conference on Learning Theory, pages 956–1006. PMLR, 2015.
  24. Fast spectral algorithms from sum-of-squares proofs: tensor decomposition and planted sparse vectors. In Proceedings of the forty-eighth annual ACM symposium on Theory of Computing, pages 178–191, 2016.
  25. Tensor decomposition for multiple-tissue gene expression experiments. Nature genetics, 48(9):1094–1100, 2016.
  26. An optimal statistical and computational framework for generalized tensor estimation. The Annals of Statistics, 50(1):1–29, 2022.
  27. On consistency and sparsity for principal components analysis in high dimensions. Journal of the American Statistical Association, 104(486):682–693, 2009.
  28. Community detection on mixture multilayer networks via regularized tensor decomposition. The Annals of Statistics, 49(6):3181–3205, 2021.
  29. Generalized power method for sparse principal component analysis. Journal of Machine Learning Research, 11(2), 2010.
  30. Iain M Johnstone. On the distribution of the largest eigenvalue in principal components analysis. The Annals of statistics, 29(2):295–327, 2001.
  31. Pca in high dimensions: An orientation. Proceedings of the IEEE, 106(8):1277–1292, 2018.
  32. Community detection in hypergraphs, spiked tensor models, and sum-of-squares. In 2017 International Conference on Sampling Theory and Applications (SampTA), pages 124–128. IEEE, 2017.
  33. Shifted power method for computing tensor eigenpairs. SIAM Journal on Matrix Analysis and Applications, 32(4):1095–1124, 2011.
  34. Tensor-structured galerkin approximation of parametric and stochastic elliptic pdes. SIAM journal on scientific computing, 33(1):364–385, 2011.
  35. The isotropic semicircle law and deformation of wigner matrices. Communications on Pure and Applied Mathematics, 66(11):1663–1749, 2013.
  36. Approximate message passing from random initialization with applications to ℤ2subscriptℤ2\mathbb{Z}_{2}blackboard_Z start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT synchronization. arXiv preprint arXiv:2302.03682, 2023.
  37. Fundamental limits of symmetric low-rank matrix estimation. Probability Theory and Related Fields, 173:859–929, 2019.
  38. Statistical and computational phase transitions in spiked tensor estimation. In 2017 IEEE International Symposium on Information Theory (ISIT), pages 511–515. IEEE, 2017.
  39. Tensor completion for estimating missing values in visual data. IEEE transactions on pattern analysis and machine intelligence, 35(1):208–220, 2012.
  40. A non-asymptotic framework for approximate message passing in spiked models. arXiv preprint arXiv:2208.03313, 2022.
  41. Tensor clustering with planted structures: Statistical optimality and computational limits. The Annals of Statistics, 50(1):584–613, 2022.
  42. Zongming Ma. Sparse principal component analysis and iterative thresholding. 2013.
  43. Spectral algorithms for tensor completion. Communications on Pure and Applied Mathematics, 71(11):2381–2425, 2018.
  44. Estimation of low-rank matrices via approximate message passing. 2021.
  45. Sum-of-squares lower bounds for sparse pca. Advances in Neural Information Processing Systems, 28, 2015.
  46. Fundamental limits of low-rank matrix estimation with diverging aspect ratios. arXiv preprint arXiv:2211.00488, 2022.
  47. A three-way model for collective learning on multi-relational data. In Icml, volume 11, pages 3104482–3104584, 2011.
  48. Exact tensor completion with sum-of-squares. In Conference on Learning Theory, pages 1619–1673. PMLR, 2017.
  49. A statistical model for tensor pca. Advances in neural information processing systems, 27, 2014.
  50. Pairwise interaction tensor factorization for personalized tag recommendation. In Proceedings of the third ACM international conference on Web search and data mining, pages 81–90, 2010.
  51. Finite sample analysis of approximate message passing algorithms. IEEE Transactions on Information Theory, 64(11):7264–7286, 2018.
  52. Tensor decomposition for signal processing and machine learning. IEEE Transactions on signal processing, 65(13):3551–3582, 2017.
  53. Iterative collaborative filtering for sparse noisy tensor estimation. In 2019 IEEE International Symposium on Information Theory (ISIT), pages 41–45. IEEE, 2019.
  54. Online and differentially-private tensor decomposition. Advances in Neural Information Processing Systems, 29, 2016.
  55. Three-way clustering of multi-tissue multi-individual gene expression data using semi-nonnegative tensor decomposition. The annals of applied statistics, 13(2):1103, 2019.
  56. Neurocognitive and neuroimaging correlates of pediatric traumatic brain injury: a diffusion tensor imaging (dti) study. Archives of Clinical Neuropsychology, 22(5):555–568, 2007.
  57. Lower bounds for the convergence of tensor power iteration on random overcomplete models. In The Thirty Sixth Annual Conference on Learning Theory, pages 3783–3820. PMLR, 2023.
  58. On polynomial time methods for exact low-rank tensor completion. Foundations of Computational Mathematics, 19(6):1265–1313, 2019.
  59. Heteroskedastic tensor clustering. arXiv preprint arXiv:2311.02306, 2023.
  60. Fairness-aware tensor-based recommendation. In Proceedings of the 27th ACM international conference on information and knowledge management, pages 1153–1162, 2018.
  61. Tensor low-rank representation for data recovery and clustering. IEEE transactions on pattern analysis and machine intelligence, 43(5):1718–1732, 2019.
  62. Tensor regression with applications in neuroimaging data analysis. Journal of the American Statistical Association, 108(502):540–552, 2013.
  63. Tensor svd: Statistical and computational limits. IEEE Transactions on Information Theory, 64(11):7311–7338, 2018.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now