Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
153 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Low-rank Tensor Estimation via Riemannian Gauss-Newton: Statistical Optimality and Second-Order Convergence (2104.12031v4)

Published 24 Apr 2021 in stat.ML, cs.LG, cs.NA, math.NA, math.OC, and stat.ME

Abstract: In this paper, we consider the estimation of a low Tucker rank tensor from a number of noisy linear measurements. The general problem covers many specific examples arising from applications, including tensor regression, tensor completion, and tensor PCA/SVD. We consider an efficient Riemannian Gauss-Newton (RGN) method for low Tucker rank tensor estimation. Different from the generic (super)linear convergence guarantee of RGN in the literature, we prove the first local quadratic convergence guarantee of RGN for low-rank tensor estimation in the noisy setting under some regularity conditions and provide the corresponding estimation error upper bounds. A deterministic estimation error lower bound, which matches the upper bound, is provided that demonstrates the statistical optimality of RGN. The merit of RGN is illustrated through two machine learning applications: tensor regression and tensor SVD. Finally, we provide the simulation results to corroborate our theoretical findings.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (116)
  1. Optimization algorithms on matrix manifolds. Princeton University Press, 2009.
  2. Tensor regression using low-rank and sparse tucker decompositions. SIAM Journal on Mathematics of Data Science, 2(4):944–966, 2020.
  3. Tensor decompositions for learning latent variable models. The Journal of Machine Learning Research, 15(1):2773–2832, 2014a.
  4. Guaranteed non-orthogonal tensor decomposition via alternating rank-1111 updates. arXiv preprint arXiv:1402.5180, 2014b.
  5. Tensor methods for additive index models under discordance and heterogeneity. arXiv preprint arXiv:1807.06693, 2018.
  6. Noisy tensor completion via the sum-of-squares hierarchy. In Conference on Learning Theory, pages 417–445, 2016.
  7. Rank 2r iterative least squares: efficient recovery of ill-conditioned low rank matrices from few entries. SIAM Journal on Mathematics of Data Science, 3(1):439–465, 2021.
  8. Multilayer tensor factorization with applications to recommender systems. The Annals of Statistics, 46(6B):3308–3333, 2018.
  9. Peter J Bickel. One-step huber estimates in the linear model. Journal of the American Statistical Association, 70(350):428–434, 1975.
  10. Nicolas Boumal. An introduction to optimization on smooth manifolds. http://sma.epfl.ch/ nboumal/#book, 2020.
  11. Low-rank matrix completion via preconditioned optimization on the Grassmann manifold. Linear Algebra and its Applications, 475:200–239, 2015.
  12. Rtrmc: A Riemannian trust-region method for low-rank matrix completion. In Advances in neural information processing systems, pages 406–414, 2011.
  13. Linear systems with a canonical polyadic decomposition constrained solution: Algorithms and applications. Numerical Linear Algebra with Applications, 25(6):e2190, 2018.
  14. Convergence analysis of Riemannian Gauss–Newton methods and its connection with the geometric condition number. Applied Mathematics Letters, 78:42–50, 2018.
  15. Reducibility and statistical-computational gaps from secret leakage. In Conference on Learning Theory, pages 648–847. PMLR, 2020.
  16. Nonconvex low-rank tensor completion from noisy data. In Advances in Neural Information Processing Systems, volume 32, pages 1863–1874, 2019.
  17. Jian-Feng Cai and Ke Wei. Exploiting the structure effectively and efficiently in low-rank matrix recovery. In Handbook of Numerical Analysis, volume 19, pages 21–51. Elsevier, 2018.
  18. Provable near-optimal low-multilinear-rank tensor recovery. arXiv preprint arXiv:2007.08904, 2020.
  19. Generalized low-rank plus sparse tensor estimation by fast Riemannian optimization. Journal of the American Statistical Association, pages 1–17, 2022.
  20. Sharp RIP bound for sparse signal and low-rank matrix recovery. Applied and Computational Harmonic Analysis, 35(1):74–93, 2013.
  21. Rop: Matrix recovery via rank-one projections. The Annals of Statistics, 43(1):102–138, 2015.
  22. Tight oracle inequalities for low-rank matrix recovery from a minimal number of noisy random measurements. IEEE Transactions on Information Theory, 57(4):2342–2359, 2011.
  23. Low-rank matrix recovery with composite optimization: good conditioning and rapid convergence. Foundations of Computational Mathematics, pages 1–89, 2021.
  24. Non-convex projected gradient descent for generalized low-rank tensor regression. The Journal of Machine Learning Research, 20(1):172–208, 2019a.
  25. Gradient descent with random initialization: Fast global convergence for nonconvex phase retrieval. Mathematical Programming, 176(1):5–37, 2019b.
  26. Riemannian dictionary learning and sparse coding for positive definite matrices. IEEE transactions on neural networks and learning systems, 28(12):2859–2871, 2016.
  27. Smoothness and periodicity of some matrix decompositions. SIAM Journal on Matrix Analysis and Applications, 22(3):772–792, 2001.
  28. Nonconvex optimization meets low-rank matrix factorization: An overview. IEEE Transactions on Signal Processing, 67(20):5239–5269, 2019.
  29. Optimization on the hierarchical tucker manifold–applications to tensor completion. Linear Algebra and its Applications, 481:131–173, 2015.
  30. A multilinear singular value decomposition. SIAM Journal on Matrix Analysis and Applications, 21(4):1253–1278, 2000.
  31. Statistical and computational limits for tensor-on-tensor association detection. In Conference on Learning Theory. PMLR, 2023.
  32. New Riemannian preconditioned algorithms for tensor completion via polyadic decomposition. SIAM Journal on Matrix Analysis and Applications, 43(2):840–866, 2022.
  33. A Newton–Grassmann method for computing the best multilinear rank-(r1subscript𝑟1r_{1}italic_r start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT, r2subscript𝑟2r_{2}italic_r start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT, r3subscript𝑟3r_{3}italic_r start_POSTSUBSCRIPT 3 end_POSTSUBSCRIPT) approximation of a tensor. SIAM Journal on Matrix Analysis and applications, 31(2):248–271, 2009.
  34. Tensor completion and low-n-rank tensor recovery via convex optimization. Inverse Problems, 27(2):025010, 2011.
  35. Greedy low-rank approximation in tucker format of solutions of tensor linear systems. Journal of Computational and Applied Mathematics, 358:206–220, 2019.
  36. Lars Grasedyck. Hierarchical singular value decomposition of tensors. SIAM Journal on Matrix Analysis and Applications, 31(4):2029–2054, 2010.
  37. A literature survey of low-rank tensor approximation techniques. GAMM-Mitteilungen, 36(1):53–78, 2013.
  38. Bayesian tensor regression. The Journal of Machine Learning Research, 18(1):2733–2763, 2017.
  39. Tensor learning for regression. IEEE Transactions on Image Processing, 21(2):816–827, 2012.
  40. Wolfgang Hackbusch. Tensor spaces and numerical tensor calculus, volume 42. Springer, 2012.
  41. A new scheme for the tensor representation. Journal of Fourier analysis and applications, 15(5):706–722, 2009.
  42. Exact clustering in tensor block model: Statistical optimality and computational limit. Journal of the Royal Statistical Society Series B: Statistical Methodology, 84(5):1666–1698, 2022a.
  43. An optimal statistical and computational framework for generalized tensor estimation. The Annals of Statistics, 50(1):1–29, 2022b.
  44. Sparse and low-rank tensor estimation via cubic sketchings. IEEE Transactions on Information Theory, 2020.
  45. A Riemannian trust-region method for low-rank tensor completion. Numerical Linear Algebra with Applications, 25(6):e2175, 2018.
  46. Most tensor problems are np-hard. Journal of the ACM (JACM), 60(6):1–39, 2013.
  47. Peter D Hoff. Multilinear tensor regression for longitudinal relational data. The Annals of Applied Statistics, 9(3):1169, 2015.
  48. Clemens Hofreither. A black-box low-rank approximation algorithm for fast matrix assembly in isogeometric analysis. Computer Methods in Applied Mechanics and Engineering, 333:311–330, 2018.
  49. On manifolds of tensors of fixed tt-rank. Numerische Mathematik, 120(4):701–731, 2012.
  50. Tensor principal component analysis via sum-of-square proofs. In Proceedings of The 28th Conference on Learning Theory, COLT, pages 3–6, 2015.
  51. Fast global convergence for low-rank matrix recovery via Riemannian gradient descent with random initialization. arXiv preprint arXiv:2012.15467, 2020.
  52. Blind deconvolution by a steepest descent algorithm on a quotient manifold. SIAM Journal on Imaging Sciences, 11(4):2757–2785, 2018.
  53. Differential-geometric Newton method for the best rank-(r1subscript𝑟1r_{1}italic_r start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT, r2subscript𝑟2r_{2}italic_r start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT, r3subscript𝑟3r_{3}italic_r start_POSTSUBSCRIPT 3 end_POSTSUBSCRIPT) approximation of tensors. Numerical Algorithms, 51(2):179–194, 2009.
  54. Best low multilinear rank approximation of higher-order tensors, based on the Riemannian trust-region scheme. SIAM Journal on Matrix Analysis and Applications, 32(1):115–135, 2011.
  55. Provable tensor factorization with missing data. In Advances in Neural Information Processing Systems, volume 27, 2014.
  56. Guaranteed rank minimization via singular value projection. In Advances in Neural Information Processing Systems, pages 937–945, 2010.
  57. Low-rank tensor completion: a Riemannian manifold preconditioning approach. In International Conference on Machine Learning, pages 1012–1021. PMLR, 2016.
  58. Matrix completion from a few entries. In 2009 IEEE International Symposium on Information Theory, pages 324–328. IEEE, 2009.
  59. Dynamical tensor approximation. SIAM Journal on Matrix Analysis and Applications, 31(5):2360–2375, 2010.
  60. Tamara G Kolda. Orthogonal tensor decompositions. SIAM Journal on Matrix Analysis and Applications, 23(1):243–255, 2001.
  61. Tensor decompositions and applications. SIAM review, 51(3):455–500, 2009.
  62. Low-rank tensor completion by Riemannian optimization. BIT Numerical Mathematics, 54(2):447–468, 2014.
  63. Preconditioned low-rank Riemannian optimization for linear systems with tensor product structure. SIAM Journal on Scientific Computing, 38(4):A2018–A2044, 2016.
  64. Parsimonious tensor response regression. Journal of the American Statistical Association, pages 1–16, 2017.
  65. Rapid, robust, and reliable blind deconvolution via nonconvex optimization. Applied and computational harmonic analysis, 47(3):893–934, 2019.
  66. Tucker tensor regression and neuroimaging analysis. Statistics in Biosciences, pages 1–26, 2018.
  67. Tensor completion made practical. In Advances in Neural Information Processing Systems, volume 33, 2020.
  68. Tensor completion for estimating missing values in visual data. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(1):208–220, 2013.
  69. High-dimensional quantile tensor regression. Journal of Machine Learning Research, 21(250):1–31, 2020.
  70. Open problem: Average-case hardness of hypergraphic planted clique detection. In Conference on Learning Theory, pages 3852–3856. PMLR, 2020.
  71. Tensor clustering with planted structures: Statistical optimality and computational limits. The Annals of Statistics, 50(1):584–613, 2022a.
  72. Tensor-on-tensor regression: Riemannian optimization, over-parameterization, statistical-computational gap, and their interplay. arXiv preprint arXiv:2206.08756, 2022b.
  73. Recursive importance sketching for rank constrained least squares: Algorithms and high-order convergence. Operations Research, 2023.
  74. Tensor product analysis of partial difference equations. Bulletin of the American Mathematical Society, 70(3):378–384, 1964.
  75. Implicit regularization in nonconvex statistical estimation: Gradient descent converges linearly for phase retrieval, matrix completion, and blind deconvolution. Foundations of Computational Mathematics, pages 1–182, 2019.
  76. Linear regression under fixed-rank constraints: a Riemannian approach. In Proceedings of the 28th international conference on machine learning, 2011.
  77. Fixed-rank matrix factorizations and Riemannian low-rank optimization. Computational Statistics, 29(3-4):591–621, 2014.
  78. Spectral algorithms for tensor completion. Communications on Pure and Applied Mathematics, 71(11):2381–2425, 2018.
  79. Square deal: Lower bounds and improved relaxations for tensor recovery. In ICML, pages 73–81, 2014.
  80. Numerical optimization. Springer Science & Business Media, 2006.
  81. Ivan V Oseledets. Tensor-train decomposition. SIAM Journal on Scientific Computing, 33(5):2295–2317, 2011.
  82. Statistical limits of spiked tensor models. Annales de l’Institut Henri Poincaré, Probabilités et Statistiques, 56(1):230–264, 2020.
  83. Second-order optimization for tensors with fixed tensor-train rank. NeurIPS workshop, OPT2020, 2020.
  84. Convex regularization for high-dimensional multiresponse tensor regression. The Annals of Statistics, 47(3):1554–1584, 2019.
  85. Tensor completion in hierarchical tensor representations. In Compressed sensing and its applications, pages 419–450. Springer, 2015.
  86. Low rank tensor recovery via iterative hard thresholding. Linear Algebra and its Applications, 523:220–262, 2017.
  87. Guaranteed minimum-rank solutions of linear matrix equations via nuclear norm minimization. SIAM review, 52(3):471–501, 2010.
  88. A statistical model for tensor pca. In Advances in Neural Information Processing Systems, pages 2897–2905, 2014.
  89. Quasi-Newton methods on Grassmannians and multilinear approximations of tensors. SIAM Journal on Scientific Computing, 32(6):3352–3393, 2010.
  90. Jun Shao. Mathematical statistics. Springer Science & Business Media, 2006.
  91. Michael Steinlechner. Riemannian optimization for high-dimensional tensor completion. SIAM Journal on Scientific Computing, 38(5):S461–S484, 2016.
  92. Guaranteed matrix completion via nonconvex factorization. In Foundations of Computer Science (FOCS), 2015 IEEE 56th Annual Symposium on, pages 270–289. IEEE, 2015.
  93. Store: sparse tensor response regression and neuroimaging analysis. The Journal of Machine Learning Research, 18(1):4908–4944, 2017.
  94. Statistical performance of convex tensor decomposition. In Advances in Neural Information Processing Systems, pages 972–980, 2011.
  95. Scaling and scalability: Provable nonconvex low-rank tensor estimation from incomplete measurements. Journal of Machine Learning Research, 23(163):1–77, 2022.
  96. Low-rank solutions of linear matrix equations via Procrustes flow. In International Conference on Machine Learning, pages 964–973, 2016.
  97. Ledyard R Tucker. Some mathematical notes on three-mode factor analysis. Psychometrika, 31(3):279–311, 1966.
  98. The geometry of algorithms using hierarchical tensors. Linear Algebra and its Applications, 439(1):133–166, 2013.
  99. Geometric methods on low-rank matrix and tensor manifolds. In Handbook of Variational Methods for Nonlinear Geometric Data, pages 261–313. Springer, 2020.
  100. Bart Vandereycken. Low-rank matrix completion by Riemannian optimization. SIAM Journal on Optimization, 23(2):1214–1236, 2013.
  101. A new truncation strategy for the higher-order singular value decomposition. SIAM Journal on Scientific Computing, 34(2):A1027–A1052, 2012.
  102. A unified computational and statistical framework for nonconvex low-rank matrix estimation. In Artificial Intelligence and Statistics, pages 981–990, 2017.
  103. Guarantees of Riemannian optimization for low rank matrix recovery. SIAM Journal on Matrix Analysis and Applications, 37(3):1198–1222, 2016.
  104. On polynomial time methods for exact low-rank tensor completion. Foundations of Computational Mathematics, pages 1–49, 2017.
  105. Statistically optimal and computationally efficient low rank tensor completion from noisy entries. The Annals of Statistics, 49(1), 2021.
  106. Learning from multiway data: Simple and efficient tensor regression. In International Conference on Machine Learning, pages 373–381. PMLR, 2016.
  107. On tensor completion via nuclear norm minimization. Foundations of Computational Mathematics, pages 1–38, 2014.
  108. Tensor svd: Statistical and computational limits. IEEE Transactions on Information Theory, 64(11):7311–7338, 2018.
  109. ISLET: Fast and optimal low-rank tensor regression via importance sketching. SIAM Journal on Mathematics of Data Science, 2(2):444–479, 2020a.
  110. Denoising atomic resolution 4d scanning transmission electron microscopy data with tensor singular value decomposition. Ultramicroscopy, 219:113123, 2020b.
  111. Tensor network factorizations: Relationships between brain structural connectomes and traits. Neuroimage, 197:330–343, 2019.
  112. A nonconvex optimization framework for low rank matrix estimation. In Advances in Neural Information Processing Systems, pages 559–567, 2015.
  113. A convergent gradient descent algorithm for rank minimization and semidefinite programming from random linear measurements. In Advances in Neural Information Processing Systems, pages 109–117, 2015.
  114. Hua Zhou. Matlab TensorReg Toolbox Version 1.0. Available online at https://hua-zhou.github.io/TensorReg/, 2017.
  115. Tensor regression with applications in neuroimaging data analysis. Journal of the American Statistical Association, 108(502):540–552, 2013.
  116. Optimal high-order tensor svd via tensor-train orthogonal iteration. IEEE Transactions on Information Theory, 68(6):3991–4019, 2022.
Citations (17)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now