Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
149 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Guaranteed Nonconvex Factorization Approach for Tensor Train Recovery (2401.02592v2)

Published 5 Jan 2024 in stat.ML, cs.LG, eess.SP, and math.OC

Abstract: In this paper, we provide the first convergence guarantee for the factorization approach. Specifically, to avoid the scaling ambiguity and to facilitate theoretical analysis, we optimize over the so-called left-orthogonal TT format which enforces orthonormality among most of the factors. To ensure the orthonormal structure, we utilize the Riemannian gradient descent (RGD) for optimizing those factors over the Stiefel manifold. We first delve into the TT factorization problem and establish the local linear convergence of RGD. Notably, the rate of convergence only experiences a linear decline as the tensor order increases. We then study the sensing problem that aims to recover a TT format tensor from linear measurements. Assuming the sensing operator satisfies the restricted isometry property (RIP), we show that with a proper initialization, which could be obtained through spectral initialization, RGD also converges to the ground-truth tensor at a linear rate. Furthermore, we expand our analysis to encompass scenarios involving Gaussian noise in the measurements. We prove that RGD can reliably recover the ground truth at a linear rate, with the recovery error exhibiting only polynomial growth in relation to the tensor order. We conduct various experiments to validate our theoretical findings.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (79)
  1. Tensor decompositions for signal processing applications: From two-way to multiway component analysis. IEEE signal processing magazine, 32(2):145–163, 2015.
  2. Tensor decomposition for signal processing and machine learning. IEEE Transactions on Signal Processing, 65(13):3551–3582, 2017.
  3. Blind PARAFAC receivers for DS-CDMA systems. IEEE Transactions on Signal Processing, 48(3):810–823, 2000.
  4. Multi-way analysis: applications in the chemical sciences. John Wiley & Sons, 2005.
  5. Unsupervised multiway data analysis: A literature survey. IEEE transactions on knowledge and data engineering, 21(1):6–20, 2008.
  6. Tensor decomposition for multiple-tissue gene expression experiments. Nature genetics, 48(9):1094–1100, 2016.
  7. Rasmus Bro. Parafac. Tutorial and applications. Chemometrics and intelligent laboratory systems, 38(2):149–171, 1997.
  8. Ledyard R Tucker. Some mathematical notes on three-mode factor analysis. Psychometrika, 31(3):279–311, 1966.
  9. Ivan V Oseledets. Tensor-train decomposition. SIAM Journal on Scientific Computing, 33(5):2295–2317, 2011.
  10. Johan Håstad. Tensor rank is np-complete. In Automata, Languages and Programming: 16th International Colloquium Stresa, Italy, July 11–15, 1989 Proceedings 16, pages 451–460. Springer, 1989.
  11. Tensor rank and the ill-posedness of the best low-rank approximation problem. SIAM Journal on Matrix Analysis and Applications, 30(3):1084–1127, 2008.
  12. Tensor decompositions and applications. SIAM review, 51(3):455–500, 2009.
  13. Nonconvex low-rank tensor completion from noisy data. Advances in neural information processing systems, 32, 2019.
  14. Jose I Latorre. Image compression and entanglement. arXiv preprint quant-ph/0510031, 2005.
  15. Efficient tensor completion for color image and video recovery: Low-rank tensor train. IEEE Transactions on Image Processing, 26(5):2466–2479, 2017.
  16. Expressive power of recurrent neural networks. arXiv preprint arXiv:1711.00811, 2017.
  17. Supervised learning with tensor networks. Advances in neural information processing systems, 29, 2016.
  18. Tensorizing neural networks. Advances in neural information processing systems, 28, 2015.
  19. Tensor-train recurrent neural networks for video classification. In International Conference on Machine Learning, pages 3891–3900. PMLR, 2017.
  20. Compressing recurrent neural network with tensor train. In 2017 International Joint Conference on Neural Networks (IJCNN), pages 4451–4458. IEEE, 2017.
  21. Long-term forecasting using tensor-train rnns. Arxiv, 2017.
  22. A tensorized transformer for language modeling. Advances in neural information processing systems, 32, 2019.
  23. Tensor methods and recommender systems. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 7(3):e1201, 2017.
  24. Tensor-train density estimation. In Uncertainty in artificial intelligence, pages 1321–1331. PMLR, 2021.
  25. Tensor train spectral method for learning of hidden markov models (hmm). Computational Methods in Applied Mathematics, 19(1):93–99, 2019.
  26. Matrix product states represent ground states faithfully. Physical review b, 73(9):094423, 2006.
  27. Matrix product states, projected entangled pair states, and variational renormalization group methods for quantum spin systems. Advances in physics, 57(2):143–224, 2008.
  28. Ulrich Schollwöck. The density-matrix renormalization group in the age of matrix product states. Annals of physics, 326(1):96–192, 2011.
  29. Efficient and feasible state tomography of quantum many-body systems. New Journal of Physics, 15(1):015024, 2013.
  30. Latent schatten tt norm for tensor completion. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 2922–2926. IEEE, 2019.
  31. Tensor completion by alternating minimization under the tensor train (tt) model. arXiv preprint arXiv:1609.05587, 2016.
  32. High-order tensor completion via gradient-based optimization under tensor train format. Signal Processing: Image Communication, 73:53–61, 2019.
  33. Low rank tensor recovery via iterative hard thresholding. Linear Algebra and its Applications, 523:220–262, 2017.
  34. Tensor completion in hierarchical tensor representations. In Compressed sensing and its applications, pages 419–450. Springer, 2015.
  35. Tensor train completion: local recovery guarantees via riemannian optimization. arXiv preprint arXiv:2110.03975, 2021.
  36. Tensor completion using low-rank tensor train decomposition by riemannian optimization. In 2019 Chinese Automation Congress (CAC), pages 3380–3384. IEEE, 2019.
  37. Provable tensor-train format tensor completion by riemannian optimization. Journal of Machine Learning Research, 23(123):1–77, 2022.
  38. A tensor-train deep computation model for industry informatics big data feature learning. IEEE Transactions on Industrial Informatics, 14(7):3197–3204, 2018.
  39. Exploiting low-rank tensor-train deep neural networks based on riemannian gradient descent with illustrations of speech processing. arXiv preprint arXiv:2203.06031, 2022.
  40. Tensor ring decomposition with rank minimization on latent space: An efficient approach for tensor completion. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 9151–9158, 2019.
  41. Designing tensor-train deep neural networks for time-varying mimo channel estimation. IEEE Journal of Selected Topics in Signal Processing, 15(3):759–773, 2021.
  42. Quantum state tomography with tensor train cross approximation. arXiv preprint arXiv:2207.06397, 2022.
  43. Low-rank solutions of linear matrix equations via procrustes flow. In International Conference on Machine Learning, pages 964–973. PMLR, 2016.
  44. Non-square matrix sensing without spurious local minima via the burer-monteiro approach. In Artificial Intelligence and Statistics, pages 65–74. PMLR, 2017.
  45. Global optimality in low-rank matrix optimization. IEEE Transactions on Signal Processing, 66(13):3614–3628, 2018.
  46. An optimal statistical and computational framework for generalized tensor estimation. The Annals of Statistics, 50(1):1–29, 2022.
  47. Stable tomography for structured quantum states. arXiv preprint arXiv:2306.09432, 2023.
  48. On manifolds of tensors of fixed tt-rank. Numerische Mathematik, 120(4):701–731, 2012.
  49. Weakly convex optimization over Stiefel manifold using Riemannian subgradient-type methods. SIAM Journal on Optimization, 31(3):1605–1634, 2021.
  50. Optimization landscape of neural networks. Mathematical Aspects of Deep Learning, page 200, 2022.
  51. A unified computational and statistical framework for nonconvex low-rank matrix estimation. In Artificial Intelligence and Statistics, pages 981–990. PMLR, 2017.
  52. How to escape saddle points efficiently. In International conference on machine learning, pages 1724–1732. PMLR, 2017.
  53. Nonconvex robust low-rank matrix recovery. SIAM Journal on Optimization, 30(1):660–686, 2020.
  54. Beyond procrustes: Balancing-free gradient descent for asymmetric low-rank matrix sensing. IEEE Transactions on Signal Processing, 69:867–877, 2021.
  55. Accelerating ill-conditioned low-rank matrix estimation via scaled gradient descent. J. Mach. Learn. Res., 22:150–1, 2021.
  56. Scaling and scalability: Provable nonconvex low-rank tensor estimation from incomplete measurements. arXiv preprint arXiv:2104.14526, Nov. 2021.
  57. On polynomial time methods for exact low-rank tensor completion. Foundations of Computational Mathematics, 19(6):1265–1313, 2019.
  58. Tensor regression with applications in neuroimaging data analysis. Journal of the American Statistical Association, 108(502):540–552, 2013.
  59. Parsimonious tensor response regression. Journal of the American Statistical Association, 112(519):1131–1146, 2017.
  60. Tensor learning for regression. IEEE Transactions on Image Processing, 21(2):816–827, 2011.
  61. Sparse and low-rank tensor estimation via cubic sketchings. In International Conference on Artificial Intelligence and Statistics, pages 1319–1330. PMLR, 2020.
  62. David L Donoho. Compressed sensing. IEEE Transactions on information theory, 52(4):1289–1306, 2006.
  63. Robust uncertainty principles: Exact signal reconstruction from highly incomplete frequency information. IEEE Transactions on information theory, 52(2):489–509, 2006.
  64. An introduction to compressive sampling. IEEE signal processing magazine, 25(2):21–30, 2008.
  65. Guaranteed minimum-rank solutions of linear matrix equations via nuclear norm minimization. SIAM review, 52(3):471–501, 2010.
  66. Iterative hard thresholding for low cp-rank tensor models. Linear and Multilinear Algebra, pages 1–17, 2021.
  67. Modulus of continuity of some conditionally sub-gaussian fields, application to stable random fields. Bernoulli, 21(3):1719–1759, 2015.
  68. Phase transitions of spectral initialization for high-dimensional non-convex estimation. Information and Inference: A Journal of the IMA, 9(3):507–541, 2020.
  69. Phase retrieval via wirtinger flow: Theory and algorithms. IEEE Transactions on Information Theory, 61(4):1985–2007, 2015.
  70. Optimal spectral initialization for signal recovery with applications to phase retrieval. IEEE Transactions on Signal Processing, 67(9):2347–2356, 2019.
  71. Andrzej Cichocki. Tensor networks for big data analytics and large-scale optimization problems. arXiv preprint arXiv:1407.3124, 2014.
  72. Small random initialization is akin to spectral learning: Optimization and generalization guarantees for overparameterized low-rank matrix reconstruction. Advances in Neural Information Processing Systems, 34:23831–23843, 2021.
  73. Algorithmic regularization in model-free overparametrized asymmetric matrix factorization. arXiv preprint arXiv:2203.02839, 2022.
  74. A validation approach to over-parameterized matrix and image recovery. arXiv preprint arXiv:2209.10675, 2022.
  75. The power of preconditioning in overparameterized low-rank matrix sensing. arXiv preprint arXiv:2302.01186, 2023.
  76. The global optimization geometry of low-rank matrix optimization. IEEE Transactions on Information Theory, 67(2):1308–1331, 2021.
  77. Tight oracle inequalities for low-rank matrix recovery from a minimal number of noisy random measurements. IEEE Transactions on Information Theory, 57(4):2342–2359, 2011.
  78. Preconditioned gradient descent for over-parameterized nonconvex matrix factorization. Advances in Neural Information Processing Systems, 34:5985–5996, 2021.
  79. Tian Tong. Scaled gradient methods for ill-conditioned low-rank matrix and tensor estimation. PhD thesis, Carnegie Mellon University, 2022.
Citations (5)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets