Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
129 tokens/sec
GPT-4o
28 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Structured Matrix Learning under Arbitrary Entrywise Dependence and Estimation of Markov Transition Kernel (2401.02520v1)

Published 4 Jan 2024 in stat.ML, cs.LG, math.ST, and stat.TH

Abstract: The problem of structured matrix estimation has been studied mostly under strong noise dependence assumptions. This paper considers a general framework of noisy low-rank-plus-sparse matrix recovery, where the noise matrix may come from any joint distribution with arbitrary dependence across entries. We propose an incoherent-constrained least-square estimator and prove its tightness both in the sense of deterministic lower bound and matching minimax risks under various noise distributions. To attain this, we establish a novel result asserting that the difference between two arbitrary low-rank incoherent matrices must spread energy out across its entries, in other words cannot be too sparse, which sheds light on the structure of incoherent low-rank matrices and may be of independent interest. We then showcase the applications of our framework to several important statistical machine learning problems. In the problem of estimating a structured Markov transition kernel, the proposed method achieves the minimax optimality and the result can be extended to estimating the conditional mean operator, a crucial component in reinforcement learning. The applications to multitask regression and structured covariance estimation are also presented. We propose an alternating minimization algorithm to approximately solve the potentially hard optimization problem. Numerical results corroborate the effectiveness of our method which typically converges in a few steps.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (69)
  1. Abbe, E. (2017). Community detection and stochastic block models: recent developments. The Journal of Machine Learning Research, 18(1), 6446–6531.
  2. Adamczak, R. (2008). A tail inequality for suprema of unbounded empirical processes with applications to markov chains. Electronic Journal of Probability, 13, 1000–1034.
  3. Flambe: Structural complexity and representation learning of low rank mdps. Advances in neural information processing systems, 33, 20095–20107.
  4. Noisy matrix decomposition via convex relaxation: Optimal rates in high dimensions. The Annals of Statistics, 40(2), 1171–1197.
  5. Berestycki, N. (2016). Mixing times of markov chains: Techniques and examples. Alea-Latin American Journal of Probability and Mathematical Statistics.
  6. Parallel and distributed computation: numerical methods.
  7. Generalized low-rank plus sparse tensor estimation by fast riemannian optimization. Journal of the American Statistical Association, (pp. 1–17).
  8. Exact matrix completion via convex optimization. Communications of the ACM, 55(6), 111–119.
  9. Robust principal component analysis? Journal of the ACM (JACM), 58(3), 1–37.
  10. Matrix completion with noise. Proceedings of the IEEE, 98(6), 925–936.
  11. Rank-sparsity incoherence for matrix decomposition. SIAM Journal on Optimization, 21(2), 572–596.
  12. Nonconvex rectangular matrix completion via gradient descent without l2,∞subscript𝑙2l_{2,\infty}italic_l start_POSTSUBSCRIPT 2 , ∞ end_POSTSUBSCRIPT regularization. IEEE Transactions on Information Theory, 66(9), 5806–5841.
  13. Harnessing structures in big data via guaranteed low-rank matrix estimation: Recent theory and fast algorithms via convex and nonconvex optimization. IEEE Signal Processing Magazine, 35(4), 14–31.
  14. Spectral methods for data science: A statistical perspective. Foundations and Trends® in Machine Learning, 14(5), 566–806.
  15. Noisy matrix completion: Understanding statistical guarantees for convex relaxation via nonconvex optimization. SIAM journal on optimization, 30(4), 3098–3121.
  16. Inference and uncertainty quantification for noisy matrix completion. Proceedings of the National Academy of Sciences, 116(46), 22931–22937.
  17. Bridging convex and nonconvex optimization in robust pca: Noise, outliers and missing data. The Annals of Statistics, 49(5), 2948–2971.
  18. Convex and nonconvex optimization are both minimax-optimal for noisy blind deconvolution under random designs. Journal of the American Statistical Association, 118(542), 858–868.
  19. Spectral mle: Top-k rank aggregation from pairwise comparisons. In International Conference on Machine Learning (pp. 371–380).: PMLR.
  20. Fast low-rank estimation by projected gradient descent: General statistical and algorithmic guarantees. arXiv preprint arXiv:1509.03025.
  21. Nearly optimal robust matrix completion. In International Conference on Machine Learning (pp. 797–805).: PMLR.
  22. Nonconvex optimization meets low-rank matrix factorization: An overview. IEEE Transactions on Signal Processing, 67(20), 5239–5269.
  23. Chung, K. L. (1967). Markov chains. Springer-Verlag, New York.
  24. Iteratively reweighted least squares minimization for sparse recovery. Communications on Pure and Applied Mathematics: A Journal Issued by the Courant Institute of Mathematical Sciences, 63(1), 1–38.
  25. High dimensional covariance matrix estimation using a factor model. Journal of Econometrics, 147(1), 186–197.
  26. Farmtest: Factor-adjusted robust multiple testing with approximate false discovery control. Journal of the American Statistical Association, 114(528), 1880–1893.
  27. Estimation of high dimensional mean regression in the absence of symmetry and light tail assumptions. Journal of the Royal Statistical Society Series B: Statistical Methodology, 79(1), 247–265.
  28. High dimensional covariance matrix estimation in approximate factor models. Annals of statistics, 39(6), 3320.
  29. Large covariance estimation by thresholding principal orthogonal complements. Journal of the Royal Statistical Society. Series B, Statistical methodology, 75(4).
  30. Principal component analysis for big data. arXiv preprint arXiv:1801.01602.
  31. Robust high dimensional factor models with applications to statistical machine learning. Statistical science: a review journal of the Institute of Mathematical Statistics, 36(2), 303.
  32. A shrinkage principle for heavy-tailed data: High-dimensional robust low-rank matrix recovery. arXiv preprint arXiv:1603.08315.
  33. A shrinkage principle for heavy-tailed data: High-dimensional robust low-rank matrix recovery. Annals of statistics, 49(3), 1239.
  34. On the optimization landscape of tensor decompositions. Advances in Neural Information Processing Systems, 30.
  35. Gross, D. (2011). Recovering low-rank matrices from few coefficients in any basis. IEEE Transactions on Information Theory, 57(3), 1548–1566.
  36. Low-rank and sparse structure pursuit via alternating minimization. In Artificial Intelligence and Statistics (pp. 600–609).: PMLR.
  37. On learning markov chains. Advances in Neural Information Processing Systems, 31.
  38. Robust matrix decomposition with sparse corruptions. IEEE Transactions on Information Theory, 57(11), 7221–7234.
  39. Bernstein’s inequality for general markov chains. arXiv preprint arXiv:1805.10721.
  40. Provably efficient reinforcement learning with linear function approximation. In Conference on Learning Theory (pp. 2137–2143).: PMLR.
  41. Pca in high dimensions: An orientation. Proceedings of the IEEE, 106(8), 1277–1292.
  42. On learning distributions from their samples. In Conference on Learning Theory (pp. 1066–1100).: PMLR.
  43. Nuclear-norm penalization and optimal rates for noisy low-rank matrix completion. The Annals of Statistics, 39(5), 2302–2329.
  44. Markov chains and mixing times, volume 107. American Mathematical Soc.
  45. Optimality of spectral clustering in the gaussian mixture model. The Annals of Statistics, 49(5), 2506–2530.
  46. Implicit regularization in nonconvex statistical estimation: Gradient descent converges linearly for phase retrieval and matrix completion. In International Conference on Machine Learning (pp. 3345–3354).: PMLR.
  47. McCullagh, P. (2019). Generalized linear models. Routledge.
  48. A bernstein type inequality and moderate deviations for weakly dependent sequences. Probability Theory and Related Fields, 151, 435–474.
  49. Model-free representation learning and exploration in low-rank mdps. arXiv preprint arXiv:2102.07035.
  50. Ndaoud, M. (2018). Sharp optimal recovery in the two-component gaussian mixture model. arXiv preprint arXiv:1812.08078.
  51. Restricted strong convexity and weighted matrix completion: Optimal bounds with noise. The Journal of Machine Learning Research, 13(1), 1665–1697.
  52. A unified framework for high-dimensional analysis of m-estimators with decomposable regularizers.
  53. Non-convex robust pca. Advances in neural information processing systems, 27.
  54. Norris, J. R. (1998). Markov chains. Number 2. Cambridge university press.
  55. Iterative Solution of Nonlinear Equations in Several Variables, volume 30. SIAM.
  56. Paulin, D. (2015). Concentration inequalities for markov chains by marton couplings and spectral methods. Electronic Journal of Probability, 20, 1–32.
  57. Estimation of high-dimensional low-rank matrices.
  58. Guaranteed matrix completion via non-convex factorization. IEEE Transactions on Information Theory, 62(11), 6535–6579.
  59. Wainwright, M. J. (2019). High-dimensional statistics: A non-asymptotic viewpoint, volume 48. Cambridge university press.
  60. Robust matrix completion with heavy-tailed noise. arXiv preprint arXiv:2206.04276.
  61. Minimax learning of ergodic markov chains. In Algorithmic Learning Theory (pp. 904–930).: PMLR.
  62. Statistical estimation of ergodic markov chain kernel over discrete state space. Bernoulli, 27(1), 532–553.
  63. Robust principal component analysis: Exact recovery of corrupted low-rank matrices via convex optimization. Advances in neural information processing systems, 22.
  64. Fast algorithms for robust pca via gradient descent. Advances in neural information processing systems, 29.
  65. Yu, B. (1997). Assouad, fano, and le cam. Festschrift for Lucien Le Cam: research papers in probability and statistics, (pp. 423–435).
  66. Dimension reduction and coefficient estimation in multivariate linear regression. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 69(3), 329–346.
  67. Spectral state compression of markov processes. IEEE transactions on information theory, 66(5), 3202–3231.
  68. Convergence analysis for rectangular matrix completion using burer-monteiro factorization and gradient descent. arXiv preprint arXiv:1605.07051.
  69. Learning markov models via low-rank optimization. Operations Research, 70(4), 2384–2398.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets