2000 character limit reached
Diagonal Gaussian Mixture Models and Higher Order Tensor Decompositions (2401.01337v1)
Published 2 Jan 2024 in math.NA and cs.NA
Abstract: This paper studies how to recover parameters in diagonal Gaussian mixture models using tensors. High-order moments of the Gaussian mixture model are estimated from samples. They form incomplete symmetric tensors generated by hidden parameters in the model. We propose to use generating polynomials to compute incomplete symmetric tensor approximations. The obtained decomposition is utilized to recover parameters in the model. We prove that our recovered parameters are accurate when the estimated moments are accurate. Using high-order moments enables our algorithm to learn Gaussian mixtures with more components. For a given model dimension and order, we provide an upper bound of the number of components in the Gaussian mixture model that our algorithm can compute.
- D. Achlioptas and F. McSherry, On spectral learning of mixtures of distributions, International Conference on Computational Learning Theory, pp. 458–469, 2005.
- A. Anandkumar, R. Ge and M. Janzamin, Analyzing tensor power method dynamics in overcomplete regime, Journal Of Machine Learning Research 18(22):1–40, 2017.
- The more, the merrier: the blessing of dimensionality for learning large Gaussian mixtures, Conference On Learning Theory, pp. 1135-1164, 2014.
- M. Belkin and K. Sinha, Polynomial learning of distribution families, 2010 IEEE 51st Annual Symposium on Foundations of Computer Science, pages 103–112, 2010.
- Smoothed analysis of tensor decompositions, Proceedings of the forty-sixth annual ACM symposium on Theory of computing, pages 594–603, 2014.
- F. Chatelin, Eigenvalues of matrices: revised edition, SIAM, 2012.
- S. Dasgupta, Learning mixtures of Gaussians, 40th Annual Symposium on Foundations of Computer Science (Cat. No. 99CB37039), pages 634–644. IEEE, 1999.
- S. Dasgupta and L. Schulman, A two-round variant of EM for Gaussian mixtures, arXiv preprint arXiv:1301.3850, 2013.
- Maximum likelihood from incomplete data via the EM algorithm, Journal of the Royal Statistical Society: Series B (Methodological) 39(1): 1–22, 1977.
- R. Ge, Q. Huang and S. M. Kakade, Learning mixtures of gaussians in high dimensions, Proceedings Of The Forty-seventh Annual ACM Symposium On Theory Of Computing, pp. 761-770, 2015.
- I. Goodfellow, Y. Bengio and A. Courville, Deep learning, MIT Press, 2016.
- B. Guo, J. Nie and Z. Yang, Learning diagonal Gaussian mixture models and incomplete tensor decompositions, Vietnam Journal of Mathematics, pages 1–26, 2021.
- D. Hsu and S. M. Kakade, Learning mixtures of spherical gaussians: moment methods and spectral decompositions, Proceedings of the 4th conference on Innovations in Theoretical Computer Science, pages 11–20, 2013.
- A. T. Kalai, A. Moitra and G. Valiant, Efficiently learning mixtures of two Gaussians, Proceedings Of The Forty-second ACM Symposium On Theory Of Computing, pp. 553-562, 2010.
- S. Karpagavalli and E. Chandra. A review on automatic speech recognition architecture and approaches. International Journal of Signal Processing, Image Processing and Pattern Recognition, 9(4):393–404, 2016.
- R. Khouja, P. A. Mattei and B. Mourrain, Tensor decomposition for learning Gaussian mixtures from moments Journal of Symbolic Computation 113, 193–210, 2022
- D. -S. Lee, Effective Gaussian mixture learning for video background subtraction, IEEE transactions on pattern analysis and machine intelligence, 27(5):827–832, 2005.
- Estimating Gaussian mixtures using sparse polynomial moment system, arXiv preprint arXiv:2106.15675, 2021.
- A. Moitra and G. Valiant, Settling the polynomial learnability of mixtures of gaussians, 2010 IEEE 51st Annual Symposium on Foundations of Computer Science, pp. 93–102, 2010.
- J. Nie, Generating polynomials and symmetric tensor decompositions Foundations of Computational Mathematics 17(2), 423–465, 2017.
- K. Pearson, Contributions to the mathematical theory of evolution, Philosophical Transactions of the Royal Society of London. A 185, 71–110, 1894.
- M. Magdon-Ismail and J. T. Purnell, Approximating the covariance matrix of GMMs with low-rank perturbations, International Conference on Intelligent Data Engineering and Automated Learning, pages 300–307, 2010.
- J. M Pereira, J. Kileel and T. G Kolda, Tensor moments of gaussian mixture models: Theory and applications, Prprint, 2022. arXiv:2202.06930
- The subspace Gaussian mixture model—a structured model for speech recognition, Computer Speech & Language 25(2), 404–439, 2011.
- R. A. Redner and H. F. Walker, Mixture densities, maximum likelihood and the EM algorithm, SIAM review 26(2), 195–239, 1984.
- D. A. Reynolds, Speaker identification and verification using Gaussian mixture speaker models, Speech communication 17(1-2), 91–108, 1995.
- A. Sanjeev and R. Kannan, Learning mixtures of arbitrary Gaussians, Proceedings of the thirty-third annual ACM symposium on Theory of computing, pp 247–257, 2001.
- S. Sherman and T. G. Kolda, Estimating higher-order moments using symmetric tensor decomposition, SIAM. J. Matrix Anal. Appl. 41(3), 1369–1387, 2020.
- S. Vempala and G. Wang, A spectral algorithm for learning mixture models, Journal of Computer and System Sciences 68(4):841–860, 2004.
- Fully unsupervised learning of Gaussian mixtures for anomaly detection in hyperspectral imagery, 2009 Ninth International Conference on Intelligent Systems Design and Applications, 596–601, 2009.
- Z. Zivkovic, Improved adaptive Gaussian mixture model for background subtraction, Proceedings of the 17th International Conference on Pattern Recognition, 2004. ICPR 2004., 2, 28–31, 2004.