TERM Model: Tensor Ring Mixture Model for Density Estimation (2312.08075v1)
Abstract: Efficient probability density estimation is a core challenge in statistical machine learning. Tensor-based probabilistic graph methods address interpretability and stability concerns encountered in neural network approaches. However, a substantial number of potential tensor permutations can lead to a tensor network with the same structure but varying expressive capabilities. In this paper, we take tensor ring decomposition for density estimator, which significantly reduces the number of permutation candidates while enhancing expressive capability compared with existing used decompositions. Additionally, a mixture model that incorporates multiple permutation candidates with adaptive weights is further designed, resulting in increased expressive flexibility and comprehensiveness. Different from the prevailing directions of tensor network structure/permutation search, our approach provides a new viewpoint inspired by ensemble learning. This approach acknowledges that suboptimal permutations can offer distinctive information besides that of optimal permutations. Experiments show the superiority of the proposed approach in estimating probability density for moderately dimensional datasets and sampling to capture intricate details.
- B. Nachman and D. Shih, “Anomaly detection with density estimation,” Physical Review D, vol. 101, no. 7, p. 075042, 2020.
- L. J. Latecki, A. Lazarevic, and D. Pokrajac, “Outlier detection with kernel density functions,” in International Workshop on Machine Learning and Data Mining in Pattern Recognition, pp. 61–75, Springer, 2007.
- R. J. Campello, D. Moulavi, and J. Sander, “Density-based clustering based on hierarchical density estimates,” in Pacific-Asia conference on knowledge discovery and data mining, pp. 160–172, Springer, 2013.
- Y.-J. Oyang, S.-C. Hwang, Y.-Y. Ou, C.-Y. Chen, and Z.-W. Chen, “Data classification with radial basis function networks based on a novel kernel density estimation algorithm,” IEEE transactions on neural networks, vol. 16, no. 1, pp. 225–236, 2005.
- G. Papamakarios, T. Pavlakou, and I. Murray, “Masked autoregressive flow for density estimation,” Advances in neural information processing systems, vol. 30, 2017.
- J. Alsing, B. Wandelt, and S. Feeney, “Massive optimal data compression and density estimation for scalable, likelihood-free inference in cosmology,” Monthly Notices of the Royal Astronomical Society, vol. 477, no. 3, pp. 2874–2885, 2018.
- J. Van Ryzin, “A histogram method of density estimation,” Communications in Statistics-Theory and Methods, vol. 2, no. 6, pp. 493–506, 1973.
- G. R. Terrell and D. W. Scott, “Variable kernel density estimation,” The Annals of Statistics, pp. 1236–1265, 1992.
- D. P. Kingma and M. Welling, “Auto-encoding variational bayes,” arXiv preprint arXiv:1312.6114, 2013.
- I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” Advances in neural information processing systems, vol. 27, 2014.
- A. Van Den Oord, N. Kalchbrenner, and K. Kavukcuoglu, “Pixel recurrent neural networks,” in International conference on machine learning, pp. 1747–1756, PMLR, 2016.
- L. Dinh, J. Sohl-Dickstein, and S. Bengio, “Density estimation using real nvp,” arXiv preprint arXiv:1605.08803, 2016.
- D. P. Kingma and P. Dhariwal, “Glow: Generative flow with invertible 1x1 convolutions,” Advances in neural information processing systems, vol. 31, 2018.
- W. Grathwohl, R. T. Chen, J. Bettencourt, I. Sutskever, and D. Duvenaud, “Ffjord: Free-form continuous dynamics for scalable reversible generative models,” arXiv preprint arXiv:1810.01367, 2018.
- Springer, 2022.
- H. Huang, Y. Liu, Z. Long, and C. Zhu, “Robust low-rank tensor ring completion,” IEEE Transactions on Computational Imaging, vol. 6, pp. 1117–1126, 2020.
- J. Liu, C. Zhu, and Y. Liu, “Smooth compact tensor ring regression,” IEEE Transactions on Knowledge and Data Engineering, vol. 34, no. 9, pp. 4439–4452, 2020.
- J. Liu, C. Zhu, Z. Long, and Y. Liu, “Tensor regression,” Foundations and Trends in Machine Learning, vol. 14, no. 4, pp. 379–565, 2021.
- F. L. Hitchcock, “The expression of a tensor or a polyadic as a sum of products,” Journal of Mathematics and Physics, vol. 6, no. 1-4, pp. 164–189, 1927.
- I. V. Oseledets, “Tensor-train decomposition,” SIAM Journal on Scientific Computing, vol. 33, no. 5, pp. 2295–2317, 2011.
- Q. Zhao, G. Zhou, S. Xie, L. Zhang, and A. Cichocki, “Tensor ring decomposition,” arXiv preprint arXiv:1606.05535, 2016.
- K. Ye and L.-H. Lim, “Tensor network ranks,” arXiv preprint arXiv:1801.02662, 2018.
- C. Li, Z. Sun, and Q. Zhao, “High-order learning model via fractional tensor network decomposition,” in First Workshop on Quantum Tensor Networks in Machine Learning, 34th Conference on Neural Information Processing Systems (NeurIPS 2020), 2020.
- H. Qiu, C. Li, Y. Weng, Z. Sun, X. He, and Q. Zhao, “On the memory mechanism of tensor-power recurrent models,” in International Conference on Artificial Intelligence and Statistics, pp. 3682–3690, PMLR, 2021.
- C. Li, J. Zeng, Z. Tao, and Q. Zhao, “Permutation search of tensor network structures via local sampling,” in International Conference on Machine Learning, pp. 13106–13124, PMLR, 2022.
- C. Li and Z. Sun, “Evolutionary topology search for tensor network decomposition,” in International Conference on Machine Learning, pp. 5947–5957, PMLR, 2020.
- D. A. Reynolds et al., “Gaussian mixture models.,” Encyclopedia of biometrics, vol. 741, no. 659-663, 2009.
- Z. Long, C. Zhu, J. Liu, and Y. Liu, “Bayesian low rank tensor ring for image recovery,” IEEE Transactions on Image Processing, vol. 30, pp. 3568–3580, 2021.
- Y. LeCun, S. Chopra, R. Hadsell, M. Ranzato, and F. Huang, “A tutorial on energy-based learning,” Predicting structured data, vol. 1, no. 0, 2006.
- G. Papamakarios, E. Nalisnick, D. J. Rezende, S. Mohamed, and B. Lakshminarayanan, “Normalizing flows for probabilistic modeling and inference,” The Journal of Machine Learning Research, vol. 22, no. 1, pp. 2617–2680, 2021.
- M. Amiridi, N. Kargas, and N. D. Sidiropoulos, “Low-rank characteristic tensor density estimation part i: Foundations,” IEEE Transactions on Signal Processing, vol. 70, pp. 2654–2668, 2022.
- T. G. Kolda and B. W. Bader, “Tensor decompositions and applications,” SIAM review, vol. 51, no. 3, pp. 455–500, 2009.
- S. Dolgov, K. Anaya-Izquierdo, C. Fox, and R. Scheichl, “Approximation and sampling of multivariate probability distributions in the tensor train decomposition,” Statistics and Computing, vol. 30, pp. 603–625, 2020.
- T. Cui and S. Dolgov, “Deep composition of tensor-trains using squared inverse rosenblatt transports,” Foundations of Computational Mathematics, vol. 22, no. 6, pp. 1863–1922, 2022.
- G. S. Novikov, M. E. Panov, and I. V. Oseledets, “Tensor-train density estimation,” in Uncertainty in artificial intelligence, pp. 1321–1331, PMLR, 2021.
- W. J. Gordon and R. F. Riesenfeld, “B-spline curves and surfaces,” in Computer aided geometric design, pp. 95–126, Elsevier, 1974.
- C. Durkan, A. Bekasov, I. Murray, and G. Papamakarios, “Neural spline flows,” Advances in neural information processing systems, vol. 32, 2019.
- C.-W. Huang, R. T. Chen, C. Tsirigotis, and A. Courville, “Convex potential flows: Universal probability distributions with optimal transport and convex optimization,” arXiv preprint arXiv:2012.05942, 2020.
- A. Wehenkel and G. Louppe, “Unconstrained monotonic neural networks,” Advances in neural information processing systems, vol. 32, 2019.
- C. Meng, Y. Song, J. Song, and S. Ermon, “Gaussianization flows,” in International Conference on Artificial Intelligence and Statistics, pp. 4336–4345, PMLR, 2020.
- M. Germain, K. Gregor, I. Murray, and H. Larochelle, “Made: Masked autoencoder for distribution estimation,” in International conference on machine learning, pp. 881–889, PMLR, 2015.
- N. De Cao, W. Aziz, and I. Titov, “Block neural autoregressive flow,” in Uncertainty in artificial intelligence, pp. 1263–1273, PMLR, 2020.
- J. Oliva, A. Dubey, M. Zaheer, B. Poczos, R. Salakhutdinov, E. Xing, and J. Schneider, “Transformation autoregressive networks,” in International Conference on Machine Learning, pp. 3898–3907, PMLR, 2018.
- A.-H. Phan, K. Sobolev, D. Ermilov, I. Vorona, N. Kozyrskiy, P. Tichavsky, and A. Cichocki, “How to train unstable looped tensor network,” arXiv preprint arXiv:2203.02617, 2022.