Hierarchical mixtures of Unigram models for short text clustering: The role of Beta-Liouville priors
Abstract: This paper presents a variant of the Multinomial mixture model tailored to the unsupervised classification of short text data. While the Multinomial probability vector is traditionally assigned a Dirichlet prior distribution, this work explores an alternative formulation based on the Beta-Liouville distribution, which offers a more flexible correlation structure than the Dirichlet. We examine the theoretical properties of the Beta-Liouville distribution, with particular focus on its conjugacy with the Multinomial likelihood. This property enables the derivation of update equations for a CAVI (Coordinate Ascent Variational Inference) algorithm, facilitating approximate posterior inference of the model parameters. In addition, we introduce a stochastic variant of the CAVI algorithm to enhance scalability. The paper concludes with empirical examples demonstrating effective strategies for selecting the Beta-Liouville hyperparameters.
- \bibcommenthead
- Amari SI (1998) Natural Gradient Works Efficiently in Learning. Neural Computation 10(2):251–276. 10.1162/089976698300017746
- Anderlucci L, Viroli C (2020) Mixtures of Dirichlet-Multinomial distributions for supervised and unsupervised classification of short text data. Advances in Data Analysis and Classification 14(4):759–770. 10.1007/s11634-020-00399-3
- Bakhtiari AS, Bouguila N (2016) A latent Beta-Liouville allocation model. Expert Systems with Applications 45:260–272. 10.1016/j.eswa.2015.09.044
- Bouguila N (2008) Clustering of Count Data Using Generalized Dirichlet Multinomial Distributions. IEEE Trans Knowl Data Eng 20(4):462–474. 10.1109/TKDE.2007.190726
- Bouguila N (2011) Count Data Modeling and Classification Using Finite Mixtures of Distributions. IEEE Trans Neural Netw 22(2):186–198. 10.1109/TNN.2010.2091428
- Bouguila N (2012) Hybrid Generative/Discriminative Approaches for Proportional Data Modeling and Classification. IEEE Transactions on Knowledge and Data Engineering 24(12):2184–2202. 10.1109/TKDE.2011.162
- Connor RJ, Mosimann JE (1969) Concepts of Independence for Proportions with a Generalization of the Dirichlet Distribution. Journal of the American Statistical Association 64(325):194–206. 10.2307/2283728
- Corsini N, Viroli C (2022) Dealing with overdispersion in multivariate count data. Computational Statistics & Data Analysis 170:107447. 10.1016/j.csda.2022.107447
- Gupta A, Kabe D (1999) On multivariate Liouville distribution. Metron LVII:173–179. URL https://ideas.repec.org/a/mtn/ancoec/99010212.html
- Gupta A, Song D (1996) Generalized Liouville Distribution. Computers & Mathematics with Applications 32(2):103–109. 10.1016/0898-1221(96)00107-1
- Gupta RD, Richards DSP (1987) Multivariate Liouville distributions. Journal of Multivariate Analysis 23(2):233–256. 10.1016/0047-259X(87)90155-2
- Lee SY (2021) Gibbs sampler and coordinate ascent variational inference: A set-theoretical review. Communications in Statistics - Theory and Methods pp 1–21. 10.1080/03610926.2021.1921214
- Nguyen D (2023) An in Depth Introduction to Variational Bayes Note. 10.2139/ssrn.4541076
- Rosenfeld R (2000) Two decades of statistical language modeling: where do we go from here? Proceedings of the IEEE 88(8):1270–1278. 10.1109/5.880083
- Sato Ma (2001) Online Model Selection Based on the Variational Bayes. Neural Computation 13(7):1649–1681. 10.1162/089976601750265045
- Silverman J (2023) RcppHungarian: Solves Minimum Cost Bipartite Matching Problems. URL https://CRAN.R-project.org/package=RcppHungarian, R package version 0.3
- Wesolowski J (1993) Some remarks on the multivariate liouville distribution. Statistics: A Journal of Theoretical and Applied Statistics 10.1080/02331888308802403
- Zitouni I (2007) Backoff hierarchical class n-gram language models: effectiveness to model unseen events in speech recognition. Computer Speech & Language 21(1):88–104. 10.1016/j.csl.2006.01.001
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.