Bayesian Nonparametric Clustering to Support Medical Decision-Making: A Variational Inference Approach

Abstract: Medical decision-making increasingly requires rapid and reliable assignment of patients to disease subtypes, as many diseases are no longer treated as single entities. For example, cancer patients may be stratified into aggressive and non-aggressive subtypes, with different treatment strategies for each group. We propose a Bayesian nonparametric approach based on a Dirichlet process mixture model for clustering individuals into disease subtypes. We implement a coordinate ascent variational inference algorithm, yielding an effective and computationally efficient alternative to Markov chain Monte Carlo (MCMC), to support medical decision-making. In synthetic experiments, we demonstrate that the proposed approach accurately assigns observations to their ground-truth clusters, achieving strong performance across evaluation metrics, such as homogeneity and completeness. Additionally, we illustrate the proposed approach achieves a substantial improvement in computational cost compared to MCMC, without sacrificing accuracy that would lead to the increased risk of misdiagnosis.