Unsupervised Learning in a General Semiparametric Clusterwise Index Distribution Model

Abstract: This study introduces a general semiparametric clusterwise index distribution model to analyze how latent clusters affect the covariate-response relationships. By employing sufficient dimension reduction to account for the effects of covariates on the cluster variable, we develop a distinct method for estimating model parameters. Building on a subjectwise representation of the underlying model, the proposed separation penalty estimation method partitions individuals and estimates cluster index coefficients. We propose a convergent algorithm for this estimation procedure and incorporate a heuristic initialization to expedite optimization. The resulting partition estimator is subsequently used to fit the cluster membership model and to construct an optimal classification rule, with both procedures iteratively updating the partition and parameter estimators. Another key contribution of our method is the development of two consistent semiparametric information criteria for selecting the number of clusters. In line with principles of classification and estimation in supervised learning, the estimated cluster structure is consistent and optimal, and the parameter estimators possess the oracle property. Comprehensive simulation studies and empirical data analyses illustrate the effectiveness of the proposed methodology.