Turtle shell clustering: A mixture approach to discriminative clustering with applications to flow cytometry and other data

Abstract: Generative approaches to clustering provide information on geometric properties of clusters, whereas discriminative approaches provide boundaries between clusters. Ideas from both approaches are incorporated to present a fully unsupervised, probabilistic, and discriminative clustering method via a regularized mutual information objective function, wherein a mixture of mixtures of Gaussian and uniform distributions is used for formulation of the conditional model. Automatic selection of the number of components is established with the introduction of the regularizing term and a merge step, similar to those applied in reversible jump Markov chain Monte Carlo methods used in Bayesian clustering. Consequently, the turtle shell method -- a fully unsupervised clustering method capable of estimating non-linear boundary lines, automatically selecting the number of components, and capturing intuitive clusters in the presence of data abnormalities such as noise and/or irregular cluster shapes -- is introduced. We test this method on various simulated and real datasets commonly explored in clustering research, and extend the analysis to datasets arising from flow cytometry experiments.

• The paper introduces an innovative turtle shell clustering method that combines generative and discriminative paradigms using a mixture-of-mixtures formulation to achieve robust, noise-resistant clustering.
• The method applies mutual information maximization with regularization and intelligent merge steps for automatic cluster number selection and improved boundary estimation.
• Evaluations demonstrate that the approach outperforms traditional Gaussian mixture models, showing enhanced accuracy and stability in real-world datasets like flow cytometry.

Turtle Shell Clustering: A Mixture Approach to Discriminative Clustering

Conceptual Foundations and Motivation

The turtle shell clustering method advances unsupervised learning by integrating generative and discriminative paradigms. Traditional model-based clustering typically adopts Gaussian mixture models (GMMs), which are effective for convex and normal-shaped clusters but often fail when clusters are non-convex, contain noise, or show irregular shapes. Discriminative clustering, in contrast, directly estimates boundaries but lacks probabilistic interpretability and robust uncertainty quantification.

This method builds upon mutual information maximization frameworks, especially the Regularized Information Maximization (RIM) model [Krause et al., 2010]. By combining robust mixture modeling (mixtures of Gaussian and uniform distributions) [Browne et al., 2011] with order selection techniques akin to reversible jump Markov chain Monte Carlo (RJMCMC), the turtle shell clustering introduces a fully unsupervised, regularized mutual information objective that achieves:

• Non-linear boundary estimation.
• Automatic selection of the number of clusters.
• Robustness to noise, density abnormalities, and irregular cluster shapes.

Importantly, this method targets problems such as flow cytometry in immunological research, where manual gating is subjective and unreliable, and automated discriminative clustering has been underdeveloped.

Methodological Framework

The core objective function optimizes mutual information between latent cluster assignments and data, penalized by regularization terms to control cluster proliferation and cluster shape:

$$F(\theta; X, \lambda) = I_\theta\{y; x\} - R_1(\theta; \lambda_1) - R_2(\theta; \lambda_2)$$

The conditional model, $p(y_{ik} | x_i; \theta_k)$, is specified as a mixture of mixtures (Gaussian + uniform) with softmax-transformed mixing proportions. The uniform component addresses irregular density, noise, and non-Gaussian structure. Regularization $R_1$ penalizes small clusters, leading to automatic removal based on size thresholds. $R_2$ penalizes divergence between Gaussian and uniform component means within each cluster, preventing pathological cluster splitting.

Cluster order selection leverages an intelligent merge step: clusters with high uncertainty are considered for merging based on similarity metrics, emulating RJMCMC protocols. Parameter optimization proceeds via L-BFGS-B, with initialization by Louvain community detection on $k$-nearest neighbor graphs. The initialization is robust, allowing effective clustering even when the initial cluster count is overestimated.

Figure 1: Histogram of data generated from a mixture of Gaussian and uniform distributions, illustrating basic density structure.

Numerical Evaluation and Comparative Results

Extensive simulation studies demonstrate the superiority of the turtle shell method compared to both EM-optimized GMMs (with BIC and ICL order selection) and simplified turtle shell models (pure Gaussian mixtures).

Key Numerical Highlights:

• Cross Simulation: Turtle shell merges crossed-over Gaussians into intuitive crossed clusters, consistently selecting the correct number of clusters (4 out of 6 Gaussian generators), outperforming GMMs which typically select more clusters and misassign intersection points.
• Mixture of Gaussians Simulation: Turtle shell reliably identifies the intuitive three-cluster solution when two generators are close together (always 3 clusters identified), while GMM/BIC estimates typically oversplit.
• Outlier Simulation: Turtle shell robustly incorporates uniform-distributed outliers into Gaussian clusters, whereas GMM-based methods arbitrarily assign outliers to separate clusters or split true groups.

On real-world datasets including flow cytometry (Levine13, HIPC), wine quality, bankruptcy, and others:

• Turtle shell achieves highest or nearly highest adjusted Rand index (ARI) across almost all datasets.
• ARI on flow cytometry datasets matches or exceeds specialized methods (PhenoGraph, cytometree).
• Robustness to initialization is observed for $k=15, 25, 45$; results are stable unless extreme $k$ values are used.

Practical and Theoretical Implications

The turtle shell clustering method substantially improves the interpretability and robustness of unsupervised clustering, particularly in domains with non-convex clusters, noise, or high-density regions. For flow cytometry, it offers a fully automated, objective alternative to manual gating, capable of modeling non-Gaussian populations critical in immunological research.

Practically, this method is adaptable to a wide array of clustering tasks beyond biomedical applications, including financial risk evaluation, anomaly detection, and high-dimensional biological data, where traditional generative models or kernel methods are ineffective.

Theoretically, the mixture-of-mixtures formulation, combined with regularized mutual information, bridges the gap between uncertainty quantification and boundary discrimination. The merge step embodies Bayesian order selection principles without requiring explicit MCMC, allowing efficient computation.

Future Directions

Potential advancements include:

• Extension to other mixture components suitable for complex or structured data (e.g., skewed, heavy-tailed distributions).
• Integration with deep generative models or GNNs for clustering on graphs or manifold data.
• Application to novel domains with heterogeneous or multimodal input spaces, including multi-omic or temporal data.

Further research could evaluate convergence properties, computational scalability on ultra-large datasets, and combine cluster purity metrics with probabilistic calibration.

Conclusion

The turtle shell clustering method establishes a discriminative, probabilistic approach robust to noise, density anomalies, and non-linear cluster boundaries. Its integration of mutual information maximization, mixture-of-mixtures modeling, and Bayesian order selection principles yields consistent high performance across diverse datasets, including challenging flow cytometry applications. Future work should explore its extension to broader mixture modeling and integration into complex data structures, paving the way for improved unsupervised learning frameworks.