Revisiting k-means: New Algorithms via Bayesian Nonparametrics (1111.0352v2)

Abstract: Bayesian models offer great flexibility for clustering applications---Bayesian nonparametrics can be used for modeling infinite mixtures, and hierarchical Bayesian models can be utilized for sharing clusters across multiple data sets. For the most part, such flexibility is lacking in classical clustering methods such as k-means. In this paper, we revisit the k-means clustering algorithm from a Bayesian nonparametric viewpoint. Inspired by the asymptotic connection between k-means and mixtures of Gaussians, we show that a Gibbs sampling algorithm for the Dirichlet process mixture approaches a hard clustering algorithm in the limit, and further that the resulting algorithm monotonically minimizes an elegant underlying k-means-like clustering objective that includes a penalty for the number of clusters. We generalize this analysis to the case of clustering multiple data sets through a similar asymptotic argument with the hierarchical Dirichlet process. We also discuss further extensions that highlight the benefits of our analysis: i) a spectral relaxation involving thresholded eigenvectors, and ii) a normalized cut graph clustering algorithm that does not fix the number of clusters in the graph.

• The paper introduces DP-means, a novel algorithm that extends k-means by dynamically creating clusters using a distance threshold controlled by a penalty parameter.
• The paper rigorously establishes an underlying objective function linking Bayesian nonparametrics to k-means, including a spectral relaxation for advanced clustering insights.
• The paper demonstrates practical scalability by extending the framework to hierarchical models, benefiting large-scale applications like image analysis and document clustering.

Revisiting K-Means: New Algorithms via Bayesian Nonparametrics

The paper presents a novel exploration of clustering algorithms by integrating Bayesian nonparametric approaches with classical k-means methods. The authors address the limitations of the classical k-means algorithm, which lacks the flexibility of Bayesian models despite its simplicity and scalability. Bayesian nonparametric models, particularly the Dirichlet process (DP) mixture model, introduce the advantage of allowing an infinite number of potential clusters, addressing one of the core limitations of k-means.

Methodological Insights

The researchers propose a new clustering algorithm, DP-means, which is formulated from the asymptotic behavior of a Gibbs sampling approach for Dirichlet process mixtures. By considering this asymptotic limit, the DP-means algorithm transforms into a hard clustering method reminiscent of k-means. The key innovation is the ability to create new clusters dynamically: a new cluster is formed whenever a data point is sufficiently distant from all existing cluster centroids, with the distance threshold controlled by a parameter, $\lambda$.

The paper further extends this framework to handle multiple datasets using hierarchical Bayesian nonparametrics via hierarchical Dirichlet processes (HDP). The result is an algorithm that not only clusters individual datasets but also allows for shared cluster structures across different datasets. The complexity of real-world datasets, often characterized by shared substructures, is thus addressed more naturally.

Theoretical Contributions

The authors make the significant theoretical contribution of identifying an underlying objective function for the proposed DP-means algorithm. This objective function resembles the k-means objective with an additional penalty term for the number of clusters, drawing a direct theoretical line between the Bayesian framework and classical k-means. The convergence properties of this algorithm are rigorously established, offering a reliable clustering process that has both the theoretical grounding and practical performance attributes.

Another noteworthy extension is the spectral relaxation of the DP-means objective. This is comparable to the k-means spectral relaxation, where the solution is found by computing eigenvectors corresponding to eigenvalues above a certain threshold. Such relaxation provides additional insights into the interplay between spectral methods and Bayesian nonparametrics, suggesting further potential for advanced methods in complex clustering tasks.

Practical Implications and Future Directions

The integration of the Bayesian nonparametric framework grants the proposed algorithms enhanced flexibility, allowing them to adapt to datasets with varying cluster counts without necessitating pre-specification. This is particularly beneficial for large-scale applications such as image analysis and document clustering.

The practical implications of this work extend into various fields where unsupervised learning is critical. For example, in computer vision tasks such as the construction of visual vocabularies, the flexibility and scalability of the DP-means and its hierarchical extension could prove invaluable.

Looking forward, several avenues offer promising potential for expanding this work: refining the scalability and efficiency of these algorithms, developing alternate relaxations and constraints, and adapting the approach to other nonparametric Bayesian models. Future research could examine the integration with more complex tree structures or hierarchical models, ultimately leading to more comprehensive frameworks capable of handling even broader classes of clustering problems.

In conclusion, the paper represents a meaningful step towards integrating the structured adaptability of Bayesian nonparametric models with the computational simplicity that underpins classical clustering. The scalability, flexibility, and theoretical grounding provided here lay a foundation for continued advancements in clustering methodologies.