Hierarchical SuperKMeans Clustering

• Hierarchical SuperKMeans is a family of algorithms that initially over-segments data using K-means and then applies hierarchical merging to uncover nested, non-convex clusters.
• The Multilevel Wasserstein Means (MWM) formulation replaces centroids with discrete measures and employs Wasserstein distances to jointly optimize local and global clustering.
• The K-means + Hierarchical Hybrid (K-mH) algorithm uses a model-based dissimilarity after over-segmentation to achieve high accuracy and scalability in large datasets.

Hierarchical SuperKMeans refers to a family of algorithms that merge the computational efficiency of K-means clustering with the flexibility of hierarchical procedures, thereby enabling the detection of complex, possibly non-spherical and multilevel clustering structures. Two principal algorithmic formulations bearing this designation are the Multilevel Wasserstein Means (MWM) (Ho et al., 2017) and the K-means + Hierarchical Hybrid (K-mH) (Peterson et al., 2017). Both share the paradigm of initial over-segmentation via K-means-like local clustering, followed by a global or hierarchical merging phase; they differ fundamentally in their mathematical formulation, objective functions, and operational scope.

1. Algorithmic Foundations and Motivation

Hierarchical SuperKMeans addresses limitations inherent in standard K-means and hierarchical agglomerative clustering. K-means presumes spherical, similarly sized clusters and demands prior knowledge of cluster number, while agglomerative clustering scales quadratically in computation and memory, hindering large-scale applications and often dissolving cluster homogeneity near the dendrogram root (Peterson et al., 2017). Hierarchical SuperKMeans remedies these deficits by initially over-partitioning the dataset into small, locally coherent groups and subsequently aggregating these entities using a secondary criterion—Wasserstein barycenter distances in the MWM formulation, or a model-based dissimilarity in K-mH. The resulting methodology is capable of identifying nested, non-convex clusters efficiently in large datasets.

2. Formulation: Multilevel Wasserstein Means

The MWM algorithm (Ho et al., 2017) recasts clustering as a joint optimization over discrete probability measures. For a hierarchically structured dataset, suppose each group $j$ is represented by its empirical distribution $P_{nj}$ over $\Theta \subset \mathbb{R}^d$. The algorithm jointly fits, for each group, a $k_j$-atomic measure $G_j \in \ell_{k_j}(\Theta)$ approximating $P_{nj}$, and globally clusters the collection $\{G_j\}_{j=1}^m$ into $M$ prototypes $\{H_i\}_{i=1}^M$ within $\mathcal{P}_2(\Theta)$. The joint objective function is: $P_{nj}$0 where $P_{nj}$1 denotes the squared $P_{nj}$2-Wasserstein distance. The “local” term quantifies the optimal-transport discrepancy between empirical and atomic measures within groups; the “global” term clusters the groups’ measures via assignment to global prototypes, also in Wasserstein space.

An alternating minimization (“Lloyd-style”) scheme is used:

1. Assignment: Each local measure $P_{nj}$3 is assigned to its nearest global prototype $P_{nj}$4 via Wasserstein distance.
2. Local update: Each $P_{nj}$5 is updated as the barycenter of the empirical sample $P_{nj}$6 and the assigned global prototype $P_{nj}$7, constrained to $P_{nj}$8 atoms.
3. Global update: Each $P_{nj}$9 is recomputed as the Wasserstein barycenter of its assigned $\Theta \subset \mathbb{R}^d$0.

This procedure generalizes classic Lloyd’s K-means, replacing centroids and Euclidean distances with discrete measures and Wasserstein geometry. In special cases (single-atom measures), barycenter steps reduce to weighted means in $\Theta \subset \mathbb{R}^d$1; otherwise, Sinkhorn or network-flow routines are employed (Ho et al., 2017).

3. K-means + Hierarchical Hybrid (K-mH) Algorithm

The K-mH algorithm (Peterson et al., 2017) is designed to both scale to large $\Theta \subset \mathbb{R}^d$2 and recover general-shaped clusters, leveraging a two-phase scheme:

1. K-means Over-segmentation: Run K-means with a large $\Theta \subset \mathbb{R}^d$3 (often $\Theta \subset \mathbb{R}^d$4), yielding a fine partition into small, roughly spherical “entities” $\Theta \subset \mathbb{R}^d$5.
2. Hierarchical Merging: Treating each $\Theta \subset \mathbb{R}^d$6 as an atomic entity, compute a tailored dissimilarity

$\Theta \subset \mathbb{R}^d$7

where $\Theta \subset \mathbb{R}^d$8 is the probability that a sample from $\Theta \subset \mathbb{R}^d$9 is closer (in Mahalanobis distance) to $k_j$0 than to $k_j$1 itself, as computed using normal approximations or noncentral $k_j$2 results. Agglomerative hierarchical clustering using single linkage and this data-driven dissimilarity then merges entities, with the desired number of clusters $k_j$3 identified via gaps in the dendrogram or consensus over multiple partitions.

A consensus partition is ultimately chosen by maximizing the average Adjusted Rand index across a grid of candidate $k_j$4, $k_j$5 pairs. The algorithm removes “scatter” clusters prior to analysis and visualizes clustering stability via a co-association matrix heatmap.

4. Theoretical Properties and Computational Aspects

The MWM algorithm enjoys the following guarantees (Ho et al., 2017):

• Monotonic descent: Each alternating step (assignment, local, and global barycenters) decreases or maintains the joint objective, ensuring convergence to a local minimum.
• Statistical consistency: As sample sizes per group $k_j$6, the empirical local measures converge in Wasserstein distance to population minimizers, with both objective values and measures converging in $k_j$7.
• Computational complexity: Given $k_j$8 groups each approximated by a $k_j$9-atom measure, each assignment step computes $G_j \in \ell_{k_j}(\Theta)$0 Wasserstein distances, with per-distance cost $G_j \in \ell_{k_j}(\Theta)$1 (network flow) or $G_j \in \ell_{k_j}(\Theta)$2 (Sinkhorn). Local/global barycenter updates maintain similar scaling, with further acceleration by constraining support sizes.

The K-mH algorithm, by contrast, achieves linearithmic scaling with respect to data size for K-means over-segmentation and quadratic scaling only in the much smaller space of "entities" $G_j \in \ell_{k_j}(\Theta)$3 during hierarchical merging (Peterson et al., 2017):

• K-means phase: $G_j \in \ell_{k_j}(\Theta)$4 for $G_j \in \ell_{k_j}(\Theta)$5 restarts.
• Hierarchical phase: $G_j \in \ell_{k_j}(\Theta)$6 (negligible for $G_j \in \ell_{k_j}(\Theta)$7).
• Empirical results report clustering of $G_j \in \ell_{k_j}(\Theta)$8 points in minutes.

5. Parameter Selection and Practical Guidance

Both methods require careful choice of tuning parameters:

Algorithm	Local parameter	Global parameter	Selection strategy
MWM	$G_j \in \ell_{k_j}(\Theta)$9 (group atoms)	$P_{nj}$0 (global)	Grid search, “elbow” method, BIC, CV
K-mH	$P_{nj}$1 (entities)	$P_{nj}$2 (clusters)	Krzanowski criterion, dendrogram gaps, heatmap block structure

Additional considerations:

• In MWM, the tradeoff between local/global clustering can be tuned by a multiplier $P_{nj}$3 on the global term.
• Both algorithms benefit from robust K-means initialization (multiple starts or k-means++ seeding).
• For MWM, the entropic regularization parameter $P_{nj}$4 in Sinkhorn computations controls speed-accuracy tradeoff.
• Variants such as MWMS (“with sharing”) restrict all group supports to a global set of atoms, amounting to a further hierarchical layer (Ho et al., 2017).

6. Empirical Performance and Illustrative Cases

Evaluation on synthetic and real datasets demonstrates the ability of both algorithms to recover non-convex and multimodal clustering structures:

• K-mH: Achieves high Adjusted Rand indices ($P_{nj}$5) on 2D “bullseye” and “banana-sphere” data, with interpretable clusterings on high-dimensional Olive Oil and digit datasets ($P_{nj}$6 ranging from 0.54 to 0.85) (Peterson et al., 2017). Visual inspection of the co-association matrix $P_{nj}$7 reveals well-separated block structures corresponding to the true clusters.
• MWM: Demonstrates scalability and flexibility across synthetic group-structured data and hierarchical corpora, with consistency properties corroborated by experiments (Ho et al., 2017).

A plausible implication is that both frameworks bridge the algorithmic gap between rigid partitioning and global shape flexibility, removing undesirable sensitivity to initialization or restrictive assumptions on distributional shape, while affording efficient deployment at scale.

7. Relation to Broader Clustering Methodology

The Hierarchical SuperKMeans framework generalizes classical K-means (Euclidean centroids, flat partitions) and hierarchical clustering (dendrogram structures, linkage criteria). In MWM, the replacement of centroids by discrete measures and Euclidean metrics by Wasserstein distances notably connects statistical clustering to optimal transport theory. K-mH’s use of a model-based, probabilistically motivated dissimilarity for merging K-means “entities” is related to evidence accumulation and consensus-based clustering, but with sharper computational and statistical guarantees (Peterson et al., 2017). These methods position themselves as alternatives to model-based approaches (e.g., Gaussian mixtures), density-based clustering (e.g., DBSCAN), and more recent optimal transport clustering frameworks.

Both MWM and K-mH exemplify the modern trend towards hybrid, scalable clustering algorithms that leverage structure, optimal transport, and model-based techniques to contend with heterogeneous data complexity and massive scale.