HD-BWDM: Robust High-Dim Cluster Validation

• HD-BWDM is a robust nonparametric framework that replaces centroid-based measures with spatial medians or medoids to counteract noise and outliers in high-dimensional data.
• It integrates advanced dimensionality reduction techniques such as random projection and PCA to maintain meaningful distance computations and ensure computational efficiency.
• Employing trimming and robust clustering strategies, HD-BWDM provides a reliable index that converges to the true cluster structure, aiding in optimal partition selection.

High-Dimensional Between-Within Distance Median (HD-BWDM) is a robust, nonparametric clustering validation and inference framework specifically designed for high-dimensional, large-scale, and contaminated data. Unlike classical clustering validity indices that rely on centroids and are sensitive to both the structure of high-dimensional space and the presence of outliers, HD-BWDM leverages medoid or spatial median–based measures, integrates advanced dimensionality reduction (such as random projection and principal component analysis), and employs robust clustering strategies to directly address the challenges posed by modern high-dimensional data environments (Baragilly et al., 15 Oct 2025).

1. Motivation and Conceptual Foundations

HD-BWDM was introduced in response to the failures of traditional centroid-based indices such as Calinski-Harabasz, Davies-Bouldin, and Silhouette in high-dimensional datasets, where the "curse of dimensionality" renders Euclidean distances nearly uniform and highly sensitive to noise or outlying data. In these regimes, both between- and within-cluster distances become less informative due to concentration phenomena, undermining the utility of standard summary statistics for validating clustering structures.

The HD-BWDM approach replaces arithmetic centroids with spatial medians (or medoids), which offer superior robustness to the distributional tails and contamination—a crucial requirement when data may include anomalous or irrelevant features. Furthermore, it introduces dimensionality reduction steps (random projections and PCA) to ensure that distance computations remain meaningful and computationally tractable as dimensionality increases.

2. HD-BWDM Framework: Methodology

The HD-BWDM methodology comprises the following critical components (Baragilly et al., 15 Oct 2025):

• Dimensionality Reduction: Implements random projections (RP) and principal component analysis (PCA) to reduce the intrinsic dimensionality. Let $X \in \mathbb{R}^{n \times d}$ denote the data matrix. PCA projects to the top $p$ principal components ($X_p = X V_p$), and RP applies a random matrix $R$ (entries $\sim N(0, 1/p)$), yielding $X_p = X R$.
• Robust Clustering via Trimming and Medoids: Utilizes trimmed clustering algorithms ("tclust") that explicitly discard a proportion $a$ of outlying points. Subsequently, each cluster is summarized by its spatial median or medoid, reinforcing resistance to skewness and outlier influence.
• Index Computation: Defines the HD-BWDM index as the ratio of the average within-cluster median distance (AWDM) to the average between-cluster median distance (ABDM), all computed in the reduced $p$-dimensional space and possibly after trimming:

$$\text{HD-BWDM}(K, p, a) = \frac{\text{AWDM}_p(K, a)}{\text{ABDM}_p(K)}$$

Here, AWDM and ABDM incorporate robust summaries of distances, and the index serves both as a stopping rule and a comparative metric for cluster validation.

• Johnson-Lindenstrauss Guarantee: The framework ensures that the random projection step preserves pairwise distances with high probability, according to the Johnson-Lindenstrauss lemma:

$$(1 - \varepsilon)\|x_i - x_j\| \leq \|R x_i - R x_j\| \leq (1 + \varepsilon)\|x_i - x_j\|$$

for a suitable choice of projection dimension $p$ dependent on $\varepsilon$ and $\log n$.

3. Theoretical Properties

HD-BWDM establishes theoretical guarantees for its robustness and statistical consistency in high-dimensional regimes:

• Consistency: Under standard mixture models and provided the projection $R$ is Johnson-Lindenstrauss, the empirical HD-BWDM converges in probability to the population BWDM* as $n \rightarrow \infty$ and $p$ increases:

$$\sqrt{p}\, \text{HD-BWDM}_n(K, p, a) \to \text{BWDM}^*(K)$$

• Convergence Rate: The estimator satisfies a sharp probabilistic error rate:

$$\left| \text{HD-BWDM}_n(K,p,a) - \text{BWDM}^*(K) \right| = O_p\left(\epsilon + n^{-1/2}\right)$$

where $O_p(\epsilon)$ is induced by the random projection distortion and $O_p(n^{-1/2})$ from sampling variability.

• Optimality in Partition Selection: Under ideal conditions (well-separated, noise-free clusters), HD-BWDM achieves a maximum at the true number of clusters $K^*$. Any merging (decreasing $K$) or overpartitioning (increasing $K$) will tend to lower the index monotonically, supporting its interpretation as a clustering validation criterion.

4. Empirical Validation and Performance

A broad range of Monte Carlo experiments is performed on synthetic high-dimensional sets (e.g., $n{=}500$, $d{=}500$, with 10% contamination):

• HD-BWDM exhibits clear differentiation between the true clustering (highest index value) and alternatives produced by regular or trimmed k-means.
• As the embedding dimension $p$ increases (e.g., 150 to 400), the HD-BWDM index monotonically increases, indicating improved preservation of the true cluster geometry.
• Trimming and medoid/spatial median choices stabilize the index, making it insensitive to heavy-tailed noise and rendering it robust to aggressive contamination scenarios.
• Variability in the index is controlled, as measured by standard deviation across replicates.

No alternative centroid-based index demonstrates similar resistance to contamination and dimensionality effects in these testing regimes.

5. Connections to Cluster Validation and Modern Robust Statistics

HD-BWDM lies at the intersection of advances in robust statistics, high-dimensional geometry, and scalable clustering validation. The principal innovations distinguishing it from previous validation criteria are:

• Use of spatial medians or medoids, leveraging their vanishing bias relative to the mean in high dimensions under weak dependence (see (Schwank et al., 18 Aug 2025)).
• Systematic dimensionality reduction with minimal distortion guarantees, in contrast to prior methods that ignore the loss of informativeness of pairwise distances at high $d$.
• Explicit trimming of clusters before calculation, which protects against both global and local outliers, a feature missing from the majority of standard indices.
• Computational efficiency suitable for large-scale settings, since the index depends only on pairwise distances and robust summaries, all computable in projected space.

These advances place HD-BWDM as a leading candidate for cluster validation in modern large-scale, high-dimensional, and potentially contaminated datasets.

6. Practical Applications and Limitations

HD-BWDM is particularly relevant for applications in:

• Genomics and proteomics, where data are high-dimensional and contaminated by noise.
• Natural language processing and text mining, with large sparse feature matrices.
• Computer vision and high-throughput imaging, where traditional cluster validation measures are unreliable.

The computational scalability achieved by working in low-dimensional projected spaces and with robust cluster summaries is directly responsible for its practical viability. However, the necessity to properly tune the trimming parameter $a$ and dimensionality $p$, as well as careful implementation of medoid/spatial median computation in massive datasets, remain active areas for further research.

7. Summary

HD-BWDM is a robust, theoretically principled, and computationally efficient validation criterion for clustering in high-dimensional settings. Its central features—robustness via medians, dimensionality reduction via random projections/PCA, and empirical consistency—address the core deficiencies of existing centroid-based indices. The framework is supported by both rigorous asymptotic theory and comprehensive simulation studies, making it a substantial advancement in cluster analysis methodology for contemporary high-dimensional data environments (Baragilly et al., 15 Oct 2025).