A Mean Curvature Approach to Boundary Detection: Geometric Insights for Unsupervised Learning

Abstract: Accurate boundary detection in high-dimensional data remains a central challenge in unsupervised learning, particularly in the presence of non-linear structures and heterogeneous densities. In this work, we introduce Mean Curvature Boundary Points (MCBP), a novel geometric framework grounded in Geometric Machine Learning that departs from traditional density-based approaches by explicitly modeling the intrinsic curvature of the data manifold. The method relies on a discrete approximation of the shape operator, estimated from local k-nearest neighbor patches, to compute pointwise mean curvature without requiring explicit manifold parametrization. The key insight of MCBP is to use mean curvature as a principled descriptor of boundary structure: high-curvature regions naturally correspond to transitions between clusters, geometric irregularities, and low-density interfaces. This yields a unified geometric interpretation of boundary, outlier, and transition points. We further introduce an adaptive percentile-based thresholding scheme that enables multiscale boundary extraction without relying on ad hoc density parameters. Beyond detection, we propose a curvature-driven data decomposition that separates samples into smooth (low-curvature) and boundary (high-curvature) subsets, effectively acting as a non-linear geometric filtering mechanism. This representation enhances cluster separability and improves the robustness of downstream unsupervised algorithms. Extensive experiments on synthetic and real-world datasets demonstrate that MCBP consistently improves clustering performance, particularly in complex and high-dimensional scenarios. These results position MCBP as a concrete contribution to Geometric Machine Learning, highlighting the potential of curvature-aware analysis as a unifying paradigm bridging differential geometry and data-driven modeling.

• The paper introduces Mean Curvature Boundary Points (MCBP), which estimates local curvature from k-nearest-neighbor covariance and quadratic patches to identify boundaries across nonlinear, anisotropic, and unevenly sampled data.
• MCBP filters high-curvature points before clustering and improves internal validation across roughly 60 OpenML datasets, raising average silhouette scores for HDBSCAN from 0.227 to 0.359 and for a hybrid pipeline from 0.143 to 0.303.
• The method offers a practical geometric alternative to density-based detection but remains heuristic, depends on neighborhood and percentile choices, and lacks formal consistency guarantees or external-label validation.

Motivation and problem setting

The paper addresses boundary point detection in multivariate datasets, arguing that prevailing density-based formulations—DBSCAN, LOF, HDBSCAN, BRIM, BPF and related methods—conflate "boundary" with "low density" and therefore fail on non-linear manifolds, anisotropic distributions, and heterogeneous sampling. The author proposes Mean Curvature Boundary Points (MCBP), a framework that replaces density estimation with a discrete estimate of mean curvature computed from local $k$-nearest neighbor patches. The central claim is that high-curvature regions constitute a principled surrogate for boundaries: they capture dense-cluster transitions, concave/convex interfaces, and sparse peripheries within a single geometric criterion, unifying outliers, transition regions, and decision interfaces.

Geometric formulation

MCBP is built from classical differential geometry. For a manifold $\mathcal{M} \subset \mathbb{R}^m$, local geometry at a point is described by the first fundamental form $\mathbf{G}$ (metric tensor), the second fundamental form $\mathbf{B}$ (bending relative to the normal space), and the shape operator $\mathbf{S} = \mathbf{G}^{-1}\mathbf{B}$, whose trace gives the mean curvature $H$. The paper reviews these objects carefully and then constructs discrete analogues directly from data:

• The metric tensor at sample $\mathbf{x}_i$ is approximated by the inverse of the empirical covariance over its patch $P_i = \{\mathbf{x}_i\} \cup \eta_i$: $\mathbb{I}_i \approx \boldsymbol{\Sigma}_i^{-1}$, inducing a Mahalanobis-type local geometry that adapts to anisotropy.
• The second fundamental form is approximated by fitting a quadratic design matrix $X_i$ (constant, linear, quadratic terms) in the eigenbasis of $\mathcal{M} \subset \mathbb{R}^m$0 and forming $\mathcal{M} \subset \mathbb{R}^m$1 from its second-order columns.
• The shape operator is then approximated as $\mathcal{M} \subset \mathbb{R}^m$2, and the curvature score is $\mathcal{M} \subset \mathbb{R}^m$3.

This construction yields a useful interpretation: large $\mathcal{M} \subset \mathbb{R}^m$4 arises when both local dispersion ($\mathcal{M} \subset \mathbb{R}^m$5) and second-order deformation ($\mathcal{M} \subset \mathbb{R}^m$6) are significant—i.e., low-density regions undergoing strong geometric bending, which is precisely where boundary points live. Boundary labels are assigned by a percentile threshold $\mathcal{M} \subset \mathbb{R}^m$7 on normalized curvature scores, giving multiscale control of boundary thickness without density-specific parameters. Notably, the estimator is extrinsic in flavor: it mixes covariance (first-order statistics) with a Hessian-derived second-order term, rather than computing intrinsic Riemannian curvature; the paper does not provide consistency guarantees for this discrete shape-operator approximation, which remains a theoretical gap.

Complexity is $\mathcal{M} \subset \mathbb{R}^m$8 for graph construction plus $\mathcal{M} \subset \mathbb{R}^m$9 for the per-sample loop (dominated by eigendecomposition), reducible to $\mathbf{G}$0 after PCA projection to intrinsic dimension $\mathbf{G}$1; per-sample computations are embarrassingly parallel.

Curvature-driven dataset filtering

Beyond detection, the paper frames MCBP as a non-linear low-pass filter: partitioning $\mathbf{G}$2 into a smooth set $\mathbf{G}$3 and a boundary set $\mathbf{G}$4 removes "high-frequency" components of the data distribution while preserving cluster cores. This decomposition supports three downstream uses evaluated empirically: clustering directly on $\mathbf{G}$5, initializing $\mathbf{G}$6-means with centroids estimated on $\mathbf{G}$7 ("S-centroids"), and hybrid pipelines combining HDBSCAN on $\mathbf{G}$8 with label propagation to $\mathbf{G}$9 via a 1-NN classifier. The filtering interpretation is the paper's main conceptual contribution, extending signal-processing intuition to non-Euclidean domains, though the analogy is asserted rather than formalized.

Experimental results

Synthetic experiments on Gaussian blobs, elliptical clusters, and two-moons validate the qualitative behavior: low-curvature estimates concentrate in dense, flat regions; high-curvature estimates mark peripheries, cluster interfaces, and strongly bent arc segments, adapting to anisotropy without global assumptions.

Real-world evaluation uses roughly 60 OpenML datasets spanning 61 to 70,000 samples and up to 10,935 features (high-dimensional sets reduced to 50 principal components). Four experimental protocols are reported using internal indices (Silhouette Coefficient, Calinski–Harabasz, Davies–Bouldin):

Protocol	Metric	Original	With MCBP
k-means++ on filtered data	SC	0.193	0.288
k-means++ on filtered data	CH	693.7	948.1
k-means++ on filtered data	DB	2.335	1.737
S-centroid initialization	SC	0.190	0.246
HDBSCAN on filtered data	SC	0.227	0.359
HDBSCAN on filtered data	CH	183.4	7,187.3
Hybrid HDBSCAN + k-means (S-centroids)	SC	0.1427	0.3025
HDBSCAN on $\mathbf{B}$0 + 1-NN on $\mathbf{B}$1	SC	0.1047	0.2308

The largest single gains appear in noisy or overlapping datasets (satimage, gas-drift, coil-20, DLBCL, tr23.wc), consistent with the hypothesis that high-curvature points are structurally ambiguous. The hybrid protocol more than doubles average SC relative to standalone HDBSCAN. The paper is candid about counterexamples: the CH index occasionally decreases despite improvements in SC and DB (digits, letter, vowel), and the DB index degrades slightly on Satellite, Engine1, heart-h, and AP_Breast_Colon—attributed to DB's sensitivity to dispersion and to interpolation errors in sparsely sampled boundary regions.

Two methodological caveats bear on interpretation. First, all evaluations rely exclusively on internal validation indices, which reward exactly the kind of variance/compactness structure that removing peripheral samples mechanically inflates; no external ground-truth comparison (e.g., ARI against class labels) is reported, so the improvement in cluster quality versus mere index inflation cannot be fully disentangled. Second, the percentile threshold $\mathbf{B}$2 is fixed at 0.75 (or tuned per experiment between 50% and 75%), meaning a quarter of the data is discarded before clustering—an aggressive preprocessing choice whose interaction with small-$\mathbf{B}$3 datasets is not systematically ablated.

Limitations and open questions

The paper concedes several dependencies. Curvature estimation requires positive-definite local covariances, which fail in degenerate neighborhoods and in regimes where dimensionality approaches sample size; PCA pre-reduction mitigates but does not eliminate this. The discrete approximation $\mathbf{B}$4 is heuristic—it is a Gram matrix of second-order design features, not a convergent estimator of the second fundamental form under any stated sampling model—and no error bounds relate $\mathbf{B}$5 to true mean curvature. Sensitivity analysis over the neighbor count $\mathbf{B}$6 and threshold percentile $\mathbf{B}$7 is limited. Open questions left by the work include whether formal convergence guarantees can be established for the discrete shape operator under manifold sampling assumptions, how the method behaves under adversarial or heavy-tailed noise, and whether external-label agreement improves alongside internal indices when classes are known.

Conclusion

MCBP reframes boundary detection as a curvature estimation problem, replacing density thresholds with a discrete shape operator computed from $\mathbf{B}$8-NN patches, and converts detection into a reusable preprocessing primitive: smooth/boundary decomposition, curvature-informed centroid initialization, and hybrid density-plus-centroid pipelines. Across dozens of OpenML datasets it yields consistent internal-index improvements for both $\mathbf{B}$9-means and HDBSCAN, with the strongest effects on noisy, overlapping, and geometrically complex data. Its evidential basis is entirely internal-validation-based and its geometric estimator lacks formal guarantees, but the paper makes a concrete case that second-order differential-geometric quantities carry information that purely density-driven criteria discard.