HDBSCAN-MC Algorithm
- HDBSCAN-MC is a family of methods that extend HDBSCAN* by enabling efficient exploration across various mpts values and incorporating uncertainty analysis.
- One branch accelerates clustering by reusing a reduced mutual reachability graph to compute over 128 hierarchies for roughly the cost of 2 HDBSCAN* runs.
- Another branch uses Monte Carlo simulations to account for astrometric uncertainties, producing robust cluster membership scores for Gaia DR3 YSOs.
Searching arXiv for the cited papers to ground the article in current records.
HDBSCAN-MC denotes a family of HDBSCAN*-derived procedures rather than a single universally standardized algorithmic object. In the arXiv literature, the label is used most explicitly for two distinct purposes: an efficient method for computing multiple HDBSCAN* hierarchies over a range of mpts values by reusing one reduced graph (Neto et al., 2017), and a Monte Carlo–propagated HDBSCAN* pipeline for uncertainty-aware cluster membership estimation in Gaia DR3 Young Stellar Object catalogs (Patel et al., 13 Jul 2025). Related work also describes accelerated, thresholded, and geometrically reconstructed HDBSCAN* variants that are sometimes interpreted as belonging to the same broader methodological space, although several of those papers do not use the exact name “HDBSCAN-MC” (McInnes et al., 2017, Malzer et al., 2019, Garcia-Pulido et al., 2022).
1. Core HDBSCAN* machinery
HDBSCAN* is a hierarchical density-based clustering method organized around the parameter mpts or minPts, interpreted as a minimum-points or smoothing factor. It can be viewed as a natural extension of DBSCAN, a hierarchical version of DBSCAN*, and an MST-based or robust single-linkage method. Its central construction is the mutual reachability graph, whose edge weights are defined from core distances and ordinary metric distances (McInnes et al., 2017, Neto et al., 2017).
For a point , the core distance with respect to mpts is
$\cd(p)=d(p, mpts\operatorname{-NN}(p)).$
For two points , the mutual reachability distance is
$\mrd(p,q)=\max\{\cd(p),\cd(q),d(p,q)\}.$
This distance inflates separations through sparse regions by incorporating local density via the core distances. In HDBSCAN*, the hierarchy is recovered from the minimum spanning tree of the complete graph weighted by mutual reachability distance. For a fixed threshold , thresholding the mutual reachability graph yields the same connected components as DBSCAN* at that scale, and the MST suffices because the hierarchy can be reconstructed by removing edges in descending order (McInnes et al., 2017, Garcia-Pulido et al., 2022).
A further HDBSCAN* layer is the condensed tree. The hierarchy is pruned using a minimum cluster size : clusters smaller than are removed, one surviving child can be treated as the continuation of its parent, and true splits occur when more than one child has size at least . Flat cluster extraction is then commonly posed as a stability-maximization problem over the condensed tree, subject to non-overlap constraints (McInnes et al., 2017, Malzer et al., 2019).
2. HDBSCAN-MC as multiple-hierarchy computation
In "Efficient Computation of Multiple Density-Based Clustering Hierarchies" (Neto et al., 2017), HDBSCAN-MC corresponds to an efficient strategy for computing all HDBSCAN* hierarchies for a range of mpts values. The motivating problem is that different mpts values can reveal different cluster structures, but naively exploring a range requires rerunning HDBSCAN* independently for each value. Since the mutual reachability graph is conceptually complete and its edge weights change with mpts, repeated MST computation is expensive (Neto et al., 2017).
The paper’s main idea is to replace the complete mutual reachability graph by a much smaller Relative Neighborhood Graph adapted to mutual reachability distance. For fixed mpts = i, the relevant graph is denoted , and the key structural guarantees are: $MST(\mrg[i]) \subseteq \#1{i},$
$\cd(p)=d(p, mpts\operatorname{-NN}(p)).$0
and therefore
$\cd(p)=d(p, mpts\operatorname{-NN}(p)).$1
These results imply that a single graph computed for the largest mpts value contains the information needed for all smaller values. The algorithm computes core distances once, constructs a Well-Separated Pairs Decomposition, builds a supergraph using Symmetric Bichromatic Closest Neighbors, optionally filters to obtain RNG**-, RNG*- or exact RNG-based variants, and then recomputes edge weights and MSTs on the same reduced graph for each mpts in the range (Neto et al., 2017).
This formulation preserves correctness of HDBSCAN* for every mpts in the range while amortizing the expensive graph construction. The paper’s headline claim is that one can obtain over one hundred hierarchies for the computational cost equivalent to running HDBSCAN* about 2 times, and, more specifically, that as many as 128 MSTs or hierarchies can be computed for the cost of naively computing about 2 (Neto et al., 2017).
3. HDBSCAN-MC as Monte Carlo uncertainty propagation
In "Identification of Outer Galaxy Cluster Members Using Gaia DR3 and Multidimensional Simulation" (Patel et al., 13 Jul 2025), HDBSCAN-MC denotes an uncertainty-aware extension of HDBSCAN* for Gaia DR3 YSO cluster membership assignment. Here the method is explicitly described as Monte Carlo simulation coupled with HDBSCAN*, designed for astrometry with significant uncertainties. The central methodological claim is that standard HDBSCAN* treats the input coordinates as exact, whereas the outer-Galaxy setting involves large parallax and proper-motion uncertainties, so a single deterministic clustering can be misleading (Patel et al., 13 Jul 2025).
The clustering is performed in a 5D feature space
$\cd(p)=d(p, mpts\operatorname{-NN}(p)).$2
where $\cd(p)=d(p, mpts\operatorname{-NN}(p)).$3 are Galactocentric positions and $\cd(p)=d(p, mpts\operatorname{-NN}(p)).$4 are projected velocities derived from proper motions and distance. The paper first constructs a Gaia-matched YSO catalog, removes problematic astrometry including sources with $\cd(p)=d(p, mpts\operatorname{-NN}(p)).$5, and reports a final Gaia-SFOG sample of $\cd(p)=d(p, mpts\operatorname{-NN}(p)).$6 YSOs (Patel et al., 13 Jul 2025).
The algorithm then generates $\cd(p)=d(p, mpts\operatorname{-NN}(p)).$7 mock catalogs. Distances are sampled from an Inverse Weibull distribution fit to the Gaia distance posterior and parameterized by the $\cd(p)=d(p, mpts\operatorname{-NN}(p)).$8, $\cd(p)=d(p, mpts\operatorname{-NN}(p)).$9, and 0 percentiles, while proper motions are sampled from Gaussian distributions using Gaia means and errors. Features are robustly scaled using medians and interquartile ranges before HDBSCAN* is run on each realization. If multiple clusters are returned in one mock catalog, only the largest or primary cluster is used for membership analysis, and a soft clustering threshold of 0.1 is adopted to avoid excluding borderline members too aggressively (Patel et al., 13 Jul 2025).
For each source 1, the method stores 2, the number of Monte Carlo realizations in which the source is identified as a cluster member, and defines the final score as
3
The paper is explicit that this is a normalized frequency-based membership score rather than a Bayesian posterior. Membership acceptance is controlled by a cluster-dependent Monte Carlo threshold 4, calibrated not analytically but through simulation (Patel et al., 13 Jul 2025).
That calibration is performed with GOLDFIST, "Generation Of cLuster anD FIeld STar," a forward model of cluster and field populations. GOLDFIST uses an analytic King profile for the cluster, uniform field YSO placement along the line of sight outside the cluster volume, Gaia-like uncertainty assignment procedures, and Differential Evolution optimization in two stages. The acceptance threshold is selected by matching simulated and observed Monte Carlo spectra and then choosing the threshold that retains about 95% of the true members in simulation. On three outer-Galaxy clusters, the method is reported to recover new members, confirm known members, and produce simulation-derived distances that, in two of the three examples, broadly agree with Gaia median distances and available kinematic estimates (Patel et al., 13 Jul 2025).
4. Computational-control variants related to the label
Several papers describe HDBSCAN*-derived procedures that are not named HDBSCAN-MC in the original text but are related in purpose: they preserve the HDBSCAN* clustering objective while modifying computation, selection, or data reduction.
In "Accelerated Hierarchical Density Clustering" (McInnes et al., 2017), the contribution is an accelerated implementation of HDBSCAN* rather than a new clustering family. The paper accelerates core-distance computation with space trees, including kd-trees, ball trees, and cover trees, and replaces naive MST construction with a dual-tree Borůvka algorithm adapted to mutual reachability distance. The paper states that the accelerated algorithm provides comparable performance to DBSCAN while supporting variable density clusters and eliminating the need for a difficult-to-tune 5 parameter (McInnes et al., 2017).
In "Geometric reconstructions of density based clusterings" (Garcia-Pulido et al., 2022), the proposed 6-7 is a multi-scale, cluster-boundary, geometry-driven reconstruction of standard 8. The key result is exact reconstruction rather than approximation: a graph 9 assembled from local pieces satisfies
$\mrd(p,q)=\max\{\cd(p),\cd(q),d(p,q)\}.$0
so the exact connected-component hierarchy is preserved. The method iteratively replaces the full dataset by reduced subsets $\mrd(p,q)=\max\{\cd(p),\cd(q),d(p,q)\}.$1, keeping boundary neighborhoods that matter for future cluster merges and discarding interiors that no longer affect correctness (Garcia-Pulido et al., 2022).
The paper explicitly states that it does not use the exact name “HDBSCAN-MC,” but that its $\mrd(p,q)=\max\{\cd(p),\cd(q),d(p,q)\}.$2-$\mrd(p,q)=\max\{\cd(p),\cd(q),d(p,q)\}.$3 can substantively motivate such an interpretation: it is a scalable, multi-stage version of HDBSCAN* whose purpose is computational control, not a different density notion. Its large-scale demonstration is the Microsoft Building Footprint Database of the United States, with about $\mrd(p,q)=\max\{\cd(p),\cd(q),d(p,q)\}.$4 points; the full U.S. $\mrd(p,q)=\max\{\cd(p),\cd(q),d(p,q)\}.$5-$\mrd(p,q)=\max\{\cd(p),\cd(q),d(p,q)\}.$6 run with $\mrd(p,q)=\max\{\cd(p),\cd(q),d(p,q)\}.$7 is reported to complete in about 24 hours, whereas standard HDBSCAN* does not terminate in that setting (Garcia-Pulido et al., 2022).
5. Selection-stage hybridization and micro-cluster control
A different usage adjacent to the HDBSCAN-MC label appears in "A Hybrid Approach To Hierarchical Density-based Cluster Selection" (Malzer et al., 2019). The method introduced there is HDBSCAN$\mrd(p,q)=\max\{\cd(p),\cd(q),d(p,q)\}.$8, described as a cluster selection rule on top of the existing HDBSCAN hierarchy and as a hybrid between DBSCAN* and HDBSCAN(eom). The paper’s motivation is variable-density data with a low minimum cluster size: small minPts or minimum cluster size can recover meaningful sparse clusters but may over-segment high-density regions into many micro-clusters (Malzer et al., 2019).
The method leaves hierarchy construction unchanged and instead introduces an additional threshold $\mrd(p,q)=\max\{\cd(p),\cd(q),d(p,q)\}.$9 at cluster-selection time. A cluster 0 is called epsilon stable if
1
and its epsilon stability is defined as
2
The same optimization structure used in HDBSCAN selection is then reused with 3 replaced by 4, making the procedure FOSC-compatible. Algorithmically, the tree is traversed bottom-up: leaves are initially selected, non-epsilon-stable clusters are replaced by the first epsilon-stable ancestor on the path, and descendants are deselected (Malzer et al., 2019).
The paper does not explicitly use the name “HDBSCAN-MC.” It states, however, that if the label is used elsewhere for a minimum-cutoff or micro-cluster-control selection rule, then HDBSCAN5 corresponds to that hybrid concept. In this interpretation, the method is not a rebuild of the hierarchy but a selection-stage modification that suppresses unwanted micro-clusters while preserving HDBSCAN’s variable-density behavior outside the thresholded branches (Malzer et al., 2019).
6. Conceptual distinctions, common misconceptions, and significance
A persistent misconception is that HDBSCAN-MC names a single canonical algorithm. The literature summarized here does not support that reading. In one line of work, it is a multi-mpts acceleration framework based on reduced mutual-reachability RNGs; in another, it is a Monte Carlo wrapper that propagates measurement uncertainty through repeated HDBSCAN* runs; in adjacent work, similar labels can refer to accelerated implementations, boundary-based geometric reconstructions, or thresholded selection rules (Neto et al., 2017, Patel et al., 13 Jul 2025).
A second misconception is that these variants alter the density concept underlying HDBSCAN*. Several of the cited papers state the opposite in explicit terms. The multi-hierarchy framework preserves HDBSCAN* correctness for every mpts in the range. The Monte Carlo method is described as not a new clustering density estimator, but an uncertainty propagation wrapper around HDBSCAN*. The geometric reconstruction method changes how the mutual-reachability graph is assembled, not the clustering objective. The thresholded HDBSCAN6 method modifies cluster selection without modifying the hierarchy itself (Neto et al., 2017, Patel et al., 13 Jul 2025, Garcia-Pulido et al., 2022, Malzer et al., 2019).
The significance of the HDBSCAN-MC family is therefore best understood operationally. One branch addresses parameter exploration by making large collections of HDBSCAN* hierarchies computationally feasible. Another addresses uncertainty by replacing a single deterministic clustering with a distribution over cluster memberships. Related branches address scalability on massive Euclidean datasets, practical runtime, or micro-cluster suppression. This suggests that “HDBSCAN-MC” functions less as the name of a single algorithm than as a descriptor for HDBSCAN*-compatible strategies that impose computational control, uncertainty propagation, or modified cluster extraction while retaining the underlying mutual-reachability and hierarchy-based framework.