Distance-based Separability Index (DSI)

• DSI is a family of metrics that quantifies separability in classical and quantum datasets by comparing intra-group and inter-group distances.
• It employs statistical measures like the KS test and geometric margin computations to yield a normalized, scale-invariant separation score.
• DSI informs clustering validation, data complexity assessment, anomaly detection, and quantum entanglement detection, complementing traditional indices.

The Distance-based Separability Index (DSI) is a family of metrics for quantifying the separability of classes or clusters in both classical and quantum datasets, based on pairwise distance statistics. Defined with variants tailored for clustering evaluation, data complexity analysis, and quantum state separability, all DSI formulations rely on principled, purely geometric or distributional comparisons of intra-group and inter-group distances. DSI is model-independent, scale-invariant under global rescaling, and provides an interpretable real-valued measure indicating the degree of group separation, maximal overlap/mixing, or quantum non-separability.

1. Formal Definitions and Variants

1.1. Classical Data DSI (Clustering and Supervised Settings)

For a dataset partitioned into $n$ clusters or classes $\{C_1,\ldots, C_n\}$, DSI employs the divergence between distributions of intra-cluster distances (ICD) and between-cluster distances (BCD). For each cluster $C_i$:

• $$\{d^{(i)}\} = \{ \|x_p - x_q\|_2 : x_p, x_q \in C_i, p < q \}$$ (ICD)
• $$\{d^{(i,\neg i)}\} = \{ \|x_p - x_q\|_2 : x_p \in C_i, x_q \notin C_i \}$$ (BCD)
• The per-cluster separability score: $S_i = \mathrm{KS}(\{d^{(i)}\}, \{d^{(i,\neg i)}\})$, with KS the two-sample Kolmogorov–Smirnov statistic, normalized to $[0,1]$.

The aggregate DSI is: $$\mathrm{DSI}(D, \{C_i\}) = \frac{1}{n} \sum_{i=1}^{n} S_i$$ Higher values indicate greater separability; $\mathrm{DSI} = 0$ indicates maximal mixing (no separability), and $\mathrm{DSI} = 1$ corresponds to perfect separation (Guan et al., 2020, Guan et al., 2021, Guan et al., 2021, Guan et al., 2020).

1.2. Absolute DSI for Cluster Margins

A distinct construction (Bagirov et al., 15 Oct 2025) defines DSI via explicit margin computations over neighboring cluster pairs using cluster centers $\{C_1,\ldots, C_n\}$0:

• For clusters $\{C_1,\ldots, C_n\}$1, $\{C_1,\ldots, C_n\}$2, let $\{C_1,\ldots, C_n\}$3.
• Adjacent sets: $\{C_1,\ldots, C_n\}$4
• Margins: $\{C_1,\ldots, C_n\}$5, then $\{C_1,\ldots, C_n\}$6.
• DSI is aggregated over neighbor-pairs: $\{C_1,\ldots, C_n\}$7

Positive margins signal separability; negative margins indicate overlap. The separability ratio $\{C_1,\ldots, C_n\}$8 gives the fraction of non-overlapping neighbor pairs.

1.3. Quantum Separability DSI

In quantum information, DSI formalizes the distance from a density matrix $\{C_1,\ldots, C_n\}$9 to the maximally mixed state, normalized by the maximal guaranteed-separable radius: $C_i$0 Here, the numerator is the Hilbert–Schmidt norm and $C_i$1 is a closed-form function of subsystem dimensions. $C_i$2 guarantees full separability; $C_i$3 indicates possible entanglement (Patel et al., 2016).

2. Theoretical Properties

• Range: All DSI definitions are normalized, with classical DSI in $C_i$4. A higher value corresponds to greater distributional separation or geometric gap.
• Scale-Invariance: DSI values remain invariant under global scaling of Euclidean distance due to KS normalization and relative geometric relationships.
• Sensitivity to Distributional Overlap: In the classical setting, DSI approaches 0 if intra- and inter-class distance distributions coincide, which occurs if and only if classes are identically distributed in the infinite-sample limit (Guan et al., 2021, Guan et al., 2020).
• Cluster Shape Robustness: DSI directly compares distance distributions and does not rely on convexity or centroid definitions, thus accommodating non-convex and heterogeneous cluster shapes (Guan et al., 2021, Bagirov et al., 15 Oct 2025).
• Quantum DSI invariance: The quantum DSI is invariant under global unitary transformations and is analytically computable for arbitrary $C_i$5-qudit states (Patel et al., 2016).

3. Algorithms and Computational Complexity

For classical DSI:

• Computation: For each cluster, compute all intra-cluster pairwise distances ($C_i$6 for $C_i$7 points), all between-cluster distances ($C_i$8), and perform the KS test.
• Aggregate Complexity: For $C_i$9 clusters, total $$\{d^{(i)}\} = \{ \|x_p - x_q\|_2 : x_p, x_q \in C_i, p < q \}$$0 distance computations and $$\{d^{(i)}\} = \{ \|x_p - x_q\|_2 : x_p, x_q \in C_i, p < q \}$$1 for sorting in KS; often reduced in practice by uniform subsampling without significant loss of accuracy (Guan et al., 2021, Guan et al., 2020).
• Absolute DSI: Requires iterating over all neighbor cluster pairs and per-point distance computations to adjacent centers; complexity is $$\{d^{(i)}\} = \{ \|x_p - x_q\|_2 : x_p, x_q \in C_i, p < q \}$$2 for $$\{d^{(i)}\} = \{ \|x_p - x_q\|_2 : x_p, x_q \in C_i, p < q \}$$3 clusters and $$\{d^{(i)}\} = \{ \|x_p - x_q\|_2 : x_p, x_q \in C_i, p < q \}$$4 data points (Bagirov et al., 15 Oct 2025).
• Quantum DSI: Involves a single matrix subtraction and Frobenius norm computation, thus $$\{d^{(i)}\} = \{ \|x_p - x_q\|_2 : x_p, x_q \in C_i, p < q \}$$5 for $$\{d^{(i)}\} = \{ \|x_p - x_q\|_2 : x_p, x_q \in C_i, p < q \}$$6.

In practical settings, subsampling and GPU-acceleration allow scalable computation even on datasets with tens of thousands of points (Guan et al., 2021, Guan et al., 2020).

4. Empirical Behavior and Validation

Comprehensive experiments have established DSI’s effectiveness and robustness:

• Clustering evaluation: On 97 synthetic and 12 real datasets, DSI achieves competitive or superior performance among established internal indices such as Dunn, Davies–Bouldin, Silhouette, WB, CVNN, and CVDD indexes. DSI ranks among the top indices by both hit-the-best (agreement with ARI-selected clusterings) and the rank-difference metric (total deviation in ranking from ARI) (Guan et al., 2020, Guan et al., 2021).
• Data complexity measurement: On synthetic benchmarks (Random, Spiral, XOR, Moons, Circles, Blobs), DSI’s 1-DSI value strictly follows the intuitive ordering of classification complexity, outperforming classical feature-based, neighborhood, linearity, and density measures in both sensitivity and consistency—e.g., N2, N4, T1 (Guan et al., 2021, Guan et al., 2020).
• Feature and model selection: DSI tracks generalization difficulty across datasets of varying separability, correlating with classifier training dynamics (e.g., accuracy drop with increasing class overlap).
• Quantum separability: Correctly identifies the separability threshold for multipartite Werner states, providing a necessary certificate for bipartite separability and supporting entanglement detection (Patel et al., 2016).

5. Comparative Analysis with Existing Indices

The main conceptual distinction between DSI and classical indices is its distributional, as opposed to extremal (min/max distance) or centroid-based, construction. DSI captures full empirical divergence between finite-sample distance distributions, not just isolated statistics, making it highly sensitive to subtle mixing.

Index	Basis	Handles Non-convex Shape	Model Free	Distributional
DSI	Distance distributions	Yes	Yes	Yes
Dunn	Extreme distances	No	Yes	No
Silhouette	Centroid/proximity	No	Yes	No
CVDD	Density separability	Partial	Yes	No
WB, CH	Variance/centroid	No	Yes	No

No single index dominates across all classes of datasets; DSI’s distinct failure modes and strengths make it a valuable addition to the methodological toolkit, especially as a complement to center-based or density-based indices (Guan et al., 2020, Guan et al., 2021, Bagirov et al., 15 Oct 2025).

6. Extensions and Applications

• GAN and sample distribution evaluation: DSI quantifies the closeness of generative samples to real data distributions, similar in spirit to metrics such as Fréchet Inception Distance, but model free and nonparametric (Guan et al., 2021).
• Outlier and anomaly detection: Local increases in DSI highlight the disruption of mixing patterns by anomalies (Guan et al., 2021).
• Feature subset evaluation and classifier selection: DSI supports ranking of dataset features and algorithm suitability by predictive difficulty prior to model training (Guan et al., 2020).
• Quantum entanglement detection: The global and partitioned DSI provides necessary separability tests and coarse-grained entanglement profiles for general quantum many-body states (Patel et al., 2016).
• Cluster-count determination: Both classical and absolute DSI variants can infer the “true” number of clusters, outperforming several traditional clustering validity indices on synthetic and real data (Bagirov et al., 15 Oct 2025).

7. Limitations and Interpretation Caveats

• Curse of dimensionality: As the ambient dimension increases, distances concentrate, potentially reducing the informativeness of empirical distance distributions. Pre-whitening or dimensional normalization may be necessary (Guan et al., 2021, Guan et al., 2020).
• Sampling biases: For imbalanced classes or rare clusters, uniform subsampling may yield high-variance estimates for DSI (Guan et al., 2021).
• Boundary complexity: DSI measures overlap/mixing, not the minimal classification boundary complexity; some nonlinearly separable, but non-overlapping, classes may yield moderate DSI (Guan et al., 2020).
• Absolute scale: DSI scores are dataset-dependent; their absolute values are not universally comparable across unrelated tasks (Guan et al., 2020).

A plausible implication is that DSI should be used as part of a broader panel of assessment tools, especially when dataset or feature space geometry is atypical.

---

References:

• "An Internal Cluster Validity Index Using a Distance-based Separability Measure" (Guan et al., 2020)
• "A Distance-based Separability Measure for Internal Cluster Validation" (Guan et al., 2021)
• "Absolute indices for determining compactness, separability and number of clusters" (Bagirov et al., 15 Oct 2025)
• "A Novel Intrinsic Measure of Data Separability" (Guan et al., 2021)
• "Data Separability for Neural Network Classifiers and the Development of a Separability Index" (Guan et al., 2020)
• "Geometric criterion for separability based on local measurement" (Patel et al., 2016)