Class Difficulty Separability Coefficient (CDSC)

• CDSC is a normalized measure of Jensen–Shannon divergence that evaluates the separability of class-conditioned difficulty score distributions.
• It employs kernel density estimation on training-dynamics scores, ensuring fair comparison across classes irrespective of sample frequency.
• High CDSC values signal the need for class-aware coreset selection to prevent biased pruning that may overlook rare but informative classes.

Class Difficulty Separability Coefficient (CDSC) is a quantitative measure of class-difficulty separability: the regime in which training-dynamics difficulty scores cluster by class rather than being mixed across classes. In its explicit formulation, CDSC is a normalized Jensen–Shannon divergence over class-conditional difficulty-score distributions estimated by kernel density estimation (KDE), with equal weighting across classes. Low CDSC indicates that classes have similar difficulty distributions and that a global, class-agnostic ranking is usually adequate; high CDSC indicates stronger class-wise clustering of difficulty and a greater risk that class-agnostic pruning will overrepresent easy majority classes while neglecting rare but informative ones (Tsai et al., 15 Jul 2025).

1. Problem setting and conceptual role

CDSC was introduced in the context of one-shot coreset selection, where a dataset is pruned while attempting to maintain or improve downstream model performance. The motivating observation is that many coreset-selection methods implicitly assume that difficulty is distributed similarly across classes. The paper argues that this assumption can be reasonable on balanced benchmarks such as CIFAR-10/100, but often breaks down in real-world domains such as network intrusion detection and medical imaging, where some classes are consistently easy and others consistently hard (Tsai et al., 15 Jul 2025).

The structural property being measured is named class-difficulty separability. Under this view, a scalar difficulty score $s(x_i)$ derived from training dynamics is not merely an example-level attribute; its class-conditional distribution may itself be informative. If classes occupy distinct regions in difficulty-score space, then a single global ranking can become systematically biased. The paper’s operational interpretation is that higher CDSC indicates stronger class-wise clustering of difficulty, and therefore a greater need for class-aware or class-proportional coreset selection (Tsai et al., 15 Jul 2025).

A common misconception is to treat CDSC as a new difficulty score. It is not introduced for that purpose. The paper states that CDSC is generic with respect to the training-dynamics score: it is computed from a scalar difficulty score already available from methods such as AUM, Forgetting, or EL2N, and is used as a diagnostic of the score distributions across classes rather than as a replacement for those scores (Tsai et al., 15 Jul 2025).

2. Formal definition

Let the dataset be

$\mathcal{X}=\{x_1,\ldots,x_N\},$

with class labels $y_i \in \{1,\ldots,C\}$, and let $s(x_i)\in \mathbb{R}$ be a scalar difficulty score computed from training dynamics. For each class $c$, define

$\mathcal{X}_c=\{x_i\in \mathcal{X}\mid y_i=c\},$

and estimate the class-conditional score distribution $P_c$ over $\{s(x_i)\mid x_i\in \mathcal{X}_c\}$ using KDE (Tsai et al., 15 Jul 2025).

The equal-weight mixture distribution is

$M=\frac{1}{C}\sum_{c=1}^C P_c.$

CDSC is then defined as

$$\delta_{\mathrm{CDSC}}= \frac{1}{\log_2 C} \left( H(M)-\frac{1}{C}\sum_{c=1}^C H(P_c) \right),$$

where

$\mathcal{X}=\{x_1,\ldots,x_N\},$0

is Shannon entropy. The paper explicitly identifies the unnormalized term as the Jensen–Shannon divergence

$\mathcal{X}=\{x_1,\ldots,x_N\},$1

so that CDSC is the JSD normalized by $\mathcal{X}=\{x_1,\ldots,x_N\},$2 (Tsai et al., 15 Jul 2025).

The computation pipeline is stated as follows: train a model and record a training-dynamics difficulty score for each example; group the scores by class; estimate $\mathcal{X}=\{x_1,\ldots,x_N\},$3 for each class using KDE; compute the class-mixture density $\mathcal{X}=\{x_1,\ldots,x_N\},$4; compute $\mathcal{X}=\{x_1,\ldots,x_N\},$5 and $\mathcal{X}=\{x_1,\ldots,x_N\},$6; then plug these quantities into the normalized JSD formula. In the paper’s experiments, CDSC is computed primarily over class-specific AUM score distributions, with Forgetting and EL2N examined in ablations (Tsai et al., 15 Jul 2025).

3. Interpretation, normalization, and edge cases

The central intuition is distributional. If classes are intermixed in difficulty space, then $\mathcal{X}=\{x_1,\ldots,x_N\},$7 for all $\mathcal{X}=\{x_1,\ldots,x_N\},$8, the Jensen–Shannon divergence is small, and CDSC is low. If each class has its own distinct difficulty profile, then the $\mathcal{X}=\{x_1,\ldots,x_N\},$9 diverge, the mixture becomes more informative than any single class distribution, and CDSC increases (Tsai et al., 15 Jul 2025).

The paper states that CDSC lies in $y_i \in \{1,\ldots,C\}$0 because of the normalization by $y_i \in \{1,\ldots,C\}$1. It further states that $y_i \in \{1,\ldots,C\}$2 when all class-conditional difficulty distributions are identical, and that $y_i \in \{1,\ldots,C\}$3 as the supports of the class distributions become mutually disjoint. Because the mixture

$y_i \in \{1,\ldots,C\}$4

uses equal class weighting regardless of dataset size, CDSC measures class-wise separability, not sample-frequency-weighted separability (Tsai et al., 15 Jul 2025).

This equal weighting is consequential. A tiny rare class influences CDSC equally with a large majority class. The paper frames this as necessary because the objective is to detect when global ranking is unsafe for rare but informative classes. A plausible implication is that CDSC is intentionally sensitive to minority-class structure even when those classes contribute little to the empirical sample mass.

The paper also indicates several implementation-level assumptions. The score distributions are treated as continuous densities through KDE, and the metric is meaningful only relative to the chosen difficulty score. In that sense, CDSC is not a purely geometric measure of raw data; it is a measure of how training-dynamics difficulty is organized across classes (Tsai et al., 15 Jul 2025).

4. Algorithmic consequences for coreset selection

CDSC is introduced as a diagnostic metric, but its main algorithmic consequence is to motivate class-proportional sampling. The paper argues that class-agnostic methods rank all points together using one global difficulty distribution. When difficulty varies substantially by class, this can over-select easy majority classes, discard rare minority classes, or even remove entire classes under aggressive pruning. An explicit example is given from CICIDS2017, where a 1% cutoff can entirely remove the Web Attack – XSS class (Tsai et al., 15 Jul 2025).

The proposed remedy is to split the data by class, allocate a per-class budget proportional to class frequency, and run the chosen difficulty-based sampling method within each class independently. The paper presents the following class-proportional sampling procedure. With total sampling rate $y_i \in \{1,\ldots,C\}$5, minimum examples per class $y_i \in \{1,\ldots,C\}$6, dataset size $y_i \in \{1,\ldots,C\}$7, class count $y_i \in \{1,\ldots,C\}$8, and sampling function $y_i \in \{1,\ldots,C\}$9,

$s(x_i)\in \mathbb{R}$0

is the total budget, and the raw per-class budget is

$s(x_i)\in \mathbb{R}$1

If $s(x_i)\in \mathbb{R}$2, budgets are reduced from larger classes until the total fits the budget while respecting the minimum $s(x_i)\in \mathbb{R}$3. For each class $s(x_i)\in \mathbb{R}$4, the class-specific subset $s(x_i)\in \mathbb{R}$5 is sampled as

$s(x_i)\in \mathbb{R}$6

and the union of the $s(x_i)\in \mathbb{R}$7 forms the final coreset (Tsai et al., 15 Jul 2025).

This framework is used to construct Hardest-CP, SW-CP, and CCS-CP. The paper emphasizes that the method is not to invent an entirely new difficulty score, but to wrap an existing score-based selector in a class-aware budget allocation. In practical terms: compute a training-dynamics score such as AUM, measure CDSC to diagnose separability, and, if separability is high, switch to class-proportional selection (Tsai et al., 15 Jul 2025).

5. Empirical behavior and reported values

The reported CDSC values span both security and medical datasets, with CIFAR-10 included as a low-separability baseline.

Dataset	CDSC
UNSW-NB15	0.067
CICIDS2017	0.114
CTU-13	0.489
DermaMNIST	0.082
BloodMNIST	0.280
CIFAR-10	0.043

The paper interprets CTU-13 (0.489) as the most difficulty-separable among the listed datasets, with BloodMNIST (0.280) also showing notable separability. UNSW-NB15, CICIDS2017, and DermaMNIST are lower, though CICIDS2017 still exhibits enough separability to induce class-agnostic failure modes under sufficiently aggressive pruning (Tsai et al., 15 Jul 2025).

The main empirical claim is that higher CDSC corresponds to larger gains from switching from a class-agnostic method to a class-proportional one, especially for accuracy, precision, and recall. The clearest case is CTU-13 at an extreme 99% pruning rate: CCS-CP shows an accuracy drop of 2.58%, precision drop of 0.49%, and recall drop of 0.19%, whereas the class-agnostic CCS baseline suffers drops of 7.59%, 4.57%, and 4.11%, respectively. For CICIDS2017 at 99.9% pruning, CCS-CP achieves a 14.32% recall gain over standard CCS, which the paper interprets as better preservation of rare attack classes (Tsai et al., 15 Jul 2025).

The paper also reports that class-proportional variants produce flatter performance curves under pruning, meaning that performance degrades more slowly as pruning becomes aggressive. This suggests that CDSC is not only descriptive but operationally predictive: low CDSC implies that class-agnostic selection is usually fine, whereas high CDSC implies brittleness under global ranking (Tsai et al., 15 Jul 2025).

6. Relation to adjacent separability metrics

CDSC belongs to a broader family of separability-oriented measures, but it is not interchangeable with all of them. In “How deep is deep enough? -- Quantifying class separability in the hidden layers of deep neural networks”, the Generalized Discrimination Value (GDV) measures class separability in a representation space without training an extra classifier and without projecting the data. GDV is computed from z-scored activation vectors using the balance between mean intra-class distance and mean inter-class distance, with normalization by $s(x_i)\in \mathbb{R}$8 to achieve dimensionality invariance. Its layerwise form $s(x_i)\in \mathbb{R}$9 tracks how class clusters evolve through network depth; more negative GDV means better separability, GDV near 0 indicates poor separability or randomized labels, and in the idealized best case GDV approaches $c$0 (Schilling et al., 2018).

The distinction is structural. GDV measures separability of representations in hidden-layer activation space, whereas CDSC measures separability of difficulty-score distributions across classes. Both are scalar indices of class separability, but they operate on different objects: activation geometry in one case, class-conditional training-dynamics statistics in the other. This suggests that CDSC should be understood as a distributional diagnostic over difficulty space rather than as a direct hidden-representation metric.

A second related measure is the cumulative spectral gradient (CSG), introduced as a dataset complexity measure derived from pairwise class overlap in an embedding space and a spectral clustering framework. CSG is a global dataset-level measure: higher CSG indicates lower separability and higher class difficulty, while the intermediate similarity matrix $c$1 exposes which classes are close or entangled. The paper reports strong correlation between CSG and CNN test error, including 0.968 Pearson correlation with AlexNet error rate on six 10-class datasets for CSG with CNN-AE + t-SNE embedding (Branchaud-Charron et al., 2019).

A further neighboring line of work does not define CDSC, but explicitly optimizes the same qualitative objective of increasing inter-class separability and decreasing intra-class spread. “Triplet Loss Based Quantum Encoding for Class Separability” trains a quantum encoding circuit with a triplet-loss objective and measures separability through average trace distances between encoded density matrices. The paper states that it does not provide a normalized coefficient framed as class difficulty, but it is clearly related to the same conceptual space (Mordacci et al., 19 Sep 2025).

Taken together, these measures delineate several separability regimes: layerwise representation separability in GDV, dataset-level overlap complexity in CSG, Hilbert-space trace-distance separation in the quantum setting, and class-conditional difficulty-distribution separability in CDSC. CDSC is specific in both its input object and its intended use: it quantifies whether training-dynamics difficulty clusters by class, and thereby diagnoses when class-agnostic coreset selection is likely to be unsafe.