Support Points in Imbalanced Classification

• Support points are a representative subset of majority class samples that minimizes energy distance to preserve the original distribution.
• The method uses gradient descent to optimize support point locations, balancing attraction to data with dispersion among points.
• This approach contrasts with mutual information-based stratified sampling, offering global distribution preservation for improved classifier performance.

Class imbalance and distributional differences in large datasets can degrade classification performance by biasing models toward the majority class and weakening performance on minority classes. “Statistical Undersampling with Mutual Information and Support Points” studies this problem through two undersampling strategies: mutual information-based stratified simple random sampling and support-points-based optimization. In both cases, the central aim is not merely to reduce majority-class cardinality, but to retain a subset that remains statistically representative of the original majority distribution, thereby minimizing information loss and improving balanced classification accuracy in imbalanced learning settings (Mak et al., 2024).

1. Problem setting and proposed undersampling framework

The paper addresses imbalanced classification under the premise that naive reduction of the majority class can distort the underlying data distribution. Its two proposed methods are deliberately statistical in orientation. One uses mutual information to form strata and then performs stratified simple random sampling. The other uses support points, computed by minimizing a distributional discrepancy, to obtain a compressed majority-class representation (Mak et al., 2024).

The support-points component is specialized to the majority class of an imbalanced dataset. Let $X=\{x_1,\dots,x_N\}$ denote the majority-class data, and let $Z=\{z_1,\dots,z_m\}$ with $m\ll N$ denote a much smaller representative set. The paper defines support points through minimization of the empirical energy distance between the empirical distribution of $X$ and the empirical distribution supported on $Z$. This makes the undersampling objective explicitly distributional rather than purely geometric or cluster-based (Mak et al., 2024).

The mutual-information component is structurally different. It computes pairwise mutual information values across dimensions, converts them to a dissimilarity matrix, uses $k$-means to define strata, and then applies stratified simple random sampling with Neyman or “optimal” allocation. The paper treats the two methods as alternative undersampling strategies based on different principles: information-theoretic grouping versus distributional matching (Mak et al., 2024).

2. Support points as a distribution-preserving majority-class summary

In this work, support points are a set of representative samples chosen from the majority class by solving an optimization problem that minimizes the energy distance between the empirical distribution of the original majority data and the empirical distribution of a much smaller set of selected points. The paper explicitly relates this construction to the original support-points framework of Mak and Joseph, where support points serve as compact representative sets for continuous distributions (Mak et al., 2016).

The energy distance between distributions $F$ and $G$ on $\mathbb{R}^d$ is

$$E(F,G)=2\mathbb{E}\|X-Y\|-\mathbb{E}\|X-X'\|-\mathbb{E}\|Y-Y'\|,$$

where $Z=\{z_1,\dots,z_m\}$0 and $Z=\{z_1,\dots,z_m\}$1. The paper notes that this distance is always nonnegative and is equal to zero if and only if $Z=\{z_1,\dots,z_m\}$2, so minimizing it is a direct way to match distributions (Mak et al., 2024).

For a majority-class dataset $Z=\{z_1,\dots,z_m\}$3 and candidate support points $Z=\{z_1,\dots,z_m\}$4, the empirical energy distance is

$Z=\{z_1,\dots,z_m\}$5

Since the term involving only the $Z=\{z_1,\dots,z_m\}$6's is constant with respect to $Z=\{z_1,\dots,z_m\}$7, the optimization problem is

$Z=\{z_1,\dots,z_m\}$8

The paper emphasizes the attraction-repulsion structure of this objective. The term

$Z=\{z_1,\dots,z_m\}$9

pulls the support points toward the majority-class cloud, while

$m\ll N$0

encourages dispersion among support points. This yields a small subset that spreads over the support of the majority distribution rather than collapsing into a few dense regions (Mak et al., 2024).

This distinguishes support points from several standard undersampling families. Random undersampling is computationally cheap but may remove rare yet important majority patterns. Cluster-based undersampling typically minimizes within-cluster variance, not a global discrepancy between the original and retained distributions. NearMiss, Tomek links, and related methods focus on local geometry near class boundaries or on cleaning overlaps. Support points instead preserve the global structure of the majority class without directly using minority points or decision-boundary information (Mak et al., 2024).

3. Optimization procedure and computational specialization

The paper uses gradient-based optimization over the support-point locations $m\ll N$1. Starting from an initial subset of majority-class points, it iteratively updates each $m\ll N$2 using the gradient of $m\ll N$3. The gradient reported is

$m\ll N$4

and the update is

$m\ll N$5

where $m\ll N$6 is the learning rate (Mak et al., 2024).

The algorithmic sketch is explicit. First, initialize $m\ll N$7 by randomly sampling $m\ll N$8 points from the majority-class dataset $m\ll N$9. Second, repeat gradient computation and gradient-descent updates for a fixed number of iterations or until convergence. Third, after convergence, map each continuous $X$0 to its nearest neighbor in the original majority set: $X$1 These nearest-neighbor points form the actual undersampled majority instances used for training (Mak et al., 2024).

The credit-card fraud experiment motivates a two-stage implementation. The dataset contains 284,807 samples with 30 features, with the majority class at about 99.82% of the data and the minority class at about 0.18%. Direct computation on the full majority class would require more than 600 GB of memory, so the paper first clusters the majority class using MiniBatchKMeans into 50 clusters, with $X$2 chosen by the elbow method, and then extracts a subset of 5,000 majority samples as a representative pre-compressed set $X$3. Support points are computed on this subset for 2,000 iterations, reaching an energy distance around 0.0102 before nearest-neighbor mapping (Mak et al., 2024).

For this reduced-stage implementation, if $X$4 is the size of the majority subset, $X$5 the number of support points, $X$6 the dimension, and $X$7 the number of iterations, the total complexity is

$X$8

This computational profile explains the need for pre-reduction before optimizing support points on large majority classes (Mak et al., 2024).

4. Mutual-information-based stratified simple random sampling

The second undersampling strategy is mutual-information-based stratified simple random sampling. The paper uses the standard mutual information definition

$X$9

with $Z$0 the joint distribution and $Z$1 the marginals (Mak et al., 2024).

Operationally, the procedure has four steps. It computes pairwise mutual information values across data dimensions, constructs a mutual-information matrix, converts that matrix to a dissimilarity matrix, runs $Z$2-means clustering on that dissimilarity to form strata, and finally performs stratified simple random sampling within those strata using Neyman or “optimal” allocation. The paper uses this method on a small breast-cancer dataset with 272 samples (Mak et al., 2024).

The contrast with support points is explicit. Mutual-information-based stratified SRS groups similar data points and then samples randomly within the resulting strata. It does not optimize a global distributional discrepancy. Support points, by contrast, are obtained by direct minimization of empirical energy distance. The paper also notes a computational asymmetry: the MI-based method, as implemented, is $Z$3 because of pairwise mutual-information computation, which makes it infeasible for large datasets; this is why it is not used on the credit-card fraud data (Mak et al., 2024).

A plausible implication is that the two methods target different forms of representativeness. MI-based sampling promotes diversity through stratification, whereas support points impose a global distribution-matching criterion over the retained majority subset. The paper treats them separately and does not explore a hybrid procedure (Mak et al., 2024).

5. Experimental evidence and representativeness diagnostics

The support-points evaluation on the credit-card fraud dataset compares the original majority class, a majority subset chosen by naive random undersampling, and a majority subset chosen by support points after nearest-neighbor mapping. The paper examines feature-wise means and standard deviations as well as Kolmogorov–Smirnov tests on marginal distributions (Mak et al., 2024).

For the three example features reported, the original, support-points, and naive-random statistics are as follows. For Feature 5, the means are $Z$4, $Z$5, and $Z$6, while the standard deviations are $Z$7, $Z$8, and $Z$9. For Feature 12, the means are $k$0, $k$1, and $k$2, and the standard deviations are $k$3, $k$4, and $k$5. For Feature 21, the means are $k$6, $k$7, and $k$8, and the standard deviations are $k$9, $F$0, and $F$1. The paper’s conclusion is that both methods approximate the original distribution fairly well; support points sometimes deviate more in variance for certain features but overall maintain good fidelity (Mak et al., 2024).

The KS tests provide stronger distributional evidence. Example results reported are: for Feature 8, KS(Support Points) $F$2 with $F$3, versus KS(Naive) $F$4 with $F$5; for Feature 14, KS(Support Points) $F$6 with $F$7, versus KS(Naive) $F$8 with $F$9; for Feature 24, KS(Support Points) $G$0 with $G$1, versus KS(Naive) $G$2 with $G$3. High $G$4-values indicate that the subset’s marginal distribution is not significantly different from the original majority distribution, and the paper notes that support points often yield slightly higher $G$5-values than naive undersampling on some features (Mak et al., 2024).

The classification results are reported using balanced accuracy, although the table labels them as “accuracy” in the paper’s discussion. Support points always match or slightly improve over random undersampling on the fraud dataset (Mak et al., 2024).

Classifier	Random undersampling	Support points
Logistic Regression	0.9261	0.9401
Random Forest	0.9295	0.9330
XGBoost	0.9225	0.9225
SVC	0.9260	0.9331

The largest gain is for Logistic Regression, at about 1.4 percentage points. SVC also improves, Random Forest improves slightly, and XGBoost is unchanged. This suggests that preserving the global majority distribution can be particularly useful for classifiers whose performance is more sensitive to the retained geometry of the training set (Mak et al., 2024).

6. Limitations, interpretation, and broader context

The paper does not derive new theorems for support points in imbalanced classification, but it relies on the fact that energy distance is a metric on distributions and on the attraction-repulsion structure of the empirical objective. This suggests that support-point undersampling is appropriate when the goal is to compress the majority class while preserving its distributional structure rather than merely its local boundary geometry (Mak et al., 2024).

Its limitations are explicit. Direct support-point optimization on very large majority classes can be infeasible, as illustrated by the more-than-600-GB memory requirement for the full fraud dataset. Gradient descent may converge to local minima, so initialization and learning rate matter. The method emphasizes global distribution matching, which is beneficial for representativeness but may miss fine local features near decision boundaries; the paper therefore notes that if local boundary behavior is crucial, boundary-focused methods may still be relevant (Mak et al., 2024).

The broader context presented in the paper connects support points to uncertainty quantification, Bayesian computation, distribution compression, kernel herding, coresets, and optimal experimental design. In the original support-points formulation, they are optimal discrete approximations that minimize energy distance to a target distribution (Mak et al., 2016). Later work extends this idea to weighted and random support points for generative modeling, again treating support points as compact representative sets derived from a distributional objective rather than a neural architecture (Zhao et al., 28 Aug 2025). Within that larger lineage, the present paper uses support points not for integration or generation, but as a principled undersampling mechanism for majority classes in imbalanced classification (Mak et al., 2024).

A plausible implication is that the paper’s contribution is methodological rather than architecture-specific. It frames undersampling as a problem of empirical distribution approximation, and then shows that this framing yields a practical alternative to random undersampling and to information-theoretic stratification when the majority class is large, heterogeneous, and expensive to process in full (Mak et al., 2024).