---
title: Support Points in Imbalanced Classification
url: https://www.emergentmind.com/topics/support-points
type: topic
---

# Support Points in Imbalanced Classification

Class imbalance and distributional differences in large datasets can degrade classification performance by biasing models toward the majority class and weakening performance on minority classes. “Statistical Undersampling with Mutual Information and Support Points” studies this problem through two undersampling strategies: mutual information-based stratified simple random sampling and support-points-based optimization. In both cases, the central aim is not merely to reduce majority-class cardinality, but to retain a subset that remains statistically representative of the original majority distribution, thereby minimizing information loss and improving balanced classification accuracy in imbalanced learning settings [2412.14527].

## 1. Problem setting and proposed undersampling framework

The paper addresses imbalanced classification under the premise that naive reduction of the majority class can distort the underlying data distribution. Its two proposed methods are deliberately statistical in orientation. One uses mutual information to form strata and then performs stratified simple random sampling. The other uses support points, computed by minimizing a distributional discrepancy, to obtain a compressed majority-class representation [2412.14527].

The support-points component is specialized to the majority class of an imbalanced dataset. Let \(X=\{x_1,\dots,x_N\}\) denote the majority-class data, and let \(Z=\{z_1,\dots,z_m\}\) with \(m\ll N\) denote a much smaller representative set. The paper defines support points through minimization of the empirical energy distance between the empirical distribution of \(X\) and the empirical distribution supported on \(Z\). This makes the undersampling objective explicitly distributional rather than purely geometric or cluster-based [2412.14527].

The mutual-information component is structurally different. It computes pairwise mutual information values across dimensions, converts them to a dissimilarity matrix, uses \(k\)-means to define strata, and then applies stratified simple random sampling with Neyman or “optimal” allocation. The paper treats the two methods as alternative undersampling strategies based on different principles: information-theoretic grouping versus distributional matching [2412.14527].

## 2. Support points as a distribution-preserving majority-class summary

In this work, support points are a set of representative samples chosen from the majority class by solving an optimization problem that minimizes the energy distance between the empirical distribution of the original majority data and the empirical distribution of a much smaller set of selected points. The paper explicitly relates this construction to the original support-points framework of Mak and Joseph, where support points serve as compact representative sets for continuous distributions [1609.01811].

The energy distance between distributions \(F\) and \(G\) on \(\mathbb{R}^d\) is
\[
E(F,G)=2\mathbb{E}\|X-Y\|-\mathbb{E}\|X-X'\|-\mathbb{E}\|Y-Y'\|,
\]
where \(X,X' \overset{\text{i.i.d.}}{\sim} F\) and \(Y,Y' \overset{\text{i.i.d.}}{\sim} G\). The paper notes that this distance is always nonnegative and is equal to zero if and only if \(F=G\), so minimizing it is a direct way to match distributions [2412.14527].

For a majority-class dataset \(X=\{x_1,\dots,x_N\}\) and candidate support points \(Z=\{z_1,\dots,z_m\}\), the empirical energy distance is
\[
\hat{E}(X,Z)=\frac{2}{Nm}\sum_{i=1}^N\sum_{j=1}^m \|x_i-z_j\|
-\frac{1}{N^2}\sum_{i,i'=1}^N \|x_i-x_{i'}\|
-\frac{1}{m^2}\sum_{j,j'=1}^m \|z_j-z_{j'}\|.
\]
Since the term involving only the \(x_i\)'s is constant with respect to \(Z\), the optimization problem is
\[
\min_Z \hat{E}(X,Z).
\]

The paper emphasizes the attraction-repulsion structure of this objective. The term
\[
\frac{2}{Nm}\sum_{i=1}^N\sum_{j=1}^m \|x_i-z_j\|
\]
pulls the support points toward the majority-class cloud, while
\[
-\frac{1}{m^2}\sum_{j,j'=1}^m \|z_j-z_{j'}\|
\]
encourages dispersion among support points. This yields a small subset that spreads over the support of the majority distribution rather than collapsing into a few dense regions [2412.14527].

This distinguishes support points from several standard undersampling families. Random undersampling is computationally cheap but may remove rare yet important majority patterns. Cluster-based undersampling typically minimizes within-cluster variance, not a global discrepancy between the original and retained distributions. NearMiss, Tomek links, and related methods focus on local geometry near class boundaries or on cleaning overlaps. Support points instead preserve the global structure of the majority class without directly using minority points or decision-boundary information [2412.14527].

## 3. Optimization procedure and computational specialization

The paper uses gradient-based optimization over the support-point locations \(z_j\). Starting from an initial subset of majority-class points, it iteratively updates each \(z_j\) using the gradient of \(\hat{E}(X,Z)\). The gradient reported is
\[
\frac{\partial \hat{E}}{\partial z_j}
=
\frac{2}{Nm}\sum_{i=1}^N \frac{z_j-x_i}{\|z_j-x_i\|}
-
\frac{2}{m^2}\sum_{j'\neq j}\frac{z_j-z_{j'}}{\|z_j-z_{j'}\|},
\]
and the update is
\[
z_j \leftarrow z_j - \eta \frac{\partial \hat{E}}{\partial z_j},
\]
where \(\eta\) is the learning rate [2412.14527].

The algorithmic sketch is explicit. First, initialize \(Z\) by randomly sampling \(m\) points from the majority-class dataset \(X\). Second, repeat gradient computation and gradient-descent updates for a fixed number of iterations or until convergence. Third, after convergence, map each continuous \(z_j\) to its nearest neighbor in the original majority set:
\[
Z_{\text{nearest}}
=
\left\{
\arg\min_{x\in X}\|x-z_j\| : j=1,\dots,m
\right\}.
\]
These nearest-neighbor points form the actual undersampled majority instances used for training [2412.14527].

The credit-card fraud experiment motivates a two-stage implementation. The dataset contains 284,807 samples with 30 features, with the majority class at about 99.82% of the data and the minority class at about 0.18%. Direct computation on the full majority class would require more than 600 GB of memory, so the paper first clusters the majority class using MiniBatchKMeans into 50 clusters, with \(k\) chosen by the elbow method, and then extracts a subset of 5,000 majority samples as a representative pre-compressed set \(X_{\text{subset}}\). Support points are computed on this subset for 2,000 iterations, reaching an energy distance around 0.0102 before nearest-neighbor mapping [2412.14527].

For this reduced-stage implementation, if \(N'\) is the size of the majority subset, \(m\) the number of support points, \(d\) the dimension, and \(T\) the number of iterations, the total complexity is
\[
O\big(T(N'md+m^2d)\big).
\]
This computational profile explains the need for pre-reduction before optimizing support points on large majority classes [2412.14527].

## 4. Mutual-information-based stratified simple random sampling

The second undersampling strategy is mutual-information-based stratified simple random sampling. The paper uses the standard mutual information definition
\[
I(X,Y)=\sum_{x,y} P_{XY}(x,y)\log\frac{P_{XY}(x,y)}{P_X(x)P_Y(y)},
\]
with \(P_{XY}\) the joint distribution and \(P_X,P_Y\) the marginals [2412.14527].

Operationally, the procedure has four steps. It computes pairwise mutual information values across data dimensions, constructs a mutual-information matrix, converts that matrix to a dissimilarity matrix, runs \(k\)-means clustering on that dissimilarity to form strata, and finally performs stratified simple random sampling within those strata using Neyman or “optimal” allocation. The paper uses this method on a small breast-cancer dataset with 272 samples [2412.14527].

The contrast with support points is explicit. Mutual-information-based stratified SRS groups similar data points and then samples randomly within the resulting strata. It does not optimize a global distributional discrepancy. Support points, by contrast, are obtained by direct minimization of empirical energy distance. The paper also notes a computational asymmetry: the MI-based method, as implemented, is \(O(n^2)\) because of pairwise mutual-information computation, which makes it infeasible for large datasets; this is why it is not used on the credit-card fraud data [2412.14527].

A plausible implication is that the two methods target different forms of representativeness. MI-based sampling promotes diversity through stratification, whereas support points impose a global distribution-matching criterion over the retained majority subset. The paper treats them separately and does not explore a hybrid procedure [2412.14527].

## 5. Experimental evidence and representativeness diagnostics

The support-points evaluation on the credit-card fraud dataset compares the original majority class, a majority subset chosen by naive random undersampling, and a majority subset chosen by support points after nearest-neighbor mapping. The paper examines feature-wise means and standard deviations as well as Kolmogorov–Smirnov tests on marginal distributions [2412.14527].

For the three example features reported, the original, support-points, and naive-random statistics are as follows. For Feature 5, the means are \(0.0036\), \(0.0006\), and \(0.0075\), while the standard deviations are \(0.9846\), \(0.9558\), and \(0.9870\). For Feature 12, the means are \(0.0102\), \(0.0240\), and \(0.0347\), and the standard deviations are \(0.9504\), \(0.9737\), and \(0.9525\). For Feature 21, the means are \(-0.0011\), \(0.0663\), and \(0.0032\), and the standard deviations are \(0.9886\), \(1.4370\), and \(0.9244\). The paper’s conclusion is that both methods approximate the original distribution fairly well; support points sometimes deviate more in variance for certain features but overall maintain good fidelity [2412.14527].

The KS tests provide stronger distributional evidence. Example results reported are: for Feature 8, KS(Support Points) \(=0.0232\) with \(p=0.9556\), versus KS(Naive) \(=0.0283\) with \(p=0.8331\); for Feature 14, KS(Support Points) \(=0.0407\) with \(p=0.4055\), versus KS(Naive) \(=0.0650\) with \(p=0.0352\); for Feature 24, KS(Support Points) \(=0.0208\) with \(p=0.9846\), versus KS(Naive) \(=0.0327\) with \(p=0.6809\). High \(p\)-values indicate that the subset’s marginal distribution is not significantly different from the original majority distribution, and the paper notes that support points often yield slightly higher \(p\)-values than naive undersampling on some features [2412.14527].

The classification results are reported using balanced accuracy, although the table labels them as “accuracy” in the paper’s discussion. Support points always match or slightly improve over random undersampling on the fraud dataset [2412.14527].

| Classifier | Random undersampling | Support points |
|---|---:|---:|
| Logistic Regression | 0.9261 | 0.9401 |
| Random Forest | 0.9295 | 0.9330 |
| XGBoost | 0.9225 | 0.9225 |
| SVC | 0.9260 | 0.9331 |

The largest gain is for Logistic Regression, at about 1.4 percentage points. SVC also improves, Random Forest improves slightly, and XGBoost is unchanged. This suggests that preserving the global majority distribution can be particularly useful for classifiers whose performance is more sensitive to the retained geometry of the training set [2412.14527].

## 6. Limitations, interpretation, and broader context

The paper does not derive new theorems for support points in imbalanced classification, but it relies on the fact that energy distance is a metric on distributions and on the attraction-repulsion structure of the empirical objective. This suggests that support-point undersampling is appropriate when the goal is to compress the majority class while preserving its distributional structure rather than merely its local boundary geometry [2412.14527].

Its limitations are explicit. Direct support-point optimization on very large majority classes can be infeasible, as illustrated by the more-than-600-GB memory requirement for the full fraud dataset. Gradient descent may converge to local minima, so initialization and learning rate matter. The method emphasizes global distribution matching, which is beneficial for representativeness but may miss fine local features near decision boundaries; the paper therefore notes that if local boundary behavior is crucial, boundary-focused methods may still be relevant [2412.14527].

The broader context presented in the paper connects support points to uncertainty quantification, Bayesian computation, distribution compression, kernel herding, coresets, and optimal experimental design. In the original support-points formulation, they are optimal discrete approximations that minimize energy distance to a target distribution [1609.01811]. Later work extends this idea to weighted and random support points for generative modeling, again treating support points as compact representative sets derived from a distributional objective rather than a neural architecture [2508.21255]. Within that larger lineage, the present paper uses support points not for integration or generation, but as a principled undersampling mechanism for majority classes in imbalanced classification [2412.14527].

A plausible implication is that the paper’s contribution is methodological rather than architecture-specific. It frames undersampling as a problem of empirical distribution approximation, and then shows that this framing yields a practical alternative to random undersampling and to information-theoretic stratification when the majority class is large, heterogeneous, and expensive to process in full [2412.14527].

Source: https://www.emergentmind.com/topics/support-points