Optimized Coreset Selection

• Optimized coreset selection is a technique that builds compact, representative data subsets to yield full-data performance with reduced computational cost.
• It leverages methods such as hardness-based, training-free submodular selection, and adaptive pruning to enhance robustness against noise and adversarial attacks.
• State-of-the-art approaches like EasyCore, SubZeroCore, HyperCore, and FAST offer formal guarantees, improved scalability, and significant accuracy gains in diverse learning tasks.

Optimized coreset selection refers to the design and application of algorithms that construct compact, representative subsets (“coresets”) of large datasets, such that training on the coreset efficiently yields model performance comparable to full-data training—even under adverse, noisy, or adversarial regimes. Recent advances formalize, analyze, and empirically validate coreset selection methods with strong theoretical guarantees and state-of-the-art empirical performance across robustness, scalability, and label-noise resilience, leading to new paradigms for efficient data-centric learning (Ramesh et al., 13 Oct 2025, Moser et al., 26 Sep 2025, Moser et al., 26 Sep 2025).

1. Hardness-based Coreset Selection and Adversarial Robustness

A principal recent breakthrough is the explicit linking of sample-wise “hardness”—quantified via the Average Input Gradient Norm (AIGN)—to adversarial vulnerability and model robustness (Ramesh et al., 13 Oct 2025). The AIGN of sample $(x_i, y_i)$ for a model $f_\theta$ trained over $T$ epochs is defined as

$$\operatorname{AIGN}(x_i) = \frac{1}{T} \sum_{t=1}^T \left\| \nabla_x \ell(f_{\theta^{(t)}}(x_i), y_i) \right\|_2$$

where $\ell(\cdot)$ is the per-sample loss. Sorting samples by low AIGN yields “easy” examples, which lie farthest from the decision boundary, exhibit high adversarial robustness, and induce wider margins when used preferentially in training.

The EasyCore algorithm proceeds by (a) accumulating per-sample input gradients over $T$ epochs, (b) computing AIGN per sample, and (c) statically selecting the lowest-AIGN fraction $f$ (commonly $5\%$ to $60\%$). Training on this coreset, under both standard and adversarial pipelines (e.g., PGD, TRADES), consistently yields up to $+7$ percentage points in adversarial accuracy compared to uniform, gradient-matching, or k-center baselines. Theoretical analysis attributes the gains to increased decision boundary margins and reduced boundary curvature, with high-AIGN example removal freeing model capacity to learn generalizable, robust features (Ramesh et al., 13 Oct 2025).

2. Submodular, Training-Free, and Density-Aware Selection

Optimized coreset methods have advanced beyond gradient or label-centric criteria to entirely training-free selection. SubZeroCore achieves this by combining submodular coverage and density weighting into a single monotone objective (Moser et al., 26 Sep 2025): $f_\theta$0 where $f_\theta$1 is a density score for candidate $f_\theta$2, computed using a closed-form expectation over $f_\theta$3-nearest neighbor radii, tuned via a single hyperparameter $f_\theta$4 controlling density–coverage trade-off.

SubZeroCore is fully label and training agnostic, requiring only embedding similarities; its greedy maximization yields a $f_\theta$5-approximation to the optimal subset. Empirically, SubZeroCore matches or surpasses gradient-based coreset methods at high pruning ratios ($f_\theta$6), with superior runtime (e.g., $f_\theta$7 min for CIFAR-10 at $f_\theta$8 vs. $f_\theta$9–$T$0 min for training-based signals) and robustness under label noise injections. These properties make SubZeroCore highly scalable and robust for large, noisy, or web-scale data (Moser et al., 26 Sep 2025).

3. Robust and Adaptive Coreset Selection Under Annotation Noise

HyperCore introduces class-wise hypersphere models for robust coreset selection in environments with noisy or ambiguous labels (Moser et al., 26 Sep 2025). For each class $T$1, a small neural network $T$2 maps samples to $T$3, and in-class samples (label $T$4) are clustered near the origin inside a hypersphere, while outliers (mislabeled, corrupted, or out-of-class points) lie outside.

Pruning is performed adaptively by maximizing Youden’s $T$5 statistic over the conformity distance distribution within and outside each class: $T$6 where TPR is the true positive rate for in-class points at radius $T$7, and FPR is the false positive rate for outliers. The optimal retention threshold $T$8 is set to maximize $T$9. This process discards ambiguous and noisy examples while keeping the informative core.

Experiments demonstrate HyperCore’s superiority under both moderate and severe label noise, low-data regimes, and imbalanced conditions: at $$\operatorname{AIGN}(x_i) = \frac{1}{T} \sum_{t=1}^T \left\| \nabla_x \ell(f_{\theta^{(t)}}(x_i), y_i) \right\|_2$$0 retention, accuracy gains over best baselines reach $$\operatorname{AIGN}(x_i) = \frac{1}{T} \sum_{t=1}^T \left\| \nabla_x \ell(f_{\theta^{(t)}}(x_i), y_i) \right\|_2$$1 points (Moser et al., 26 Sep 2025). The method is highly parallelizable and efficiently scalable.

4. Distribution Matching, Topological Constraints, and Frequency-Domain Methods

FAST leverages graph spectral theory and distributional matching in the frequency domain for DNN-free optimized coreset selection (Cui et al., 22 Nov 2025). The task is formulated as minimizing the characteristic function distance (CFD) between the empirical distribution of the full dataset $$\operatorname{AIGN}(x_i) = \frac{1}{T} \sum_{t=1}^T \left\| \nabla_x \ell(f_{\theta^{(t)}}(x_i), y_i) \right\|_2$$2 and candidate coreset $$\operatorname{AIGN}(x_i) = \frac{1}{T} \sum_{t=1}^T \left\| \nabla_x \ell(f_{\theta^{(t)}}(x_i), y_i) \right\|_2$$3: $$\operatorname{AIGN}(x_i) = \frac{1}{T} \sum_{t=1}^T \left\| \nabla_x \ell(f_{\theta^{(t)}}(x_i), y_i) \right\|_2$$4 where $$\operatorname{AIGN}(x_i) = \frac{1}{T} \sum_{t=1}^T \left\| \nabla_x \ell(f_{\theta^{(t)}}(x_i), y_i) \right\|_2$$5 is the empirical characteristic function evaluated at frequency $$\operatorname{AIGN}(x_i) = \frac{1}{T} \sum_{t=1}^T \left\| \nabla_x \ell(f_{\theta^{(t)}}(x_i), y_i) \right\|_2$$6.

To overcome the “vanishing phase gradient” in mid/high frequency $$\operatorname{AIGN}(x_i) = \frac{1}{T} \sum_{t=1}^T \left\| \nabla_x \ell(f_{\theta^{(t)}}(x_i), y_i) \right\|_2$$7 regimes, FAST introduces an attenuated phase-decoupled CFD loss, with targeted penalty in regions where only the phase conveys essential distributional information. Topology is preserved by extracting Laplacian eigenvectors from a multi-scale graph on $$\operatorname{AIGN}(x_i) = \frac{1}{T} \sum_{t=1}^T \left\| \nabla_x \ell(f_{\theta^{(t)}}(x_i), y_i) \right\|_2$$8, and the selection is constrained to align in manifold space by Hungarian matching and Laplacian regularization. Progressive Discrepancy-Aware Sampling schedules frequency sampling from low to high to match global structure prior to local refinement.

Benchmarks show FAST achieves up to $$\operatorname{AIGN}(x_i) = \frac{1}{T} \sum_{t=1}^T \left\| \nabla_x \ell(f_{\theta^{(t)}}(x_i), y_i) \right\|_2$$9 accuracy gains over SOTA DNN-free methods, $\ell(\cdot)$0 CPU speedup, and a $\ell(\cdot)$1 reduction in power consumption, with cross-model generalization loss $\ell(\cdot)$2 (Cui et al., 22 Nov 2025).

5. Coreset Selection for Structured and Non-classification Tasks

Optimized coreset selection has been adapted to object detection, continual learning, and dynamic systems identification. The CSOD framework for object detection aggregates classwise and imagewise RoI features, then applies a submodular facility-location objective balancing representativeness and diversity (Lee et al., 2024). Greedy maximization yields a performance boost of $\ell(\cdot)$3 points in AP$\ell(\cdot)$4 over random selection on Pascal VOC, with scalability to BDD100k and COCO2017.

Online coreset selection in dynamic systems uses geometric penetration criteria and polyhedral volume reduction guarantees for system identification, maintaining vanishing selection rates ($\ell(\cdot)$5 over $\ell(\cdot)$6 timesteps) while ensuring convergence of the feasible parameter set (Li et al., 28 Jun 2025).

For rehearsal-based continual learning, OCS maximizes gradient similarity scores online for current adaptation and memory affinity, yielding strong empirical results across standard, imbalanced, and noisy benchmarks (Yoon et al., 2021).

6. Theoretical Guarantees and Empirical Performance

State-of-the-art optimized coreset selection methods exhibit formal approximation guarantees and efficiency bounds. SubZeroCore and CSOD achieve provable $\ell(\cdot)$7 approximations to their objectives via greedy selection (Moser et al., 26 Sep 2025, Lee et al., 2024). Distribution-matching methods (Cui et al., 22 Nov 2025, Kokot et al., 28 Apr 2025) show poly-logarithmic or exponential sample size reduction for lossless compression under smooth divergences and RKHS spectral decay, with matching error rates to random sampling.

Hardness-based selection (Ramesh et al., 13 Oct 2025) demonstrates statistically significant robustness gains under adversarial training, outperforming model-dependent, gradient-based, and dynamic coreset methods by $\ell(\cdot)$8 points adv-accuracy (CIFAR-100, PGD-20). Robust selection methods such as HyperCore adaptively prune noisy data and outperform fixed-ratio selectors under rising label noise (Moser et al., 26 Sep 2025). FAST and SubZeroCore offer massive reductions in compute and power costs while preserving or improving downstream accuracy (Cui et al., 22 Nov 2025, Moser et al., 26 Sep 2025).

7. Integration and Deployment Considerations

Optimized coreset selection algorithms are highly modular. EasyCore requires only a single pre-training gradient accumulation run for AIGN computation; its scores are transferable across architectures and training protocols (Ramesh et al., 13 Oct 2025). SubZeroCore can operate on raw embeddings from any pretrained model, needing no labels or network training, and is suitable for large-scale noisy or unlabeled data (Moser et al., 26 Sep 2025). HyperCore trains small MLPs per class and requires no tuning of coreset size, automatically adjusting via Youden’s $\ell(\cdot)$9 statistic (Moser et al., 26 Sep 2025). FAST only needs Laplacian graph construction and frequency library sampling, enabling efficient execution on edge devices (Cui et al., 22 Nov 2025).

Empirical guidance recommends coreset fractions in the range $T$0–$T$1, with class-balancing and feature-space diversity constraints applied for high-imbalance or multi-modal settings. All leading approaches exceed or match baseline performance across classification, detection, continual learning, and regression tasks, with the added benefit of substantial computational savings.

---

Optimized coreset selection now offers rigorous, scalable, and robust frameworks for reducing data and computational burdens in deep learning, with theoretical guarantees, superior noisiness resilience, and ease of integration into diverse data-centric pipelines (Ramesh et al., 13 Oct 2025, Moser et al., 26 Sep 2025, Moser et al., 26 Sep 2025, Cui et al., 22 Nov 2025).