Coreset Sampling Strategy: Concepts & Coverage

• Coreset sampling strategy is a method for selecting a representative subset of data using semantic scores derived from concept bottlenecks and stratified coverage to preserve training utility.
• It leverages LLM-based semantic signals to compute sample difficulty via a fast linear bottleneck, achieving significant speed-ups compared to full-model approaches.
• The strategy employs stratified sampling to balance representation across score strata, ensuring robust downstream performance even at high pruning rates.

Coreset sampling strategy denotes the procedure by which a learning system selects a small subset of examples intended to preserve the training utility of a much larger dataset. In contemporary coreset selection, the central design choice is the sampling signal: some methods use training dynamics, some use geometric or submodular coverage, some use classwise diversity, and some optimize a continuous weighting objective. A recent line of work replaces downstream-model-dependent signals with concept-alignment scores derived from LLMs and visual-language encoders, then applies stratified coverage-centric sampling to the resulting score distribution, yielding a coreset that is efficiently computable, interpretable at the level of human-understandable attributes, and usable even for unlabeled data (Mehra et al., 23 Feb 2025).

1. Problem setting and selection criteria

Coreset Selection (CS) aims to identify a subset of the training dataset that achieves model performance comparable to using the entire dataset. In many state-of-the-art CS methods, the subset is selected using scores whose computation requires training the downstream model on the entire dataset first and recording changes in the model’s behavior on samples as it trains. The reported limitations of this paradigm are twofold: the scores are inefficient to compute, and they are hard to interpret because they do not indicate whether a sample is difficult to learn in general or only for a specific downstream model.

A second design constraint arises at high pruning rates. Coverage-centric work shows that one-shot methods which simply rank examples by importance can suffer a catastrophic accuracy drop at high pruning rates, and can perform worse than random sampling because of worse data coverage. That literature introduces a distribution-cover perspective through the $p$-partial $r$-cover and the associated $AUC_{pr}$, with

$$AUC_{pr}(S')=\int_0^1 r(p)\,dp = \mathbb E_{x\sim P_\mu}[d(x,S')],$$

thereby making coverage an explicit property of the selected subset rather than an incidental by-product of importance ranking (Zheng et al., 2022).

Within this design space, the concept-bottleneck strategy in (Mehra et al., 23 Feb 2025) is defined by three coupled choices: a model-agnostic semantic score, a margin-based estimate of sample difficulty, and a stratified sampling rule intended to preserve coverage across the difficulty spectrum.

2. LLM-based concept bottlenecks

The concept-bottleneck construction begins with a labeled dataset

$\mathcal D=\{(x_i,y_i)\}$

of $n$ examples and $N$ classes. For each class label name $\ell$, a LLM such as LLaVA is prompted with: “Can you give distinct attributes for $\langle \ell\rangle$? Output a comma-separated list.” The LLM returns a short list of keyword attributes; the description gives “airplane” with attributes such as “wing tips” and “nose cone.” The procedure keeps $k$ discriminative attributes per class, plus the class name itself, yielding a concept set

$r$0

Each concept $r$1 is embedded by a pre-trained text encoder $r$2, for example CLIP’s text tower, to produce a $r$3-dimensional vector. Stacking these vectors yields the concept-embedding matrix

$r$4

A fixed pre-trained visual encoder $r$5, such as CLIP’s image tower, is then used to compute concept similarities: $r$6 This vector is the sample’s dot-product similarity to each concept. A linear concept-bottleneck layer $r$7 is trained so that

$r$8

approximates the ground-truth label; the final softmax is omitted when measuring margins. The parameters are learned by minimizing the cross-entropy objective

$r$9

via SGD for $AUC_{pr}$0 epochs. Because $AUC_{pr}$1 is only a single linear layer on top of fixed features, the training cost is reported as very cheap, including minutes on ImageNet (Mehra et al., 23 Feb 2025).

This construction makes the sampling signal explicitly semantic: the intermediate representation is not tied to the hidden states of a future downstream ResNet or ViT, but to similarities between visual features and textual attributes.

3. Difficulty estimation by area under the margin

Once the linear bottleneck head is trained, sample difficulty is defined through the Area-Under-the-Margin (AUM) of that bottleneck predictor. At epoch $AUC_{pr}$2, the temporary predictor is $AUC_{pr}$3. For each labeled sample $AUC_{pr}$4, the margin is

$AUC_{pr}$5

The sample’s AUM is the average margin across training epochs: $AUC_{pr}$6 Lower AUM implies a harder sample.

The reported interpretation is that difficulty is the alignment between a sample’s visual features and semantic attributes, not quirks of a particular downstream model. The method never trains or observes any downstream model of interest; all scores come from the simple concept-bottleneck head $AUC_{pr}$7. Because $AUC_{pr}$8 is fixed and derived from human-understandable concepts, the AUM is described as model-agnostic. This suggests a distinction between semantic difficulty and architecture-specific difficulty: the former is estimated from the bottleneck geometry, whereas the latter is deliberately excluded from the scoring pipeline (Mehra et al., 23 Feb 2025).

4. Stratified coverage-centric sampling

Given scores $AUC_{pr}$9, pruning ratio $$AUC_{pr}(S')=\int_0^1 r(p)\,dp = \mathbb E_{x\sim P_\mu}[d(x,S')],$$0, cutoff $$AUC_{pr}(S')=\int_0^1 r(p)\,dp = \mathbb E_{x\sim P_\mu}[d(x,S')],$$1, and number of bins $$AUC_{pr}(S')=\int_0^1 r(p)\,dp = \mathbb E_{x\sim P_\mu}[d(x,S')],$$2, the sampling rule is a coverage-centric stratified procedure.

The algorithm is:

1. Sort all examples by increasing $$AUC_{pr}(S')=\int_0^1 r(p)\,dp = \mathbb E_{x\sim P_\mu}[d(x,S')],$$3, with the hardest first.
2. Remove the top $$AUC_{pr}(S')=\int_0^1 r(p)\,dp = \mathbb E_{x\sim P_\mu}[d(x,S')],$$4 hardest samples to guard against label noise.
3. Bin the remaining $$AUC_{pr}(S')=\int_0^1 r(p)\,dp = \mathbb E_{x\sim P_\mu}[d(x,S')],$$5 examples into $$AUC_{pr}(S')=\int_0^1 r(p)\,dp = \mathbb E_{x\sim P_\mu}[d(x,S')],$$6 equal-mass score-quantile bins $$AUC_{pr}(S')=\int_0^1 r(p)\,dp = \mathbb E_{x\sim P_\mu}[d(x,S')],$$7.
4. Let $$AUC_{pr}(S')=\int_0^1 r(p)\,dp = \mathbb E_{x\sim P_\mu}[d(x,S')],$$8 be the target coreset size and initialize $$AUC_{pr}(S')=\int_0^1 r(p)\,dp = \mathbb E_{x\sim P_\mu}[d(x,S')],$$9.
5. Repeat until $\mathcal D=\{(x_i,y_i)\}$0 has $\mathcal D=\{(x_i,y_i)\}$1 examples:
   • pick the bin $\mathcal D=\{(x_i,y_i)\}$2 with fewest unselected examples;
   • allocate a budget

   $\mathcal D=\{(x_i,y_i)\}$3

• sample $\mathcal D=\{(x_i,y_i)\}$4 points uniformly from $\mathcal D=\{(x_i,y_i)\}$5, add them to $\mathcal D=\{(x_i,y_i)\}$6, remove them from $\mathcal D=\{(x_i,y_i)\}$7, and reduce $\mathcal D=\{(x_i,y_i)\}$8 by $\mathcal D=\{(x_i,y_i)\}$9.

1. Return $n$0.

The broader CCS literature frames this type of procedure as an alternative to pure importance pruning. Rather than keeping only the highest-scoring points, it enforces representation across score strata, with a hard cutoff to suppress the most problematic examples. In the earlier CCS formulation, the score range is partitioned into equal-width intervals and the method jointly considers overall data coverage upon a distribution as well as the importance of each example; the concept-bottleneck method instantiates the same coverage-centric principle using score-quantile bins over semantic AUM scores (Mehra et al., 23 Feb 2025, Zheng et al., 2022).

A common misconception is that coreset sampling is equivalent to selecting the “hardest” points. The coverage-centric analysis rejects that equivalence: at high pruning rates, removing all low-difficulty examples can damage coverage of dense regions, enlarge $n$1, and degrade downstream accuracy.

5. Computational profile and label-free extension

Training-dynamics approaches are reported to cost

$n$2

for example computing AUM on the full ResNet-34 trained from scratch, which is described as hours on ImageNet. The concept-bottleneck strategy uses one linear layer over fixed CLIP features. On ImageNet, concept-bottleneck training is reported as approximately $n$3 min versus $n$4 h for the full model, yielding a $n$5 speed-up.

A further extension eliminates the need for human labels. Pseudo-labels are obtained by zero-shot CLIP: $n$6 The method then computes $n$7 exactly as in the labeled setting and runs the same CCS procedure. The description states that no human labels are needed.

This label-free variant is significant because it preserves the same three-stage structure—semantic representation, bottleneck AUM, and stratified sampling—while shifting supervision from ground-truth labels to VLM pseudo-labels. A plausible implication is that the sampling strategy is separable from the annotation regime: the scoring mechanism depends on a concept-aligned bottleneck, not on the source of the label itself (Mehra et al., 23 Feb 2025).

6. Empirical behavior and transfer

The reported experiments cover CIFAR-10, CIFAR-100, and ImageNet-1K, with downstream test accuracy or F1 measured for models trained on coresets at pruning rates $n$8. For labeled CS on CIFAR-10/100 and ImageNet-1K, the downstream models are ResNet-18 on CIFAR and ResNet-34 on ImageNet. Random subsets are reported to fall off steeply at high pruning. Training-dynamics-based methods such as Forgetting Score and full-model AUM reach approximately $n$9 at $N$0 CIFAR-10 and approximately $N$1 at $N$2 CIFAR-10. The concept+AUM+CCS method obtains $N$3 on CIFAR-10, matches or exceeds dynamics-based AUM on ImageNet at low pruning, and dramatically outperforms random at high pruning, including a $N$4 gain at $N$5.

For label-free CS on CIFAR-10/100, the method’s Ours-LF coresets outperform random, with $N$6 versus $N$7 at $N$8 CIFAR-10, and are competitive with ELFS. On ImageNet, Ours-LF is reported to be better than Random at all $N$9 and to exceed ELFS (DINO variant) at smaller pruning rates. The same concept-derived coresets also transfer across ResNet-18, ResNet-34, and ResNet-50 with consistent gains over random. Additional experiments on AffectNet and BloodMNIST report that concept-coresets outperform random in both cases (Mehra et al., 23 Feb 2025).

These results place the method in a specific empirical niche: it is not presented as uniformly dominant at every regime, but as especially effective when pruning is aggressive and when the cost of full-model score computation is prohibitive.

7. Relation to other coreset sampling paradigms

The broader literature shows that coreset sampling strategy is a heterogeneous design space rather than a single canonical recipe. Coverage-centric one-shot selection uses importance scores, but counters their failure at high pruning by stratification across score strata; on CIFAR-10 at $\ell$0, reported ResNet18 accuracies include Random $\ell$1, Forgetting $\ell$2, AUM $\ell$3, Moderate $\ell$4, CCS with AUM $\ell$5, and CCS with Forgetting $\ell$6 (Zheng et al., 2022).

A different paradigm is classwise diversity preservation. In the intra-class diversity method, each class is embedded by a pretrained feature extractor, dimension-reduced via PCA, clustered by $\ell$7-Medoids, and sampled uniformly within each cluster so that both inter-class balance and intra-class diversity are retained. On the Peripheral Blood Cell dataset, the $\ell$8 coreset results report RS $\ell$9 Acc$\langle \ell\rangle$0 and IS $\langle \ell\rangle$1 Acc$\langle \ell\rangle$2 for both a custom CNN and a ResNet-101-v2 transfer-learning setup (Ashraf et al., 23 Sep 2025).

Open-set SSL induces yet another geometry. SimCore defines a facility-location-style objective

$\langle \ell\rangle$3

with cosine similarity in latent space, and greedily mines a subset of the open-set that is semantically closest to the target dataset. Across $\langle \ell\rangle$4 fine-grained datasets, it reports an average $\langle \ell\rangle$5 linear-probe accuracy gain over SSL on $\langle \ell\rangle$6 alone, versus $\langle \ell\rangle$7 when naively using the full open-set $\langle \ell\rangle$8 (Kim et al., 2023).

Continuous optimization approaches replace subset search by probabilistic inclusion variables. Probabilistic bilevel coreset selection introduces Bernoulli masks $\langle \ell\rangle$9, optimizes the expected outer loss

$k$0

and updates $k$1 by an unbiased score-function estimator without implicit differentiation. The reported empirical setting includes data summarization, label noise, class imbalance, continual learning, streaming, and feature selection, with advantages especially in more challenging label-noise and class-imbalance scenarios (Zhou et al., 2023).

Classical clustering theory emphasizes sensitivity and diversity. For $k$2-means, sensitivity sampling yields worst-case coresets of size $k$3, and for $k$4 cost-stable data it yields $k$5, while remaining oblivious to the stability parameter in the sampling rule itself (Bansal et al., 2024). DPP-based coresets introduce repulsive sampling through an $k$6-ensemble, and the corresponding theory shows lower variance than iid sampling with the same marginals, together with applications to $k$7-means and linear regression (Tremblay et al., 2018).

Uniform-sampling results define a separate thread. Stable coresets show that a uniform sample of size $k$8 yields, with high constant probability, a stable coreset for $k$9-median in $r$00 under the $r$01 metric (Carmel et al., 26 Sep 2025). The ring-decomposition meta-theorem for constrained clustering reduces coreset construction to a bounded number of ring instances and yields, for $r$02-median, coreset sizes $r$03 in $r$04 and $r$05 in $r$06 using uniform sampling (Braverman et al., 2022).

Distribution-matching formulations depart from subset scoring altogether. Wasserstein measure coresets minimize

$r$07

by SGD on a discrete measure supported on representative points, thereby controlling broad function classes through $r$08 or $r$09 (Claici et al., 2018). CO$r$10 treats generic smooth divergences through a functional Taylor expansion, reduces coreset selection to MMD minimization under a Hadamard operator, and for the Sinkhorn divergence obtains asymptotically lossless coresets with $r$11 under exponential kernel-spectrum decay (Kokot et al., 28 Apr 2025).

This suggests that “coreset sampling strategy” is best understood as a composite object with at least four orthogonal axes: the score or objective being optimized, the geometry in which representativeness is defined, the weighting or subset-selection mechanism, and the invariances demanded by the downstream task. Within that taxonomy, LLM-based concept bottlenecks occupy the regime of semantic, model-agnostic scoring combined with coverage-centric stratified sampling.