---
title: Stratified Coreset Sampling
url: https://www.emergentmind.com/topics/stratified-coreset-sampling
type: topic
---

# Stratified Coreset Sampling

Stratified coreset sampling refers to the construction of coresets—a weighted subset of the data that approximates the result of a learning or optimization problem—by partitioning the dataset into meaningful strata (layers), sampling within these strata, and assigning appropriate weights. Such methods offer robustness against outliers, improved representation across data subpopulations, and statistical guarantees at reduced sample size and computational overhead.

## 1. Principles of Data Stratification and Layer Definition

Stratification is achieved by partitioning input data according to key statistics, model dynamics, or semantically meaningful scores. In robust clustering or regression, layered sampling [2002.11904] proceeds by defining nested regions around an initial solution:

For $k$-median/means with outliers, the data $P\subset\mathbb{R}^d$ is partitioned into layers
- $H_0$ includes points within a small radius $r$ of centers $\tilde C$,
- $H_i$ spans $(2^{i-1}r,\, 2^{i}r]$ (dyadic rings) around centers, for $i=1,\dots,N$,
- $H_{out}$ collects the (1+1/ε)z farthest points as a dedicated outlier stratum.

For linear regression, slabs about an initial hyperplane $\tilde h$ define layers $H_i$ by bounding residuals. In concept-bottleneck-based sampling [2502.16733], data is partitioned into deciles or quantile bins of concept-driven margin scores (AUM), potentially excising the top-$\beta$ hardest examples. In class-difficulty-separable scenarios [2507.10904], stratification occurs across classes, with further stratification within each class based on difficulty.

## 2. Stratified Coreset Construction Algorithms

Sampling within layers exploits statistical concentration inequalities (Hoeffding, Chernoff):
- For each stratum $H_i$, $m_i$ samples are drawn uniformly (without replacement) and assigned weight $w(p)=n_i/m_i$ ($n_i$ is stratum population).
- Outliers are wholly included with unit weight to prevent under-sampling.
- Aggregated samples from all strata form the weighted coreset $S$.

Key pseudocode (as in [2002.11904] and [2312.09885]) appears below:

```pseudo
for each stratum H_i:
    sample m_i points uniformly without replacement
    assign weight w(p) = n_i / m_i to sampled points
include all points in H_out with weight 1
aggregate all weighted samples as coreset S
```

In class-proportional selection [2507.10904]:
- Total coreset size quota is allocated proportionally to class sizes ($B'_c = \max(\lfloor \alpha n_c \rfloor, m)$ for class $c$),
- Sampling is performed using methods like Hardest-CP, Sliding-Window-CP, or CCS-CP, independently per class.

In concept-stratified selection [2502.16733], bins are defined over concept-aligned AUM scores (e.g., $b=10$), with iterative coverage to ensure each stratum is represented.

## 3. Theoretical Guarantees and Statistical Properties

Stratified sampling inherits concentration from per-stratum sampling bounds, with union bounds controlling global approximation error across all strata. For robust clustering, layered sampling yields [2002.11904]:

- For $k$-median/means with outliers, the weighted coreset $S$ satisfies
$$
|K_1^{-z}(S, C) - K_1^{-z}(P, C)| \leq \varepsilon(K_1^{-z}(P,\tilde{C}) + L)
$$
for any solution within a local neighborhood $\tilde{C}\pm L$.

- Coreset size is $O((kd/\varepsilon^2)\,\mathrm{polylog}(d/\varepsilon\delta)) + (1+1/\varepsilon)z$.

For non-decomposable objectives (F1, MCC), stratified coresets achieve weak $\varepsilon$-guarantees on metrics for all classifiers above a performance threshold, with sample complexities that match lower bounds:
$$
O\left( \frac{d+\log(1/\delta)}{\varepsilon^2} \right)
$$
per class, where $d$ is VC-dimension and $\delta$ is the confidence parameter [2312.09885].

Uniform stratification by distance rings yields $\tilde{O}(k^3\varepsilon^{-6})$ size coresets for capacitated/fair $k$-median/means and $\tilde{O}(\varepsilon^{-1.5})$ for the geometric median in $\mathbb{R}^2$ [2209.01901]. Stratification by concept-difficulty yields balanced coverage and resilience to mislabeled outliers, with empirical robustness at high pruning rates [2502.16733].

## 4. Stratification Schemes: Class, Concept, and Difficulty

Stratification schemes vary with problem context:
- **Class-based**: Each class receives a quota and is sampled independently, preventing majority class dominance and mitigating minority class underrepresentation [2507.10904].
- **Concept-difficulty**: Samples are binned by interpretable model-agnostic difficulty scores (AUM via concept bottlenecks) [2502.16733]; this enables coverage across semantic complexity and works for both labeled and unlabeled data.
- **Distance/residual-based**: Layers are built with exponentially growing radii or residual slabs about an initial solution to absorb variance and isolate outliers [2002.11904].

| Stratification Principle | Layer Definition                       | Guarantee Type                         |
|-------------------------|----------------------------------------|----------------------------------------|
| Class-proportional      | Per-class quotas, stratify difficulty  | Preserves class proportions, robust to imbalance [2507.10904] |
| Concept/difficulty      | AUM quantile bins, outlier cutoff      | Balanced semantic coverage, outlier control [2502.16733] |
| Distance/residual       | Dyadic metric rings/slabs, outlier bin | Additive error for robust objectives, trims outliers [2002.11904] |

## 5. Empirical Results and Practical Implementation

Stratified coreset sampling demonstrates improved data efficiency and stability under aggressive pruning. Representative empirical results include:
- Class-proportional CCS-CP maintains >97% accuracy, precision, and recall even at 99% pruning, outperforming class-agnostic CCS (accuracy drop: 2.58% vs. 7.59%) on CTU-13 [2507.10904].
- Concept-stratified coresets achieve 84.6% accuracy (90% pruning) on CIFAR-10, compared to 79.1% for random sampling under the same protocol [2502.16733].
- Stratified coresets match or outperform leverage-score or k-means-based coresets in non-decomposable settings (F1, MCC), with 10–100× runtime advantage [2312.09885].

Key practical recommendations:
- Measure class-difficulty separability ($\delta_{CD}$) to guide stratification.
- Allocate sampling budgets to minority/rare strata for variance reduction.
- Use empirically-determined stratification parameters (number of bins, outlier cutoff).
- Parallelize stratified sampling across strata for scalability to large datasets.

## 6. Relation to Classical Stratified Sampling

Stratified coreset sampling derives from classical stratified sampling but adapts to machine learning through target-aware strata boundaries, adaptive allocation, and coreset weight assignment:

Similarities:
- Data partitioned so that target statistics (e.g., loss, metric) are bounded within layers.
- Uniform subsampling and weighting within strata yield unbiased estimators.
- Per-stratum concentration ensures global approximation.

Differences:
- Strata are frequently adaptive and guided by empirical model characteristics, not fixed population features.
- Outlier layers are explicitly constructed and always fully represented.
- Guarantees target not just mean estimation but complex optimization objectives (robust clustering, non-decomposable metrics, assignment-preserving clustering).

This tailored stratification yields coresets suitable for robust optimization, structured datasets, and fairness/constrained settings, with sample sizes independent of data size for many tasks [2002.11904, 2209.01901].

## 7. Applications and Extensions

Stratified coreset sampling underpins scalable algorithms for:
- Robust $k$-median/means clustering with/without outliers [2002.11904],
- Linear regression under outlier contamination [2002.11904],
- Non-decomposable supervised classification objectives (F1, MCC) [2312.09885],
- Model-agnostic image dataset pruning using concept bottlenecks [2502.16733],
- Fair and capacitated clustering and Wasserstein barycenter computation [2209.01901],
- High-stakes, imbalanced domains (network security, medical imaging) [2507.10904].

A plausible implication is that stratified coreset sampling generalizes classical sampling theory to contemporary large-scale, high-dimensional, and structured data, providing an adaptive mechanism for efficient and effective data reduction across a spectrum of learning and optimization tasks.

Source: https://www.emergentmind.com/topics/stratified-coreset-sampling