---
title: Coreset Subsampling Overview
url: https://www.emergentmind.com/topics/coreset-subsampling
type: topic
---

# Coreset Subsampling Overview

Coreset subsampling is a central framework for dataset reduction across computational statistics, machine learning, signal processing, Bayesian inference, and numerical linear algebra. The core idea is to construct a small, weighted subset (the "coreset") of a much larger dataset such that key model or inference tasks performed on the coreset approximate those on the full data according to predefined guarantees. Coreset subsampling strategies leverage probabilistic, combinatorial, geometric, convex, or submodular properties of the data and the corresponding optimization problems, enabling algorithmic and statistical speedups, scalability to massive data, memory and energy savings, and new theoretical insights into data summarization.

## 1. Coreset Fundamentals: Definitions, Problem Statements, and Applicability

A coreset for a problem is a (typically small) weighted subset whose objective, or cost, approximates that of the full data to provable accuracy. Let $P$ be a weighted dataset, $Q$ a family of queries (e.g., model parameters, clusters), and $f:P \times Q \to \mathbb{R}_{\ge 0}$ a loss/cost function. A weighted subset $(C, u)$ of $P$ is an $\varepsilon$-coreset if for all $q \in Q$:
\[
\Big| \sum_{p \in P} w(p) f(p, q) - \sum_{c \in C} u(c) f(c, q) \Big| \leq \varepsilon \sum_{p \in P} w(p) f(p, q)
\]
This structure covers mean/variance estimation [2111.03046], $k$-means/median clustering, $k$-line clustering, subspace approximation, SVMs [2002.06469], kernel density estimation [1709.04453], low-rank factorization [1907.01433, 2206.10240], and Bayesian inference [2310.17063, 2211.02377, 2203.09675].

Core applications include:
- Efficient model training and validation with reduced data
- Fast hyperparameter and architecture sweeps
- Streaming, distributed, and federated learning
- Accelerated optimization and Bayesian inference
- Robustness to noisy or adversarial data

Coreset size typically depends on data dimension, model complexity, error tolerance, and the "sensitivity" structure of the specific problem.

## 2. Sensitivity, Importance, and Diversity Sampling Frameworks

**Sensitivity sampling** is foundational for many coreset constructions [1612.00889, 1907.01433]. For each data point $p \in P$, its sensitivity
\[
\sigma(p) = \sup_{q \in Q} \frac{w(p) f(p, q)}{\sum_{p' \in P} w(p') f(p', q)}
\]
quantifies its maximal influence on the objective. The sum $t = \sum_{p \in P} \sigma(p)$ governs sample complexity: sampling $N = O((t/\varepsilon^2) (d \log t + \log(1/\delta)))$ points with probability proportional to $\sigma(p)/t$ and appropriate rescales yields an $\varepsilon$-coreset for cost functions of VC-dimension $d$ [1612.00889]. This framework unifies theoretical guarantees for $k$-clustering [2004.06263], SVM [2002.06469], regression [2206.10240], density estimation [2011.04907], and many “near-convex” problems [2006.05482].

**Importance sampling** generalizes sensitivity sampling with heuristic or problem-specific weights (leverage scores, gradient magnitudes, combined influence metrics). **Diversity-based sampling** (e.g., Determinantal Point Processes, DPPs) introduces negative correlations to reduce redundancy [1803.08700], strictly lowering estimator variance and often yielding superior subsample efficiency, especially in clustering and regression.

## 3. Submodular, Geometric, and Modern Non-Sensitivity-Based Coresets

Recent advances address the empirical and computational limitations of sensitivity and importance sampling for high-dimensional, nonconvex, or deep learning settings. **Submodular maximization**—specifically facility location and related functions—yields robust streaming-compatible greedy algorithms with $(1-1/e)$-optimality guarantees for set selection, commonly used in SubZeroCore [2509.21748] and deep coreset libraries [2204.08499]. These methods synthesize density, coverage, and diversity criteria in the coreset objective and leverage scalable $k$-nearest neighbor search and lazy greedy maximization.

**Geometric partition/aggregation** approaches (such as ring decomposition for clustering [2209.01901]) can, sometimes surprisingly, enable pure uniform sampling or VC-based approximations with coreset size independent of $n$. These methods are particularly effective for constrained clustering (capacitated, fair, or Wasserstein barycenter), and yield smaller, sometimes optimal $\varepsilon$-dependent coresets in low dimensions.

## 4. Specialized and Advanced Coreset Constructions

**Bayesian coresets** recast posterior inference as a data summarization problem, optimizing weighted KL-divergence between the full and coreset posterior [2203.09675, 2310.17063, 2211.02377]. Greedy variational, quasi-Newton, and even MCMC-based joint sample-weighted schemes are established, with explicit high-probability guarantees, control of approximation error in total variation or two-moment KL, and extension to intractable BNNs and other models.

For **kernel density estimation** and general smooth divergences (including Sinkhorn), Carathéodory or kernel quadrature-based strategies [2011.04907, 2504.20194] enable coreset construction via moment or maximum mean discrepancy minimization. These methods achieve minimax-optimal $L_2$ risk and, especially for Sinkhorn divergence, achieve sublinear ($m = o(n)$) coreset size with rigorous statistical control.

**Element-wise core-sets** [2206.10240, 2509.18024] select large-magnitude entries per column (rather than rows), optimally exploiting numerical sparsity in regression or matrix factorization (e.g., ALS for recommender systems), providing notable computation and accuracy speedups in very high dimensions.

## 5. Empirical Performance, Complexity, and Practical Choices

Empirical evaluations consistently support the theoretical speedup and compression of coreset subsampling, but reveal context-dependent trade-offs and the need for careful baseline testing:
- For mean, clustering, regression, and graphical estimation, sensitivity or diversity-based coresets produce 5–50$\times$ lower error than uniform at equal size, especially in high-variance or highly redundant data regimes [1907.01433, 2002.06469, 1709.04453, 1803.08700, 2206.10240, 1906.01021].
- In deep learning, especially under moderate budgets and robust architectures, random or stratified sampling is often surprisingly competitive [2204.08499, 2301.06163]. Training-free geometric or submodular schemes (e.g., SubZeroCore [2509.21748]) increasingly outperform gradient or error-based baselines at extreme pruning.
- Quasi-Newton and variational refinement for Bayesian coresets can attain near full-data posterior accuracy with $10$–$1000\times$ speedup for moderate $M$, but communication/storage costs remain challenging for extreme scales [2203.09675, 2310.17063, 2211.02377].

Generic complexity is at most a small multiple of original data passes ($O(nd)$ to $O(\mathrm{nnz}(X) + rp^2)$, or $O(K[NM^2 + M^3 + SN])$ for Bayesian MCMC/VI) unless advanced approximate nearest neighbor or randomized algebraic routines are used.

## 6. Extensions: Streaming, Distributed, Budget-Aware, and Robust Coresets

Merge-and-reduce paradigms [1612.00889, 1907.01433] enable scalable streaming and distributed coresets: per-block summaries are coresetized locally, then recursively merged and re-coresetized, maintaining polylogarithmic size, communication, and error. Robust variants—such as median-of-means aggregation in linear regression [2206.10240] or cost-aware greedy schemes for graph summarization [1906.01021]—adapt to outliers, nonuniform costs, and adversarial contamination. Element- or block-wise selection is particularly effective in networks, tensor decompositions, and large-scale collaborative filtering [2509.18024].

## 7. Limitations, Open Problems, and Future Directions

Despite theoretical and empirical success, practical deployment of coresets reveals several gaps:
- Sensitivity bounds are sometimes too loose to outperform uniform sampling, particularly in loosely regularized or low-variance models [2301.06163].
- Many advanced coresets have substantial model/hyperparameter dependency or require pretraining or nontrivial feature engineering [2509.21748, 2204.08499].
- Finite-sample and sharp minimax bounds for high-dimensional, nonconvex, or composite objectives remain open [2504.20194, 2211.02377].
- Developing fully automatic, adaptive, or data-driven coreset size selectors and deeper integration into iterative machine learning pipelines (e.g., coreset MCMC, dataset distillation) are active lines of research.

Further connections of coreset construction to kernel quadrature, moment and score matching, discrepancy theory, and randomized numerical linear algebra continue to deepen and broaden the scope of efficient and theoretically principled data summarization.

---

**Selected references**: [1612.00889], [1803.08700], [1709.04453], [2002.06469], [2004.06263], [1907.01433], [2203.09675], [2206.10240], [2310.17063], [2211.02377], [2011.04907], [2209.01901], [1906.01021], [2509.21748], [2204.08499], [2301.06163], [2006.05482], [2509.18024], [2504.20194].

Source: https://www.emergentmind.com/topics/coreset-subsampling