---
title: Greedy Coreset Subsampling
url: https://www.emergentmind.com/topics/greedy-coreset-subsampling
type: topic
---

# Greedy Coreset Subsampling

Greedy coreset subsampling refers to a class of algorithmic frameworks that use greedy selection strategies to construct compact, representative subsets ("coresets") for large-scale data problems, typically under cardinality or fidelity constraints. These methods aim to preserve a quantitative objective—such as data coverage, statistical error, or optimization loss—while operating at a fraction of the cost and size of the original dataset. The greedy paradigm is characterized by iterative selection: at each step, the item which maximally improves the objective (e.g., marginal gain) is added to the coreset, sometimes with further local or distributed refinements. Recent research establishes theoretical approximation guarantees, distributed variants, and practical scalability for greedy coreset subsampling across numerous domains, including matrix approximation, clustering, determinant maximization, Bayesian inference, and training data pruning.

## 1. Greedy Coreset Subsampling: Core Principles and Problem Formulations

Greedy coreset subsampling arises in diverse data summarization tasks, all unified by the need to select a small weighted subset that suffices for approximating an algorithmic objective over the full dataset. In the prototypical Column Subset Selection (CSS) problem, given an $m\times n$ data matrix $A=[A_1\;\dots\;A_n]$ and a target rank $k<n$, the task is to choose $k$ columns $S$ maximizing
\[
f_A(S) = \lVert \Pi_{A[S]} A\rVert_F^2 = \sum_{i=1}^n \lVert \mathrm{proj}(A_i \mid \mathrm{span}\{A_j:j\in S\})\rVert_2^2.
\]
The selected set $S$ acts as a coreset enabling downstream linear tasks with controlled error. A $(1-\varepsilon)$-approximate coreset of size $r$ satisfies $f_A(S) \geq (1-\varepsilon) \max_{|T|=k} f_A(T)$, with $r \approx k/(\sigma_{\min}(OPT)\varepsilon)$, where $\sigma_{\min}(OPT)$ is the smallest squared singular value of the optimal $m\times k$ submatrix [1605.08795].

Analogous greedy formulations are central in:
- Determinant maximization (selecting $k$ vectors maximizing $\det(X_S^TX_S)$) [2309.15286],
- $k$-center clustering with outliers (selecting centers minimizing max-radius for inlier covering) [1901.08219, 2301.02814],
- Mean estimation on graphs or sensor networks [1906.01021],
- Data pruning for deep learning or instruction tuning [2407.15235, 2509.21748].

In each case, the coreset is designed so any algorithm solving the full problem instance can be applied to the subsample with theoretical fidelity guarantees.

## 2. Greedy Algorithms: Single-Machine, Distributed, Randomized, and Submodular Extensions

The canonical greedy routine iteratively selects the candidate that yields maximum marginal gain in the target objective. In the CSS problem, starting with $S=\emptyset$, for $r$ steps,
\[
v = \arg\max_{b\in B\setminus S} f_A(S\cup\{b\}),
\]
and $v$ is added to $S$ [1605.08795]. The determinant maximization problem uses a similar loop, maximizing the volume of the current set [2309.15286]. For $k$-center clustering with $z$ outliers, a farthest-point greedy augmented to ignore extreme outliers is employed [1901.08219, 2301.02814].

Recent research advances include:
- **Randomized or "lazier-than-lazy" variants,** sampling candidate subsets per iteration to reduce computational cost at slight accuracy tradeoff (expectation bounds still hold) [1605.08795].
- **Composable/distributed algorithms,** partitioning data across $\ell$ workers, running local greedy coresets, and then merging via further rounds of greedy selection, with provable approximation factors depending on condition numbers of the objective matrix or local optimality gaps [1605.08795, 2309.15286, 2301.02814].
- **Submodular maximization context,** in which the greedy method achieves a $(1-1/e)$-approximation for monotone submodular objectives under cardinality constraints. Many practical objectives (e.g., facility location, coverage) are amenable to this structure, as are unifications of coverage/density [2509.21748].
- **Weakly submodular/approximate-submodular cases,** in complex problems such as semi-supervised learning, the greedy selection function is shown to be approximately submodular, retaining meaningful approximation bounds [2106.07760, 2407.15235].

## 3. Theoretical Guarantees and Approximation Bounds

State-of-the-art greedy coreset subsampling provides:
- **CSS (Frobenius mass):** Theorem 3.1 of [1605.08795] states that greedy selection with $r = 16k/(\sigma_{\min}(OPT_k)\varepsilon)$ steps yields $f_A(S_r) \geq (1-\varepsilon)f_A(OPT_k)$; this bound is tight up to a constant.
- **Determinant maximization:** Greedy yields an $O(k)^{3k}$-composable coreset; any set that is $(1+\sqrt{k})$-locally optimal under swaps produces such a coreset. The key local-optimality lemma shows that the volume increase from a single swap is at most $1+\sqrt{k}$ [2309.15286].
- **$k$-center clustering:** In doubling metrics (dimension $\rho$), greedy with $O((2/\mu)^\rho k)$ centers yields a $\mu$-coreset [1901.08219, 2301.02814]. For general metrics, the method gives a 2-approximation with $O(k/ \varepsilon + z)$ size.
- **Bayesian inference:** Greedy geodesic ascent (GIGA) achieves geometric convergence of the approximation error: at coreset size $M$, the residual $\|\mathcal L(w) - \mathcal L\|$ decays as $O(\nu^M)$ for some $\nu<1$ [1802.01737]; information-geometric and Riemannian variants further optimize the KL divergence to the exact posterior [1906.03329].
- **Semi-supervised learning and other weakly submodular settings:** Approximate submodularity parameter $\alpha$ controls the approximation: the stochastic greedy approach achieves $(1-e^{-(1-\alpha)}-\varepsilon)$ approximation [2106.07760].

Greedy methods are often (up to logarithmic or condition number factors) near-optimal among polynomial-time algorithms for these classes of objectives.

## 4. Practical Implementations, Scaling, and Empirical Results

Efficient greedy coreset construction employs both algorithmic and numerical optimization:
- **Marginal gain computation:** Classical greedy runs $O(rn)$ marginal gain evaluations, but random projections, projection-cost presketching, and lazy evaluations reduce costs to $O(n\log(1/\delta) / (\sigma_{\min}(OPT)\varepsilon))$ [1605.08795].
- **Maintenance of orthogonal bases:** In determinant maximization or CSS, maintaining a QR factorization enables $O(knd + kd^2)$ time implementations [2309.15286].
- **Distributed and composable strategies:** Each partition builds a local coreset; further greedy aggregation is performed centrally, controlling overall communication and memory footprints [1605.08795, 2301.02814].
- **Selection in non-vectorial domains:** In graphs, greedy selection operates on spectral/diffusion embeddings or random-walk projections, and adds cost constraints as in sensor placement [1906.01021, 2405.17404].
- **Gradient-based objectives:** For instruction tuning or SSL, gradients or model update directions parameterize data utility; greedy routines operate in the projected gradient space, sometimes after clustering [2407.15235].

Empirically, greedy coreset methods have been shown to:
- Match or closely approximate full-data performance in regression, SVM training, clustering, and GNN training using only $5–30\%$ of the data [1605.08795, 2405.17404, 2407.15235].
- Yield 3–10$\times$ reductions in overall wall-time for large-scale learning tasks [1605.08795, 2106.07760].
- Remain robust under the presence of outliers, class imbalance, and low-homophily in graph data [2301.02814, 2405.17404].

## 5. Variations: Submodular, Information-Geometric, and Weakly Submodular Greedy Coreset Methods

Beyond classic greedy maximization, several structural variations exist:
- **Submodular greedy:** Facility location or coverage objectives for deep learning pruning use submodular greedy algorithms, with extensions for integrating density or representativeness (e.g., SubZeroCore) [2509.21748].
- **Geometric and Riemannian greedy:** GIGA and its Riemannian extensions operate by greedy alignment on the unit sphere or under the Fisher information metric, crucial for Bayesian coresets and mean estimation on graphs [1802.01737, 1906.03329, 1906.01021].
- **Weakly submodular maximization:** For objectives not strictly submodular but satisfying approximate submodularity, e.g., in RETRIEVE for SSL, the greedy algorithm retains controlled approximation factors [2106.07760].
- **Clustering in feature or gradient space before greedy selection:** Improves balance and coverage in heterogenous data (e.g., TAGCOS) [2407.15235].

These generalizations maintain the greedy character while addressing objective smoothness, curvature, or domain-specific structural constraints.

## 6. Limitations, Extensions, and Open Challenges

While greedy coreset subsampling is well-understood in several cases, limitations and future directions include:
- **Dependence on objective curvature/condition number:** In CSS, bounds scale inversely with smallest singular value of the optimal submatrix, and in determinant maximization, the gap can be $k^{O(k)}$ [1605.08795, 2309.15286].
- **Computational cost in high-dimensional settings:** All-pairs similarity computations or large-scale QR updates may dominate unless structure is exploited (e.g., random projections, sketching) [1605.08795, 2509.21748].
- **Model-based vs. model-free selection:** Some methods assume access to model gradients or feature spaces (e.g., TAGCOS, RETRIEVE), while others are fully data-driven or training-free (e.g., SubZeroCore).
- **Non-monotone or highly non-modular objectives:** Theory for greedy maximization in non-monotone or more general supermodular regimes remains open.

Empirical evidence and ablation studies suggest that, with appropriate design, greedy coresets are both scalable and robust to a range of real-world complexities.

## 7. Applications Across Domains

Contemporary greedy coreset subsampling is deployed in:
- **Feature selection and dimensionality reduction** via CSS and volume maximization [1605.08795, 2309.15286].
- **Clustering and anomaly detection,** including distributed and outlier-robust $k$-center clustering [1901.08219, 2301.02814].
- **Graph-based learning** and mean estimation in sensor networks [1906.01021, 2405.17404].
- **Deep learning dataset pruning,** model-in-the-loop data selection, and instruction tuning [2407.15235, 2106.07760, 2509.21748].
- **Bayesian inference acceleration** via summarizing log-likelihoods or sufficient statistics [1802.01737, 1906.03329].

The greedy coreset paradigm provides a modular, theoretically-underpinned toolkit for principled data reduction in large-scale data analysis, statistical learning, and distributed computing.

Source: https://www.emergentmind.com/topics/greedy-coreset-subsampling