---
title: Data Selection via Learning Percentage
url: https://www.emergentmind.com/topics/data-selection-via-learning-percentage
type: topic
---

# Data Selection via Learning Percentage

Data selection via learning percentage refers to the formalized process of selecting a subset of data sources or samples, often under a budget constraint, such that the chosen subset guarantees a prescribed proportion of the achievable learning performance. Originating in Bayesian learning and extending to empirical risk minimization, curriculum learning, subset selection for large-scale pretraining, and active learning, the “learning percentage” is operationalized as a threshold ρ or as a fixed fraction of samples, with the aim of minimizing cost or label budget while achieving (or closely matching) the full-data predictive power. This concept has become central in the design and analysis of data-centric learning systems across theory and practice.

## 1. Formalization of the Data Selection Problem

The canonical problem is to select a subset \( S \subset \Omega \) of sources or data points (with \( |\Omega| = n \)) that incurs minimum total cost while guaranteeing that a monotonic “learning-gain” function \( f(S) \) meets or exceeds a target threshold \( \rho \) (the learning percentage). In mathematical terms [2011.10712]:
\[
\min_{S \subseteq \Omega} \sum_{i \in S} c_i \quad \text{subject to} \quad f(S) \geq \rho
\]
where:
- \( c_i > 0 \) denotes the cost associated with item/source \( i \)
- \( f \) is a nonnegative, normalized (i.e., \( f(\varnothing)=0 \)), monotone, submodular function quantifying the utility or performance
- \( \rho \) specifies the target fraction of full achievable performance

In Bayesian data source selection [2011.10712], \( f(S) = z(S) \) is a normalized error-bound coverage, also interpretable as information gain, mutual information, or variance reduction. In supervised learning, \( f \) may capture accuracy, explained variance, representation coverage, or diversity. Solutions adhere to cardinality (|S| ≤ k, with p = k/n specifying the fraction) or value-oriented constraints (f(S) ≥ ρ).

## 2. Theoretical Properties and Algorithmic Hardness

Data selection via learning percentage is, in general, computationally intractable. For example, in the Bayesian setting the problem is shown to be NP-hard via a reduction from Set Cover—even when all costs are unitary [2011.10712]. The key insight is that requiring the error on a particular hypothesis to reach zero is equivalent to covering all elements in a universe, which directly maps to classical set cover.

Despite this hardness, the objective admits a favorable structure: for a wide class of performance measures, including those derived from mutual information, variance reduction, or information gain, the set function \( f \) is monotonic and submodular. This renders the problem amenable to algorithms with provable approximation guarantees.

## 3. Submodular Set Cover and Greedy Approximations

When \( f \) is monotone submodular, the data selection task becomes a submodular set cover problem. The canonical greedy algorithm iteratively selects the item with the highest normalized marginal gain per unit cost, updating the subset until the learning threshold is met [2011.10712, 1901.01151]. Greedy methods are accompanied by strong guarantees: if \( S^* \) is optimal and \( M = \max_{j} f(\{j\}) \), then the cost of the greedy solution \( h(S_\mathrm{greedy}) \) satisfies
\[
h(S_\mathrm{greedy}) \leq (1 + \ln M) h(S^*)
\]
[2011.10712]. Approximations for subset selection over coverage/dispersion functions (Facility-Location, Disparity-Min) yield (1−1/e) and 1/2 multiplicative factors for the achievable learning value [1901.01151]. Fast threshold-based variants accelerate runtime with minimal degradation in guarantee: a (1/(1−ε)) multiplicative factor, with query complexity scaling as \( O((n/\epsilon)\ln(n/\epsilon)) \) [2011.10712].

## 4. Learning Percentage in Modern Data- and Label-Efficient Training

The learning percentage framework directly informs practical selection in massive data settings. In large-scale LLM pretraining [2512.24265], policy-gradient based mask learning enforces a strict selection fraction (e.g., exactly 10% of 15T tokens in FineWeb) while optimizing a joint quality-diversity objective:
\[
R(\theta) = \mathbb{E}_{M \sim p_\theta}\left[\alpha Q(M) + \beta D(M)\right]
\]
Sampling is constrained to \( \sum_{i=1}^N M_i = S \) (fraction learned is \( S/N \)), and policy gradients ensure parameter updates favor high expected utility subject to the selection percentage. Hard constraints on subset size avoid the need for penalty terms or soft targets. With this approach, a 10% (“learning percentage”) subset suffices to improve downstream LLM performance over quality-only or diversity-only baselines [2512.24265].

In curriculum and active learning, adaptive policies (reinforcement learning [1702.08635], curriculum progression based on learner ability [2011.00080], or batch-level subset selection [1901.01151]) govern the fraction of data presented at each stage. Stagewise or epoch-based methods dynamically expand or contract the percentage of included data, often as a function of an estimated learning curve, reward, or model confidence.

## 5. Performance Bounds and Statistical Guarantees

Learning-theoretic analyses formalize how small a learning percentage suffices for various ERMs. For mean estimation in ℝ, the worst-case relative risk after optimal n-point selection is at most \( 1 + 1/(2n-1) \), and for linear classification in ℝ^d, error is zero as soon as n > d [2504.14572]. In strictly convex stochastic optimization, n > d points suffice to match population loss.

In high-dimensional or weakly supervised regimes, optimal sampling probabilities (π(x)) and the associated learning performance as a function of γ = n/N (the “learning percentage”) are derived. Soft subsampling with tuned exponents (e.g., π(x) ∝ [uncertainty score(x)]^α), with α ≈ 0.5, achieves within 1–2% of full-data error for γ ≈ 0.4–0.6, even with weak surrogate models [2309.14563]. Non-reweighted deterministic selection can outperform reweighted unbiased schemes, especially as γ decreases toward the practical regime.

Typical empirical results reveal that for diverse computer vision and NLP tasks, 10–40% of the original data (selected by submodular, uncertainty, or RL-based policies) achieves 95–100% of full-data performance [1901.01151, 2506.01085, 2512.24265]. In some cases, carefully chosen fractions even improve out-of-sample accuracy over the full set, producing a negative gap.

## 6. Practical Algorithms and Selection Criteria

Algorithmic instantiations span a spectrum:
- Greedy and accelerated greedy set function maximization (Facility-Location, Dispersion, coverage functions) [2011.10712, 1901.01151]
- Policy-gradient mask learning with hard subset budget constraints [2512.24265]
- Soft or thresholded uncertainty-based selection with tunable exponents, sometimes guided by influence function or curvature calculations [2309.14563]
- Reinforcement learning policies that optimize for rapid convergence or data efficiency [1702.08635]
- Dynamic data inclusion based on learner ability relative to difficulty, thresholded at each stage [2011.00080]
Practical implementation includes pilot runs to estimate accuracy vs. percentage curves, tuning of the balance between diversity and representation, and interpolation between random/sample-hardness-driven and submodular objectives [1901.01151, 2506.01085].

## 7. Empirical Insights and Recommendations

- For multi-class and high-redundancy datasets, representation-oriented selection (Facility-Location) is superior; for binary, highly redundant data, diversity (Dispersion) is preferable [1901.01151]
- Validating learning percentage via curves or cross-validation identifies plateau points (knee p*) beyond which additional percentage yields diminishing accuracy returns
- In extremely large-scale or resource-constrained scenarios, pre-filtering (e.g., via quality scores) before joint selection further accelerates subset learning [2512.24265]
- Non-reweighted, thresholded selection strategies are robust, and the optimal fraction is problem-dependent but often low (10–50%) [2309.14563]
- In true weak supervision or surrogate-limited scenarios, even marginally informative surrogates suffice for near-optimal data selection [2309.14563]
- Data selection via learning percentage unifies set cover, sample compression, coresets, and curriculum learning in a data-centric learning theory [2504.14572]

## Table: Methods and Guarantees for Data Selection via Learning Percentage

| Paper / Framework          | Selection Mechanism    | Typical Guarantee / Regime                     |
|---------------------------|-----------------------|------------------------------------------------|
| [2011.10712] (Bayesian SC)   | Submodular greedy/fast-greedy | Cost ≤ (1+ln M)·Opt, runs in O(n²); fast variant: O((n/ε)ln(n/ε))  |
| [2512.24265] (DATAMASK)      | Policy-grad mask, hard budget  | ≈10% of tokens, +3.2%/1.9% vs. baselines (LLMs) |
| [1901.01151] (Submodular subset, active)  | Facility-Location, Dispersion, greedy | 95–100% accuracy at 10–50% subset, (1–1/e) or 1/2 approx. factor   |
| [2309.14563] (Weak supervision) | Uncertainty/influence-thresholded | <2% error gap at γ ≈ 0.4–0.6, optimal π(x) yields minimum excess risk |
| [2011.00080] (Curriculum)     | IRT-based ability threshold    | Dynamic %S grows/shrinks per epoch; fastest convergence            |
| [1702.08635] (RL/NDF)         | RL-learned filtering           | Up to 50% instance reduction, matches full-data accuracy           |

Data selection via learning percentage thus provides a principled, theory-grounded, and empirically validated methodology for cost-efficient, high-performance model training across a diverse range of modern ML settings. The dominant open questions concern further tightening of selection bounds in various regimes, development of practical near-linear-time algorithms, and extensions to non-Euclidean, nonconvex, or pretrained representation scenarios.

Source: https://www.emergentmind.com/topics/data-selection-via-learning-percentage