Computational Budget-Aware Data Selection

• CADS is a framework that selects training data under explicit computational, monetary, and annotation constraints to optimize model performance.
• It formulates the selection task as a bilevel or multi-stage optimization problem, incorporating budget-aware utility functions and techniques such as policy gradients and submodular maximization.
• Empirical results show that CADS can reduce data labeling and computational costs while maintaining or even enhancing predictive accuracy across diverse application domains.

Computational Budget-Aware Data Selection (CADS) refers to the principled selection of training examples or variables under explicit computational, monetary, or annotation cost constraints, so as to maximize model performance while respecting resource budgets. Unlike conventional data selection, which assumes unconstrained resources, CADS tightly couples the selection process with quantifiable budgetary limits, ensuring that the cost of data acquisition, labeling, or processing is treated as a first-class objective together with predictive efficacy. Recent research demonstrates that incorporating computational budget constraints fundamentally alters both optimal data selection strategies and the achievable trade-offs between efficiency and learning quality.

1. Foundational Concepts and Motivation

The CADS paradigm is motivated by scenarios where training costs—be it monetary cost of human labeling, computational expense of model training, time delays for acquiring or annotating data, or memory/storage constraints—place hard limits on the amount and type of data that can be used in machine learning workflows. Classical approaches either ignore such constraints or address them indirectly through heuristics, often leading to suboptimal, inefficient, or unscalable data selection (Wan et al., 19 Oct 2025, Ntoulas et al., 2013, Yin et al., 21 Oct 2024).

A central tenet of CADS is that data selection policy and resource budget must be solved jointly, not as independent considerations. Empirical studies have shown that data selection strategies which ignore the budget can perform inconsistently, with no method universally dominating across varying constraint regimes (Wan et al., 19 Oct 2025, Yin et al., 21 Oct 2024).

2. Key Methodological Formulations

The current literature frames CADS as a bilevel or multi-stage optimization problem:

• Bilevel Optimization: The outer level seeks to minimize validation loss with respect to data selection decisions (usually represented as discrete masks or probabilistic weights), while the inner level performs model training limited by a strict compute or annotation budget (Wan et al., 19 Oct 2025, Yin et al., 21 Oct 2024).
• Budget-Aware Utility Function: The objective is articulated as:

$$\min_{\mathbf{s}} \mathbb{E}_{\mathbf{m} \sim p(\mathbf{m}|\mathbf{s})}\bigl[\mathcal{L}_{\mathrm{val}}(\theta_C(\mathbf{m}))\bigr]$$

with $\theta_C(\mathbf{m})$ denoting model parameters after training on the subset encoded by mask $\mathbf{m}$ with limited budget $C$. The budget constraints apply to number of selected examples, annotator time, or total floating-point operations (FLOPs) (Ntoulas et al., 2013, Yin et al., 21 Oct 2024, Wan et al., 19 Oct 2025).

• Greedy and Submodular Maximization: In data stream and distributed regimes, selection is executed via an online thresholding policy grounded in submodular maximization, offering constant-factor approximation to the offline maximum under budget (Werner et al., 2022).
• Penalty-Based Relaxation: To avoid expensive differentiation through non-converged inner loops (due to budget constraints), the inner optimization objective is replaced with a surrogate penalty based on a learned function of subset size, thereby retaining efficiency and effectiveness (Wan et al., 19 Oct 2025).

3. Algorithmic Strategies

CADS encompasses a spectrum of approaches, tailored to application and resource structure:

• Feature Coverage Algorithms: Selects examples to maximize coverage of discrete attribute–value pairs. Greedy set-cover style algorithms iteratively pick instances contributing maximal feature coverage gain, ensuring that with reduced labeling, diverse and informative regions of feature space are represented (Ntoulas et al., 2013).
• Cost-Sensitive Model Schedules: Constructs model schedules via ensemble paths (importance, cost, normalized importance, L₁ regularization) and filters out models not on the Pareto front of accuracy versus cost, enabling rapid lookup of the best model under a fixed budget (Yan et al., 2019).
• Bilevel Optimization with Policy Gradient: Uses a probabilistic reparameterization of the data selection decision distribution (Bernoulli or source-level weights) and Hessian-free policy gradient, thus avoiding heavy second-order computations. The key gradient estimator is:

$$\nabla_{\mathbf{s}} \mathbb{E}_{p(\mathbf{m}|\mathbf{s})}\left[\mathcal{L}_{\mathrm{val}}(\theta_C(\mathbf{m}))\right] \approx \mathbb{E}_{p(\mathbf{m}|\mathbf{s})}\left[\mathcal{L}_{\mathrm{val}}(\theta_C(\mathbf{m}))\, \nabla_{\mathbf{s}} \log p(\mathbf{m}|\mathbf{s})\right]$$

(Wan et al., 19 Oct 2025).

• Hybrid Annotation Selection: Selects not only which examples to label but also the annotation granularity (e.g., weak vs. strong supervision in object detection) according to uncertainty and predicted increment in model performance, under fixed annotation budget constraints (Pardo et al., 2019).
• Scaling Law–Guided Source Selection: Uses pilot samples and optimal transport distances to model and extrapolate the expected gain in validation performance as a function of budget allocation over multiple partial data sources, enabling efficient prediction-based acquisition (Kang et al., 2023).

4. Empirical Performance and Benchmark Validation

CADS frameworks have demonstrated substantial savings in data labeling and computational cost without compromising, and often even improving, upon predictive performance achieved with the full dataset:

Study / Approach	Dataset/Domain	Budget Fraction Used	Performance Degradation vs. Full Data
Greedy Feature Coverage	UCI Spam, Twitter	18.5% – 29.2%	Nearly none (precision, recall, F₁)
BAOD	PASCAL VOC	87.2%	None; +2.0 mAP at 100% budget
Budget-Aware Adapters	Visual Decathlon, CIFAR	10–50% FLOPs	Negligible or stable
Online Submodular/DMGT	ImageNet, MNIST	Fractional selection	+5–20% on class-balanced accuracy
Bilevel CADS (penalty relaxation)	CIFAR-10, MNIST	14–29%	Up to 14.42% gain over baselines

Experiments verify that on benchmarks such as ImageNet, Tiny-ImageNet, and challenging NLP/LLM finetuning tasks, CADS can outperform existing methods under various compute, labeling, or storage constraints (Ntoulas et al., 2013, Werner et al., 2022, Yin et al., 21 Oct 2024, Wan et al., 19 Oct 2025).

5. Trade-Offs, Theoretical Guarantees, and Considerations

• Data Quantity vs. Diversity: Budget constraints favor selection of examples that maximize coverage or model uncertainty, as opposed to simple random or cheapest-first selection. The feature-coverage and active learning principles ensure that the selected subset preserves diversity and informativeness even as size shrinks (Ntoulas et al., 2013, Pardo et al., 2019).
• Approximation Guarantees: Online, distributed, and submodular maximization–based CADS algorithms provide quantifiable constant-factor guarantees (e.g., via Dynamic Marginal Gain Thresholding) for selected subset value relative to best offline budgeted subset (Werner et al., 2022).
• Model Evaluation Coupling: In bilevel optimization CADS, the selection is explicitly tied to model performance after resource-constrained training, not just a utility function on the data.
• Inner Optimization and Differentiation: Efficient policy gradient estimators and the novel penalty-based surrogates reduce computation and memory costs commonly associated with full bilevel or meta-learning optimizations (Wan et al., 19 Oct 2025). Correct handling of inner non-optimality is crucial, as resource-constrained training generally yields non-stationary solutions.

6. Application Domains and Practical Implications

The CADS family of techniques has found application in:

• Industrial-scale machine learning with expensive human annotation (e.g., web spam detection, content moderation) (Ntoulas et al., 2013).
• Cost-sensitive feature selection and variable measurement in scientific, medical, or IoT systems (Yan et al., 2019, Ming et al., 2023).
• Object detection under annotation cost constraints, where mixed strong/weak labeling and active teacher–student approaches yield optimal mAP per dollar (Pardo et al., 2019).
• Multi-domain visual recognition and resource-constrained deployment of large networks (Berriel et al., 2019).
• Online distributed learning from massive or streaming sources, where storage and transfer costs severely limit set size (Werner et al., 2022).
• Budget-aware few-shot and active learning in domains where each label or digitized input is expensive (Yan et al., 2022).
• LLM fine-tuning with trade-offs between the cost of intelligent subset selection (e.g., gradient-based scoring) and eventual training cost, with the best method depending on training-to-selection model scale ratio (Yin et al., 21 Oct 2024, Wang et al., 12 Jun 2025).

In all these settings, CADS provides analytical and empirical methodologies for practitioners to allocate limited resources most effectively.

7. Impact, Open Challenges, and Future Directions

CADS has established that neither budget-agnostic nor purely greedy selection strategies are universally optimal across all constraint regimes (Wan et al., 19 Oct 2025, Yin et al., 21 Oct 2024). Integrating compute costs into data selection policy exposes new opportunities and challenges:

• Algorithmic Adaptivity: Designing CADS algorithms that adaptively choose between quantity–diversity–quality trade-offs as the budget scales. Empirical results show that as budgets change, optimal selection strategies shift—e.g., leveraging weaker or more uncertain examples under tight constraints, but incorporating a broader base at higher budgets (Wan et al., 19 Oct 2025).
• Optimization Scalability: Efficiently solving the CADS bilevel, stochastic, or online optimization, especially for massive or non-i.i.d. datasets.
• Theoretical Limits: Expanding analytical guarantees (multiplicative or additive approximation bounds) beyond specific learning problems. Open research probes exact scaling behaviors for intermediate budget regimes and explores the connection to sample compression, coreset, and active learning theory (Hanneke et al., 20 Apr 2025).
• Multi-modal and Multi-objective Scenarios: Extending CADS ideas to simultaneous optimization over multiple annotation types (e.g., strong/weak labels, multimodal features) and in settings with both compute and monetary constraints (Pardo et al., 2019, Yang et al., 15 Oct 2024).
• Robustness to Annotation Noise and Confounders: Recent advances show that budget-aware selection can simultaneously act as a filter for noisy, redundant, or confounded samples, further amplifying efficiency and performance benefits (Yang et al., 15 Oct 2024, Ji et al., 2 Mar 2025).

CADS thus provides a mathematically formal and empirically validated framework for resource-constrained machine learning, with broad applicability and significant potential for further development in both theory and large-scale practice.