Towards a statistical theory of data selection under weak supervision (2309.14563v2)

Abstract: Given a sample of size $N$, it is often useful to select a subsample of smaller size $n<N$ to be used for statistical estimation or learning. Such a data selection step is useful to reduce the requirements of data labeling and the computational complexity of learning. We assume to be given $N$ unlabeled samples ${{\boldsymbol x}i}{i\le N}$, and to be given access to a `surrogate model' that can predict labels $y_i$ better than random guessing. Our goal is to select a subset of the samples, to be denoted by ${{\boldsymbol x}i}{i\in G}$, of size $|G|=n<N$. We then acquire labels for this set and we use them to train a model via regularized empirical risk minimization. By using a mixture of numerical experiments on real and synthetic data, and mathematical derivations under low- and high- dimensional asymptotics, we show that: $(i)$~Data selection can be very effective, in particular beating training on the full sample in some cases; $(ii)$~Certain popular choices in data selection methods (e.g. unbiased reweighted subsampling, or influence function-based subsampling) can be substantially suboptimal.

Towards a Statistical Theory of Data Selection Under Weak Supervision

The paper discusses the importance of data selection in the context of statistical estimation and machine learning, especially when dealing with large datasets where labeling or computational resources are limited. The authors propose a theoretical framework for understanding and optimizing data selection using a surrogate model, with the aim of retaining a smaller subset of samples to be used for training, referred to as weak supervision. The report covers both low-dimensional and high-dimensional asymptotic settings, providing insights into unbiased and biased data selection methodologies.

Key Insights and Findings

1. Unbiased vs. Biased Data Selection: The paper highlights the limitations of unbiased data selection, where each sample is weighted inversely to its selection probability to ensure an unbiased estimate. The authors show that biased data selection schemes, which involve different weighting strategies, can outperform unbiased ones significantly. Particularly in low-dimensional settings, optimal biased selection can reduce generalization error substantially compared to its unbiased counterpart.
2. Dependence on Subsampling Fraction: Traditional data selection methods typically do not account for the subsampling fraction when computing selection probabilities. This paper demonstrates that optimal selection probabilities should depend substantially on the target subsampling fraction. In both theory and experimental results, ignoring this dependence can lead to suboptimal performance.
3. Role of Surrogate Models: The surrogate model plays a crucial role in determining which samples are worth retaining. However, simply plugging the surrogate model’s predictions into the selection process may not be optimal, as indicated by the constructed scenarios where a less accurate surrogate model can sometimes lead to better data selection.
4. Practical Implications of Data Selection: The research uncovers several scenarios where data selection not only retains performance comparable to full data training but can even surpass it. This is shown through both theoretical derivation and empirical evidence, indicating that careful selection can lead to models with reduced test error compared to those trained on the full dataset.
5. Theoretical and Empirical Validation: The theory stands up to rigorous quantitative analysis of various subsampling strategies, both in low and high-dimensional spaces. Simulations, including logistic regression experiments using synthetic and real-world data, validate the theoretical predictions, showing excellent alignment between theory and practice.

Theoretical Framework and Models

• Low-Dimensional Regime: In this setting, the report clarifies how traditional statistical assumptions apply, focusing on scenarios where the number of dimensions is fixed as the sample size grows. The authors use asymptotic analysis to derive optimal subsampling techniques and explore the impact of these selections on the generalization error.
• High-Dimensional Regime: As data grows in both samples and dimensions, the paper shifts its focus to generalized linear models, leveraging techniques like Gaussian comparisons to detail the behavior of subsampling strategies in this regime. The density of relevant derivatives and the impact of surrogate models are explored in depth.

Implications and Future Directions

The implications of this research extend both theoretically and practically:

• Improved Algorithms: The insights from this paper can guide the development of more efficient learning algorithms that maintain high performance when trained on significantly reduced datasets, offering savings in computational cost and time.
• Active Learning and Experimental Design: The strategies outlined can be applied to more complex settings within active learning and experimental design, where data selection is critical for achieving specific performance goals under resource constraints.
• Broader Applications in AI: The results suggest potential applications in AI fields requiring efficient data utilization and quick model adaptation, including domains like autonomous driving where rapid decision-making with partial information is essential.

Future research may look into refining these methodologies, exploring how they interact with different model architectures and data distributions, and expanding on the integration of imperfect surrogate models through a deeper minimax analysis approach.