OrderDP: A Theoretically Guaranteed Lossless Dynamic Data Pruning Framework

Abstract: Data pruning (DP), as an oft-stated strategy to alleviate heavy training burdens, reduces the volume of training samples according to a well-defined pruning method while striving for near-lossless performance. However, existing approaches, which commonly select highly informative samples, can lead to biased gradient estimation compared to full-dataset training. Furthermore, the analysis of this bias and its impact on final performance remains ambiguous. To address these challenges, we propose OrderDP, a plug-and-play framework that aims to obtain stable, unbiased, and near-lossless training acceleration with theoretical guarantees. Specifically, OrderDP first randomly selects a subset and then chooses the top-$q$ samples, where unbiasedness is established with respect to a surrogate loss. This ensures that OrderDP conducts unbiased training in terms of the surrogate objective. We further establish convergence and generalization analyses, elucidating how OrderDP affects optimal performance and enables well-controlled acceleration while ensuring guaranteed final performance. Empirically, we evaluate OrderDP against comprehensive baselines on CIFAR-10, CIFAR-100, and ImageNet-1K, demonstrating competitive accuracy, stable convergence, and exact control -- all with a simpler design and faster runtime, while reducing training cost by over 40%. Delivering both strong performance and computational efficiency, our method serves as a robust and easily adaptable tool for data-efficient learning. The code is publicly available at https://github.com/shengze-xu/OrderDP.

• The paper introduces OrderDP, an SGD-compatible dynamic data pruning framework that reduces training cost by up to 42% while ensuring near-lossless performance.
• The paper leverages a novel surrogate loss optimization technique to provide unbiased gradient estimates and precise control over pruning ratios.
• Empirical evaluations on CIFAR-10, CIFAR-100, and ImageNet-1K demonstrate OrderDP’s robustness across various optimizers and architectures.

The OrderDP Framework: Lossless Dynamic Data Pruning with Theoretical Guarantees

Motivation and Problem Definition

The exponential increase in dataset and model sizes has escalated the computational and financial costs associated with deep neural network training. Data pruning (DP) seeks to address this by reducing the effective training set size while ideally maintaining model performance—termed "near-lossless" pruning. Existing DP techniques are typically static (pre-training sample selection) or dynamic (in-training adaptive selection). While dynamic approaches such as InfoBatch attempt unbiased sample selection through gradient rescaling, lingering issues include unstable optimization dynamics, persistent gradient bias, and practical difficulties in achieving exact control over pruning ratios.

OrderDP: Methodology

OrderDP introduces an SGD-compatible dynamic pruning method that achieves unbiased gradient estimation with respect to a rigorously defined surrogate loss, $\mathcal{L}_q$, rather than the empirical risk directly. At each iteration, OrderDP uniformly samples a candidate batch and selects the top-$q$ samples (by instantaneous loss) for gradient computation and parameter updates. Only retained samples have their loss scores refreshed. This approach combines score-based ranking with unbiased subgradient estimation and allows explicit control of the pruning ratio via two parameters: exploration size $s$ (candidate pool size) and exploitation size $q$ (retained count).

The core algorithmic steps include:

• Randomly sample a candidate subset $S$ from the entire dataset.
• Rank $S$ by per-sample loss, selecting the top-$q$ for SGD updates.
• Refresh scores for only the retained samples, minimizing redundancy in computation.
• The exact retained data fraction is determined by the product $(q/s) \cdot (s/|D|)$.

Unlike prior dynamic DP methods, OrderDP does not require large-scale annealing or rescaling, and can tune the pruning ratio precisely.

Theoretical Analysis

OrderDP establishes rigorous guarantees on both optimization and generalization:

• Unbiased Surrogate Loss Optimization: While the gradient estimate remains biased w.r.t. the empirical risk, it is unbiased for the surrogate loss $\mathcal{L}_q$, where pruning-induced bias is explicitly modeled through closed-form weights $y_j$ assigned to the $q$0-th ranked loss in the batch. These $q$1 weights form a non-uniform, normalized distribution derived from order statistics, allowing the pruning mechanism to be exactly analyzed.
• Convergence Rates: For convex, Lipschitz-continuous per-sample losses, OrderDP achieves the same $q$2 convergence rate as standard mini-batch SGD; this holds despite discarding a large fraction of training data at each iteration.
• Generalization Bounds: The generalization error is controlled by a bounded bias term (determined by the deviation of $q$3 from uniformity) and a vanishing optimization term. As the pruning becomes less aggressive ($q$4), the bias term contracts, reducing to zero in the standard SGD regime.

The analysis transparently shows how pruning aggressiveness trades off against distributional bias, providing explicit formulas for selecting $q$5 and $q$6 to reach a desired accuracy-efficiency compromise.

Empirical Evaluation

Comprehensive experiments on CIFAR-10, CIFAR-100, and ImageNet-1K demonstrate that OrderDP:

• Matches full-dataset accuracy up to normal stochastic fluctuations (within 0.1%) at moderate (30–40%) pruning ratios.
• Maintains the smallest accuracy degradation compared to all static and dynamic baselines as pruning aggressiveness increases.
• Achieves over 40% reduction in training cost and runtime, with consistently stable convergence.
• Demonstrates optimizer and architecture robustness, with comparable gains for SGD, AdamW, LARS, and LAMB optimizers, and for both CNNs and Vision Transformers.
• Maintains full dataset “coverage” across training: all samples are eventually used, avoiding long-term exclusion of any datapoint.

Strong numerical results include achieving lossless pruning up to 42% on CIFAR-10/100 and ImageNet-1K, outperforming InfoBatch and other state-of-the-art DP algorithms at all tested pruning ratios.

Practical and Theoretical Implications

OrderDP’s contributions have several implications for both practice and theory:

• The ability to rigorously control the pruning ratio and bias means practitioners can "budget" compute precisely without empirical guesswork.
• Plug-and-play integration with standard training pipelines, absence of large-scale sorting/annealing overheads, and effectiveness across architectures and optimizers suggest immediate applicability in resource-constrained or large-scale training.
• The explicit theory connecting selection bias, surrogate loss, and generalization allows direct extension to adaptive schedules, robust variants (e.g., under label noise), or more nuanced selection rules (e.g., mixing hard/easy samples or min-$q$7 variants).

Limitations and Future Work

OrderDP is currently evaluated on moderate-size benchmarks in supervised vision tasks. Further investigation is warranted along the following axes:

• Extension to very large-scale models, streaming/continual learning, and cross-modality (e.g., NLP, multi-modal).
• Adaptive or learnable pruning ratio schedules.
• Integration of noise-robust or outlier-robust variants, particularly for datasets with label corruption or heavy-tailed loss distributions.
• Studying implications under distribution shift or for self-supervised objectives.

Conclusion

OrderDP is a dynamic data pruning framework that achieves lossless (or near-lossless) training data reduction with precise, plug-and-play efficiency gains. Its core strength lies in providing explicit theoretical control: unbiased surrogate loss optimization, provable convergence and generalization, and exact prune ratio specification. Empirically, it matches or outperforms all contemporary baselines in both accuracy and computational efficiency, while avoiding the instability and practical limitations of prior dynamic data pruning approaches. The generality of the underlying theory and design enables straightforward adaptation to a range of domains and future algorithmic advancements.

(2606.08574)