Batch Loss Score for Dynamic Data Pruning

Abstract: Dynamic data pruning accelerates deep learning by selectively omitting less informative samples during training. While per-sample loss is a common importance metric, obtaining it can be challenging or infeasible for complex models or loss functions, often requiring significant implementation effort. This work proposes the Batch Loss Score (BLS), a computationally efficient alternative using an Exponential Moving Average (EMA) of readily available batch losses to assign scores to individual samples. We frame the batch loss, from the perspective of a single sample, as a noisy measurement of its scaled individual loss, with noise originating from stochastic batch composition. It is formally shown that the EMA mechanism functions as a first-order low-pass filter, attenuating high-frequency batch composition noise. This yields a score approximating the smoothed and persistent contribution of the individual sample to the loss, providing a theoretical grounding for BLS as a proxy for sample importance. BLS demonstrates remarkable code integration simplicity (\textbf{three-line injection}) and readily adapts existing per-sample loss-based methods (\textbf{one-line proxy}). Its effectiveness is demonstrated by enhancing two such methods to losslessly prune \textbf{20\%-50\%} of samples across \textit{14 datasets}, \textit{11 tasks} and \textit{18 models}, highlighting its utility and broad applicability, especially for complex scenarios where per-sample loss is difficult to access. Code is available at https://github.com/mrazhou/BLS.

• The paper presents Batch Loss Score (BLS), which uses an EMA of the mean batch loss as a proxy for estimating per-sample importance to enable dynamic data pruning.
• The methodology is validated across 14 datasets and multiple tasks, achieving 20–50% data pruning with negligible accuracy loss compared to per-sample loss techniques.
• The EMA-based approach effectively separates low-frequency sample signals from high-frequency noise, offering a simple, plug-and-play solution without complex pipeline modifications.

Batch Loss Score for Dynamic Data Pruning: A Technical Summary

Motivation and Context

Dynamic data pruning accelerates deep neural network (DNN) training by focusing computational resources on informative samples. The dominant strategy in existing approaches is to estimate sample importance via per-sample loss values, $l_i(t)$, under the premise that larger losses signal more difficult or informative examples. However, direct, general access to $l_i(t)$ within modern deep learning pipelines is often infeasible. This difficulty escalates in scenarios involving complex or multi-part losses, e.g., object detection, where the standard practice is to compute only an aggregated batch loss. The "Batch Loss Score for Dynamic Data Pruning" paper addresses this by introducing the Batch Loss Score (BLS), which utilizes the universally accessible mean batch loss signal for importance estimation and seamlessly integrates it into existing pruning frameworks.

The Batch Loss Score (BLS) Mechanism

BLS maintains a per-sample score, $s_i(t)$, for each data point, but updates this score only when the sample appears in a mini-batch. At each such occurrence, the score is updated using an exponential moving average (EMA) of the observed batch loss,

$s_i(t) = \alpha s_i(t-1) + (1-\alpha) L(B_t, t),$

where $L(B_t, t)$ is the mean batch loss and $\alpha$ is a decay parameter. This mechanism is highly efficient, requiring a three-line code change as opposed to the significant engineering overhead required by per-sample loss instrumentation.

Figure 1: Score acquisition complexity: BLS can be implemented with three lines of code, while InfoBatch requires 33+ lines and intrusive pipeline modifications.

Moreover, the overall workflow of BLS presents a stark contrast in simplicity compared to extracting and using per-sample losses, particularly in complex tasks such as detection or multi-task learning.

Figure 2: Workflow comparison: BLS enables a black-box integration, versus the high manual complexity of per-sample loss extraction.

Signal-Processing Interpretation and Theoretical Guarantees

The EMA scheme is not simply a heuristic. The paper establishes that the batch loss seen by a sample during training can be decomposed as the sum of a (scaled) true sample loss and stochastic noise arising from random batch composition. Under the hypothesis that the "sample signal" is persistently low-frequency while batch composition noise is high-frequency, the EMA acts as a first-order low-pass filter, selectively smoothing persistent trends in sample importance while attenuating batch noise.

This frequency separation hypothesis is strongly corroborated via power spectral density (PSD) estimates from empirical training runs, in which the energy of the batch noise is concentrated at higher frequencies compared to the scaled sample loss signal.

Figure 3: The batch composition noise component in the loss exhibits much higher power at high frequencies than the sample signal, confirming the effectiveness of low-pass filtering by EMA.

Hence, BLS-propagated scores yield robust, temporally smoothed proxies for direct per-sample loss trends, suitable for plug-and-play use in existing pruning or curriculum learning algorithms.

Empirical Results Across Modalities and Architectures

The paper provides extensive empirical validation on 14 datasets, 11 task types, and 18 model architectures, integrating BLS with canonical per-sample loss-based pruning schemes (InfoBatch and SeTa). Strong claims are supported by the following findings:

• On large-scale visual captioning (e.g., ToCa, SS1M), scene text recognition (MJ+ST), classification (CIFAR10/100, ImageNet), object detection, semantic segmentation, multi-view stereo, vision-language, and generative modeling (VAE, DDPM), BLS-based systems prune 20–50% of training samples with negligible performance degradation.
• In many cases, BLS-based pruning achieves indistinguishable accuracy/metric scores compared to per-sample loss-informed pruning, substantiating the substitution of direct loss access.
• BLS maintains architectural agnosticism and its performance generalizes robustly to CNNs, Transformers (ViT, Swin), state space models (Vim), as well as joint vision-language backbones, with no architecture-specific tuning required.
• Empirical ablation studies demonstrate that removing the EMA step (i.e., using instantaneous batch loss) consistently degrades the accuracy and pruning effectiveness of the score; thus, the temporal smoothing afforded by the EMA is essential.

Hyperparameter Sensitivity and Ablation

A sensitivity study on the EMA decay parameter $\alpha$ manifests a predictable trade-off: low $\alpha$ gives highly responsive but noisy scores (lower accuracy, higher pruning), while high $\alpha$ produces overly sluggish, conservative scores (lower pruning, baseline-level accuracy). The optimal trade-off occurs at intermediate values (generally $\alpha$ in $l_i(t)$0 across tasks).

Figure 4: Pruning ratio and accuracy exhibit optimal trade-offs at intermediate $l_i(t)$1 values; both under- and over-smoothing harm pruning effectiveness or accuracy.

Integration and Practical Engineering Implications

BLS's core strength is its seamless integration with existing frameworks. It does not require custom loss definitions, architecture-specific code, or modifications to training control flow: the approach is implemented in three lines and operates as a direct "one-line proxy" for InfoBatch or SeTa, with almost no computational overhead (overhead $l_i(t)$2s per 1M samples). Notably, the methodology directly sidesteps the intractable engineering challenge of extracting representative per-sample losses from pipelines such as YOLO, where losses are composites of multi-stage structured objectives, and "sample hardness" is not easily mapped to a unique scalar per sample.

Implications and Future Directions

Theoretical implications include a robust justification for batch-level, temporally-smoothed statistics as general-purpose sample importance proxies. This opens avenues for data pruning, curriculum learning, and dataset selection in settings where per-sample granularity is fundamentally infeasible. Moreover, the practical utility is increased by the universality of batch loss signals: BLS can be applied to any problem where batch-wise training is used, spanning from standard classification to structured prediction, multimodal inputs, and even semi-supervised learning.

Future directions include exploring adaptive or learnable EMA parameters, extending the framework to unsupervised/data-distillation settings, and investigating non-linear or higher-order filtering schemes for further improved separation of signal and noise in alternative training regimes.

Conclusion

The Batch Loss Score constitutes a robust, scalable mechanism for dynamic sample pruning and selection that operates solely on mean batch loss signals. Its theoretical grounding, empirical validation, plug-and-play integration, and negligible cost collectively make it a compelling alternative to per-sample loss-based pipelines—particularly in settings where per-sample losses are conceptually ambiguous, computationally prohibitive, or engineering-intractable. The generality of the BLS representation positions it as a foundational tool for broad adoption in efficient large-scale deep neural network training.

Reference: "Batch Loss Score for Dynamic Data Pruning" (2604.04681).