Beyond Loss Values: Robust Dynamic Pruning via Loss Trajectory Alignment

Abstract: Existing dynamic data pruning methods often fail under noisy-label settings, as they typically rely on per-sample loss as the ranking criterion. This could mistakenly lead to preserving noisy samples due to their high loss values, resulting in significant performance drop. To address this, we propose AlignPrune, a noise-robust module designed to enhance the reliability of dynamic pruning under label noise. Specifically, AlignPrune introduces the Dynamic Alignment Score (DAS), which is a loss-trajectory-based criterion that enables more accurate identification of noisy samples, thereby improving pruning effectiveness. As a simple yet effective plug-and-play module, AlignPrune can be seamlessly integrated into state-of-the-art dynamic pruning frameworks, consistently outperforming them without modifying either the model architecture or the training pipeline. Extensive experiments on five widely-used benchmarks across various noise types and pruning ratios demonstrate the effectiveness of AlignPrune, boosting accuracy by up to 6.3\% over state-of-the-art baselines. Our results offer a generalizable solution for pruning under noisy data, encouraging further exploration of learning in real-world scenarios. Code is available at: https://github.com/leonqin430/AlignPrune.

• The paper introduces AlignPrune, leveraging a dynamic alignment score to differentiate clean from noisy samples during training.
• It employs temporal analysis of loss trajectories to robustly prune data without altering the underlying model or training schedule.
• It demonstrates significant accuracy improvements across benchmarks while maintaining efficiency and integration with existing methods.

Robust Dynamic Data Pruning under Label Noise via Loss Trajectory Alignment

Introduction and Motivation

Dynamic data pruning has emerged as a key paradigm for enhancing the efficiency of deep learning model training by adaptively excluding samples with low utility. However, the presence of noisy labels—a ubiquitous property of web-scale and weakly-supervised datasets—undermines traditional dynamic pruning algorithms. These methods, often reliant on per-sample loss values for sample selection, misidentify noisy instances as "important" due to their consistently large losses, thus retaining them and degrading downstream performance.

This work addresses the challenge by introducing AlignPrune, a plug-and-play module that replaces loss-based sample rankings with a robust, trajectory-level alignment metric—the Dynamic Alignment Score (DAS). DAS leverages the temporal consistency of loss decay for clean samples, contrasting it against erratic dynamics from noisy samples. AlignPrune is demonstrated as architecture- and training-agnostic, integrating seamlessly into state-of-the-art pruning algorithms while achieving significant robustness gains under diverse noise settings.

Figure 1: (a) Motivation: Dynamic pruning shows greater robustness to label noise than static pruning. (b) Issue: Loss-based pruning retains noisy, high-loss samples. (c) Key insight: Clean vs. noisy samples exhibit different loss trajectory behaviors.

Methodology

The core methodological innovation of AlignPrune is DAS, which quantifies the alignment between a sample's loss trajectory and a reference clean trajectory via temporal correlation—defaulting to the Pearson coefficient for computational efficiency and invariance to scale. Clean samples tend to display monotonically decreasing loss over time, whereas noisy samples are typified by non-monotonic, oscillatory loss curves.

Algorithmic details:

• For each epoch $t$ and sample $i$, the windowed loss trajectory $\mathbf{v}_i^{(t)}$ is constructed.
• A small, held-out, clean reference set's mean loss trajectory $\mathbf{v}_{ref}^{(t)}$ is maintained.
• DAS for each sample is computed as $\rho(\mathbf{v}_i^{(t)}, \mathbf{v}_{ref}^{(t)})$.
• Pruning is applied by discarding samples with the lowest DAS, thus favoring alignment with clean learning dynamics.

  Figure 2: AlignPrune framework: existing dynamic pruning methods use loss values for ranking, while AlignPrune uses DAS to exclude noisy samples.

Notably, AlignPrune does not alter the underlying gradient computation, model, or training schedule. This level of modularity ensures seamless integration with methods such as InfoBatch [qin2024infobatch] and SeTa [zhou2025scale], yielding unbiased training dynamics while enhancing noise robustness.

Experimental Results

Extensive evaluations are conducted across five benchmarks (CIFAR-100N, CIFAR-10N, WebVision, Clothing-1M, and ImageNet-1K) and multiple types and levels of label noise. Key numerical results include:

• CIFAR-100N: At a 30% pruning ratio, InfoBatch+AlignPrune boosts accuracy by up to +6.3% over baselines under severe noise, achieving +2.7% on average.
• CIFAR-10N: SeTa+AlignPrune yields persistent gains in both real and synthetic noise configurations.
• WebVision & Clothing-1M: Substantial improvements in both accuracy and training efficiency, maintaining superiority even with minimal reference data.
• ImageNet-1K: Robust gains are observed across both ConvNeXt and Transformer backbones under synthetic noisy-label conditions.

  Figure 3: Performance comparison versus baselines on (a) WebVision and (b) Clothing-1M; AlignPrune consistently outperforms dynamic pruning methods without sacrificing efficiency.

Additionally, experiments demonstrate that AlignPrune preserves performance for clean-label scenarios, indicating that the method does not over-prune hard legitimate samples. Ablation studies confirm that performance is stable across a wide range of hyperparameters, including the DAS trajectory window and pruning ratios.

Figure 4: Ablation on pruning probability $r$ shows performance under varying degrees of aggressiveness in sample exclusion.

Figure 5: Even with only 0.1% of the reference set used as clean anchor, AlignPrune maintains effectiveness.

Practical and Theoretical Implications

The AlignPrune module introduces a robust temporal perspective to data pruning, moving the field past snapshot-based heuristics that struggle under distribution shift or noise. Conceptually, this approach provides a general template for trajectory-level sample selection, leveraging the statistical regularity of clean sample learning dynamics.

Practically, AlignPrune is especially consequential for large-scale, weakly-labeled scenarios, such as those underpinning modern vision and language foundation models. Since no architectural or loop modifications are required, alignments can be deployed with minimal engineering cost for both new and legacy systems. The method’s low sensitivity to reference set size and composition further reduces the cost of securing strong clean sets, enabling the use of pseudo-clean automatically curated anchors.

Theoretically, aligning loss trajectories with robust reference signals aligns with assumptions of consistency in memorization dynamics, potentially informing future advances in curriculum learning, noise-robust optimization, or adaptive coreset design.

Future Directions

• Extended meta-learning: Trajectory-level information could inform sophisticated meta-learners or controllers that adaptively re-weight or re-sample data to further improve generalization under labeling or distributional uncertainty.
• Multimodal and self-supervised extensions: Similar alignment techniques may be adapted for sequential or multimodal representations, self-supervised signals, or other proxy statistics.
• Resource-efficient annotation: Model-driven selection of small clean anchors, possibly via active learning, could couple with trajectory alignment for ultra-low-resource robust training.

Conclusion

The AlignPrune module, based on loss trajectory alignment, provides a robust and generalizable solution for dynamic data pruning under noisy labels. Experiments demonstrate significant improvements in accuracy and efficiency over state-of-the-art baselines across a range of pruning ratios, architectures, and noise settings. The modularity, noise robustness, and low reliance on explicit clean labels render AlignPrune especially attractive for scaling robust deep learning in real-world, large-scale applications (2604.07306).