Improving Resistance to Noisy Label Fitting by Reweighting Gradient in SAM (2411.17132v1)

Abstract: Noisy labels pose a substantial challenge in machine learning, often resulting in overfitting and poor generalization. Sharpness-Aware Minimization (SAM), as demonstrated in Foret et al. (2021), improves generalization over traditional Stochastic Gradient Descent (SGD) in classification tasks with noisy labels by implicitly slowing noisy learning. While SAM's ability to generalize in noisy environments has been studied in several simplified settings, its full potential in more realistic training settings remains underexplored. In this work, we analyze SAM's behavior at each iteration, identifying specific components of the gradient vector that contribute significantly to its robustness against noisy labels. Based on these insights, we propose SANER (Sharpness-Aware Noise-Explicit Reweighting), an effective variant that enhances SAM's ability to manage noisy fitting rate. Our experiments on CIFAR-10, CIFAR-100, and Mini-WebVision demonstrate that SANER consistently outperforms SAM, achieving up to an 8% increase on CIFAR-100 with 50% label noise.

• The paper presents SANER, a gradient reweighting approach that curtails noisy label overfitting through component-wise analysis.
• Empirical results demonstrate up to an 8% accuracy increase on CIFAR-100 with 50% label noise, confirming SANER’s robustness.
• SANER seamlessly integrates with various SAM-based optimizers and architectures, improving generalization in diverse noisy environments.

The paper "Improving Resistance to Noisy Label Fitting by Reweighting Gradient in SAM" focuses on enhancing machine learning performance in environments with noisy labels. The key challenge addressed is that noisy labels often lead to overfitting, which hinders generalization capabilities of deep neural networks. The paper introduces a new approach called SANER (Sharpness-Aware Noise-Explicit Reweighting), which is designed to improve upon the existing Sharpness-Aware Minimization (SAM) methodology.

Key Concepts:

• Noisy Labels: These are inaccurate labels in datasets that can mislead the training process, particularly in large-scale datasets with human annotation errors.
• SAM (Sharpness-Aware Minimization): An optimizer that helps improve model generalization by finding flat minima in the loss landscape, making it naturally robust to noisy labels. It does this by modifying the gradient to avoid sharp minima, which are associated with overfitting.
• Gradient Reweighting in SANER: The proposed SANER method involves analyzing SAM's gradient at each iteration to identify components that contribute to robustness against noise. SANER dynamically reweights these gradient components, reducing those that correspond to noisy labels.

Main Contributions:

1. Component-Wise Gradient Analysis: The paper investigates the behavior of SAM gradients by breaking them down into component-wise analyses. It identifies that specific gradient components are pivotal in resisting noisy label fitting. SANER leverages this by re-weighting these components to enhance robustness against noise.
2. Empirical Evaluation: Experiments conducted on datasets such as CIFAR-10, CIFAR-100, and Mini-WebVision demonstrate that SANER consistently outperforms SAM by achieving better accuracy rates, notably with substantial improvements in scenarios with high noise—up to an 8% increase in accuracy on CIFAR-100 with 50% label noise.
3. Control of Noisy Fitting: By focusing on component-wise gradient reweighting, SANER specifically targets and reduces the fitting rate of noisy samples without significantly impacting the learning of clean samples. This is achieved by further diminishing certain SAM gradient components that are related to noise during training iterations.
4. Generalization Enhancement: The paper shows that the optimized learning achieved by SANER, through its noise-reduction strategy, leads to improved test accuracy, effectively managing overfitting in contexts with extensive noise.
5. Extensibility to SAM Variants: SANER is validated as a versatile solution that can be integrated effectively with other SAM-based optimizers, demonstrating performance improvements even when combined with variations like ASAM, GSAM, FSAM, and VaSSO.

Experimental Observations:

• Robustness Across Different Architectures: SANER's effectiveness was evaluated across various architectures beyond ResNet18, including ResNet34, DenseNet121, and WideResNet structures, affirming its broad applicability and robustness.
• Adaptability to Real-World and Synthetic Noises: The performance was consistent not only in controlled synthetic noise scenarios but also in real-world noisy datasets, underscoring SANER's capability to generalize well in diverse environments.
• Scheduler for Enhanced Performance: A linear scheduler is proposed to gradually adjust the reweighting during early training epochs. This adjustment helps stabilize the network's focus on clean noise-free learning before fully deploying the noise reduction strategy.

The paper concludes that SANER successfully mitigates the impact of noisy labels more effectively than SAM by using a strategic gradient reweighting mechanism, thus delivering improved model robustness and accuracy in noisy training environments.