Active Negative Loss: A Robust Framework for Learning with Noisy Labels (2412.02373v1)

Abstract: Deep supervised learning has achieved remarkable success across a wide range of tasks, yet it remains susceptible to overfitting when confronted with noisy labels. To address this issue, noise-robust loss functions offer an effective solution for enhancing learning in the presence of label noise. In this work, we systematically investigate the limitation of the recently proposed Active Passive Loss (APL), which employs Mean Absolute Error (MAE) as its passive loss function. Despite the robustness brought by MAE, one of its key drawbacks is that it pays equal attention to clean and noisy samples; this feature slows down convergence and potentially makes training difficult, particularly in large-scale datasets. To overcome these challenges, we introduce a novel loss function class, termed Normalized Negative Loss Functions (NNLFs), which serve as passive loss functions within the APL framework. NNLFs effectively address the limitations of MAE by concentrating more on memorized clean samples. By replacing MAE in APL with our proposed NNLFs, we enhance APL and present a new framework called Active Negative Loss (ANL). Moreover, in non-symmetric noise scenarios, we propose an entropy-based regularization technique to mitigate the vulnerability to the label imbalance. Extensive experiments demonstrate that the new loss functions adopted by our ANL framework can achieve better or comparable performance to state-of-the-art methods across various label noise types and in image segmentation tasks. The source code is available at: https://github.com/Virusdoll/Active-Negative-Loss.

• The paper introduces NNLFs that replace MAE in the APL framework, boosting convergence in noisy label settings.
• It improves model robustness by focusing on memorized clean samples to prevent overfitting even under high noise.
• Empirical results on benchmarks like CIFAR-10 and Clothing-1M show ANL's superior performance over GCE and SCE losses.

An Analysis of Active Negative Loss Within Noisy Label Learning

Deep supervised learning has made significant progress across various domains, especially in tasks such as image classification and segmentation. However, a critical vulnerability remains: the susceptibility to noisy labels, which can considerably degrade the performance of over-parameterized Deep Neural Networks (DNNs). Addressing this challenge is imperative, notably because acquiring clean, accurately annotated datasets is often costly and time-consuming. The paper proposes an innovative approach to enhance learning using noisy labels through a novel framework termed Active Negative Loss (ANL).

Framework Overview

The paper identifies limitations inherent in the Active Passive Loss (APL) framework, primarily attributed to the Mean Absolute Error (MAE) used as the passive loss function. The employment of MAE, despite its noise-robustness, results in prolonged convergence times, which impacts the training efficacy on large-scale datasets. ANL addresses this by introducing Normalized Negative Loss Functions (NNLFs) as a replacement for MAE in the APL framework. NNLFs effectively counter the drawbacks of training efficiency via the Active Negative Loss framework, merging NNLFs with normalized active loss functions.

Contribution Highlights

1. Introduction of NNLFs: The paper presents NNLFs, which are derived using complementary label learning and "vertical flipping" of the loss function. These loss functions are normalized to make them robust to label noise.
2. Enhanced Robustness: ANL improves robustness by focusing on memorized clean samples within the training set. It alleviates overfitting even with high levels of label noise, contrasting significantly with MAE-centered APL frameworks, which suffer training inefficiencies.
3. Addressed Class Imbalance: In scenarios featuring non-symmetric label noise, ANL incorporates an entropy-based regularization strategy to tackle label imbalance. This ensures a balanced output from the model and further fortifies model robustness.
4. Superior Empirical Performance: Across several benchmark datasets (CIFAR-10, CIFAR-100, WebVision, Animal-10N, and Clothing-1M) with varying noise scenarios, ANL consistently outperforms existing loss functions such as GCE and SCE, offering enhanced resilience in synthetic and real-world noise conditions.

Implications and Speculations

From a practical perspective, ANL advances the noise-tolerant learning capabilities required for effectively deploying DNNs in noisy-label environments. Theoretically, it prompts a re-evaluation of passive loss functions employed in learning frameworks, showcasing a path forward in improving convergence without sacrificing robustness.

The introduction of NNLFs and the improved ANL framework invite further exploration into even more adaptable and scalable noise-robust loss functions. It will be vital to investigate the generalization potential of ANL-derived methods across various tasks beyond image-based datasets, such as in NLP and other domains where label noise is prevalent.

Conclusion

Overall, the research delineates a stepwise approach for upgrading passive loss functions in the presence of noisy data through the ANL framework. Its effectiveness has been empirically validated, encouraging subsequent research to build upon these foundational concepts and extending the same to other classifications contexts within AI and machine learning. While the present work offers a compelling performance improvement, the quest for achieving optimal noise-tolerant training across broader application domains continues.