Learning from Noisy Labels with Deep Neural Networks: A Survey (2007.08199v7)

Abstract: Deep learning has achieved remarkable success in numerous domains with help from large amounts of big data. However, the quality of data labels is a concern because of the lack of high-quality labels in many real-world scenarios. As noisy labels severely degrade the generalization performance of deep neural networks, learning from noisy labels (robust training) is becoming an important task in modern deep learning applications. In this survey, we first describe the problem of learning with label noise from a supervised learning perspective. Next, we provide a comprehensive review of 62 state-of-the-art robust training methods, all of which are categorized into five groups according to their methodological difference, followed by a systematic comparison of six properties used to evaluate their superiority. Subsequently, we perform an in-depth analysis of noise rate estimation and summarize the typically used evaluation methodology, including public noisy datasets and evaluation metrics. Finally, we present several promising research directions that can serve as a guideline for future studies. All the contents will be available at https://github.com/songhwanjun/Awesome-Noisy-Labels.

• The paper presents a comprehensive survey of 62 robust methods addressing the challenge of learning from noisy labels in deep neural networks.
• It categorizes techniques into five groups—robust architecture, regularization, loss functions, loss adjustments, and sample selection—with detailed comparative analyses.
• The survey elucidates practical implications and future directions, including strategies for instance-dependent noise, multi-label challenges, and model fairness.

Learning from Noisy Labels with Deep Neural Networks: A Survey

The paper "Learning from Noisy Labels with Deep Neural Networks: A Survey," authored by Hwanjun Song, Minseok Kim, Dongmin Park, Yooju Shin, and Jae-Gil Lee, presents a comprehensive review of state-of-the-art methodologies addressing the challenge of learning from noisy labels in deep neural networks (DNNs). This area of research has proven critical for modern machine learning applications, given the prevalence of mislabeled data in real-world datasets.

Abstract and Motivation

The abstract highlights the issue of noisy labels which can significantly degrade the generalization performance of DNNs. As such, the focus is on robust training methods that enable learning under conditions of label noise. The survey covers 62 robust training methods, categorizing them into five distinct groups based on their methodological differences.

Methodological Categorization

The survey categorizes robust training methods into the following five primary groups:

1. Robust Architecture: Methods under this category involve architectural modifications, such as adding noise adaptation layers to the DNNs.
   • Key Methods: Noise Model, Probabilistic Noise Model, and contrastive-additive noise networks.
2. Robust Regularization: Regularization techniques to explicitly or implicitly counteract the overfitting to noisy labels.
   • Key Methods: Bilevel Learning, Annotator Confusion Estimation, and Adversarial Training.
3. Robust Loss Function: Designing loss functions that are inherently noise-robust.
   • Key Methods: Robust Mean Absolute Error (MAE), Generalized Cross Entropy (GCE), and Symmetric Cross Entropy (SCE).
4. Loss Adjustment: Techniques to adjust the loss during training by estimating confidence in labels.
   • Categories: Loss Correction, Loss Reweighting, and Label Refurbishment.
   • Key Methods: Backward Loss Correction, Importance Reweighting, and Dynamic Bootstrapping.
5. Sample Selection: Identifying and selecting true-labeled examples for training.
   • Key Methods: Co-teaching, Iterative Trimmed Loss Minimization (ITLM), and DivideMix.

In-depth Analyses

The survey delves deeply into specific aspects of noise-robust training methods. For instance, it provides a thorough discussion on:

• Noise rate estimation techniques, crucial for effectively applying certain robust training methodologies.
• Detailed experimental design considerations when validating these methods on benchmark datasets.
• Evaluation metrics for robust DNN training accuracy and label precision metrics.

Numerical Results and Comparative Analysis

The paper systematically compares these methods regarding six properties:

1. Flexibility in supporting various DNN architectures.
2. Elimination of the need for pre-training models.
3. Ability to explore the entire dataset without discarding noisy samples.
4. Independence from supervision (auxiliary data or parameters).
5. Effectiveness under high noise rates.
6. Handling of complex (instance-dependent) noise types.

Implications and Future Directions

The practical implications of this survey are substantial. Techniques such as robust loss functions and architectures are pivotal for domains where data labeling is prone to significant noise. For example, medical data labeling often suffers from annotation inconsistencies, and robust methods ensure model reliability.

Theoretical advancements from this paper pave the way for fruitful future directions, including:

• Instance-dependent Label Noise: Shifting focus towards realistic scenarios where the noise depends on features.
• Multi-label Learning: Addressing the unique challenges posed by noisy multi-label datasets.
• Class Imbalance and Noise: Developing methods to simultaneously handle imbalanced classes and label noise.
• Fair and Robust Training: Ensuring fairness in model predictions alongside robustness, especially in scenarios with biased noisy data.
• Efficient Learning Pipelines: Optimizing the computational efficiency of robust training methods to handle large-scale data.

Conclusion

This survey serves as a foundational reference for researchers aiming to understand and develop methods for managing noisy labels in DNNs. By methodically categorizing and comparing robust training methods, the paper offers both a comprehensive overview of the current state and a roadmap for future research in this vital area of machine learning.