Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labels (1804.06872v3)

Abstract: Deep learning with noisy labels is practically challenging, as the capacity of deep models is so high that they can totally memorize these noisy labels sooner or later during training. Nonetheless, recent studies on the memorization effects of deep neural networks show that they would first memorize training data of clean labels and then those of noisy labels. Therefore in this paper, we propose a new deep learning paradigm called Co-teaching for combating with noisy labels. Namely, we train two deep neural networks simultaneously, and let them teach each other given every mini-batch: firstly, each network feeds forward all data and selects some data of possibly clean labels; secondly, two networks communicate with each other what data in this mini-batch should be used for training; finally, each network back propagates the data selected by its peer network and updates itself. Empirical results on noisy versions of MNIST, CIFAR-10 and CIFAR-100 demonstrate that Co-teaching is much superior to the state-of-the-art methods in the robustness of trained deep models.

• The paper introduces Co-teaching, a dual-network method that selects small-loss samples to mitigate the adverse effects of extremely noisy labels.
• It leverages the phenomenon where deep networks first learn clean patterns before memorizing noise, improving performance on MNIST, CIFAR-10, and CIFAR-100.
• Empirical results demonstrate significant accuracy gains over state-of-the-art methods under both symmetric and pair noise conditions.

Overview of the Paper "Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labels"

The paper "Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labels" introduces a novel deep learning paradigm to address the challenge of training with extremely noisy labels. Deep neural networks inherently possess a high capacity for memorization, which includes the potential memorization of noisy labels, thereby degrading their performance significantly. This research proposes the "Co-teaching" mechanism, which leverages two neural networks to mitigate the detrimental effects of noisy supervision.

The core idea of Co-teaching is the simultaneous training of two networks that iteratively teach each other by selecting samples with possibly clean labels from a mini-batch. Each network filters and selects a subset of data with small loss values from its perspective, which is then used to update the other network. This cross-updating mechanism aims to reduce the risk of overfitting to noisy labels by continuously exchanging information about likely clean instances between the two networks.

Major Contributions

1. Paradigm Shift in Handling Noisy Labels: The Co-teaching method diverges from the traditional reliance on estimating the noise transition matrix by instead focusing on a sample selection strategy based on loss values. This approach alleviates the dependency on accurately modeling noise, which can be particularly challenging in scenarios with a large number of classes or severely noisy data.
2. Empirical Validation: Extensive experiments demonstrate that Co-teaching achieves superior robustness under both extremely noisy and moderately noisy conditions. The method was evaluated on MNIST, CIFAR-10, and CIFAR-100 datasets with imposed noise, showing substantial improvements over state-of-the-art methods like MentorNet, Decoupling, and other traditional techniques.
3. Insightful Utilization of Memorization Effects: The work leverages the observation that deep networks first learn clean patterns before memorizing noisy data. By dynamically adjusting the proportion of data considered for training based on loss values, Co-teaching effectively utilizes this memorization epoch to favor clean label learning, thus improving generalization.

Results

The experimental results were substantial:

• MNIST: Under a symmetric noise rate of 50%, Co-teaching achieved an accuracy of ~91.32%, significantly outperforming baselines where accuracies plummeted below 80%. For pair noise at 45%, the method reached ~87.63%, compared to ~80.88% for the next best approach, MentorNet.
• CIFAR-10: At 50% symmetric noise, Co-teaching reached an accuracy of 74.02%, while other methods, such as F-correction, reached 59.83%. With pair noise at 45%, it achieved ~72.62%.
• CIFAR-100: At 50% symmetry noise, Co-teaching resulted in 41.37% accuracy, while the next best, MentorNet, achieved ~39.00%. With pair noise, Co-teaching showed an accuracy of ~34.81%.

Technical Details

The Co-teaching algorithm follows a structured strategy:

1. Initialization: Two networks, $f$ and $g$, are initialized.
2. Mini-Batch Processing: For each mini-batch, both networks select a subset of instances with small losses.
3. Cross Update: Each network backpropagates using the small-loss instances selected by the other network, thus filtering out noisy labels through mutual teaching.
4. Dynamic Adjustment: Over epochs, the proportion of instances considered (denoted $R(T)$) dynamically decreases, effectively reducing the impact of noisy labels as training progresses.

Implications and Future Work

Practically, Co-teaching offers a robust framework for deploying deep learning models in environments plagued by label noise, such as crowdsourced data labeling or insufficient quality control in data collection practices. By effectively managing noise without necessitating additional auxiliary models or complex noise estimation mechanisms, Co-teaching simplifies the training pipeline and enhances model reliability.

Theoretically, the paper opens several avenues for future research, especially in understanding the bounds and behavior of noise handling in deeply non-convex spaces typical of deep neural networks. Prospective studies might investigate the further theoretical guarantees for Co-teaching and extend its applicability to more complex scenarios, including multi-label settings and positive-unlabeled learning contexts.

In conclusion, this research presents an insightful and practical solution for robust learning in noisy environments, balancing elegance and efficiency in approach. As deep learning models continue to deploy in increasingly diverse and uncontrolled environments, techniques like Co-teaching will be pivotal in sustaining performance and reliability.