Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
153 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Regroup Median Loss for Combating Label Noise (2312.06273v1)

Published 11 Dec 2023 in cs.LG and cs.CV

Abstract: The deep model training procedure requires large-scale datasets of annotated data. Due to the difficulty of annotating a large number of samples, label noise caused by incorrect annotations is inevitable, resulting in low model performance and poor model generalization. To combat label noise, current methods usually select clean samples based on the small-loss criterion and use these samples for training. Due to some noisy samples similar to clean ones, these small-loss criterion-based methods are still affected by label noise. To address this issue, in this work, we propose Regroup Median Loss (RML) to reduce the probability of selecting noisy samples and correct losses of noisy samples. RML randomly selects samples with the same label as the training samples based on a new loss processing method. Then, we combine the stable mean loss and the robust median loss through a proposed regrouping strategy to obtain robust loss estimation for noisy samples. To further improve the model performance against label noise, we propose a new sample selection strategy and build a semi-supervised method based on RML. Compared to state-of-the-art methods, for both the traditionally trained and semi-supervised models, RML achieves a significant improvement on synthetic and complex real-world datasets. The source code of the paper has been released.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (50)
  1. Unsupervised Label Noise Modeling and Loss Correction. In Proceedings of ICML, June 9-15, 2019, Long Beach, CAL, USA, 312–321.
  2. A Closer Look at Memorization in Deep Networks. In Proceedings of ICML - Volume 70, 233–242. JMLR.org.
  3. Understanding and improving early stopping for learning with noisy labels. In Proceedings of NeuIPS, December 6-14, 2021, virtual, 24392–24403.
  4. MixMatch: A Holistic Approach to Semi-Supervised Learning. In Proceedings of NeuIPS, December 8-14, 2019, Vancouver, BC, Canada, 5050–5060.
  5. Catoni, O. 2012. Challenging the empirical mean and empirical variance: a deviation study. In Annales de l’IHP Probabilités et statistiques, 1148–1185.
  6. Learning with Instance-Dependent Label Noise: A Sample Sieve Approach. In Proceedings of ICLR, May 3-7, 2021, Virtual Event, Austria.
  7. Improving Generalization of Deep Neural Network Acoustic Models with Length Perturbation and N-best Based Label Smoothing. arXiv preprint arXiv:2203.15176.
  8. Classification in the presence of label noise: a survey. IEEE transactions on neural networks and learning systems, 25: 845–869.
  9. Girshick, R. 2015. Fast r-cnn. In Proceedings of the CVPR, June 7-12, 2015, Boston, MA, USA, 1440–1448.
  10. Towards Understanding Deep Learning from Noisy Labels with Small-Loss Criterion. In Proceedings of IJCAI, August 19-27, 2021, Virtual Event, Montreal, Canada, 2469–2475.
  11. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In Proceedings of NeuIPS, December 3-8, 2018, Montréal, Canada, 8536–8546.
  12. Mask r-cnn. In Proceedings of the ICCV, October 22-29, 2017, Venice, Italy, 2961–2969.
  13. Learning with Neighbor Consistency for Noisy Labels. In Proceedings of CVPR, June 18-24, 2022, New Orleans, LA, USA, 4662–4671.
  14. Deep Bilevel Learning. In Proceedings of CVPR, September 8-14, 2018, Munich, Germany, 632–648.
  15. UNICON: Combating Label Noise Through Uniform Selection and Contrastive Learning. In Proceedings of the CVPR, June 18-24, 2022, New Orleans, LA, USA, 9676–9686.
  16. Imagenet classification with deep convolutional neural networks. Communications of the ACM, 60(6): 84–90.
  17. Robust machine learning by median-of-means: Theory and practice. The Annals of Statistics, 48(2): 906 – 931.
  18. Dividemix: Learning with noisy labels as semi-supervised learning. In Proceedings of ICLR, April 26-30, 2020, Addis Ababa, Ethiopia, 824–832.
  19. Learning to Learn From Noisy Labeled Data. In Proceedings of CVPR, June 16-20, 2019, Long Beach, CA, USA, 5051–5059.
  20. Selective-Supervised Contrastive Learning with Noisy Labels. In Proceedings of CVPR, June 18-24, 2022, New Orleans, LA, USA, 316–325.
  21. Webvision database: Visual learning and understanding from web data. arXiv preprint arXiv:1708.02862.
  22. Provably end-to-end label-noise learning without anchor points. In Proceedings of ICML, July 18-24, 2021, Virtual Event, USA, 6403–6413.
  23. Early-Learning Regularization Prevents Memorization of Noisy Labels. In Proceedings of NeuIPS, December 6-12, 2020, virtual, USA, 20331–20342.
  24. When does label smoothing help? In Proceedings of NeuIPS, December 8-14, 2019, Vancouver, BC, Canada, 4696–4705.
  25. Problem complexity and method efficiency in optimization. Siam Review, 27(2): 264–266.
  26. Multi-objective interpolation training for robustness to label noise. In Proceedings of the CVPR, June 19-25, 2021, virtual, USA, 6606–6615.
  27. Making Deep Neural Networks Robust to Label Noise: A Loss Correction Approach. In Proceedings of CVPR, July 21-26, 2017, Honolulu, HI, USA, 2233–2241.
  28. Rasmussen, C. 1999. The infinite Gaussian mixture model. In Proceedings of NeuIPS, November 29 - December 4, 1999, Denver, Colorado, USA, volume 12, 554–560.
  29. You only look once: Unified, real-time object detection. In Proceedings of ICCV, June 27-30, 2016, Las Vegas, NV, USA, 779–788.
  30. Learning from Noisy Labels with Deep Neural Networks: A Survey. CoRR, abs/2007.08199.
  31. When ot meets mom: Robust estimation of Wasserstein distance. In Proceedings of AISTATS, April 13 - 15, 2021, Virtual Conference, USA, 136–144.
  32. Going deeper with convolutions. In Proceedings of CVPR, June 7-12, 2015, Boston, MA, USA, 1–9.
  33. Joint Optimization Framework for Learning With Noisy Labels. In Proceedings of CVPR, June 18-22, 2018, Salt Lake City, UT, USA, 5552–5560.
  34. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. In Proceedings of NeuIPS, December 4-9, 2017, Long Beach, CA, USA, 1195–1204.
  35. Learning with symmetric label noise: The importance of being unhinged. In Proceedings of NeuIPS, December 7-12, 2015, Montreal, Quebec, Canada, 10–18.
  36. Sample selection with uncertainty of losses for learning with noisy labels. arXiv preprint arXiv:2106.00445.
  37. Part-dependent label noise: Towards instance-dependent label noise. In Proceedings of NeuIPS, December 6-12, 2020, Virtual Event, USA, 7597–7610.
  38. Are anchor points really indispensable in label-noise learning? In Proceedings of NeuIPS, December 8-14, 2019, Vancouver, BC, Canada, 6835–6846.
  39. Learning from massive noisy labeled data for image classification. In Proceedings of CVPR, June 7-12, 2015, Boston, MA, USA, 2691–2699.
  40. L_DMI: A Novel Information-theoretic Loss Function for Training Deep Nets Robust to Label Noise. In Proceedings of NeuIPS, December 8-14, 2019, Vancouver, BC, Canada, 6222–6233.
  41. Jo-SRC: A contrastive approach for combating noisy labels. In Proceedings of CVPR, June 19-25, 2021, virtual, USA, 5192–5201.
  42. Probabilistic End-To-End Noise Correction for Learning With Noisy Labels. In Proceedings of CVPR, June 16-20, 2019, Long Beach, CA, USA, 7017–7025.
  43. How does disagreement help generalization against label corruption? In Proceedings of ICML, June 9-15, 2019, Long Beach, California, USA, 7164–7173.
  44. Understanding deep learning requires rethinking generalization. In Proceedings of ICLR.
  45. Delving deep into label smoothing. IEEE Transactions on Image Processing, 30: 5984–5996.
  46. Context encoding for semantic segmentation. In Proceedings of CVPR, June 18-22, 2018, Salt Lake City, UT, USA, 7151–7160.
  47. Accelerating very deep convolutional networks for classification and detection. IEEE transactions on pattern analysis and machine intelligence, 38(10): 1943–1955.
  48. Learning noise transition matrix from only noisy labels via total variation regularization. In Proceedings of ICML, July 18-24, 2021, Virtual Event, USA, 12501–12512.
  49. A second-order approach to learning with instance-dependent label noise. In Proceedings of CVPR, June 19-25, 2021, virtual Event, USA, 10113–10123.
  50. Crowdsourcing the annotation of rumourous conversations in social media. In Proceedings of the WWW, May 18-22, 2015, Florence, Italy, 347–353.
Citations (1)
View on Semantic Scholar

Summary

  • The paper introduces a novel Regroup Median Loss approach that improves clean sample selection by regrouping mean and median loss values.
  • It combines a unique sample selection strategy with robust loss estimation to effectively diminish the influence of noisy labels.
  • Empirical results show the method outperforms existing techniques on both synthetic and real-world datasets, enhancing model accuracy.

Regroup Median Loss: A Novel Approach for Enhuring Robustness to Label Noise

Introduction

Label noise in large-scale datasets poses a significant challenge in training accurate deep learning models. The traditional small-loss assumption, positing that clean samples contribute to smaller loss values compared to noisy samples, has been a cornerstone in designing methods to mitigate the effects of label noise. However, the overlap in loss values between clean and noisy samples limits the efficiency of these approaches. To address these shortcomings, we introduce the Regroup Median Loss (RML) method. RML enhances the selection of clean samples and offers a robust loss estimation strategy that diminishes the influence of noisy labels on model training.

RML Methodology

RML is articulated around two main processes: a novel sample selection strategy and a robust loss estimation method.

Sample Selection Strategy

The sample selection strategy is based on a novel loss processing method aimed at diminishing the selection probability of noisy samples. In essence, for each training sample, other samples with the same observed label are selected, and their losses are processed using a formula that significantly reduces the likelihood of choosing noisy samples. This step ensures that the selected samples are predominantly clean, hence offering a purer dataset for loss estimation.

Robust Loss Estimation

The second part of RML involves combining the mean and median loss through a strategic regrouping approach to achieve a robust loss estimation for noisy samples. By dividing selected samples into groups, calculating the mean loss for each group, and obtaining the median of these mean losses, RML ensures a stable and more robust loss estimation. This strategy corrects distorted losses effectively, mitigating the impact of label noise on model training.

Theoretical Analysis and Practical Implications

A theoretical analysis demonstrates the robustness of the Regroup Median Loss, providing a foundation for its efficacy in combating label noise. The empirical results underscore RML's superiority over state-of-the-art methods across various datasets, including those with synthetic and real-world label noise. RML not only improves the accuracy of models trained on datasets with a high rate of label noise but also enhances model generalization capabilities.

Future Directions

The development of RML opens new avenues for research into methods for dealing with label noise in large-scale datasets. Future work could explore the integration of RML with other noise-robust training techniques, advanced semi-supervised learning models, and its application to a broader range of tasks beyond image classification. Additionally, further exploration into optimizing the selection and regrouping strategies within RML could yield even more significant improvements in model robustness to label noise.

Conclusion

In conclusion, the Regroup Median Loss method presents a formidable approach to enhancing the robustness of deep learning models to label noise. By innovatively selecting clean samples and providing a robust loss estimation, RML significantly outperforms existing methods in mitigating the effects of label noise. Its theoretical underpinning and remarkable empirical results establish RML as a valuable addition to the toolbox for training deep learning models in the presence of label noise.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown