Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
Gemini 2.5 Pro
GPT-5
GPT-4o
DeepSeek R1 via Azure
2000 character limit reached

Dirichlet-based Per-Sample Weighting by Transition Matrix for Noisy Label Learning (2403.02690v1)

Published 5 Mar 2024 in cs.LG and cs.CV

Abstract: For learning with noisy labels, the transition matrix, which explicitly models the relation between noisy label distribution and clean label distribution, has been utilized to achieve the statistical consistency of either the classifier or the risk. Previous researches have focused more on how to estimate this transition matrix well, rather than how to utilize it. We propose good utilization of the transition matrix is crucial and suggest a new utilization method based on resampling, coined RENT. Specifically, we first demonstrate current utilizations can have potential limitations for implementation. As an extension to Reweighting, we suggest the Dirichlet distribution-based per-sample Weight Sampling (DWS) framework, and compare reweighting and resampling under DWS framework. With the analyses from DWS, we propose RENT, a REsampling method with Noise Transition matrix. Empirically, RENT consistently outperforms existing transition matrix utilization methods, which includes reweighting, on various benchmark datasets. Our code is available at \url{https://github.com/BaeHeeSun/RENT}.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (57)
  1. Why resampling outperforms reweighting for correcting sampling bias with stochastic gradients. arXiv preprint arXiv:2009.13447, 2020.
  2. A closer look at memorization in deep networks. In International Conference on Machine Learning, pp.  233–242. PMLR, 2017.
  3. From noisy prediction to true label: Noisy prediction calibration via generative model. In International Conference on Machine Learning, pp.  1277–1297. PMLR, 2022.
  4. Deep k-nn for noisy labels. In International Conference on Machine Learning, pp.  540–550. PMLR, 2020.
  5. Confidence scores make instance-dependent label-noise learning possible. In International Conference on Machine Learning, pp.  825–836. PMLR, 2021.
  6. Learning imbalanced datasets with label-distribution-aware margin loss. Advances in neural information processing systems, 32, 2019.
  7. Rsmote: A self-adaptive robust smote for imbalanced problems with label noise. Information Sciences, 553:397–428, 2021.
  8. Noise against noise: stochastic label noise helps combat inherent label noise. In International Conference on Learning Representations, 2020.
  9. Class-dependent label-noise learning with cycle-consistency regularization. In Advances in Neural Information Processing Systems, 2022.
  10. Learning with instance-dependent label noise: A sample sieve approach. In International Conference on Learning Representations, 2020.
  11. Generalization bounds for neural networks via approximate description length. Advances in Neural Information Processing Systems, 32, 2019.
  12. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp.  248–255. Ieee, 2009.
  13. George C. Runger Douglas C. Montgomery. Applied statistics and probability for engineers. John Wiley & Sons, 2013. ISBN 9781118539712.
  14. Co-teaching: Robust training of deep neural networks with extremely noisy labels. Advances in neural information processing systems, 31, 2018.
  15. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.  770–778, 2016.
  16. Uncertainty-aware learning against label noise on imbalanced datasets. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp.  6960–6969, 2022.
  17. Methods of reducing sample size in monte carlo computations. Journal of the Operations Research Society of America, 1(5):263–278, 1953.
  18. Not all samples are created equal: Deep learning with importance sampling. In International conference on machine learning, pp.  2525–2534. PMLR, 2018.
  19. Fine samples for learning with noisy labels. Advances in Neural Information Processing Systems, 34:24137–24149, 2021.
  20. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
  21. Combined cleaning and resampling algorithm for multi-class imbalanced data with label noise. Knowledge-Based Systems, 204:106223, 2020.
  22. Learning multiple layers of features from tiny images. 2009.
  23. Robust inference via generative classifiers for handling noisy labels. In International Conference on Machine Learning, pp.  3763–3772. PMLR, 2019.
  24. Dividemix: Learning with noisy labels as semi-supervised learning. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL https://openreview.net/forum?id=HJgExaVtwr.
  25. Provably end-to-end label-noise learning without anchor points. In International Conference on Machine Learning, pp.  6403–6413. PMLR, 2021.
  26. Do we need to penalize variance of losses for learning with label noise? arXiv preprint arXiv:2201.12739, 2022.
  27. Early-learning regularization prevents memorization of noisy labels. Advances in Neural Information Processing Systems, 33, 2020.
  28. Classification with noisy labels by importance reweighting. IEEE Transactions on pattern analysis and machine intelligence, 38(3):447–461, 2015.
  29. Identifiability of label noise transition matrix. In International Conference on Machine Learning, pp.  21475–21496. PMLR, 2023.
  30. Normalized loss functions for deep learning with noisy labels. In International conference on machine learning, pp.  6543–6553. PMLR, 2020.
  31. PC Mahalanobis. Mahalanobis distance. In Proceedings National Institute of Science of India, volume 49, pp.  234–256, 1936.
  32. Label-noise robust diffusion models. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=HXWTXXtHNl.
  33. Adding gradient noise improves learning for very deep networks. arXiv preprint arXiv:1511.06807, 2015.
  34. Making deep neural networks robust to label noise: A loss correction approach. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.  1944–1952, 2017.
  35. Donald B Rubin. Using the sir algorithm to simulate posterior distributions. Bayesian statistics, 3:395–402, 1988.
  36. Bayesian statistics without tears: a sampling–resampling perspective. The American Statistician, 46(2):84–88, 1992.
  37. Joint optimization framework for learning with noisy labels. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.  5552–5560, 2018.
  38. Proselflc: Progressive self label correction for training robust deep neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  752–761, 2021.
  39. Symmetric cross entropy for robust learning with noisy labels. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.  322–330, 2019.
  40. Combating noisy labels by agreement: A joint training method with co-regularization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  13726–13735, 2020.
  41. Open-set label noise can improve robustness against inherent label noise. Advances in Neural Information Processing Systems, 34:7978–7992, 2021a.
  42. Open-set label noise can improve robustness against inherent label noise. Advances in Neural Information Processing Systems, 34:7978–7992, 2021b.
  43. Learning with noisy labels revisited: A study using real-world human annotations. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=TBWA6PLJZQm.
  44. Are anchor points really indispensable in label-noise learning? Advances in Neural Information Processing Systems, 32:6838–6849, 2019.
  45. Part-dependent label noise: Towards instance-dependent label noise. Advances in Neural Information Processing Systems, 33, 2020.
  46. Learning from massive noisy labeled data for image classification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.  2691–2699, 2015.
  47. Estimating instance-dependent bayes-label transition matrix using a deep neural network. In International Conference on Machine Learning, pp.  25302–25312. PMLR, 2022.
  48. Dual t: Reducing estimation error for transition matrix in label-noise learning. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp.  7260–7271. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper/2020/file/512c5cad6c37edb98ae91c8a76c3a291-Paper.pdf.
  49. Instance-dependent label-noise learning under a structural causal model. Advances in Neural Information Processing Systems, 34, 2021.
  50. How does disagreement help generalization against label corruption? In International Conference on Machine Learning, pp.  7164–7173. PMLR, 2019.
  51. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107–115, 2021a.
  52. When noisy labels meet long tail dilemmas: A representation calibration method. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp.  15890–15900, October 2023.
  53. Learning noise transition matrix from only noisy labels via total variation regularization. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp.  12501–12512. PMLR, 18–24 Jul 2021b. URL https://proceedings.mlr.press/v139/zhang21n.html.
  54. Generalized cross entropy loss for training deep neural networks with noisy labels. In 32nd Conference on Neural Information Processing Systems (NeurIPS), 2018.
  55. Error-bounded correction of noisy labels. In International Conference on Machine Learning, pp.  11447–11457. PMLR, 2020.
  56. Clusterability as an alternative to anchor points when learning with noisy labels. In International Conference on Machine Learning, pp.  12912–12923. PMLR, 2021.
  57. Detecting corrupted labels without training a model to predict. In International Conference on Machine Learning, pp.  27412–27427. PMLR, 2022.
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Dice Question Streamline Icon: https://streamlinehq.com

Follow-up Questions

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now