Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash 100 tok/s
Gemini 2.5 Pro 58 tok/s Pro
GPT-5 Medium 29 tok/s
GPT-5 High 29 tok/s Pro
GPT-4o 103 tok/s
GPT OSS 120B 480 tok/s Pro
Kimi K2 215 tok/s Pro
2000 character limit reached

Uncertainty-aware self-training with expectation maximization basis transformation (2405.01175v1)

Published 2 May 2024 in cs.CV and cs.AI

Abstract: Self-training is a powerful approach to deep learning. The key process is to find a pseudo-label for modeling. However, previous self-training algorithms suffer from the over-confidence issue brought by the hard labels, even some confidence-related regularizers cannot comprehensively catch the uncertainty. Therefore, we propose a new self-training framework to combine uncertainty information of both model and dataset. Specifically, we propose to use Expectation-Maximization (EM) to smooth the labels and comprehensively estimate the uncertainty information. We further design a basis extraction network to estimate the initial basis from the dataset. The obtained basis with uncertainty can be filtered based on uncertainty information. It can then be transformed into the real hard label to iteratively update the model and basis in the retraining process. Experiments on image classification and semantic segmentation show the advantages of our methods among confidence-aware self-training algorithms with 1-3 percentage improvement on different datasets.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (60)
  1. Semi-supervised logistic regression. In ECAI, pages 390–394.
  2. Language models are few-shot learners. arXiv preprint arXiv:2005.14165.
  3. Open set domain adaptation for image and action recognition. IEEE transactions on pattern analysis and machine intelligence, 42(2):413–429.
  4. Semi-supervised learning (chapelle, o. et al., eds.; 2006)[book reviews]. IEEE Transactions on Neural Networks, 20(3):542–542.
  5. Progressive feature alignment for unsupervised domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 627–636.
  6. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE transactions on pattern analysis and machine intelligence, 40(4):834–848.
  7. Domain adaptive faster r-cnn for object detection in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3339–3348.
  8. No more discrimination: Cross city adaptation of road scene segmenters. In Proceedings of the IEEE International Conference on Computer Vision, pages 1992–2001.
  9. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3213–3223.
  10. Maximum likelihood from incomplete data via the em algorithm. Journal of the Royal Statistical Society: Series B (Methodological), 39(1):1–22.
  11. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee.
  12. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
  13. Domain-adversarial training of neural networks. The journal of machine learning research, 17(1):2096–2030.
  14. Semi-supervised learning by entropy minimization. In CAP, pages 281–296.
  15. Revisiting self-training for neural sequence generation. arXiv preprint arXiv:1909.13788.
  16. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778.
  17. Distilling the knowledge in a neural network.
  18. Cycada: Cycle-consistent adversarial domain adaptation. In International conference on machine learning, pages 1989–1998. PMLR.
  19. Cross-domain weakly-supervised object detection through progressive domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5001–5009.
  20. Unsupervised visual domain adaptation: A deep max-margin gaussian process approach. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4380–4390.
  21. Temporal ensembling for semi-supervised learning. arXiv preprint arXiv:1610.02242.
  22. Lee, D.-H. et al. (2013). Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on challenges in representation learning, ICML, volume 3.
  23. Expectation-maximization attention networks for semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9167–9176.
  24. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740–755. Springer.
  25. Learning transferable features with deep adaptation networks. In International conference on machine learning, pages 97–105. PMLR.
  26. Conditional adversarial domain adaptation. arXiv preprint arXiv:1705.10667.
  27. Unsupervised domain adaptation with residual transfer networks. arXiv preprint arXiv:1602.04433.
  28. Deep transfer learning with joint adaptation networks. In International conference on machine learning, pages 2208–2217. PMLR.
  29. Smooth neighbors on teacher graphs for semi-supervised learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8896–8905.
  30. Moon, T. K. (1996). The expectation-maximization algorithm. IEEE Signal processing magazine, 13(6):47–60.
  31. Uncertainty-aware self-training for text classification with few labels. arXiv preprint arXiv:2006.15315.
  32. Uncertainty-aware self-training for text classification with few labels. CoRR, abs/2006.15315.
  33. Learning with noisy labels. In NIPS, volume 26, pages 1196–1204.
  34. Visda: A synthetic-to-real benchmark for visual domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 2021–2026.
  35. Pinheiro, P. O. (2018). Unsupervised domain adaptation with similarity learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8004–8013.
  36. Improving language understanding by generative pre-training.
  37. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9.
  38. Training deep neural networks on noisy labels with bootstrapping. arXiv preprint arXiv:1412.6596.
  39. On bayesian analysis of mixtures with an unknown number of components (with discussion). Journal of the Royal Statistical Society: series B (statistical methodology), 59(4):731–792.
  40. Playing for data: Ground truth from computer games. In European conference on computer vision, pages 102–118. Springer.
  41. The synthia dataset: A large collection of synthetic images for semantic segmentation of urban scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3234–3243.
  42. Adapting visual category models to new domains. In European conference on computer vision, pages 213–226. Springer.
  43. Asymmetric tri-training for unsupervised domain adaptation. In International Conference on Machine Learning, pages 2988–2997. PMLR.
  44. Adversarial dropout regularization. arXiv preprint arXiv:1711.01575.
  45. Maximum classifier discrepancy for unsupervised domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3723–3732.
  46. Generate to adapt: Aligning domains using generative adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8503–8512.
  47. Scudder, H. (1965). Probability of error of some adaptive pattern-recognition machines. IEEE Transactions on Information Theory, 11(3):363–371.
  48. Improving dataset distillation. CoRR, abs/1910.02551.
  49. Training convolutional networks with noisy labels. arXiv preprint arXiv:1406.2080.
  50. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. arXiv preprint arXiv:1703.01780.
  51. Learning to adapt structured output space for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7472–7481.
  52. Adversarial discriminative domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7167–7176.
  53. Advent: Adversarial entropy minimization for domain adaptation in semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2517–2526.
  54. Dataset distillation. CoRR, abs/1811.10959.
  55. Yarowsky, D. (1995). Unsupervised word sense disambiguation rivaling supervised methods. In 33rd annual meeting of the association for computational linguistics, pages 189–196.
  56. Simultaneous edge alignment and learning. In Proceedings of the European Conference on Computer Vision (ECCV), pages 388–404.
  57. Fully convolutional adaptation networks for semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6810–6818.
  58. Unsupervised domain adaptation for semantic segmentation via class-balanced self-training. In Proceedings of the European conference on computer vision (ECCV), pages 289–305.
  59. Confidence regularized self-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5982–5991.
  60. Confidence regularized self-training. CoRR, abs/1908.09822.
List To Do Tasks Checklist Streamline Icon: https://streamlinehq.com

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Ai Generate Text Spark Streamline Icon: https://streamlinehq.com

Paper Prompts

Sign up for free to create and run prompts on this paper using GPT-5.

List Streamline Icon: https://streamlinehq.com Explore 10 Community Prompts
Dice Question Streamline Icon: https://streamlinehq.com

Follow-up Questions

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets