Papers
Topics
Authors
Recent
Search
2000 character limit reached

Erasing the Bias: Fine-Tuning Foundation Models for Semi-Supervised Learning

Published 20 May 2024 in cs.LG | (2405.11756v1)

Abstract: Semi-supervised learning (SSL) has witnessed remarkable progress, resulting in the emergence of numerous method variations. However, practitioners often encounter challenges when attempting to deploy these methods due to their subpar performance. In this paper, we present a novel SSL approach named FineSSL that significantly addresses this limitation by adapting pre-trained foundation models. We identify the aggregated biases and cognitive deviation problems inherent in foundation models, and propose a simple yet effective solution by imposing balanced margin softmax and decoupled label smoothing. Through extensive experiments, we demonstrate that FineSSL sets a new state of the art for SSL on multiple benchmark datasets, reduces the training cost by over six times, and can seamlessly integrate various fine-tuning and modern SSL algorithms. The source code is available at https://github.com/Gank0078/FineSSL.

Authors (2)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (61)
  1. Pseudo-labeling and confirmation bias in deep semi-supervised learning. In IJCNN, pp.  1–8, 2020.
  2. Learning with pseudo-ensembles. NeurIPS, 27, 2014.
  3. Mixmatch: A holistic approach to semi-supervised learning. NeurIPS, 32, 2019.
  4. Food-101–mining discriminative components with random forests. In ECCV, pp.  446–461, 2014.
  5. Learning imbalanced datasets with label-distribution-aware margin loss. NeurIPS, 32, 2019.
  6. Open-world semi-supervised learning. In International Conference on Learning Representations, 2022.
  7. Unlabelled data improves bayesian uncertainty calibration under covariate shift. In ICML, pp.  1392–1402, 2020.
  8. Revisiting parameter-efficient tuning: Are we really there yet? arXiv preprint, 2022a.
  9. Softmatch: Addressing the quantity-quality trade-off in semi-supervised learning. arXiv preprint, 2023.
  10. Adaptformer: Adapting vision transformers for scalable visual recognition. NeurIPS, 35:16664–16678, 2022b.
  11. A simple framework for contrastive learning of visual representations. In ICML, pp.  1597–1607, 2020a.
  12. Big self-supervised models are strong semi-supervised learners. NeurIPS, 33:22243–22255, 2020b.
  13. Imagenet: A large-scale hierarchical image database. In CVPR, pp.  248–255, 2009.
  14. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint, 2020.
  15. Semi-supervised learning by entropy minimization. NeurIPS, 17, 2004.
  16. On calibration of modern neural networks. In International conference on machine learning, pp.  1321–1330. PMLR, 2017.
  17. Robust semi-supervised learning when not all classes have labels. In Neural Information Processing Systems, 2022.
  18. Facing the elephant in the room: Visual prompt tuning or full finetuning? arXiv preprint, 2024.
  19. Deep residual learning for image recognition. In CVPR, pp.  770–778, 2016.
  20. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.  9729–9738, 2020.
  21. Parameter-efficient transfer learning for nlp. In ICML, pp.  2790–2799, 2019.
  22. Lora: Low-rank adaptation of large language models. arXiv preprint, 2021.
  23. Scaling up visual and vision-language representation learning with noisy text supervision. In ICML, pp.  4904–4916, 2021.
  24. Visual prompt tuning. In ECCV, pp.  709–727, 2022.
  25. Distribution aligning refinery of pseudo-label for imbalanced semi-supervised learning. NeurIPS, 33:14567–14579, 2020.
  26. Learning multiple layers of features from tiny images. 2009.
  27. Fine-tuning can distort pretrained features and underperform out-of-distribution. arXiv preprint, 2022.
  28. Temporal ensembling for semi-supervised learning. arXiv preprint, 2016.
  29. Contrastive regularization for semi-supervised learning. In CVPR, pp.  3911–3920, 2022.
  30. Lee, D.-H. et al. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In ICML Workshop, pp.  896, 2013.
  31. Scaling & shifting your features: A new baseline for efficient model tuning. Advances in Neural Information Processing Systems, 35:109–123, 2022.
  32. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. NeurIPS, 35:1950–1965, 2022.
  33. Enhancing clip with clip: Exploring pseudolabeling for limited-label prompt tuning. Advances in Neural Information Processing Systems, 36:60984–61007, 2023.
  34. Long-tail learning via logit adjustment. arXiv preprint, 2020.
  35. When does label smoothing help? NeurIPS, 32, 2019.
  36. Learning transferable visual models from natural language supervision. In ICML, pp.  8748–8763, 2021.
  37. In defense of pseudo-labeling: An uncertainty-aware pseudo-label selection framework for semi-supervised learning. arXiv preprint, 2021.
  38. Regularization with stochastic transformations and perturbations for deep semi-supervised learning. NeurIPS, 29, 2016.
  39. Parameter-efficient long-tailed recognition. arXiv preprint, 2023.
  40. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. NeurIPS, 33:596–608, 2020.
  41. A realistic evaluation of semi-supervised learning for fine-grained classification. In CVPR, pp.  12966–12975, 2021.
  42. On mixup training: Improved calibration and predictive uncertainty for deep neural networks. NeurIPS, 32, 2019.
  43. Attention is all you need. NeurIPS, 30, 2017.
  44. Self-tuning for data-efficient deep learning. In ICML, pp.  10738–10748, 2021.
  45. Debiased learning from naturally imbalanced pseudo-labels. In CVPR, pp.  14647–14657, 2022a.
  46. Usb: A unified semi-supervised learning benchmark for classification. NeurIPS, 35:3938–3961, 2022b.
  47. Freematch: Self-adaptive thresholding for semi-supervised learning. arXiv preprint, 2022c.
  48. Towards realistic long-tailed semi-supervised learning: Consistency is all you need. In CVPR, pp.  3469–3478, 2023.
  49. Semi-supervised vision transformers. In European conference on computer vision, pp.  605–620. Springer, 2022.
  50. Self-training with noisy student improves imagenet classification. In CVPR, pp.  10687–10698, 2020.
  51. Dash: Semi-supervised learning with dynamic thresholding. In ICML, pp.  11525–11536, 2021.
  52. Class-aware contrastive semi-supervised learning. In CVPR, pp.  14421–14430, 2022a.
  53. Parameter-efficient tuning makes a good classification head. arXiv preprint, 2022b.
  54. Bridging the gap: Learning pace synchronization for open-world semi-supervised learning. arXiv preprint, 2023.
  55. Wide residual networks. arXiv preprint arXiv:1605.07146, 2016.
  56. Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models. arXiv preprint arXiv:2106.10199, 2021.
  57. Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling. NeurIPS, 34:18408–18419, 2021.
  58. Improving calibration for long-tailed recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.  16489–16498, 2021.
  59. When semi-supervised learning meets transfer learning: Training strategies, models and datasets. arXiv preprint, 2018.
  60. Learning to prompt for vision-language models. International Journal of Computer Vision, 130(9):2337–2348, 2022.
  61. Generalized logit adjustment: Calibrating fine-tuned models by removing label bias in foundation models. arXiv preprint, 2023.
Citations (4)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up