Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
120 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
46 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

One-bit Supervision for Image Classification: Problem, Solution, and Beyond (2311.15225v1)

Published 26 Nov 2023 in cs.CV

Abstract: This paper presents one-bit supervision, a novel setting of learning with fewer labels, for image classification. Instead of training model using the accurate label of each sample, our setting requires the model to interact with the system by predicting the class label of each sample and learn from the answer whether the guess is correct, which provides one bit (yes or no) of information. An intriguing property of the setting is that the burden of annotation largely alleviates in comparison to offering the accurate label. There are two keys to one-bit supervision, which are (i) improving the guess accuracy and (ii) making good use of the incorrect guesses. To achieve these goals, we propose a multi-stage training paradigm and incorporate negative label suppression into an off-the-shelf semi-supervised learning algorithm. Theoretical analysis shows that one-bit annotation is more efficient than full-bit annotation in most cases and gives the conditions of combining our approach with active learning. Inspired by this, we further integrate the one-bit supervision framework into the self-supervised learning algorithm which yields an even more efficient training schedule. Different from training from scratch, when self-supervised learning is used for initialization, both hard example mining and class balance are verified effective in boosting the learning performance. However, these two frameworks still need full-bit labels in the initial stage. To cast off this burden, we utilize unsupervised domain adaptation to train the initial model and conduct pure one-bit annotations on the target dataset. In multiple benchmarks, the learning efficiency of the proposed approach surpasses that using full-bit, semi-supervised supervision.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (75)
  1. Training connectionist networks with queries and selective sampling. In Advances in neural information processing systems. 566–573.
  2. Mixmatch: A holistic approach to semi-supervised learning. In Advances in Neural Information Processing Systems. 5050–5060.
  3. Curriculum Labeling: Revisiting Pseudo-Labeling for Semi-Supervised Learning. arXiv preprint arXiv:2001.06001 (2020).
  4. Homm: Higher-order moment matching for unsupervised domain adaptation. In Proceedings of the AAAI conference on artificial intelligence, Vol. 34. 3422–3429.
  5. Negative sampling in semi-supervised learning. In International Conference on Machine Learning. PMLR, 1704–1714.
  6. A simple framework for contrastive learning of visual representations. In International conference on machine learning. PMLR, 1597–1607.
  7. Big self-supervised models are strong semi-supervised learners. arXiv preprint arXiv:2006.10029 (2020).
  8. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04297 (2020).
  9. Imagenet: A large-scale hierarchical image database. In Computer Vision and Pattern Recognition.
  10. Semi-supervised learning in gigantic image collections. In Advances in neural information processing systems. 522–530.
  11. Selecting influential examples: Active learning with expected model output changes. In European Conference on Computer Vision. Springer, 562–577.
  12. Born again neural networks. arXiv preprint arXiv:1805.04770 (2018).
  13. Deep bayesian active learning with image data. In International Conference on Machine Learning. PMLR, 1183–1192.
  14. Consistency-based semi-supervised active learning: Towards minimizing labeling cost. In European Conference on Computer Vision. Springer, 510–526.
  15. Xavier Gastaldi. 2017. Shake-shake regularization. arXiv preprint arXiv:1705.07485 (2017).
  16. Unsupervised representation learning by predicting image rotations. arXiv preprint arXiv:1803.07728 (2018).
  17. Yves Grandvalet and Yoshua Bengio. 2005. Semi-supervised learning by entropy minimization. In Advances in neural information processing systems. 529–536.
  18. Multimodal semi-supervised learning for image classification. In 2010 IEEE Computer society conference on computer vision and pattern recognition. IEEE, 902–909.
  19. Explanation Consistency Training: Facilitating Consistency-Based Semi-Supervised Learning with Interpretability. (2021).
  20. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 9729–9738.
  21. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. 770–778.
  22. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 (2015).
  23. Bayesian active learning for classification and preference learning. arXiv preprint arXiv:1112.5745 (2011).
  24. One-bit Supervision for Image Classification. Advances in Neural Information Processing Systems 33 (2020).
  25. Creating Something from Nothing: Unsupervised Knowledge Distillation for Cross-Modal Hashing. arXiv preprint arXiv:2004.00280 (2020).
  26. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition. 4700–4708.
  27. Label propagation for deep semi-supervised learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 5070–5079.
  28. Nlnl: Negative learning for noisy labels. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 101–110.
  29. Batchbald: Efficient and diverse batch acquisition for deep bayesian active learning. In Advances in Neural Information Processing Systems. 7024–7035.
  30. Learning multiple layers of features from tiny images. (2009).
  31. Featmatch: Feature-based augmentation for semi-supervised learning. In European Conference on Computer Vision. Springer, 479–495.
  32. Samuli Laine and Timo Aila. 2016. Temporal ensembling for semi-supervised learning. arXiv preprint arXiv:1610.02242 (2016).
  33. Colorization as a proxy task for visual understanding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 6874–6883.
  34. Deep learning. Nature 521, 7553 (2015), 436–444.
  35. Dong-Hyun Lee. 2013. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on challenges in representation learning, ICML, Vol. 3. 2.
  36. David D Lewis and William A Gale. 1994. A sequential algorithm for training text classifiers. In SIGIR’94. Springer, 3–12.
  37. Semantic concentration for domain adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 9102–9111.
  38. Latent structured active learning. In Advances in Neural Information Processing Systems. 728–736.
  39. Tomasz Malisiewicz and Alyosha Efros. 2009. Beyond categories: The visual memex model for reasoning about object relationships. In Advances in neural information processing systems. 1222–1230.
  40. Virtual adversarial training: a regularization method for supervised and semi-supervised learning. IEEE transactions on pattern analysis and machine intelligence 41, 8 (2018), 1979–1993.
  41. Mehdi Noroozi and Paolo Favaro. 2016. Unsupervised learning of visual representations by solving jigsaw puzzles. In European Conference on Computer Vision. Springer, 69–84.
  42. Representation learning by learning to count. In Proceedings of the IEEE International Conference on Computer Vision. 5898–5906.
  43. Boosting self-supervised learning via knowledge transfer. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 9359–9367.
  44. We don’t need no bounding-boxes: Training object class detectors using only human verification. In Proceedings of the IEEE conference on computer vision and pattern recognition. 854–863.
  45. Training object class detectors with click supervision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.
  46. Learning features by watching objects move. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2701–2710.
  47. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE/CVF international conference on computer vision. 1406–1415.
  48. Bayesian batch active learning as sparse subset approximation. In Advances in Neural Information Processing Systems. 6356–6367.
  49. Deep co-training for semi-supervised image recognition. In Proceedings of the european conference on computer vision (eccv). 135–152.
  50. Semi-supervised learning with ladder networks. In Advances in neural information processing systems. 3546–3554.
  51. Sachin Ravi and Hugo Larochelle. 2016. Optimization as a model for few-shot learning. (2016).
  52. In defense of pseudo-labeling: An uncertainty-aware pseudo-label selection framework for semi-supervised learning. arXiv preprint arXiv:2101.06329 (2021).
  53. Fitnets: Hints for thin deep nets. arXiv preprint arXiv:1412.6550 (2014).
  54. Imagenet large scale visual recognition challenge. International Journal of Computer Vision 115, 3 (2015), 211–252.
  55. Ozan Sener and Silvio Savarese. 2017. Active learning for convolutional neural networks: A core-set approach. arXiv preprint arXiv:1708.00489 (2017).
  56. Weishi Shi and Qi Yu. 2019. Integrating Bayesian and Discriminative Sparse Kernel Machines for Multi-class Active Learning. In Advances in Neural Information Processing Systems. 2282–2291.
  57. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. arXiv preprint arXiv:2001.07685 (2020).
  58. Unsupervised domain adaptation through self-supervision. arXiv preprint arXiv:1909.11825 (2019).
  59. Test-time training with self-supervision for generalization under distribution shifts. In International Conference on Machine Learning. PMLR, 9229–9248.
  60. Antti Tarvainen and Harri Valpola. 2017. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. In Advances in Neural Information Processing Systems.
  61. Adversarial discriminative domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition. 7167–7176.
  62. Cost-effective active learning for deep image classification. IEEE Transactions on Circuits and Systems for Video Technology 27, 12 (2016), 2591–2600.
  63. Unsupervised data augmentation for consistency training. arXiv preprint arXiv:1904.12848 (2019).
  64. Self-training with noisy student improves imagenet classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 10687–10698.
  65. Hierarchical Semantic Aggregation for Contrastive Representation Learning. arXiv preprint arXiv:2012.02733 (2020).
  66. Deep interactive object selection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 373–381.
  67. Knowledge distillation in generations: More tolerant teachers educate better students. arXiv preprint arXiv:1805.05551 (2018).
  68. Donggeun Yoo and In So Kweon. 2019. Learning loss for active learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.
  69. Tangent-normal adversarial regularization for semi-supervised learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 10676–10684.
  70. Sergey Zagoruyko and Nikos Komodakis. 2016. Wide residual networks. arXiv preprint arXiv:1605.07146 (2016).
  71. Central moment discrepancy (cmd) for domain-invariant representation learning. arXiv preprint arXiv:1702.08811 (2017).
  72. S4l: Self-supervised semi-supervised learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 1476–1485.
  73. Liheng Zhang and Guo-Jun Qi. 2020. Wcp: Worst-case perturbations for semi-supervised deep learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 3912–3921.
  74. Colorful image colorization. In European conference on computer vision. Springer, 649–666.
  75. Object Recognition with and without Objects. In International Joint Conference on Artificial Intelligence.
Citations (2)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now