Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
194 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
46 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Few Shot Semantic Segmentation: a review of methodologies, benchmarks, and open challenges (2304.05832v2)

Published 12 Apr 2023 in cs.CV and cs.AI

Abstract: Semantic segmentation, vital for applications ranging from autonomous driving to robotics, faces significant challenges in domains where collecting large annotated datasets is difficult or prohibitively expensive. In such contexts, such as medicine and agriculture, the scarcity of training images hampers progress. Introducing Few-Shot Semantic Segmentation, a novel task in computer vision, which aims at designing models capable of segmenting new semantic classes with only a few examples. This paper consists of a comprehensive survey of Few-Shot Semantic Segmentation, tracing its evolution and exploring various model designs, from the more popular conditional and prototypical networks to the more niche latent space optimization methods, presenting also the new opportunities offered by recent foundational models. Through a chronological narrative, we dissect influential trends and methodologies, providing insights into their strengths and limitations. A temporal timeline offers a visual roadmap, marking key milestones in the field's progression. Complemented by quantitative analyses on benchmark datasets and qualitative showcases of seminal works, this survey equips readers with a deep understanding of the topic. By elucidating current challenges, state-of-the-art models, and prospects, we aid researchers and practitioners in navigating the intricacies of Few-Shot Semantic Segmentation and provide ground for future development.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (142)
  1. Flamingo: a Visual Language Model for Few-Shot Learning. In Advances in Neural Information Processing Systems.
  2. Few-shot semantic segmentation via mask aggregation. Neural Processing Letters (2024).
  3. On the texture bias for few-shot cnn segmentation. In IEEE/CVF Winter conference on Applications of Computer Vision (WACV).
  4. Learning representations by maximizing mutual information across views. Advances in neural information processing systems (2019).
  5. Segnet: A deep convolutional encoder-decoder architecture for image segmentation. IEEE Transactions on pattern analysis and machine intelligence (2017).
  6. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258 (2021).
  7. Few-shot segmentation without meta-learning: A good transductive inference is all you need?. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
  8. Language models are few-shot learners. Advances in neural information processing systems (2020).
  9. Language models are few-shot learners. In Conference on Neural Information Processing Systems (NeurIPS).
  10. Robust vessel segmentation in fundus images. International journal of biomedical imaging (2013).
  11. Lung Segmentation in Chest Radiographs Using Anatomical Atlases With Nonrigid Registration. IEEE Transactions on Medical Imaging (2014).
  12. A theoretical analysis of the number of shots in few-shot learning. arXiv preprint arXiv:1909.11722 (2019).
  13. More than the Sum of Its Parts: Ensembling Backbone Networks for Few-Shot Segmentation. arXiv preprint arXiv:2402.06581 (2024).
  14. Prototype-based Incremental Few-Shot Semantic Segmentation. In British Machine Vision Conference (BMVC).
  15. Few-shot semantic segmentation: a review on recent approaches. Neural Computing and Applications (2023).
  16. Pixel Matching Network for Cross-Domain Few-Shot Segmentation. In IEEE/CVF Winter Conference on Applications of Computer Vision (WACV).
  17. Apanet: adaptive prototypes alignment network for few-shot semantic segmentation. arXiv preprint arXiv:2111.12263 (2021).
  18. Encoder-decoder with atrous separable convolution for semantic image segmentation. In European Conference on Computer Vision (ECCV).
  19. Xinlei Chen and Kaiming He. 2021. Exploring simple siamese representation learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
  20. Skin Lesion Analysis Toward Melanoma Detection 2018: A Challenge Hosted by the International Skin Imaging Collaboration (ISIC). ArXiv (2019).
  21. The cityscapes dataset for semantic urban scene understanding. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
  22. Deepglobe 2018: A challenge to parse the earth through satellite images. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops.
  23. ImageNet: A large-scale hierarchical image database. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
  24. Nanqing Dong and Eric P. Xing. 2018. Few-Shot Semantic Segmentation with Prototype Learning. In British Machine Vision Conference (BMVC).
  25. Discriminative Unsupervised Feature Learning with Convolutional Neural Networks. In Advances in Neural Information Processing Systems.
  26. The PASCAL Visual Object Classes Challenge 2012 (VOC2012) Results. http://www.pascal-network.org/challenges/VOC/voc2012/workshop/index.html.
  27. VSE++: Improving Visual-Semantic Embeddings with Hard Negatives. (2018).
  28. DARNet: Bridging Domain Gaps in Cross-Domain Few-Shot Segmentation with Dynamic Adaptation. ArXiv abs/2312.04813 (2023).
  29. Self-support Few-Shot Semantic Segmentation. In European Conference on Computer Vision (ECCV).
  30. An Ensemble Classification-Based Approach Applied to Retinal Blood Vessel Segmentation. IEEE Transactions on Biomedical Engineering (2012).
  31. SimPropNet: Improved Similarity Propagation for Few-shot Image Segmentation. In International Joint Conference on Artificial Intelligence (IJCAI).
  32. A survey on deep learning techniques for image and video semantic segmentation. Applied Soft Computing (2018).
  33. Albert Gu and Tri Dao. 2023. Mamba: Linear-Time Sequence Modeling with Selective State Spaces. arXiv preprint arXiv:2312.00752 (2023).
  34. A review of semantic segmentation using deep neural networks. International journal of multimedia information retrieval (2018).
  35. A Strong Baseline for Generalized Few-Shot Semantic Segmentation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
  36. Language models are general-purpose interfaces. arXiv preprint arXiv:2206.06336 (2022).
  37. Simultaneous detection and segmentation. In European Conference on Computer Vision (ECCV).
  38. Deep residual learning for image recognition. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
  39. Meta-learning initializations for image segmentation. arXiv preprint arXiv:1912.06290 (2019).
  40. GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium. In Neural Information Processing Systems.
  41. Cost aggregation with 4d convolutional swin transformer for few-shot segmentation. In European Conference on Computer Vision (ECCV).
  42. Locating blood vessels in retinal images by piecewise threshold probing of a matched filter response. IEEE Transactions on Medical Imaging (2000).
  43. Attention-based multi-context guiding for few-shot semantic segmentation. In AAAI conference on artificial intelligence.
  44. A survey of self-supervised and few-shot object detection. IEEE Transactions on Pattern Analysis and Machine Intelligence (2022).
  45. Rethinking Federated Learning With Domain Shift: A Prototype View. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
  46. RestNet: Boosting Cross-Domain Few-Shot Segmentation with Residual Transformation Network. In British Machine Vision Conference (BMVC).
  47. MSANet: Multi-Similarity and Attention Guidance for Boosting Few-Shot Segmentation. arXiv preprint arXiv:2206.09667 (2022).
  48. Automatic Tuberculosis Screening Using Chest Radiographs. IEEE Transactions on Medical Imaging (2014).
  49. Suvarna Kadam and Vinay Vaidya. 2020. Review and analysis of zero, one and few shot learning approaches. In Intelligent Systems Design and Applications (ISDA).
  50. Dahyun Kang and Minsu Cho. 2022. Integrative few-shot learning for classification and segmentation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
  51. Andrej Karpathy and Li Fei-Fei. 2015. Deep visual-semantic alignments for generating image descriptions. In 2015 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
  52. A style-based generator architecture for generative adversarial networks. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
  53. Analyzing and improving the image quality of stylegan. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
  54. Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL).
  55. Diederik P Kingma and Max Welling. 2013. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 (2013).
  56. Segment Anything. arXiv:2304.02643 (2023).
  57. Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models. ArXiv abs/1411.2539 (2014).
  58. Learning what not to segment: A new perspective on few-shot segmentation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
  59. Beyond the Prototype: Divide-and-conquer Proxies for Few-shot Segmentation. In International Joint Conference on Artificial Intelligence (IJCAI).
  60. Cross-Domain Few-Shot Semantic Segmentation. In European Conference on Computer Vision (ECCV).
  61. A Survey on Semantic Segmentation. In 2018 IEEE International Conference on Data Mining Workshops (ICDMW).
  62. Adaptive prototype learning and allocation for few-shot segmentation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
  63. Fss-1000: A 1000-class dataset for few-shot segmentation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
  64. Few-shot Semantic Segmentation with Self-supervision from Pseudo-classes. British Machine Vision Conference (BMVC) (2021).
  65. Look into Person: Joint Body Parsing & Pose Estimation Network and a New Benchmark. IEEE Transactions on Pattern Analysis and Machine Intelligence (2019).
  66. Microsoft COCO: Common Objects in Context. In European Conference on Computer Vision (ECCV).
  67. Harmonic Feature Activation for Few-Shot Semantic Segmentation. IEEE Transactions on Image Processing (2021).
  68. Fecanet: Boosting few-shot semantic segmentation with feature-enhanced context-aware network. IEEE Transactions on Multimedia (2023).
  69. Learning Orthogonal Prototypes for Generalized Few-Shot Semantic Segmentation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
  70. Harmonizing Base and Novel Classes: A Class-Contrastive Approach for Generalized Few-Shot Segmentation. ArXiv (2023).
  71. Few-shot Segmentation with Optimal Transport Matching and Message Flow. IEEE Transactions on Multimedia (2022).
  72. Crnet: Cross-reference networks for few-shot segmentation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
  73. Intermediate Prototype Mining Transformer for Few-Shot Semantic Segmentation. In Advances in Neural Information Processing Systems (NeurIPS).
  74. Part-aware prototype network for few-shot semantic segmentation. In European Conference on Computer Vision (ECCV).
  75. Matcher: Segment Anything with One Shot Using All-Purpose Feature Matching. In The Twelfth International Conference on Learning Representations (ICLR).
  76. Fully convolutional networks for semantic segmentation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
  77. Prediction Calibration for Generalized Few-Shot Semantic Segmentation. IEEE Transactions on Image Processing (2023).
  78. Simpler is better: Few-shot semantic segmentation with classifier weight transformer. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
  79. Timo Lüddecke and Alexander Ecker. 2022. Image Segmentation Using Text and Image Prompts. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
  80. SEGIC: Unleashing the Emergent Correspondence for In-Context Segmentation. ArXiv (2023).
  81. Hypercorrelation squeeze for few-shot segmentation. In IEEE/CVF international conference on computer vision (ICCV).
  82. Tom M Mitchell and Tom M Mitchell. 1997. Machine learning. McGraw-hill New York.
  83. Foundations of machine learning. MIT press.
  84. Khoi Nguyen and Sinisa Todorovic. 2019. Feature weighting and boosting for few-shot segmentation. In IEEE/CVF International Conference on Computer Vision (ICCV).
  85. Atsuro Okazawa. 2022. Interclass Prototype Relation for Few-Shot Segmentation. In European Conference on Computer Vision (ECCV).
  86. DINOv2: Learning Robust Visual Features without Supervision. Transactions on Machine Learning Research (2024).
  87. Learning transferable visual models from natural language supervision. In International conference on machine learning. PMLR.
  88. Improving Language Understanding by Generative Pre-Training. (2018).
  89. Few-shot segmentation propagation with guided networks. arXiv preprint arXiv:1806.07373 (2018).
  90. PACO: Parts and Attributes of Common Objects. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
  91. Learning Deep Representations of Fine-Grained Visual Descriptions. In 2016 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
  92. Visual Semantic Segmentation Based on Few/Zero-Shot Learning: An Overview. arXiv preprint arXiv:2211.08352 (2022).
  93. Stochastic backpropagation and approximate inference in deep generative models. In International Conference on Machine Learning (ICML).
  94. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical Image Computing and Computer-assisted Intervention (MICCAI).
  95. GANORCON: Are Generative Models Useful for Few-shot Segmentation?. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
  96. One-Shot Learning for Semantic Segmentation. In British Machine Vision Conference (BMVC).
  97. Dense cross-query-and-support attention weighted mask aggregation for few-shot segmentation. In European Conference on Computer Vision (ECCV).
  98. Mennatullah Siam and Boris Oreshkin. 2019. Adaptive masked weight imprinting for few-shot segmentation. (2019).
  99. AMP: Adaptive Masked Proxies for Few-Shot Segmentation. In IEEE/CVF International Conference on Computer Vision (ICCV).
  100. Karen Simonyan and Andrew Zisserman. 2015. Very Deep Convolutional Networks for Large-Scale Image Recognition. In International Conference on Learning Representations (ICLR).
  101. Prototypical networks for few-shot learning. Advances in neural information processing systems (2017).
  102. Ridge-based vessel segmentation in color images of the retina. IEEE Transactions on Medical Imaging (2004).
  103. Attentional prototype inference for few-shot segmentation. Pattern Recognition (2023).
  104. Generalized few-shot semantic segmentation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
  105. Prior guided feature enrichment network for few-shot segmentation. IEEE transactions on pattern analysis and machine intelligence (2020).
  106. Repurposing gans for one-shot semantic part segmentation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
  107. The HAM10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions. Scientific Data (2018).
  108. Attention is all you need. Advances in neural information processing systems (2017).
  109. Matching Networks for One Shot Learning. In Advances in Neural Information Processing Systems.
  110. Variational prototype inference for few-shot semantic segmentation. In IEEE/CVF Winter Conference on Applications of Computer Vision (WACV).
  111. Few-shot semantic segmentation with democratic attention networks. In European Conference on Computer Vision (ECCV).
  112. LoveDA: A Remote Sensing Land-Cover Dataset for Domain Adaptive Semantic Segmentation. In Neural Information Processing Systems Track on Datasets and Benchmarks (NeurIPS).
  113. Panet: Few-shot image semantic segmentation with prototype alignment. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
  114. Images Speak in Images: A Generalist Painter for In-Context Visual Learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
  115. SegGPT: Towards Segmenting Everything In Context. In IEEE/CVF International Conference on Computer Vision (ICCV).
  116. Generalizing from a few examples: A survey on few-shot learning. ACM computing surveys (CSUR) (2020).
  117. CRIS: CLIP-Driven Referring Image Segmentation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
  118. iSAID: A Large-scale Dataset for Instance Segmentation in Aerial Images. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops.
  119. Learning meta-class memory for few-shot semantic segmentation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
  120. CLIP Driven Few-Shot Panoptic Segmentation. IEEE Access (2023).
  121. Scale-aware graph neural network for few-shot semantic segmentation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
  122. Few-Shot Semantic Segmentation With Cyclic Memory Network. In IEEE/CVF International Conference on Computer Vision (ICCV).
  123. Few-Shot Semantic Segmentation with Cyclic Memory Network. In IEEE/CVF International Conference on Computer Vision (ICCV).
  124. Few-shot semantic segmentation with cyclic memory network. In IEEE/CVF International Conference on Computer Vision (2021).
  125. Multilevel Features-Guided Network for Few-Shot Segmentation. MDPI Electronics (2022).
  126. Prototype mixture models for few-shot semantic segmentation. In European Conference on Computer Vision (ECCV).
  127. Brinet: Towards bridging the intra-class and inter-class gaps in one-shot segmentation. arXiv preprint arXiv:2008.06226 (2020).
  128. A new local transformation module for few-shot segmentation. In MultiMedia Modeling: 26th International Conference, MMM 2020, Daejeon, South Korea, January 5–8, 2020, Proceedings, Part II 26.
  129. Self-Guided and Cross-Guided Learning for Few-Shot Segmentation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
  130. Self-guided and cross-guided learning for few-shot segmentation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
  131. Pyramid graph networks with connection attentions for region-based one-shot semantic segmentation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
  132. Canet: Class-agnostic segmentation networks with iterative refinement and attentive few-shot learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
  133. Few-shot segmentation via cycle-consistent transformer. Advances in Neural Information Processing Systems (2021).
  134. Personalize Segment Anything Model with One Shot. In The Twelfth International Conference on Learning Representations (ICLR).
  135. Rich Embedding Features for One-Shot Semantic Segmentation. IEEE Transactions on Neural Networks and Learning Systems (2022).
  136. Sg-one: Similarity guidance network for one-shot semantic segmentation. IEEE transactions on cybernetics (2020).
  137. Datasetgan: Efficient labeled data factory with minimal human effort. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
  138. Pyramid scene parsing network. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
  139. Advancing Incremental Few-Shot Semantic Segmentation via Semantic-Guided Relation Alignment and Adaptation. In MultiMedia Modeling.
  140. ZegCLIP: Towards adapting CLIP for zero-shot semantic segmentation. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2023).
  141. Self-Supervised Tuning for Few-Shot Segmentation. In International Joint Conference on Artificial Intelligence (IJCAI).
  142. Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model. arXiv preprint arXiv:2401.09417 (2024).

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now