Papers
Topics
Authors
Recent
Search
2000 character limit reached

An Alternative to WSSS? An Empirical Study of the Segment Anything Model (SAM) on Weakly-Supervised Semantic Segmentation Problems

Published 2 May 2023 in cs.CV | (2305.01586v2)

Abstract: The Segment Anything Model (SAM) has demonstrated exceptional performance and versatility, making it a promising tool for various related tasks. In this report, we explore the application of SAM in Weakly-Supervised Semantic Segmentation (WSSS). Particularly, we adapt SAM as the pseudo-label generation pipeline given only the image-level class labels. While we observed impressive results in most cases, we also identify certain limitations. Our study includes performance evaluations on PASCAL VOC and MS-COCO, where we achieved remarkable improvements over the latest state-of-the-art methods on both datasets. We anticipate that this report encourages further explorations of adopting SAM in WSSS, as well as wider real-world applications.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (53)
  1. Weakly supervised learning of instance segmentation with inter-pixel relations. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2209–2218, Long Beach, CA, USA, 2019. Computer Vision Foundation / IEEE.
  2. Learning pixel-level semantic affinity with image-level supervision for weakly supervised semantic segmentation. pages 4981–4990. Computer Vision Foundation / IEEE, 2018.
  3. What’s the point: Semantic segmentation with point supervision. pages 549–565, 2016.
  4. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. 40(4):834–848, 2017.
  5. Self-supervised image-specific prototype exploration for weakly supervised semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4288–4298, June 2022.
  6. Class re-activation maps for weakly-supervised semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 969–978, 2022.
  7. Out-of-candidate rectification for weakly supervised semantic segmentation, 2023.
  8. Boxsup: Exploiting bounding boxes to supervise convolutional networks for semantic segmentation. pages 1635–1643, 2015.
  9. Imagenet: A large-scale hierarchical image database. pages 248–255. Computer Vision Foundation / IEEE, 2009.
  10. Weakly supervised semantic segmentation by pixel-to-prototype contrast. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4320–4329, 2022.
  11. Employing multi-estimations for weakly-supervised semantic segmentation. pages 332–348, 2020.
  12. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  13. Integral object mining via online attention accumulation. pages 2070–2079, 2019.
  14. L2g: A simple local-to-global knowledge transfer framework for weakly supervised semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16886–16896, June 2022.
  15. Simple does it: Weakly supervised instance and semantic segmentation. pages 876–885. Computer Vision Foundation / IEEE, 2017.
  16. Discriminative region suppression for weakly-supervised semantic segmentation. pages 1754–1761, 2021.
  17. Segment anything, 2023.
  18. Unlocking the potential of ordinary classifier: Class-specific adversarial erasing framework for weakly supervised semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 6994–7003, October 2021.
  19. Reducing information bottleneck for weakly supervised semantic segmentation. Advances in Neural Information Processing Systems, 34, 2021.
  20. Anti-adversarially manipulated attributions for weakly and semi-supervised semantic segmentation. pages 4071–4080. Computer Vision Foundation / IEEE, 2021.
  21. Weakly supervised semantic segmentation using out-of-distribution data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16897–16906, 2022.
  22. Bbam: Bounding box attribution map for weakly supervised semantic and instance segmentation, 2021.
  23. Railroad is not a train: Saliency as pseudo-pixel supervision for weakly supervised semantic segmentation. pages 5495–5505. Computer Vision Foundation / IEEE, 2021.
  24. Expansion and shrinkage of localization for weakly-supervised semantic segmentation. arXiv preprint arXiv:2209.07761, 2022.
  25. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International Conference on Machine Learning, pages 12888–12900. PMLR, 2022.
  26. Uncertainty estimation via response scaling for pseudo-mask noise mitigation in weakly-supervised semantic segmentation. arXiv preprint arXiv:2112.07431, 2021.
  27. Pseudo-mask matters in weakly-supervised semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 6964–6973, October 2021.
  28. Scribblesup: Scribble-supervised convolutional networks for semantic segmentation. pages 3159–3167. Computer Vision Foundation / IEEE, 2016.
  29. Clip is also an efficient segmenter: A text-driven approach for weakly supervised semantic segmentation, 2023.
  30. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. 2023.
  31. Fully convolutional multi-class multiple instance learning. arXiv preprint arXiv:1412.7144, 2014.
  32. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PMLR, 2021.
  33. Weakly-supervised semantic segmentation with visual words learning and hybrid pooling. International Journal of Computer Vision, 130(4):1127–1144, 2022.
  34. Learning affinity from attention: End-to-end weakly-supervised semantic segmentation with transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16846–16855, 2022.
  35. Box-driven class-wise region masking and filling rate guided loss for weakly supervised semantic segmentation. pages 3136–3145. Computer Vision Foundation / IEEE, 2019.
  36. Context decoupling augmentation for weakly supervised semantic segmentation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 7004–7014, 2021.
  37. Ecs-net: Improving weakly supervised semantic segmentation by using connections between class activation maps. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 7283–7292, October 2021.
  38. 3d guided weakly supervised semantic segmentation. In Proceedings of the Asian Conference on Computer Vision, 2020.
  39. Inferring the class conditional response map for weakly supervised semantic segmentation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 2878–2887, 2022.
  40. Getam: Gradient-weighted element-wise transformer attention map for weakly-supervised semantic segmentation. arXiv preprint arXiv:2112.02841, 2021.
  41. On regularized losses for weakly-supervised cnn segmentation. pages 507–522, 2018.
  42. Learning random-walk label propagation for weakly-supervised semantic segmentation. pages 7158–7166. Computer Vision Foundation / IEEE, 2017.
  43. Weakly-supervised semantic segmentation by iteratively mining common object features. pages 1354–1362. Computer Vision Foundation / IEEE, 2018.
  44. Self-supervised equivariant attention mechanism for weakly supervised semantic segmentation. pages 12275–12284. Computer Vision Foundation / IEEE, 2020.
  45. Object region mining with adversarial erasing: A simple classification to semantic segmentation approach. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1568–1576, 2017.
  46. Embedded discriminative attention mechanism for weakly supervised semantic segmentation. pages 16765–16774. Computer Vision Foundation / IEEE, 2021.
  47. C2am: Contrastive learning of class-agnostic activation map for weakly supervised object localization and semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 989–998, June 2022.
  48. Leveraging auxiliary tasks with affinity learning for weakly supervised semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 6984–6993, October 2021.
  49. Multi-class token transformer for weakly supervised semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4310–4319, 2022.
  50. Non-salient region object mining for weakly supervised semantic segmentation. pages 2623–2632. Computer Vision Foundation / IEEE, 2021.
  51. Causal intervention for weakly-supervised semantic segmentation. arXiv preprint arXiv:2009.12547, 2020.
  52. Complementary patch for weakly supervised semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 7242–7251, October 2021.
  53. Splitting vs. merging: Mining object regions with discrepancy and intersection loss for weakly supervised semantic segmentation. 2020.
Citations (19)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up

GitHub

  1. GitHub - weixuansun/wsss_sam (17 stars)