Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
175 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Deep Instruction Tuning for Segment Anything Model (2404.00650v2)

Published 31 Mar 2024 in cs.CV

Abstract: Recently, Segment Anything Model (SAM) has become a research hotspot in the fields of multimedia and computer vision, which exhibits powerful yet versatile capabilities on various (un) conditional image segmentation tasks. Although SAM can support different types of segmentation prompts, we note that, compared to point- and box-guided segmentations, it performs much worse on text-instructed tasks, e.g., referring image segmentation (RIS). In this paper, we argue that deep text instruction tuning is key to mitigate such shortcoming caused by the shallow fusion scheme in its default light-weight mask decoder. To address this issue, we propose two simple yet effective deep instruction tuning (DIT) methods for SAM, one is end-to-end and the other is layer-wise. With minimal modifications, DITs can directly transform the image encoder of SAM as a stand-alone vision-language learner in contrast to building another deep fusion branch, maximizing the benefit of its superior segmentation capability. Extensive experiments on three highly competitive benchmark datasets of RIS show that a simple end-to-end DIT can improve SAM by a large margin, while the layer-wise DIT can further boost the performance to state-of-the-art with much less data and training expenditures. Our code is released at: https://github.com/wysnzzzz/DIT.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (69)
  1. Segnet: A deep convolutional encoder-decoder architecture for image segmentation. TPAMI, 2017.
  2. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. TPAMI, 2017.
  3. Transvg: End-to-end visual grounding with transformers. In ICCV, 2021.
  4. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv, 2018.
  5. Vlt: Vision-language transformer and query generation for referring segmentation. TPAMI, 2022.
  6. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv, 2020.
  7. Encoder fusion network with co-attention embedding for referring image segmentation. In CVPR, 2021.
  8. Ppt: Pre-trained prompt tuning for few-shot learning. arXiv, 2021.
  9. Ptr: Prompt tuning with rules for text classification. AI Open, 2022.
  10. Mask r-cnn. In ICCV, 2017.
  11. Segmentation from natural language expressions. In ECCV, 2016.
  12. Beyond one-to-one: Rethinking the referring image segmentation. In ICCV, 2023.
  13. Referring image segmentation via cross-modal progressive comprehension. In CVPR, 2020.
  14. Visual prompt tuning. In ECCV, 2022.
  15. Locate then segment: A strong pipeline for referring image segmentation. In CVPR, 2021.
  16. Referitgame: Referring to objects in photographs of natural scenes. In EMNLP, 2014.
  17. Restr: Convolution-free referring image segmentation using transformers. In CVPR, 2022.
  18. Vilt: Vision-and-language transformer without convolution or region supervision. In ICML, 2021.
  19. Panoptic segmentation. In CVPR, 2019.
  20. Segment anything. arXiv, 2023.
  21. Visual genome: Connecting language and vision using crowdsourced dense image annotations. IJCV, 2017.
  22. Lisa: Reasoning segmentation via large language model. arXiv, 2023.
  23. Read-only prompt optimization for vision-language few-shot learning. In ICCV, 2023.
  24. Grounded language-image pre-training. In CVPR, 2022.
  25. Referring transformer: A one-step approach to multi-task visual grounding. NeurIPS, 2021.
  26. Referring image segmentation via recurrent refinement networks. In CVPR, 2018.
  27. Attention-guided unified network for panoptic segmentation. In CVPR, 2019.
  28. Mail: A unified mask-image-language trimodal network for referring image segmentation. arXiv, 2021.
  29. Microsoft coco: Common objects in context. In ECCV, 2014.
  30. Recurrent multimodal interaction for referring image segmentation. In ICCV, 2017.
  31. Instance-specific feature propagation for referring segmentation. ACM MM, 2022.
  32. Gres: Generalized referring expression segmentation. In CVPR, 2023a.
  33. Gres: Generalized referring expression segmentation. In CVPR, 2023b.
  34. Learning to assemble neural module tree networks for visual grounding. In ICCV, 2019.
  35. Polyformer: Referring image segmentation as sequential polygon generation. In CVPR, 2023c.
  36. Polyformer: Referring image segmentation as sequential polygon generation. In CVPR, 2023d.
  37. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM CSUR, 2023e.
  38. P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks. arXiv, 2021.
  39. Universal segmentation at arbitrary granularity with language instruction. arXiv, 2023f.
  40. Fully convolutional networks for semantic segmentation. In CVPR, 2015.
  41. Cascade grouped attention network for referring expression segmentation. In ACM MM, 2020a.
  42. Multi-task collaborative network for joint referring expression comprehension and segmentation. In CVPR, 2020b.
  43. Towards efficient visual adaption via structural re-parameterization. arXiv, 2023.
  44. Generation and comprehension of unambiguous object descriptions. In CVPR, 2016.
  45. Dynamic multimodal instance segmentation guided by natural language queries. In ECCV, 2018.
  46. Modeling context between objects for referring expression understanding. In ECCV, 2016.
  47. Training language models to follow instructions with human feedback. NeurIPS, 2022.
  48. U-net: Convolutional networks for biomedical image segmentation. In MICCAI, 2015.
  49. Conditional convolutions for instance segmentation. In ECCV, 2020.
  50. Llama: Open and efficient foundation language models. arXiv, 2023.
  51. Attention is all you need. NeurIPS, 2017.
  52. Solov2: Dynamic and fast instance segmentation. NeurIPS, 2020.
  53. Cris: Clip-driven referring image segmentation. In CVPR, 2022.
  54. Finetuned language models are zero-shot learners. arXiv, 2021.
  55. Segment every reference object in spatial and temporal spaces. In ICCV, 2023a.
  56. Approximated prompt tuning for vision-language pre-trained models. arXiv, 2023b.
  57. Segformer: Simple and efficient design for semantic segmentation with transformers. NeurIPS, 2021.
  58. Meta compositional referring expression segmentation. In CVPR, 2023a.
  59. Bridging vision and language encoders: Parameter-efficient tuning for referring image segmentation. In ICCV, 2023b.
  60. Universal instance perception as object discovery and retrieval. In CVPR, 2023.
  61. Bottom-up shift and reasoning for referring image segmentation. In CVPR, 2021.
  62. Lavt: Language-aware vision transformer for referring image segmentation. In CVPR, 2022.
  63. Semantics-aware dynamic localization and refinement for referring image segmentation. arXiv, 2023.
  64. Modeling context in referring expressions. In ECCV, 2016.
  65. Unleashing text-to-image diffusion models for visual perception. arXiv, 2023.
  66. Conditional prompt learning for vision-language models. In CVPR, 2022a.
  67. Learning to prompt for vision-language models. IJCV, 2022b.
  68. Seqtr: A simple yet universal network for visual grounding. In ECCV, 2022.
  69. Segment everything everywhere all at once. NeurIPS, 2024.
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now