Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
119 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

CAT-SAM: Conditional Tuning for Few-Shot Adaptation of Segment Anything Model (2402.03631v3)

Published 6 Feb 2024 in cs.CV

Abstract: The recent Segment Anything Model (SAM) has demonstrated remarkable zero-shot capability and flexible geometric prompting in general image segmentation. However, SAM often struggles when handling various unconventional images, such as aerial, medical, and non-RGB images. This paper presents CAT-SAM, a ConditionAl Tuning network that adapts SAM toward various unconventional target tasks with just few-shot target samples. CAT-SAM freezes the entire SAM and adapts its mask decoder and image encoder simultaneously with a small number of learnable parameters. The core design is a prompt bridge structure that enables decoder-conditioned joint tuning of the heavyweight image encoder and the lightweight mask decoder. The bridging maps the prompt token of the mask decoder to the image encoder, fostering synergic adaptation of the encoder and the decoder with mutual benefits. We develop two representative tuning strategies for the image encoder which leads to two CAT-SAM variants: one injecting learnable prompt tokens in the input space and the other inserting lightweight adapter networks. Extensive experiments over 11 unconventional tasks show that both CAT-SAM variants achieve superior target segmentation performance consistently even under the very challenging one-shot adaptation setup. Project page: https://xiaoaoran.github.io/projects/CAT-SAM

Definition Search Book Streamline Icon: https://streamlinehq.com
References (53)
  1. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716–23736, 2022.
  2. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020.
  3. Adaptformer: Adapting vision transformers for scalable visual recognition. Advances in Neural Information Processing Systems, 35:16664–16678, 2022.
  4. Sam-adapter: Adapting segment anything in underperformed scenes. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, pages 3367–3375, 2023a.
  5. Sam fails to segment anything?–sam-adapter: Adapting sam in underperformed scenes: Camouflage, shadow, and more. arXiv preprint arXiv:2304.09148, 2023b.
  6. Boundary iou: Improving object-centric image segmentation evaluation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15334–15342, 2021.
  7. Sam-med2d. arXiv preprint arXiv:2308.16184, 2023.
  8. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
  9. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.
  10. Clip-adapter: Better vision-language models with feature adapters. International Journal of Computer Vision, pages 1–15, 2023.
  11. Parameter-efficient transfer learning for nlp. In International Conference on Machine Learning, pages 2790–2799. PMLR, 2019.
  12. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2021.
  13. Fully convolutional networks for multisource building extraction from an open aerial and satellite imagery data set. IEEE Transactions on geoscience and remote sensing, 57(1):574–586, 2018.
  14. Segment anything is not always perfect: An investigation of sam on different real-world applications. arXiv preprint arXiv:2304.05750, 2023.
  15. Scaling up visual and vision-language representation learning with noisy text supervision. In International conference on machine learning, pages 4904–4916. PMLR, 2021.
  16. Visual prompt tuning. In European Conference on Computer Vision, pages 709–727. Springer, 2022.
  17. Segment anything in high quality. arXiv preprint arXiv:2306.01567, 2023.
  18. Maple: Multi-modal prompt learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19113–19122, 2023.
  19. Segment anything. arXiv preprint arXiv:2304.02643, 2023.
  20. Foodsam: Any food segmentation. IEEE Transactions on Multimedia, pages 1–14, 2023.
  21. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691, 2021.
  22. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461, 2019.
  23. Semantic-sam: Segment and recognize anything at any granularity. arXiv preprint arXiv:2307.04767, 2023a.
  24. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597, 2023b.
  25. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190, 2021.
  26. Deep interactive thin object selection. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 305–314, 2021.
  27. Explicit visual prompting for low-level structure segmentations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19434–19445, 2023.
  28. Segment anything in medical images. arXiv preprint arXiv:2304.12306, 2023.
  29. V-net: Fully convolutional neural networks for volumetric medical image segmentation. In 2016 fourth international conference on 3D vision (3DV), pages 565–571. Ieee, 2016.
  30. Volodymyr Mnih. Machine Learning for Aerial Image Labeling. PhD thesis, University of Toronto, 2013.
  31. OpenAI. Gpt-4 technical report, 2023.
  32. Kvasir: A multi-class image dataset for computer aided gastrointestinal disease detection. In Proceedings of the 8th ACM on Multimedia Systems Conference, pages 164–169, 2017.
  33. Highly accurate dichotomous image segmentation. In European Conference on Computer Vision, pages 38–56. Springer, 2022.
  34. Improving language understanding by generative pre-training. 2018.
  35. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.
  36. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PMLR, 2021.
  37. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485–5551, 2020.
  38. Learning multiple visual domains with residual adapters. Advances in neural information processing systems, 30, 2017.
  39. Efficient parametrization of multi-domain deep neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8119–8127, 2018.
  40. Development of a digital image database for chest radiographs with and without a lung nodule: receiver operating characteristic analysis of radiologists’ detection of pulmonary nodules. American Journal of Roentgenology, 174(1):71–74, 2000.
  41. The marine debris dataset for forward-looking sonar semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3741–3749, 2021.
  42. Large-scale training of shadow detectors with noisily-annotated shadow examples. In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part VI 14, pages 816–832. Springer, 2016.
  43. Sam-clip: Merging vision foundation models towards semantic and spatial understanding. arXiv preprint arXiv:2310.15308, 2023.
  44. Hrsid: A high-resolution sar images dataset for ship detection and instance segmentation. Ieee Access, 8:120234–120254, 2020.
  45. Side adapter network for open-vocabulary semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2945–2954, 2023.
  46. Recognize any regions. arXiv preprint arXiv:2311.01373, 2023.
  47. Towards high-resolution salient object detection. In Proceedings of the IEEE/CVF international conference on computer vision, pages 7234–7243, 2019.
  48. Faster segment anything: Towards lightweight sam for mobile applications. arXiv preprint arXiv:2306.14289, 2023a.
  49. Customized segment anything model for medical image segmentation. arXiv preprint arXiv:2304.13785, 2023.
  50. Personalize segment anything model with one shot. arXiv preprint arXiv:2305.03048, 2023b.
  51. Fast segment anything. arXiv preprint arXiv:2306.12156, 2023.
  52. Conditional prompt learning for vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16816–16825, 2022a.
  53. Learning to prompt for vision-language models. International Journal of Computer Vision, 130(9):2337–2348, 2022b.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (8)
  1. Aoran Xiao (24 papers)
  2. Weihao Xuan (14 papers)
  3. Heli Qi (9 papers)
  4. Yun Xing (14 papers)
  5. Ruijie Ren (5 papers)
  6. Xiaoqin Zhang (39 papers)
  7. Shijian Lu (151 papers)
  8. Ling Shao (244 papers)
Citations (5)
View on Semantic Scholar

Summary

An In-Depth Analysis of "Conditional Tuning Network for Few-Shot Adaptation of Segmentation Anything Model"

The paper "Conditional Tuning Network for Few-Shot Adaptation of Segmentation Anything Model" introduces the CAT-SAM, a robust framework aimed at enhancing the adaptability of the Segment Anything Model (SAM) for domains with limited training data availability. This work systematically addresses the challenges of few-shot learning in segmentation tasks across diverse image modalities, emphasizing both technical intricacies and empirical efficacy.

SAM, lauded for its zero-shot segmentation capabilities, encounters significant performance degradation when applied to specialized domains like aerial or medical imagery that deviate from its training distribution. Such degradation results primarily due to SAM's reliance on large annotated datasets for supervised adaptation, a requirement deemed impractical in data-scarce scenarios. The proposed CAT-SAM framework illuminates an innovative pathway by introducing a conditional tuning mechanism that simultaneously caters to the adaptation of both the image encoder and mask decoder within SAM.

Core Contributions

  1. Decoder-Conditioned Joint Tuning: The paper proposes a novel tuning strategy that forms a synergistic linkage between SAM's image encoder and mask decoder. This strategy is operationalized through a prompt bridge that effectively mitigates the tuning imbalance inherent due to the size disparity between the encoder and decoder modules. This approach elegantly reconciles the parameter-efficient learning philosophy with practical domain adaptation.
  2. Integration with Prompt Tuning Methods: CAT-SAM is further realized in two variants—CAT-SAM-T employing prompt tokens and CAT-SAM-A with lightweight adapter networks. These variants leverage the prompt bridge to conditionally guide adaptation, ensuring the efficient interplay between domain-specific feature extraction and zero-shot potential retention.
  3. Comprehensive Experimental Validation: The evaluation spans 11 datasets covering both RGB and non-RGB imaging domains, providing a factual basis for the effectiveness of CAT-SAM. Remarkably, even within the confines of a one-shot setup, CAT-SAM exhibits marked improvements over existing methodologies like HQ-SAM. The paper presents strong numerical evidence—across tasks like building, road, polyp, and intricate structural segmentation—solidifying CAT-SAM's primacy in few-shot segmentation paradigm.

Implications and Future Prospects

The dual strategies within CAT-SAM underscore a pivotal shift towards more flexible and scalable model architectures in segmentation, particularly in fields where data acquisition is costly or infeasible. By circumventing the exhaustive dependency on annotated datasets, this work pushes the boundaries in domain adaptation, paving the way for broader applicability in real-world applications including autonomous navigation and medical diagnostics.

Theoretical foundations laid by the decoder-conditioned joint tuning suggest potential enhancements through further exploration of hyperparameter tuning and network architecture designs. The versatility seen in CAT-SAM's robust handling of non-RGB imagery such as Sonar and SAR should spark interest in extending this framework to multimodal fusion techniques.

Moreover, the continuous adaptation and learning approach embodied in CAT-SAM holds great promise for future AI systems that require incremental learning without catastrophic forgetting. As such, advancing this methodology could significantly contribute to the development of AI agents capable of seamlessly transferring learning across various domains.

In conclusion, the paper offers a meticulous and technically sound strategy to elevate the segmentation performance while reducing dependency on extensive data annotations. By proposing a conditional tuning network, the authors not only replicate impressive empirical results but also inspire a new direction for adapting foundational models in heterogeneous data environments.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown
Github Logo Streamline Icon: https://streamlinehq.com

GitHub

  1. CAT-SAM
Youtube Logo Streamline Icon: https://streamlinehq.com

YouTube