SAM2-Adapter: Evaluating & Adapting Segment Anything 2 in Downstream Tasks: Camouflage, Shadow, Medical Image Segmentation, and More (2408.04579v2)

Abstract: The advent of large models, also known as foundation models, has significantly transformed the AI research landscape, with models like Segment Anything (SAM) achieving notable success in diverse image segmentation scenarios. Despite its advancements, SAM encountered limitations in handling some complex low-level segmentation tasks like camouflaged object and medical imaging. In response, in 2023, we introduced SAM-Adapter, which demonstrated improved performance on these challenging tasks. Now, with the release of Segment Anything 2 (SAM2), a successor with enhanced architecture and a larger training corpus, we reassess these challenges. This paper introduces SAM2-Adapter, the first adapter designed to overcome the persistent limitations observed in SAM2 and achieve new state-of-the-art (SOTA) results in specific downstream tasks including medical image segmentation, camouflaged (concealed) object detection, and shadow detection. SAM2-Adapter builds on the SAM-Adapter's strengths, offering enhanced generalizability and composability for diverse applications. We present extensive experimental results demonstrating SAM2-Adapter's effectiveness. We show the potential and encourage the research community to leverage the SAM2 model with our SAM2-Adapter for achieving superior segmentation outcomes. Code, pre-trained models, and data processing protocols are available at http://tianrun-chen.github.io/SAM-Adaptor/

• The paper demonstrates that the SAM2-Adapter significantly improves segmentation by integrating task-specific adapters into the advanced SAM2 model.
• The methodology leverages multi-resolution Transformers and inputs like patch embeddings and high-frequency features for precise camouflaged and shadow detection.
• Experimental results show superior performance in camouflaged object, shadow, and polyp segmentation, setting new benchmarks for challenging tasks.

SAM2-Adapter: Evaluating Adapting Segment Anything 2 in Downstream Tasks

The paper "SAM2-Adapter: Evaluating Adapting Segment Anything 2 in Downstream Tasks: Camouflage, Shadow, Medical Image Segmentation, and More" by Tianrun Chen et al. presents a comprehensive evaluation of the Segment Anything 2 (SAM2) model enhanced by the SAM2-Adapter. Building on the previously proposed SAM-Adapter, this paper aims to assess and improve the performance of SAM2, a more advanced foundation model, in intricate segmentation tasks such as camouflaged object detection, shadow detection, and medical image segmentation.

Introduction

The introduction highlights the transformative impact of foundation models like SAM on AI research, particularly in image segmentation. Despite the success of the original SAM in various segmentation scenarios, the model faced challenges in low-level structural segmentation tasks. To mitigate these limitations, the SAM-Adapter was introduced, significantly enhancing SAM’s performance on challenging tasks such as camouflaged object detection. With the release of SAM2, which boasts an improved architecture and a larger training corpus, the authors revisit these challenges and introduce the SAM2-Adapter to further advance segmentation task performance.

Methodology

Leveraging SAM2 as the Backbone

The core of SAM2-Adapter lies in its integration with the advanced SAM2 model, specifically its multi-resolution hierarchical Transformer architecture. The methodology centers around harnessing the powerful image encoder and mask decoder of SAM2, with enhancements provided by a series of specialized adapters. These adapters allow for the injection of task-specific knowledge into the model, thereby improving its ability to handle various downstream tasks.

Input Task-Specific Information

The approach involves two types of visual knowledge, patch embedding ($F_{pe}$) and high-frequency components ($F_{hfc}$), which are beneficial in capturing intricate details necessary for tasks like camouflage and shadow detection. The authors employ a multi-adapter configuration tailored to different stages of SAM2, thereby leveraging the hierarchical features for more precise segmentation outcomes.

Experimental Setup

The authors conducted extensive experiments across multiple datasets to evaluate the performance of SAM2-Adapter:

• Camouflaged Object Detection: Datasets such as COD10K, CHAMELEON, and CAMO were utilized.
• Shadow Detection: The ISTD dataset was employed.
• Polyp Segmentation: The kvasir-SEG dataset served as the basis for this medical imaging task.

Results and Discussion

The results indicate that SAM2-Adapter consistently outperforms SAM and other state-of-the-art methods across all evaluated tasks and metrics. Key findings include:

1. Camouflaged Object Detection: SAM2-Adapter achieved higher S-measure, E-measure, and lower MAE compared to previous methods, including the original SAM and SAM2 models.
2. Shadow Detection: Improved Balance Error Rate (BER) demonstrated the effectiveness of SAM2-Adapter in accurately identifying shadow regions.
3. Polyp Segmentation: Enhanced mean Dice score (mDice) and mean Intersection-over-Union (mIoU) metrics underscored the model’s robustness in medical image segmentation tasks.

Implications

The paper illustrates that while SAM2 improves upon SAM, challenges in downstream tasks persist. SAM2-Adapter addresses these challenges by integrating multi-resolution hierarchical features and providing a more precise and robust framework for segmentation tasks. The findings have significant implications for the application of foundation models in specialized fields such as medical imaging, where precision is paramount.

Future Directions

The exploration of SAM2-Adapter opens avenues for further research into integrating more sophisticated knowledge adaptation techniques and extending the framework’s applicability to other domains. The adaptability and composability of SAM2-Adapter make it a compelling tool for researchers seeking to enhance segmentation performance across diverse applications.

Conclusion

SAM2-Adapter marks a significant advancement in the adaptation of foundation models for specific downstream tasks, demonstrating notable improvements in performance and establishing new benchmarks in segmentation tasks. The release of the paper's code, pre-trained models, and data processing protocols encourages the research community to build on these findings and further explore the potential of large pre-trained models in specialized applications.

The SAM2-Adapter effectively leverages the SAM2 model’s strengths, delivering superior segmentation outcomes and offering valuable insights into the future of image segmentation research.

For further details and access to the provided resources, visit the project page: SAM2-Adapter.