Medical SAM Adapter: Adapting Segment Anything Model for Medical Image Segmentation (2304.12620v7)

Abstract: The Segment Anything Model (SAM) has recently gained popularity in the field of image segmentation due to its impressive capabilities in various segmentation tasks and its prompt-based interface. However, recent studies and individual experiments have shown that SAM underperforms in medical image segmentation, since the lack of the medical specific knowledge. This raises the question of how to enhance SAM's segmentation capability for medical images. In this paper, instead of fine-tuning the SAM model, we propose the Medical SAM Adapter (Med-SA), which incorporates domain-specific medical knowledge into the segmentation model using a light yet effective adaptation technique. In Med-SA, we propose Space-Depth Transpose (SD-Trans) to adapt 2D SAM to 3D medical images and Hyper-Prompting Adapter (HyP-Adpt) to achieve prompt-conditioned adaptation. We conduct comprehensive evaluation experiments on 17 medical image segmentation tasks across various image modalities. Med-SA outperforms several state-of-the-art (SOTA) medical image segmentation methods, while updating only 2\% of the parameters. Our code is released at https://github.com/KidsWithTokens/Medical-SAM-Adapter.

Overview of "Medical SAM Adapter: Adapting Segment Anything Model for Medical Image Segmentation"

The paper "Medical SAM Adapter: Adapting Segment Anything Model for Medical Image Segmentation" addresses the challenges faced in applying the Segment Anything Model (SAM) in the field of medical image segmentation. Despite SAM's efficacy in general image segmentation tasks, its performance is notably suboptimal for medical images due to a lack of domain-specific knowledge. The authors propose a novel adaptation framework, the Medical SAM Adapter (Med-SA), which incorporates medical domain expertise into SAM using a parameter-efficient fine-tuning (PEFT) method. This adaptation enhances SAM's capability to handle complexities inherent in medical imaging, such as low contrast and intricate tissue boundaries, by updating merely 2% of the model's parameters.

Methodological Contributions

Med-SA introduces two key techniques to modify the SAM architecture effectively:

1. Space-Depth Transpose (SD-Trans): Designed to adapt the 2D SAM to handle 3D medical image data, SD-Trans transposes spatial dimensions to depth, facilitating the processing of 3D information common in medical imaging modalities like MRI and CT.
2. Hyper-Prompting Adapter (HyP-Adpt): This technique enables prompt-conditioned adaptation by generating weight maps from visual prompts (e.g., clinician annotations), thereby enhancing the model’s capacity to interactively adapt to user inputs in real-time.

Together, these innovations allow Med-SA to achieve significant improvements over SAM and previously established methods while maintaining computational efficiency.

Experimental Results

The authors rigorously evaluated Med-SA across 17 medical image segmentation tasks involving diverse modalities such as CT, MRI, ultrasound, and others. Notably, Med-SA displayed superior performance on the BTCV abdominal multi-organ segmentation benchmark, outperforming state-of-the-art systems like Swin-UNetr by 2.9% in Dice score. Additionally, Med-SA excelled in medical domains such as optic disc/cup, brain tumor, thyroid nodule, and melanoma segmentation, illustrating robust generalization across various medical imaging scenarios.

Implications and Future Directions

The proposed Med-SA framework exemplifies a significant step in effectively integrating generic segmentation models into specialized fields such as medical imaging. By leveraging techniques from both computer vision and natural language processing, the paper opens avenues for more extensive applications of foundational models in medical contexts. The paper’s findings suggest that future developments in AI could focus on refining PEFT techniques and further exploring domain-specific adaptations to enhance the versatility and accuracy of large vision models in clinical settings.

Conclusion

In conclusion, the Medical SAM Adapter stands as a promising adaptation of the SAM model tailored for the unique challenges of medical image segmentation. Its ability to substantially outperform existing solutions with minimal parameter updates signals a meaningful advancement in the quest for efficient, scalable medical imaging solutions. The methodologies introduced will likely inspire further research into adaptive models across diverse application domains, underscoring the potential of PEFT strategies in improving the accessibility and applicability of AI technologies in healthcare.