Path-SAM2: Transfer SAM2 for digital pathology semantic segmentation (2408.03651v2)

Abstract: The semantic segmentation task in pathology plays an indispensable role in assisting physicians in determining the condition of tissue lesions. With the proposal of Segment Anything Model (SAM), more and more foundation models have seen rapid development in the field of image segmentation. Recently, SAM2 has garnered widespread attention in both natural image and medical image segmentation. Compared to SAM, it has significantly improved in terms of segmentation accuracy and generalization performance. We compared the foundational models based on SAM and found that their performance in semantic segmentation of pathological images was hardly satisfactory. In this paper, we propose Path-SAM2, which for the first time adapts the SAM2 model to cater to the task of pathological semantic segmentation. We integrate the largest pretrained vision encoder for histopathology (UNI) with the original SAM2 encoder, adding more pathology-based prior knowledge. Additionally, we introduce a learnable Kolmogorov-Arnold Networks (KAN) classification module to replace the manual prompt process. In three adenoma pathological datasets, Path-SAM2 has achieved state-of-the-art performance.This study demonstrates the great potential of adapting SAM2 to pathology image segmentation tasks. We plan to release the code and model weights for this paper at: https://github.com/simzhangbest/SAM2PATH

• The paper introduces a novel integration of the UNI encoder into SAM2 to boost semantic segmentation performance in pathological images.
• The paper demonstrates the effective use of a learnable KAN module that automates prompt generation, reducing manual intervention.
• The paper achieves improved segmentation metrics, such as DSC and IOU, across public datasets, indicating robust performance in digital pathology.

Overview of Path-SAM2: Semantic Segmentation in Digital Pathology

The paper on Path-SAM2 presents a novel approach for addressing challenges in semantic segmentation in digital pathology by leveraging advancements in foundation models. The authors propose Path-SAM2, a model that enhances the Segment Anything Model 2 (SAM2) framework by integrating additional pathological encoders and a unique classification module based on Kolmogorov-Arnold Networks (KAN). This research aims to significantly improve the segmentation accuracy of pathological images, an essential task in the digital pathology domain critical for disease diagnosis and prognosis.

Path-SAM2 incorporates the UNI model, a self-supervised encoder pre-trained on over 100 million diagnostic H&E-stained histology slides, to infuse SAM2 with more domain-specific pathological information. The integration is achieved by utilizing a joint encoder configuration that fuses features from both SAM2 and UNI, thereby enhancing the model's ability to handle pathology-specific segmentation tasks.

Moreover, the paper introduces a learnable KAN classification module that replaces traditional, manual prompt processes in segmentation tasks. The KAN architecture promises better interpretability and more effective parameter utilization compared to standard MLP-based classification methods. By using this module, the segmentation process becomes less dependent on manual intervention, offering more precise segmentation results.

Key Contributions

The primary contributions of this paper can be summarized as follows:

1. Integration of UNI Encoder: The incorporation of the largest existing pre-trained pathology model, UNI, into SAM2 significantly boosts the model's ability to assimilate domain-specific knowledge, demonstrating an improved capacity for semantic segmentation in pathological images.
2. Innovative Use of KAN: Introducing a KAN classification module into the SAM2 architecture represents a noteworthy shift toward automated prompt generation, potentially reducing the manual effort required during segmentation tasks.
3. Proven Segmentation Performance: The model exhibits strong quantitative results across various public datasets used for pathology segmentation tasks, achieving notable improvements in both Dice Similarity Coefficient (DSC) and Intersection Over Union (IOU) metrics compared to existing baselines.

Implications and Future Developments

In practice, Path-SAM2's approach to enhancing digital pathology workflows holds promise for more accurate and efficient disease diagnosis. The integration of a pathology-specific encoder not only propels the capabilities of SAM-related models but also underscores the importance of domain-specific pre-training when generalizing large foundation models to specialized tasks like medical imaging.

The use of KAN for learnable prompts suggests a future direction in which models can adapt more dynamically to varied inputs without additional manual supervision, paving the way for autonomous systems in digital diagnostics.

Looking forward, this research encourages exploration into extending the Path-SAM2 framework to other areas in the medical field. The integration of specialized pre-trained models, alongside innovative computational modules like KAN, exemplifies a potent strategy for advancing AI's role in healthcare. Such developments could facilitate the transition towards fully automated analysis systems, which are expected to have significant implications for resource-constrained environments by mitigating the workload on medical professionals and improving patient care strategies.

The release of the code and model weights, as indicated in the paper, will further enable validation, collaborative development, and broader application of these findings, fostering continued progress in AI-driven medical research.