Multi-dimension unified Swin Transformer for 3D Lesion Segmentation in Multiple Anatomical Locations (2309.01823v1)

Abstract: In oncology research, accurate 3D segmentation of lesions from CT scans is essential for the modeling of lesion growth kinetics. However, following the RECIST criteria, radiologists routinely only delineate each lesion on the axial slice showing the largest transverse area, and delineate a small number of lesions in 3D for research purposes. As a result, we have plenty of unlabeled 3D volumes and labeled 2D images, and scarce labeled 3D volumes, which makes training a deep-learning 3D segmentation model a challenging task. In this work, we propose a novel model, denoted a multi-dimension unified Swin transformer (MDU-ST), for 3D lesion segmentation. The MDU-ST consists of a Shifted-window transformer (Swin-transformer) encoder and a convolutional neural network (CNN) decoder, allowing it to adapt to 2D and 3D inputs and learn the corresponding semantic information in the same encoder. Based on this model, we introduce a three-stage framework: 1) leveraging large amount of unlabeled 3D lesion volumes through self-supervised pretext tasks to learn the underlying pattern of lesion anatomy in the Swin-transformer encoder; 2) fine-tune the Swin-transformer encoder to perform 2D lesion segmentation with 2D RECIST slices to learn slice-level segmentation information; 3) further fine-tune the Swin-transformer encoder to perform 3D lesion segmentation with labeled 3D volumes. The network's performance is evaluated by the Dice similarity coefficient (DSC) and Hausdorff distance (HD) using an internal 3D lesion dataset with 593 lesions extracted from multiple anatomical locations. The proposed MDU-ST demonstrates significant improvement over the competing models. The proposed method can be used to conduct automated 3D lesion segmentation to assist radiomics and tumor growth modeling studies. This paper has been accepted by the IEEE International Symposium on Biomedical Imaging (ISBI) 2023.

The paper "Multi-dimension unified Swin Transformer for 3D Lesion Segmentation in Multiple Anatomical Locations" presents a novel model for the segmentation of lesions in 3D computed tomography (CT) scans, a crucial task in oncology research for analyzing tumor growth and progression. The challenge addressed by the authors arises from the lack of comprehensive 3D annotations, given that radiologists typically mark the largest transverse area of a lesion in 2D RECIST (Response Evaluation Criteria In Solid Tumors) slices and only a limited number of lesions in full 3D for research purposes. This results in a dataset with plentiful unlabeled 3D volumes and labeled 2D images, but scarce labeled 3D volumes, posing a significant difficulty for training effective 3D segmentation models.

To tackle this, the authors propose the Multi-Dimension Unified Swin Transformer (MDU-ST), a model that integrates a Swin Transformer encoder with a Convolutional Neural Network (CNN) decoder. The Swin Transformer, known for its shifted-window mechanism, enables the model to process both 2D and 3D inputs and learn the respective semantic information using the same encoder framework.

The methodology is structured into three distinct stages for leveraging available data:

1. Self-Supervised Learning: The first stage involves pretraining the Swin Transformer encoder using self-supervised tasks on the vast amount of unlabeled 3D lesion volumes. This step aims to capture the underlying anatomical patterns of lesions without the necessity for labeled data.
2. 2D Fine-Tuning: In the second stage, the pretrained Swin Transformer encoder is fine-tuned on 2D RECIST slices to learn detailed slice-level segmentation. This process helps the model incorporate specific features and nuances required for accurate 2D lesion delineation.
3. 3D Fine-Tuning: The final stage includes fine-tuning the encoder further using labeled 3D volumes to perform 3D segmentation. This training is crucial for integrating volumetric information and refining the 3D accuracy of the model.

To evaluate the performance of their approach, the authors use metrics such as the Dice Similarity Coefficient (DSC) and the Hausdorff Distance (HD) on a dataset comprising 593 lesions from various anatomical regions. The results demonstrate that the proposed MDU-ST model significantly outperforms existing competition, indicating its potential efficacy in automated 3D lesion segmentation tasks.

Overall, the MDU-ST framework effectively bridges the gap between the abundance of unlabeled 3D CT scans and the limited labeled data. Its application can substantially support radiomics and tumor growth modeling studies, potentially enhancing the precision and automation of lesion segmentation in clinical and research settings.