Interactive Medical Image Segmentation using Deep Learning with Image-specific Fine-tuning (1710.04043v1)

Abstract: Convolutional neural networks (CNNs) have achieved state-of-the-art performance for automatic medical image segmentation. However, they have not demonstrated sufficiently accurate and robust results for clinical use. In addition, they are limited by the lack of image-specific adaptation and the lack of generalizability to previously unseen object classes. To address these problems, we propose a novel deep learning-based framework for interactive segmentation by incorporating CNNs into a bounding box and scribble-based segmentation pipeline. We propose image-specific fine-tuning to make a CNN model adaptive to a specific test image, which can be either unsupervised (without additional user interactions) or supervised (with additional scribbles). We also propose a weighted loss function considering network and interaction-based uncertainty for the fine-tuning. We applied this framework to two applications: 2D segmentation of multiple organs from fetal MR slices, where only two types of these organs were annotated for training; and 3D segmentation of brain tumor core (excluding edema) and whole brain tumor (including edema) from different MR sequences, where only tumor cores in one MR sequence were annotated for training. Experimental results show that 1) our model is more robust to segment previously unseen objects than state-of-the-art CNNs; 2) image-specific fine-tuning with the proposed weighted loss function significantly improves segmentation accuracy; and 3) our method leads to accurate results with fewer user interactions and less user time than traditional interactive segmentation methods.

• The paper introduces an interactive segmentation framework that fine-tunes CNNs on each test image to enhance accuracy.
• It integrates user inputs like bounding boxes and scribbles to adapt the model for unseen object classes.
• Experimental results on fetal MRI and brain tumor MR scans show improved robustness and reduced user interaction time.

Interactive Medical Image Segmentation using Deep Learning with Image-specific Fine-tuning

The paper "Interactive Medical Image Segmentation using Deep Learning with Image-specific Fine-tuning" by Guotai Wang et al. presents a novel framework that seeks to address the limitations of current CNN-based segmentation models in clinical practice. The research highlights three primary challenges: inadequate accuracy for unseen classes, the need for image-specific adaptation, and the inefficiency in resource demands during real-time user interaction.

Overview

The proposed framework integrates CNNs into a user-interactive pipeline with bounding box and scribble inputs for segmentation. This method is termed BIFSeg. The central concept is to adapt CNN models to specific test images through fine-tuning, both in supervised and unsupervised modes, thereby enhancing segmentation accuracy with minimal user input.

Key Contributions

1. Framework Development: A deep learning-based interactive segmentation framework for both 2D and 3D images incorporating CNNs into a binary segmentation pipeline.
2. Image-specific Fine-tuning: The introduction of this fine-tuning allows the CNN to adapt to each test image individually. This adaptability is achieved with or without additional user scribbles.
3. Weighted Loss Function: This considers network-based and interaction-based uncertainties during fine-tuning, consequently improving robustness.
4. Unseen Object Segmentation: BIFSeg can handle segmentation of objects absent from the training phase, distinguishing it from other segmentation approaches.

Experimental Results

Two specific applications were explored: segmenting multiple organs from fetal MRI and brain tumors from MR sequences. The results indicate:

• The model demonstrated robustness in segmenting unseen object classes, surpassing standard CNNs in accuracy.
• Image-specific fine-tuning consistently improved segmentation outcomes over CRF refinement and other baseline methods.
• The interactive segmentation framework reduced user interaction time and effort compared to traditional methods.

Implications and Future Directions

This research has substantial implications for medical imaging practices, potentially streamlining clinical workflows by reducing reliance on annotated training sets and enabling more flexible model deployment. The ability to manage previously unseen classes without extensive retraining is notable and could facilitate broader applicability across diverse medical imaging challenges.

Future research could delve into optimizing the framework for real-time applications, exploring alternative loss functions to capture a wider range of uncertainties, and extending this approach to other complex medical datasets.

Overall, the framework presented promotes a significant advance in interactive medical image segmentation, providing a practical solution for clinical adaptation and unseen object segmentation challenges.