Evaluation of Algorithms for Multi-Modality Whole Heart Segmentation: An Open-Access Grand Challenge (1902.07880v1)

Abstract: Knowledge of whole heart anatomy is a prerequisite for many clinical applications. Whole heart segmentation (WHS), which delineates substructures of the heart, can be very valuable for modeling and analysis of the anatomy and functions of the heart. However, automating this segmentation can be arduous due to the large variation of the heart shape, and different image qualities of the clinical data. To achieve this goal, a set of training data is generally needed for constructing priors or for training. In addition, it is difficult to perform comparisons between different methods, largely due to differences in the datasets and evaluation metrics used. This manuscript presents the methodologies and evaluation results for the WHS algorithms selected from the submissions to the Multi-Modality Whole Heart Segmentation (MM-WHS) challenge, in conjunction with MICCAI 2017. The challenge provides 120 three-dimensional cardiac images covering the whole heart, including 60 CT and 60 MRI volumes, all acquired in clinical environments with manual delineation. Ten algorithms for CT data and eleven algorithms for MRI data, submitted from twelve groups, have been evaluated. The results show that many of the deep learning (DL) based methods achieved high accuracy, even though the number of training datasets was limited. A number of them also reported poor results in the blinded evaluation, probably due to overfitting in their training. The conventional algorithms, mainly based on multi-atlas segmentation, demonstrated robust and stable performance, even though the accuracy is not as good as the best DL method in CT segmentation. The challenge, including the provision of the annotated training data and the blinded evaluation for submitted algorithms on the test data, continues as an ongoing benchmarking resource via its homepage (\url{www.sdspeople.fudan.edu.cn/zhuangxiahai/0/mmwhs/}).

• The paper demonstrates that deep learning approaches achieve Dice scores above 0.9, outperforming traditional multi-atlas methods on CT data despite occasional variability.
• It analyzes 12 algorithms from an open-access challenge, comparing deep learning architectures like 3D U-Net with robust atlas-based techniques for whole heart segmentation.
• Findings highlight challenges in MRI segmentation quality and suggest future improvements through larger datasets, data augmentation, and hybrid methodological approaches.

Evaluation of Algorithms for Multi-Modality Whole Heart Segmentation: An Open-Access Grand Challenge

The paper presents a comprehensive evaluation of algorithms for multi-modality whole heart segmentation (WHS) as part of the Multi-Modality Whole Heart Segmentation (MM-WHS) challenge, associated with MICCAI 2017. This challenge serves as an open-access benchmarking platform that provides clinical datasets and evaluation frameworks for algorithm development and comparison. The objective is to automate the segmentation of whole heart substructures, crucial for clinical applications. The challenge encompasses 120 three-dimensional cardiac images, equally split between CT and MRI modalities, obtained from clinical environments.

Overview of the Challenge and Methodologies

Whole heart segmentation aims to delineate critical substructures like the left ventricle (LV), right ventricle (RV), left atrium (LA), right atrium (RA), myocardium of LV, ascending aorta (AO), and pulmonary artery (PA). The difficulty of WHS stems from cardiac anatomical complexity, shape variability, and varied image quality. The challenge evaluates twelve selected algorithms submitted by research teams, split almost evenly between deep learning (DL) and traditional multi-atlas segmentation (MAS) frameworks.

The paper provides a detailed analysis of these twelve methods, with a focus on contrasting deep learning-based approaches and conventional MAS methods. In general, the DL methods employed architectures like 3D U-Net, fully convolutional networks (FCN), and variants that incorporate prior anatomical knowledge to address large volumetric data challenge. MAS methods, by comparison, leverage atlas registration and selection techniques ensuring robust but slower processing.

Performance and Implications

The results indicate that DL-based methods have achieved higher segmentation accuracy, particularly on CT datasets. The best performing methods achieved Dice scores above 0.9, indicating the algorithm's high fidelity in discerning cardiac substructures. However, these DL models also showed more variability with occasional poor results due to overfitting, likely attributed to limited training data.

In contrast, the MAS-based methods, while not achieving top-tier accuracy, offered robustness across cases and generated more plausible anatomical shapes. The MRI datasets, inherently more challenging due to image quality and anatomical variability, resulted in comparatively lower segmentation scores across all methods. This emphasizes the need for methodological advancements, especially in MRI-based segmentation.

Discussion and Future Scope

The challenge underscores the growing maturity and adaptability of DL frameworks to medical image analysis. Nevertheless, this paper reveals limitations, as exemplified by MRI variability and occasional unrealistic DL outcomes. Future work in WHS algorithm development could benefit from larger datasets, potentially through techniques like data augmentation or synthetic data generation via Generative Adversarial Networks (GANs). Hybrid approaches combining DL techniques with shape priors or other anatomical constraints could enhance segmentation accuracy and consistency.

Moreover, the combination of ensemble learning strategies could potentially leverage the strengths of both DL and MAS methods. The MM-WHS challenge provides an ongoing resource for stimulating such collaborative improvements, laying the groundwork for fully automated, clinically adaptable whole heart segmentation solutions.