Translating and Segmenting Multimodal Medical Volumes with Cycle- and Shape-Consistency Generative Adversarial Network (1802.09655v2)

Abstract: Synthesized medical images have several important applications, e.g., as an intermedium in cross-modality image registration and as supplementary training samples to boost the generalization capability of a classifier. Especially, synthesized computed tomography (CT) data can provide X-ray attenuation map for radiation therapy planning. In this work, we propose a generic cross-modality synthesis approach with the following targets: 1) synthesizing realistic looking 3D images using unpaired training data, 2) ensuring consistent anatomical structures, which could be changed by geometric distortion in cross-modality synthesis and 3) improving volume segmentation by using synthetic data for modalities with limited training samples. We show that these goals can be achieved with an end-to-end 3D convolutional neural network (CNN) composed of mutually-beneficial generators and segmentors for image synthesis and segmentation tasks. The generators are trained with an adversarial loss, a cycle-consistency loss, and also a shape-consistency loss, which is supervised by segmentors, to reduce the geometric distortion. From the segmentation view, the segmentors are boosted by synthetic data from generators in an online manner. Generators and segmentors prompt each other alternatively in an end-to-end training fashion. With extensive experiments on a dataset including a total of 4,496 CT and magnetic resonance imaging (MRI) cardiovascular volumes, we show both tasks are beneficial to each other and coupling these two tasks results in better performance than solving them exclusively.

• The paper proposes a novel GAN framework integrating cycle- and shape-consistency losses to synthesize and segment 3D medical images.
• It employs an end-to-end 3D CNN architecture with alternating training of generators and segmentors to mutually reinforce both tasks.
• Experimental evaluation on CT and MRI volumes shows notable segmentation improvements with Dice scores reaching 74.4% and 73.2% respectively.

Translating and Segmenting Multimodal Medical Volumes with GANs

The paper introduces a sophisticated approach to synthesize and segment medical imaging data from different modalities, specifically addressing the challenges inherent in 3D medical imaging. The proposed method employs a Cycle- and Shape-Consistency Generative Adversarial Network (GAN) to tackle the dual tasks of cross-modality image synthesis and volume segmentation effectively.

Key Contributions

The paper outlines a methodology that enhances the translation and segmentation of multimodal medical volumes by integrating generative modeling and segmentation networks. This work leverages the capabilities of GANs to synthesize realistic 3D images by focusing on crucial aspects like anatomical structure preservation and addressing the lack of paired training data in medical imaging tasks.

Detailed Methodology

The authors propose a framework composed of cycle-consistency and shape-consistency losses to ensure that synthetic medical images are realistic and anatomically consistent. While the cycle-consistency loss aids in maintaining the identity of images during domain translation, it is limited by potential geometric distortions. To address this, the authors introduce a shape-consistency scheme supported by segmentors that enforces anatomical structure preservation, a novel approach in the context of unpaired medical data synthesis.

The method employs an end-to-end 3D convolutional neural network (CNN) architecture that concurrently enhances image synthesis and segmentation tasks. A unique perspective of this work is how the generators and segmentors are trained alternately, allowing them to benefit mutually from each other's optimizations.

Experimental Evaluation

The approach is robustly evaluated with a dataset comprising 4,496 cardiovascular 3D volumes from CT and MRI modalities. The proposed method demonstrates significant improvements in medical image segmentation when coupled with synthetic data, outperforming traditional isolated approaches. Notably, the integration of synthetic data in an online fashion provides consistent performance enhancements over offline augmentation techniques.

Numerical Results

The quality of segmentation is quantitatively assessed using Dice scores, showing a notable improvement when synthetic data is used in conjunction with real data. For instance, the proposed method yields a 74.4% and 73.2% Dice score for CT and MRI segmentations respectively, showing a marked increase compared to baseline and offline augmented strategies.

Implications and Future Work

This research holds potential for significant implications in clinical settings, particularly in improving imaging diagnostics where data scarcity is a concern. The ability to synthesize realistic cross-modality medical images enhances the available dataset without the burdensome requirement of paired data, facilitating better-informed clinical decisions.

Theoretically, this work opens avenues for future research in enhancing GAN architectures to address domain-specific challenges such as multi-organ systems and pathology-specific segmentations. Additionally, reducing the distribution gap between synthetic and real data remains an open research area following the findings of this paper.

The paper presents a substantial contribution to the field of medical imaging, particularly in optimizing segmentation networks by leveraging synthetic data generated through GAN architectures. The integration of cycle- and shape-consistency frameworks marks a significant step towards effective multimodal medical data synthesis and utilization.