Missing MRI Pulse Sequence Synthesis using Multi-Modal Generative Adversarial Network (1904.12200v3)

Abstract: Magnetic resonance imaging (MRI) is being increasingly utilized to assess, diagnose, and plan treatment for a variety of diseases. The ability to visualize tissue in varied contrasts in the form of MR pulse sequences in a single scan provides valuable insights to physicians, as well as enabling automated systems performing downstream analysis. However many issues like prohibitive scan time, image corruption, different acquisition protocols, or allergies to certain contrast materials may hinder the process of acquiring multiple sequences for a patient. This poses challenges to both physicians and automated systems since complementary information provided by the missing sequences is lost. In this paper, we propose a variant of generative adversarial network (GAN) capable of leveraging redundant information contained within multiple available sequences in order to generate one or more missing sequences for a patient scan. The proposed network is designed as a multi-input, multi-output network which combines information from all the available pulse sequences, implicitly infers which sequences are missing, and synthesizes the missing ones in a single forward pass. We demonstrate and validate our method on two brain MRI datasets each with four sequences, and show the applicability of the proposed method in simultaneously synthesizing all missing sequences in any possible scenario where either one, two, or three of the four sequences may be missing. We compare our approach with competing unimodal and multi-modal methods, and show that we outperform both quantitatively and qualitatively.

• The paper introduces MM-GAN, a multi-modal generative adversarial network that synthesizes missing MRI sequences using implicit conditioning and curriculum learning.
• It employs a UNet-based generator and a PatchGAN discriminator to leverage redundancy among sequences, significantly reducing mean squared error and boosting SSIM compared to prior methods.
• The approach promises efficient clinical integration by generating complete datasets from incomplete scans, streamlining diagnostics and treatment planning.

Missing MRI Pulse Sequence Synthesis Using Multi-Modal Generative Adversarial Network

The paper "Missing MRI Pulse Sequence Synthesis using Multi-Modal Generative Adversarial Network" by Anmol Sharma and Ghassan Hamarneh presents a novel approach to address the challenges of missing MRI sequences. The proposed solution employs a multi-modal generative adversarial network (MM-GAN) that synthesizes the missing sequences by leveraging available information from the existing ones. This methodology is particularly significant in medical imaging, where MRI scans are utilized for diagnostics and treatment planning but can be incomplete due to various constraints.

Methodology

The core of the paper is the introduction of a multi-input, multi-output network designed to generate missing MRI pulse sequences. Unlike traditional unimodal synthesis methods, MM-GAN can simultaneously synthesize multiple missing sequences in any scenario where up to three of the four sequences are missing. It leverages the redundancy and complementarity in existing sequences to ensure the generation of other required sequences in a single forward pass. The architecture integrates a UNet-based generator and a PatchGAN discriminator, tailored for the synthesis tasks.

The authors introduce implicit conditioning (IC) and curriculum learning (CL) strategies to enhance the model's performance. IC includes techniques such as imputing zero images for missing sequences and selective loss computation, which simplifies the training process and improves synthesis accuracy. CL is employed to gradually introduce the network to more complex scenarios, thereby stabilizing and optimizing learning.

Evaluation and Results

The authors validate their methodology using two brain MRI datasets, ISLES2015 and BraTS2018, covering a spectrum of clinical scenarios. MM-GAN demonstrates superior performance in synthesizing missing sequences compared to state-of-the-art methods like REPLICA and MM-Synthesis. Specifically, the paper showcases MM-GAN's ability to significantly reduce mean squared error (MSE) and maintain structural similarity index metric (SSIM) scores, which are critical indicators of image synthesis quality.

Results illustrate that multi-modal synthesis offers a substantial edge over single-modal techniques. The use of multiple sequences in the input not only improves synthesis fidelity but also decreases the computational overhead compared to maintaining multiple unimodal models. This single model approach to synthesizing all possible scenarios without retraining highlights the potential efficiency gains in practical clinical settings.

Implications and Future Directions

The research highlights the potential for MM-GAN to transform medical imaging protocols by filling in missing MRI data, thus preserving the quality of downstream analysis pipelines without needing to alter them fundamentally. This synthesis could especially benefit automated systems relying on complete datasets to function accurately.

The authors suggest future work could extend to refining the up-scaling capabilities showcased, where the model could enhance the resolution of MRI scans. Another potential is further integration into clinical decision-making processes, enabling more seamless usage of incomplete datasets. The paper opens avenues for making MRI imaging more robust in diverse clinical conditions where resource and time constraints prevent acquiring all necessary sequences.

In conclusion, this paper contributes significantly to the domain of medical image synthesis, pushing the boundaries of what can be achieved through generative networks and paving the way for versatile, efficient patient diagnostics and treatment planning methodologies.