Hi-Net: Hybrid-fusion Network for Multi-modal MR Image Synthesis (2002.05000v1)

Published 11 Feb 2020 in cs.CV and eess.IV

Abstract: Magnetic resonance imaging (MRI) is a widely used neuroimaging technique that can provide images of different contrasts (i.e., modalities). Fusing this multi-modal data has proven particularly effective for boosting model performance in many tasks. However, due to poor data quality and frequent patient dropout, collecting all modalities for every patient remains a challenge. Medical image synthesis has been proposed as an effective solution to this, where any missing modalities are synthesized from the existing ones. In this paper, we propose a novel Hybrid-fusion Network (Hi-Net) for multi-modal MR image synthesis, which learns a mapping from multi-modal source images (i.e., existing modalities) to target images (i.e., missing modalities). In our Hi-Net, a modality-specific network is utilized to learn representations for each individual modality, and a fusion network is employed to learn the common latent representation of multi-modal data. Then, a multi-modal synthesis network is designed to densely combine the latent representation with hierarchical features from each modality, acting as a generator to synthesize the target images. Moreover, a layer-wise multi-modal fusion strategy is presented to effectively exploit the correlations among multiple modalities, in which a Mixed Fusion Block (MFB) is proposed to adaptively weight different fusion strategies (i.e., element-wise summation, product, and maximization). Extensive experiments demonstrate that the proposed model outperforms other state-of-the-art medical image synthesis methods.

Authors (5)

Tao Zhou (398 papers)
Huazhu Fu (185 papers)
Geng Chen (115 papers)
Jianbing Shen (96 papers)
Ling Shao (244 papers)

Citations (235)

View on Semantic Scholar

Summary

Hi-Net: Hybrid-Fusion Network for Multi-modal MR Image Synthesis

This paper introduces a novel network architecture, Hi-Net, which addresses the challenge of synthesizing missing modalities in multi-modal Magnetic Resonance Imaging (MRI) datasets. The Hi-Net framework consists of several interconnected modules: a modality-specific network, a multi-modal fusion network, and an overview network. Together, they facilitate the generation of target-modalities from a subset of available MRI modalities. This paper presents a technical exposition of Hi-Net, demonstrating its capabilities in enhancing the performance of medical image synthesis tasks.

Core Components of Hi-Net

The authors propose a modular approach to the problem of missing MRI modalities:

Modality-Specific Network: Each modality receives its distinct network to retain individual modality features, leveraging an autoencoder structure to foster robust high-level feature learning. This network aims to preserve unique modality characteristics while serving as an input for subsequent fusion.
Multi-modal Fusion Network: The paper introduces an innovative layer-wise multi-modal fusion strategy that effectively integrates features across modalities. A key contribution is the Mixed Fusion Block (MFB), which adaptively weights three prevalent fusion strategies: element-wise summation, product, and maximization, thereby dynamically refining the fusion process.
Multi-modal Synthesis Network: Functioning as a generator within a GAN framework, this network synthesizes the missing modality images. The generator is designed to exploit fused features to produce realistic target-modality images, while the discriminator distinguishes between synthesized and authentic images.

Methodology and Experimentation

The methodology espoused in this paper is grounded in a rigorous experimental setup using the BraTs2018 dataset. The paper benchmarks its performance against contemporary synthesis models like Pix2pix and cycleGAN. The empirical results are presented with various metrics: Peak Signal-to-Noise Ratio (PSNR), Normalized Mean Squared Error (NMSE), and Structural Similarity Index Measurement (SSIM).

The Hi-Net demonstrates superior performance across the metrics:

Achieved a PSNR of 25.05 for the $T_1 + T_2 \rightarrow \text{Flair}$ task, outperforming other methods.
Exhibited a NMSE reduction to 0.0258 for the same task.
Attained a SSIM score of 0.8909, surpassing its competitors.

These results underscore Hi-Net's efficacy in synthesizing target modalities more precisely by effectively capitalizing on the interdependencies between different MRI modalities.

Implications and Future Directions

The work has significant implications for clinical settings, particularly where patient imaging is frequently incomplete due to constraints like resource limitations or patient dropout. By enhancing data availability through synthetic modalities, Hi-Net presents a valuable asset for improving diagnostic precision and guiding interventions.

Furthermore, the integration of Hi-Net into broader medical imaging frameworks can yield substantial benefits in personalized medicine, particularly as a tool for augmenting datasets to fine-tune other diagnostic models. Looking ahead, refining the fusion mechanics with advances in machine learning could enhance the generalizability and applicability of such synthesis networks. Future research might explore scaling Hi-Net's architecture for use across additional imaging modalities beyond MRI or its adaption into real-time clinical usage scenarios.

Overall, this in-depth exploration of Hi-Net illustrates a meticulously designed approach to improving the robustness and effectiveness of multi-modal medical image synthesis.

PDF Markdown

Related Papers

Find Related Papers