Brain MR Image Synthesis with Multi-contrast Self-attention GAN

Abstract: Accurate and complete multi-modal Magnetic Resonance Imaging (MRI) is essential for neuro-oncological assessment, as each contrast provides complementary anatomical and pathological information. However, acquiring all modalities (e.g., T1c, T1n, T2, T2f) for every patient is often impractical due to time, cost, and patient discomfort, potentially limiting comprehensive tumour evaluation. We propose 3D-MC-SAGAN (3D Multi-Contrast Self-Attention generative adversarial network), a unified 3D multi-contrast synthesis framework that generates high-fidelity missing modalities from a single T2 input while explicitly preserving tumour characteristics. The model employs a multi-scale 3D encoder-decoder generator with residual connections and a novel Memory-Bounded Hybrid Attention (MBHA) block to capture long-range dependencies efficiently, and is trained with a WGAN-GP critic and an auxiliary contrast-conditioning branch to produce T2f, T1n, and T1c volumes within a single unified network. A frozen 3D U-Net-based segmentation module introduces a segmentation-consistency constraint to preserve lesion morphology. The composite objective integrates adversarial, reconstruction, perceptual, structural similarity, contrast-classification, and segmentation-guided losses to align global realism with tumour-preserving structure. Extensive evaluation on 3D brain MRI datasets demonstrates that 3D-MC-SAGAN achieves state-of-the-art quantitative performance and generates visually coherent, anatomically plausible contrasts with improved distribution-level realism. Moreover, it maintains tumour segmentation accuracy comparable to fully acquired multi-modal inputs, highlighting its potential to reduce acquisition burden while preserving clinically meaningful information.

• The paper introduces a unified conditional 3D GAN with a multi-scale encoder-decoder and Memory-Bounded Hybrid Attention to synthesize T2f, T1n, and T1c images from T2 input.
• Experimental evaluations demonstrate superior performance in PSNR, SSIM, MSE, and tumor Dice scores, surpassing baselines and even matching real modality segmentation accuracy.
• Incorporating segmentation-consistency loss and hybrid attention mechanisms ensures preservation of global anatomy and tumor morphology while reducing acquisition burden.

Brain MR Image Synthesis with Multi-Contrast Self-Attention GAN: Technical Summary

Motivation and Problem Statement

Comprehensive neuro-oncological assessment with MRI relies on acquiring multiple modalities—T1n, T2, T2f, and T1c—that provide complementary anatomical and pathological contrasts. However, routine acquisition of all modalities for every patient is constrained by time, cost, comfort, and sometimes, technical limitations. This presents a critical need for methods capable of synthesizing missing MRI contrasts, ensuring complete diagnostic information with reduced acquisition burden. Prior deep learning-based synthesis strategies, especially those based on 2D conditional GANs and pairwise translation approaches, suffer from local realism bias, poor global coherence, and lack of explicit tumor-preserving constraints. Most models are either restricted to one-to-one mapping or are computationally infeasible when extended to high-resolution 3D volumetric data.

Model Architecture and Methodology

3D-MC-SAGAN proposes a conditional 3D GAN for unified multi-contrast MRI synthesis. Key architectural elements are:

• Multi-Scale 3D Encoder-Decoder Generator: This backbone processes entire 3D brain volumes (not 2D slices or patches), mitigating through-plane discontinuities and supporting preservation of global anatomy and tumor morphology.
• Memory-Bounded Hybrid Attention (MBHA) Block: To address computational bottlenecks in full-resolution 3D self-attention, the MBHA adaptively downsamples queries and keys/values to fit explicit memory budgets ($T_q$, $T_{kv}$, $T_\text{attn}$). When volumetric size exceeds affordance, MBHA defaults to SE-only gating, ensuring robust operation without OOM errors. The block fuses spatial non-locality with global channel recalibration and is inserted throughout the encoder and decoder.
• Domain Conditioning for Unified Multi-Contrast Synthesis: The generator is conditioned on a learned one-hot code, allowing it to synthesize T2f, T1n, or T1c target volumes from a single T2 input within a unified parametric model.
• Patch-Based Critic (WGAN-GP): A conditional critic estimates Wasserstein distances and contrast-classification accuracy, with spectral normalization for stability. Adversarial and auxiliary losses align the distributions and enforce correct domain mapping.
• Frozen 3D U-Net Segmentor: Supervision incorporates a pretrained, fixed segmentor. Segmentation consistency loss (Soft Dice and BCE) ensures synthesized contrasts retain tumor boundaries and appearance.
• Composite Objective: The generator minimizes a weighted sum of: adversarial, domain-classification, segmentation-consistency, weighted $\ell_1$ reconstruction (with tumor emphasis), 3D perceptual (MedicalNet), and MS-SSIM losses. This loss cocktail is tuned via random search to optimize both image fidelity and downstream segmentation.

Experimental Results

Data and Protocol: Trained/evaluated on BraTS 2023 (1251 subjects, all standard contrasts, tumor masks). Inputs are T2; outputs are synthesized T2f, T1n, and T1c. Performance is measured by PSNR, SSIM, MSE for direct image similarity, MedicalNet Fréchet Distance (MFD) for distributional similarity, and Dice for downstream tumor segmentation.

Quantitative Results:

• Fidelity: 3D-MC-SAGAN achieves the highest mean PSNR, SSIM, and lowest MSE across all target modalities versus all state-of-the-art 2D/3D/patch/diffusion baselines. Example: On T2 $\rightarrow$ T1c, 3D-MC-SAGAN achieves 28.34 (PSNR), 0.953 (SSIM), and 0.0016 (MSE), outperforming alternatives.
• Distributional Realism: 3D-MC-SAGAN yields the lowest MFD, indicating that the distribution of generated volumes more closely matches real data in MedicalNet feature space.
• Downstream Segmentation: When the synthesizer's outputs are combined with T2 and passed to a segmentation model, tumor Dice reaches 0.8631—even surpassing the Dice with all real modalities (0.8618)—while all other generative methods show notable performance drops.
• Ablations: Removal of MBHA, perceptual, and segmentation-consistency terms significantly hurts both fidelity and segmentation metrics. The segmentation-consistency loss is especially crucial, with Dice dropping from 0.8631 to 0.8237 without it.

Discussion and Implications

Advancements:

• 3D-MC-SAGAN successfully unifies multi-contrast MR synthesis in a single 3D network with contrast-conditional architecture, obviating the need for modality-pairwise models and capturing three different contrasts concurrently. This reduces complexity and deployment burden.
• MBHA enables effective, scalable self-attention for large 3D volumes, mitigating high memory costs and providing robust long-range context crucial for anatomical and pathological structure preservation.

Numerical Claims:

• The paper asserts that the proposed framework achieves "state-of-the-art" performance over all tested benchmarks and demonstrates the only generative approach to preserve or improve tumour segmentation accuracy relative to real data.

Contradictory/Surprising Findings:

• The model's synthesis is so structurally faithful that downstream segmentation is at least as robust as with ground-truth modalities—a claim not supported by any baseline, suggesting that previous models fail to adequately preserve clinically actionable anatomy even when apparent per-pixel fidelity is high.

Theoretical and Practical Significance

By incorporating explicit tumor-consistency via segmentation supervision, memory-aware attention, and feature-level perceptual guidance, 3D-MC-SAGAN transcends the limitations of pixel-level losses and achieves robust pathology-preserving multi-modal MRI synthesis. The results indicate that it is possible to reduce the acquisition protocol to a single T2 sequence while reconstructing radiologically viable surrogates for the missing contrasts. This has significant clinical implications: efficient resource use, reduced patient burden, and the potential for retrospective harmonization of incomplete or legacy datasets.

On a theoretical level, the work advances 3D self-attention engineering for medical data and demonstrates how hybrid loss functions and supervision from frozen networks can guide generative models toward preservation of task-critical semantic structures beyond intensity similarity.

Future Directions

Areas for further development include:

• Extending MBHA with dynamic budget allocation or alternative sparse/self-similarity mechanisms to process even larger spatial domains.
• Investigating transfer learning to support adaptation across scanner domains, patient populations, and imaging protocols.
• Integrating uncertainty quantification to enable risk-aware deployment in clinical pipelines.
• Exploring the combination of this approach with new generative paradigms such as diffusion or flow-based models, but with explicit semantic constraints.

Conclusion

3D-MC-SAGAN represents a significant advance in volumetric brain MR image synthesis. Through joint attention mechanisms, domain-conditional synthesis, and explicit segmentation consistency, the framework produces multi-contrast MR volumes from single-modality input that match or exceed ground-truth data in terms of both image fidelity and support for automated tumor segmentation. This positions 3D-MC-SAGAN as a compelling candidate for adoption in clinical workflow augmentation, diagnostic pre-processing, and retrospective data curation.