Multimodal Masked Autoencoder Pre-training for 3D MRI-Based Brain Tumor Analysis with Missing Modalities (2505.00568v2)

Abstract: Multimodal magnetic resonance imaging (MRI) constitutes the first line of investigation for clinicians in the care of brain tumors, providing crucial insights for surgery planning, treatment monitoring, and biomarker identification. Pre-training on large datasets have been shown to help models learn transferable representations and adapt with minimal labeled data. This behavior is especially valuable in medical imaging, where annotations are often scarce. However, applying this paradigm to multimodal medical data introduces a challenge: most existing approaches assume that all imaging modalities are available during both pre-training and fine-tuning. In practice, missing modalities often occur due to acquisition issues, specialist unavailability, or specific experimental designs on small in-house datasets. Consequently, a common approach involves training a separate model for each desired modality combination, making the process both resource-intensive and impractical for clinical use. Therefore, we introduce BM-MAE, a masked image modeling pre-training strategy tailored for multimodal MRI data. The same pre-trained model seamlessly adapts to any combination of available modalities, extracting rich representations that capture both intra- and inter-modal information. This allows fine-tuning on any subset of modalities without requiring architectural changes, while still benefiting from a model pre-trained on the full set of modalities. Extensive experiments show that the proposed pre-training strategy outperforms or remains competitive with baselines that require separate pre-training for each modality subset, while substantially surpassing training from scratch on several downstream tasks. Additionally, it can quickly and efficiently reconstruct missing modalities, highlighting its practical value. Code and trained models are available at: https://github.com/Lucas-rbnt/BM-MAE

Multimodal Masked Autoencoder Pre-training for 3D MRI-Based Brain Tumor Analysis with Missing Modalities

The paper introduces a novel approach to tackle the challenges of analyzing brain tumor images from 3D MRI scans when faced with missing modalities. This approach, termed Brain Multimodal Masked Autoencoder (BM-MAE), provides a self-supervised pre-training strategy that is both efficient and effective in handling incomplete modality scenarios, an issue often encountered in clinical practice.

Multimodal MRI is crucial in evaluating brain tumors, offering valuable insights that guide surgical planning, treatment monitoring, and biomarker identification. Typically, standard sequences such as T1-weighted, contrast-enhanced T1-weighted (T1c), T2-weighted, and Fluid Attenuated Inversion Recovery (FLAIR) are acquired, each contributing unique information to provide a comprehensive assessment. However, due to various factors, including acquisition limitations and protocol inconsistencies, some modalities might be unavailable, obstructing accurate analysis.

BM-MAE is designed to handle such situations elegantly. It is based on the Masked Autoencoder (MAE) approach but moves a step further by processing multimodal inputs. The model can be pre-trained across all modalities available and subsequently fine-tuned on any subset without requiring multiple architectures or retraining. This flexibility is particularly advantageous in medical imaging where data scarcity, often compounded by missing modalities, presents significant hurdles.

Methodology and Results

The methodology centers on self-supervised learning techniques to capture rich, transferable representations from multimodal MRI data. In the BM-MAE architecture, modalities are encoded into a shared latent space using a Vision Transformer-based encoder. Unlike conventional setups that require all modalities for both training and inference, the BM-MAE framework masks random patches across modalities, thereby reducing dependency on full modality availability and decreasing computational costs.

The encoder utilizes cortaneously learned representations to reconstruct the masked patches during pre-training. This design promotes generalization across different configurations, proving versatile in the absence of specific modality channels during fine-tuning.

Empirical evaluations across tasks such as glioma segmentation, subtyping for lower-grade gliomas versus glioblastoma, and survival prediction demonstrate BM-MAE’s efficacy. The experimental results reveal consistent improvements in model performance when BM-MAE pre-trained weights are initialized compared to training from scratch or using other pre-training methods like SimCLR or standard MAE. Notably, performances in segmentation and classification tasks showed statistically significant improvements, accentuating the importance of capturing cross-modal interactions.

Furthermore, BM-MAE's reconstruction abilities were illustrated, showcasing the model's potential in reconstructing missing modalities from available ones. This underlines the utility of the pre-training phase in learning representative features beyond mere memorization.

Implications and Future Directions

The implications of BM-MAE for clinical and research applications are multifold. It enables computational efficiency and ease of use in environments constrained by data availability, positioning it as a valuable tool in the integration of AI into medical imaging workflows. The architecture can be adapted for use in scenarios beyond brain tumor analysis, suggesting broader applicability across medical diagnostics involving multimodal imaging.

Future directions for BM-MAE could explore further enhancing reconstruction capabilities or incorporating advanced fusion techniques to refine modal interactions. Additionally, extending the model's pre-training to accommodate emerging imaging modalities, or integrating genetic and histopathological data, could provide a more holistic approach to diagnosis and treatment planning.

In conclusion, BM-MAE represents a robust framework for multimodal medical image analysis, addressing inherent challenges linked to missing data. Its application has promising potential to streamline and elevate AI-powered solutions in healthcare, contributing positively toward personalized medicine and improved patient outcomes.