Multimodal Cardiovascular MR Imaging

• Multimodal CMR imaging is a comprehensive technique that integrates multiple pulse sequences and views to provide detailed assessments of cardiac anatomy, function, and tissue characteristics.
• It employs advanced reconstruction methodologies, including deep learning and physics-informed models, to enhance image quality and accelerate data acquisition.
• Large, standardized k-space datasets enable rigorous benchmarking, clinical validation, and improved diagnostic accuracy through objective performance metrics.

Multimodal cardiovascular magnetic resonance (CMR) imaging refers to the integrated acquisition, reconstruction, and analysis of cardiac MR data capturing diverse physical tissue properties, functional states, and anatomical planes. By leveraging multiple pulse sequences (modalities) and anatomical views, multimodal CMR provides comprehensive insight into cardiac morphology, function, perfusion, tissue composition, and flow. The field encompasses developments in sequence design, high-throughput k-space acquisition, advanced reconstruction (especially deep learning–based and physics-informed frameworks), data fusion, and joint interpretation strategies, and is increasingly supported by large, protocol-diverse datasets and universal machine learning approaches.

1. Modalities, Anatomical Coverage, and Acquisition Protocols

Multimodal CMR integrates several validated imaging contrasts within a single clinical workflow. The CMRxRecon2024 dataset specifies six main pulse sequences:

• Cine imaging (TrueFISP): Seven prescribed anatomical views, including long axis (LAX-2CH, 3CH, 4CH), short axis (SAX), LVOT, aortic transverse (Tra), and sagittal (Sag). Used for dynamic assessment of ventricular volumes, ejection fraction, wall motion; characterized by high blood–myocardium contrast and bright blood lumen.
• T1 mapping (MOLLI-FLASH): SAX views, providing pixel-wise native and post-contrast T1 relaxation maps for tissue characterization (fibrosis, infiltration).
• T2 mapping (T2prep-FLASH): SAX; quantitative maps sensitive to myocardial edema (e.g., myocarditis, infarction).
• Myocardial tagging (SPAMM-TrueFISP): SAX, visualizing grid/line "tag" patterns to enable calculation of regional myocardial strain and torsion.
• Phase-contrast flow (Venc-TrueFISP, “Flow2D”): Transverse plane, encoding through-plane velocities for quantifying valvular and vessel flow.
• Black-blood imaging (TSE): SAX; used for visualization of vessel wall and thrombus, characterized by blood signal nulling.

Data are acquired from full-heart coverage, with multi-channel (30–34) coil arrays, and retrospectively ECG-gated to sample multiple phases of the cardiac cycle. CMRxRecon2024 encompasses 330 healthy volunteers, with all subjects undergoing the full protocol across multiple modalities and anatomical views (Wang et al., 2024).

2. Dataset Resources and Standardization

Recent progress is driven by public release of large, standardized k-space CMR datasets. CMRxRecon2024 (Wang et al., 2024) and similar resources (e.g., MMCMR-427K (Wang et al., 25 Dec 2025); CMR×Recon (Wang et al., 2023)) provide:

• Raw multi-coil k-space data at scale, spanning all major clinical modalities, anatomical views, and multiple undersampling trajectories (Cartesian uniform, Gaussian variable-density, pseudo-radial).
• Rigorously annotated splits: e.g., CMRxRecon2024 (200 subjects training, 60 validation, 70 testing), ensuring diversity of sampling masks and imaging protocols.
• Detailed scan parameters (FOV, slice thickness, temporal resolution) and demographic metadata.
• Tools for processing, benchmarking, and DICOM-compatible export.
• Centralized performance metrics (PSNR, SSIM, RMSE, reconstruction time, radiologist scores) (Wang et al., 2024, Wang et al., 25 Dec 2025, Wang et al., 2023).

These datasets enable fair benchmarking of universal and contrast-specific reconstruction algorithms under a wide array of acceleration factors (4× to 24×), directly promoting methodological innovation and reproducibility.

3. Imaging Models and Reconstruction Methodologies

Multimodal CMR relies on solving the inverse problem for undersampled multi-coil acquisitions:

$y = F_u x + n$

where $y$ is the acquired k-space, $F_u$ is the undersampled, coil-wise Fourier encoding (incorporating sensitivity maps), $x$ is the unknown complex image, and $n$ is additive noise. Reconstruction is formulated as:

$\min_x \|F_u x - y\|_2^2 + \lambda R(x)$

with $R(x)$ a regularizer, chosen according to the modality and application:

Regularizer Type	Expression	Typical Use
ℓ₁-wavelet sparsity	$\| \Psi x \|_1$	General anatomical sparsity
Total Variation (TV)	$\| \nabla x \|_1$	Piecewise-smooth images
Low-rank	Nuclear-norm of dynamic (Casorati) matrix	Dynamic cine, mapping
Learned prior (deep net)	Implicit in θ($x$; $W$)	Universal, modality-specific

Universal machine learning frameworks leverage multi-domain, multi-contrast architectures, including:

• Shared encoder–decoder backbones with modality adapters (e.g., FiLM layers),
• Multi-branch networks for contrast/view specialization with shared lower layers,
• Cross-task consistency and perceptual losses,
• Adversarial domain adaptation for protocol-agnostic generalization,
• Physics-driven unrolled optimization (e.g., variational networks with iterative data consistency and regularization) (Wang et al., 2024, Wang et al., 25 Dec 2025). A typical variational network update at iteration $k$:

$$x^{(k+1)} = x^{(k)} - \tau F_u^H(F_u x^{(k)} - y) - \theta_k(x^{(k)}; W_k)$$

where $\theta_k(\cdot; W_k)$ is a learnable CNN prior and $\tau$ is a step size parameter.

4. Multimodal Representation Learning and Fusion Architectures

Beyond reconstruction, multimodal CMR supports advanced representation learning and clinical analytics:

• CMRformer (Qiu et al., 2023): Joint learning of CMR "videos" and free-text radiology reports via dual-encoder contrastive transformers. The visual encoder implements divided space–time attention (akin to Timesformer), while the text encoder is DistilBERT. The objective aligns image–text pairs via a symmetric temperature-softmax contrastive loss across large clinical datasets.
• Content-based retrieval and disease classification are supported directly from these multimodal representations, with notable gains when combining CINE and LGE, leveraging all four standard long- and short-axis views.
• Multi-modal architectures can ingest imaging, text, and tabular (EHR) features, with fusion strategies such as early, intermediate, late, or hybrid linear combination, as demonstrated in interpretable pipelines for hemodynamics assessment (Tripathi et al., 2024). Here, tensor-based spatio-temporal features are extracted from CMR, and key EHR predictors identified via graph attention networks, all combined by linear SVMs to guarantee interpretability.

5. Evaluation Metrics and Clinical Validation

Comprehensive assessment of multimodal CMR pipelines includes both quantitative fidelity metrics and clinical applicability:

• Quantitative metrics: PSNR, SSIM within specified regions of interest (myocardium, blood pool), RMSE for parametric maps (T1, T2), and slice- or subject-wise Dice coefficient for segmentation.
• Subjective and diagnostic metrics: Radiologist grades (blurring, artifact, interpretability), net benefit in decision curve analysis for clinical triage, and correlation with clinical measurements (e.g., Pearson correlation for myocardial infarct size, ejection fraction, or hemodynamic markers).
• Universal deep models, especially those trained on protocol- and view-diverse datasets, outperform classical compressed sensing by 2–5 dB in PSNR or 0.05–0.1 in SSIM, with sub-second inference per slice, even at high acceleration (Wang et al., 2024, Wang et al., 25 Dec 2025).
• Clinical studies confirm the preservation of key cardiac phenotypes (e.g., LVEF PCC > 0.97 @ 8× acceleration), reliable quantification of myocardial biomarkers (LGE mass, T1/T2 mapping), and reader confidence equivalent to fully sampled images (Wang et al., 25 Dec 2025, Tripathi et al., 2024).

6. Emerging Foundation Models and Universal Frameworks

The paradigm is shifting toward foundation models—large-scale, generalist architectures trained across modalities, protocols, and patient metadata:

• CardioMM on MMCMR-427K (Wang et al., 25 Dec 2025): Text-aware, physics-informed unrolled reconstruction model integrating CLIP-text metadata and k-space semantics, achieving robust zero-shot generalization across centers, scanners, and unseen acceleration patterns.
• ViTa (Zhang et al., 17 Apr 2025): Unified visual–tabular transformer embedding full 3D+T cine stacks and 100+ patient attributes for context-aware segmentation, cardiac phenotyping, and disease risk stratification via a CLIP-style contrastive backbone.
• Foundations established by these models promise rapid exam protocols (<5 min with 24× acceleration), cross-site harmonization, and universal downstream task support, including population-scale disease classification and individualized cardiac health assessment.

7. Technical Challenges and Future Directions

Outstanding challenges include:

• Reliable multimodal registration, particularly for unaligned sequences and pathologies with large geometric changes.
• Modality-specific and cross-protocol artifacts (motion, undersampling, domain shifts) requiring robust, adaptive priors.
• Efficient learning from weak or sparse supervision (text reports, structured EHRs) and data from heterogeneous imaging environments.
• Transitioning from supervised, task-specific algorithms to scalable, self-supervised pretraining and universally deployable models.
• Clinical validation over multi-center, multi-vendor datasets, and elucidation of model interpretability and regulatory compliance.

A plausible implication is that universal, metadata-aware, physics-informed foundation models—enabled by large multi-modal k-space resources—will drive the next generation of clinically accessible, high-throughput CMR, supporting not only reconstruction but also integrated phenotyping, diagnostics, and outcome prediction (Wang et al., 25 Dec 2025, Zhang et al., 17 Apr 2025, Wang et al., 2024, Tripathi et al., 2024).