OmniMRI: Unified MRI Interpretation

• OmniMRI is a unified vision-language model that integrates MRI reconstruction, segmentation, abnormality detection, and report generation into a consolidated workflow.
• It employs a dual-encoder and dual-decoder Transformer architecture with a multi-stage training paradigm over diverse datasets to build robust cross-modal representations.
• The model streamlines fragmented radiology processes by merging imaging and clinical language data, enhancing diagnostic consistency and workflow efficiency.

OmniMRI describes a unified, generalist vision–language foundation model for comprehensive MRI interpretation that is trained on heterogeneous, large-scale datasets and designed to perform tasks across the complete MRI workflow—including acquisition, image reconstruction, segmentation, abnormality detection, diagnostic suggestion, and report generation. Unlike traditional modality-specific systems, OmniMRI consolidates fragmented pipelines, integrates imaging with clinical language data, and employs a multi-stage training paradigm to build transferable representations and robust cross-modal reasoning capabilities (He et al., 24 Aug 2025).

1. Model Architecture and Training Paradigm

OmniMRI is constructed with a dual-encoder and dual-decoder architecture centered around an autoregressive multimodal Transformer backbone. The system comprises:

• A Swin Transformer-based vision encoder for hierarchical visual feature extraction from MRI data (2D slices/3D volumes).
• A lightweight clinical language encoder/tokenizer that processes structured metadata, radiologist annotations, reports, and task-based prompts.

The outputs (tokens) from both encoders are combined and input into a unified Transformer (adapted from Qwen2.537), which employs multimodal self-attention and a mixture-of-experts feedforward network to enhance parameter efficiency and scalability. The model uses two output heads:

• A diffusion-based image decoder for pixel-level tasks (e.g., image reconstruction, anatomical segmentation).
• A text decoder for semantic outputs, including diagnostic suggestions and radiology report generation.

Training proceeds in four explicit stages:

1. Self-supervised vision pretraining on raw image-only MRI data using masked image modeling, patch-level contrastive objectives, and instance discrimination clustered by acquisition context.
2. Vision-language alignment via a CLIP-style contrastive loss, projecting text into the same space as visual features, optimizing for high cosine similarity between paired image and text tokens:

$$L_\text{align} = -\log\left(\frac{\exp(\text{sim}(v,t))}{\sum_j \exp(\text{sim}(v,t_j))}\right)$$

1. Unified multimodal pretraining where interleaved visual and language tokens are modeled with a next-token prediction objective.
2. Multi-task instruction tuning that exposes the model to instruction-response pairs, casting workflow stages (e.g., segmentation, report writing) as prompted tasks for the language decoder, with paired groundtruth.

This sequence establishes OmniMRI’s capacity for high-fidelity visual encoding, cross-modal association, and robust instruction-following across the MRI workflow.

2. Training Corpus Scale, Diversity, and Construction

OmniMRI is trained on a corpus compiled from 60 public datasets and extensive generative augmentation techniques, resulting in:

• 47,917 patients
• 70,580 scans
• 224,194 MRI volumes
• Over 19 million MRI slices

The dataset exhibits diversity in demographics (ages 1–87, balanced sex ratios), scanner vendors (Siemens, GE, Philips), field strength (1.5T, 3T), anatomical coverage (brain, breast, knee, prostate, and others), and sequence protocols (T1, T2, FLAIR, PD, diffusion).

Three data modalities serve distinct training roles:

• Image-only data is reserved for vision pretraining, enabling anatomical and contrast-based representation learning without annotation bias.
• Paired vision-text data is constructed by hierarchical templates and generative augmentation, standardizing radiological language for imaging findings, tissue character, and diagnostic impressions (utilizing locally hosted Qwen-VL for synthetic annotation).
• Instruction-response data recasts various segmentation, reconstruction, detection, and reporting tasks into a unified prompt-based format for instruction tuning.

Such heterogeneity and annotation depth build robust cross-modal correspondences essential for generalist MRI interpretation.

3. Scope of Capabilities and Qualitative Performance

OmniMRI is evaluated for its ability to consolidate the tasks traditionally covered by fragmented, sequential pipelines. Within a single model framework, it demonstrates:

• MRI Reconstruction: Recovery of high-fidelity images from undersampled acquisitions, with suppression of aliasing artifacts across multiple organs and controversial acceleration rates.
• Anatomical and Pathological Segmentation: Precise delineation of structures (e.g., whole-brain, cartilage, prostate) and pathological features with sharply resolved masks.
• Abnormality Detection: Localization of subtle clinical findings such as bone marrow edema or meniscal tears through bounding box annotation.
• Diagnostic Suggestion: Generation of differential diagnosis lists that synthesize image features and structured clinical reasoning akin to radiologist discourse.
• Report Generation: Automated composition of radiology reports that emulate human language structure, including anatomical context and follow-up recommendations.

Qualitative examples illustrate sharp boundary segmentation, fine structural reconstruction from alias-contaminated k-space, and report outputs reflecting clinical normativity. This suggests that the model generalizes across anatomy, imaging protocol, and task formalism.

4. Integration of Imaging and Language in Clinical Context

OmniMRI’s design addresses the routine reliance of radiologists on both imaging and language. The model’s ability to interpret imaging findings and generate clinical reports offers the potential to streamline radiology workflow, unify previously siloed stages, and reduce interpretive variability.

By integrating structured language prompts and hierarchical annotation with imaging features, OmniMRI establishes a symbiotic relationship between visual findings and clinical decision-making, potentially enhancing diagnostic consistency and throughput.

However, the model’s developers note the necessity of further quantitative benchmarking and prospective radiologist validation to substantiate these qualitative gains across heterogeneous clinical environments, acquisition protocols, and patient populations.

5. Challenges, Limitations, and Future Directions

While OmniMRI demonstrates wide-ranging capability, the paper acknowledges several outstanding challenges:

• Quantitative benchmarking: Systematic, multi-institutional evaluation remains pending to characterize performance at scale.
• Generalizability: The model faces potential variability in scanner type, imaging protocol, and rare pathology representation, implying a need for continued fine-tuning.
• Prospective clinical validation: Integration with workflow and direct radiologist feedback is essential for deployment readiness.

Planned research directions include:

• Expansion to additional anatomical regions and imaging protocols.
• Refinement of multimodal pretraining and instruction tuning methods.
• Systematic validation across institutions and radiologist cohorts.
• Investigation of deployment pathways for seamless radiology system incorporation.

A plausible implication is that foundation models such as OmniMRI may eventually redefine MRI interpretation as a unified, scalable, data-driven workflow.

6. Mathematical Formulations and Optimization Framework

Key mathematical expressions central to OmniMRI include:

• The contrastive alignment loss for vision–language correspondence:

$$L_\text{align} = -\log\left(\frac{\exp(\text{sim}(v,t))}{\sum_j \exp(\text{sim}(v,t_j))}\right)$$

• Masked image modeling and patch-level contrastive objectives for vision self-supervision.
• Next-token autoregressive modeling for joint multimodal sequence learning.

Optimization is performed over joint visual–language-token sequences, with prospective expansion to accommodate new input modalities and further system scaling.

7. Impact and Significance

OmniMRI has established a foundation model paradigm for MRI analysis, offering unification of imaging and clinical language across acquisition, reconstruction, segmentation, detection, diagnosis, and reporting. Its multi-stage, multimodal training strategy over a diverse, large-scale corpus supports robust, cross-task generalizability. Within the context of clinical radiology practice, OmniMRI holds potential to mitigate workflow fragmentation, enhance diagnostic consistency, and support scalable interpretation across heterogeneous clinical contexts.

These attributes align OmniMRI with broader trends in medical AI toward generalist, foundational architectures capable of leveraging both imaging and semantic data for comprehensive end-to-end analysis, as documented in (He et al., 24 Aug 2025).