Vision-Language Model-Guided Deep Unrolling Enables Personalized, Fast MRI

Abstract: Magnetic Resonance Imaging (MRI) is a cornerstone in medicine and healthcare but suffers from long acquisition times. Traditional accelerated MRI methods optimize for generic image quality, lacking adaptability for specific clinical tasks. To address this, we introduce PASS (Personalized, Anomaly-aware Sampling and reconStruction), an intelligent MRI framework that leverages a Vision-LLM (VLM) to guide a deep unrolling network for task-oriented, fast imaging. PASS dynamically personalizes the imaging pipeline through three core contributions: (1) a deep unrolled reconstruction network derived from a physics-based MRI model; (2) a sampling module that generates patient-specific $k$-space trajectories; and (3) an anomaly-aware prior, extracted from a pretrained VLM, which steers both sampling and reconstruction toward clinically relevant regions. By integrating the high-level clinical reasoning of a VLM with an interpretable, physics-aware network, PASS achieves superior image quality across diverse anatomies, contrasts, anomalies, and acceleration factors. This enhancement directly translates to improvements in downstream diagnostic tasks, including fine-grained anomaly detection, localization, and diagnosis.

• The paper introduces PASS, a closed-loop MRI framework that leverages vision-language model guidance for adaptive sampling and deep unrolling reconstruction.
• It significantly enhances image quality metrics like PSNR and SSIM, particularly in anomaly regions, outperforming conventional and learning-based methods.
• PASS demonstrates robust performance across multiple datasets and acceleration rates, promising improved diagnostic precision and reduced scan times.

Vision-LLM-Guided Deep Unrolling Enables Personalized, Fast MRI

Introduction and Motivation

The slow acquisition times in MRI constrain clinical throughput and diagnostic workflow by increasing motion artifact susceptibility and patient discomfort. While advances in parallel imaging and compressed sensing have reduced scan durations, most accelerated MRI techniques persistently optimize for generic image quality metrics such as PSNR or SSIM, neglecting the explicit clinical value of improved visibility and characterization of subject-specific pathologies. Model-based deep unrolling methods provide a balance between interpretability and performance but fail to incorporate downstream diagnostic priors or personalize acquisition and reconstruction.

The paper "Vision-LLM-Guided Deep Unrolling Enables Personalized, Fast MRI" (2604.06849) introduces PASS (Personalized Anomaly-aware Sampling and reconStruction), a closed-loop MRI framework driven by a pretrained vision-LLM (VLM). PASS bridges the gap between knowledge-rich, foundation models and physics-aware deep learning, directly optimizing both k-space sampling and image reconstruction for patient-tailored, anomaly-centric clinical utility rather than only generic image quality.

PASS Framework: Closed-Loop, VLM-Guided MRI

PASS tightly integrates three novel components:

• An adaptive sampling module that allocates k-space measurements via a patient-specific, anomaly-aware policy.
• A deep unrolled reconstruction network conditioned on VLM priors, facilitating iterative, physics-instructed restoration with emphasis on regions of clinical interest.
• A VLM fine-tuned from CLIP, whose anomaly localization and semantic features both guide k-space sampling and enhance the targeted reconstruction pipeline.

The sequential pipeline utilizes low-resolution ACS regions to generate a prior image, from which the VLM infers anomaly attention maps. These spatial maps condition sampling mask selection and propagate as soft spatial weights through every stage of the deep unrolling network, thus enabling consistent prioritization of pathology-related details from acquisition to final reconstruction.

Figure 1: Schematic of PASS, which integrates a fine-tuned VLM for adaptive sampling and reconstruction in a closed-loop MRI system.

Quantitative and Qualitative Results under Diverse Protocols

PASS was evaluated on the fastMRI benchmark comprising brain (T1w, FLAIR) and knee (PD) datasets, with two acceleration rates ($4\times$, $8\times$). Competing methods included model-based (GRAPPA, MoDL), model-unrolling (ISTA-Net), transformer-based (SwinMR), and generative (Reflow, Nail, PDAC) approaches.

PASS displayed superior PSNR and SSIM metrics across all anatomical sites, contrasts, and acceleration rates. Notably, the observed PSNR gain for T1w brain at $4\times$ acceleration was up to $19.53$ dB over baselines, and the advantage persisted at high acceleration, highlighting stable generalization. The method proved robust against data heterogeneity and lesion variability, establishing clear quantitative superiority.

Figure 2: PASS achieves consistently higher PSNR/SSIM across acceleration rates, anatomies, and contrasts compared with alternative methods.

Qualitative inspection confirmed that PASS preserves fine anatomical structure and reduces error magnitudes, especially in regions with abnormal signal, outperforming both classical and contemporary learning-based methods.

Figure 3: Visual and error map comparison demonstrates sharper pathological detail and reduced reconstruction artifacts for PASS.

Lesion-Focused Evaluation

Traditional global metrics fail to capture the diagnostic fidelity necessary in pathology-rich MRI. PASS was evaluated on a comprehensive spectrum of both frequent and rare lesions using the fastMRI+ annotated dataset. In every lesion category, the lesion-focused PSNR/SSIM of PASS exceeded all reference methods, with up to $16.92$ dB improvement in localized PSNR. The performance advantage was invariant to lesion prevalence, confirming strong clinical robustness and transferability.

Figure 4: PASS provides superior reconstruction quality across both frequent and rare lesion types in T1w brain data.

Adaptive, Patient-Specific k-Space Sampling

Unlike global or fixed sampling strategies, the VLM-guided adaptive module in PASS leverages anomaly maps to prioritize high-frequency k-space sampling over lesion regions. Ablation comparisons elucidated three effects: 1) optimized sampling (LOUPE, PASS) clearly outperformed static masks, 2) architectures with end-to-end, task-driven losses outperformed those using generic image loss, and 3) subject-level adaptivity yielded additional gains compared to population-optimal patterns.

Visualization of acquired masks showed consistent sampling near the anatomy core but dynamically increased coverage in lesion-relevant spatial frequencies.

Figure 5: Adaptive sampling in PASS increases k-space density in regions linked to clinical anomalies, providing measurable PSNR/SSIM gains over other strategies.

Clinical Task-Specific Assessment

Transitioning from image fidelity to clinical efficacy, PASS was benchmarked on VLM-based anomaly detection and fine-grained lesion classification. The results demonstrate that improved voxelwise PSNR/SSIM does not always correlate with diagnostic AUC or accuracy. PASS, however, delivered the highest AUC values and, notably, outperformed fully-sampled ground-truth in some multi-label classification tasks—indicating that VLM-guided reconstruction enhances key semantic cues while suppressing reconstruction-induced artifact or noise.

Figure 6: PASS leads to improved anomaly detection AUC and higher lesion-wise classification accuracy, demonstrating direct diagnostic impact.

Methodological Design and Implementation

The core technical innovations of PASS include:

• Fine-tuning of a frozen CLIP backbone by adding lightweight adapters for pixel-level (PLA) and image-level (ILA) supervision, resulting in accurate anomaly-localization attention maps even in undersampled images.
• Iterative deep unrolling architecture, with integrated physics-based data consistency (solved exactly) and anomaly-prior regularization at each stage.
• Two-stage optimization for k-space mask learning, with initial LOUPE-driven global optimization followed by an anomaly-aware, subject-specific adjustment informed by the VLM.
• Custom training protocols (PyTorch, Adam, batch size 8, 300 epochs) with careful balancing between reconstruction, lesion-focused, and sampling mask losses.

Implications and Future Directions

Theoretical implications are substantial: PASS is the first framework to embed high-level, semantic, patient-adaptive priors directly into the sampling and reconstruction pipeline, effectively closing the loop between acquisition and clinical interpretation. This revises long-standing assumptions in MRI that fidelity to raw measurements should be prioritized over downstream utility.

Practically, PASS poses a replicable blueprint for integrating foundation models as soft deciders across all imaging steps, not just post hoc analysis. Positive transfer to rare pathologies and high acceleration rates suggest potential value in time-sensitive, resource-limited, or population-rich clinical environments.

However, several limitations remain. The VLM is still initialized from a CLIP model pretrained on natural images; radiology-centric large-scale pretraining has the potential to further improve anomaly sensitivity and downstream generalizability. The current adaptive sampling module leverages low-frequency information from ACS, potentially limiting capture of sub-resolution or subtle pathology. Furthermore, all validation is retrospective; prospective scanner integration and multi-site evaluation is required for clinical translation.

Conclusion

PASS establishes a transformative paradigm for MRI, unifying VLM-guided semantic guidance with physics-informed deep learning for personalized, task-aware imaging. The approach achieves measurable improvements in both conventional and clinical task-based metrics, surpassing both traditional and advanced learning-based strategies in the preservation of pathology-relevant details. By coupling acquisition and reconstruction to patient-level clinical needs through adaptive, anomaly-aware priors, PASS charts a path toward subject-specific, high-value diagnostic imaging that transcends the limitations of generic, k-space-agnostic MRI acceleration.

(2604.06849)