Med-MLLMs: Medical Multimodal LLMs

• Med-MLLMs are advanced AI systems that combine modality-specific encoders with large language models to interpret image, text, and biomedical data.
• They employ projection-based, query-based, and cross-attention techniques to achieve joint representations essential for tasks like report generation, visual QA, and diagnosis.
• Emerging challenges such as data scarcity, multimodal alignment issues, and model hallucinations drive ongoing research in evaluation frameworks, interpretability, and clinical safety.

Medical Multimodal LLMs (Med-MLLMs) are advanced Transformer-based systems that extend the capabilities of unimodal LLMs by integrating specialized modality encoders—most commonly for visual data—enabling unified understanding and generation across textual, imaging, and structured biomedical modalities. Through architectural advances, tailored pretraining strategies, and new evaluation paradigms, Med-MLLMs are transforming medical AI from narrowly focused, task-specific systems to generalist, context-aware engines capable of comprehensive clinical reasoning and decision support (Xiao et al., 2024, Ye et al., 29 Apr 2025).

1. Definition, Motivation, and Paradigm Shift

Med-MLLMs are characterized by the incorporation of one or more modality-specific encoders (typically vision Transformers or CNNs) attached to a pretrained text LLM backbone. The encoders transform non-textual data such as radiological images, pathology slides, or 1D biomedical signals into high-dimensional representations, while the LLM processes and integrates these embeddings alongside natural language inputs. The unified model can interpret multimodal clinical inputs, generate structured medical reports, and reason across image–text domains without ad hoc specialist modules.

Paradigm Shift: Medical AI has evolved from unimodal, task-specific models (e.g., segmentation networks, symbolic expert systems) toward generalist Med-MLLMs capable of cross-modal understanding and few-shot adaptation. This enables new application scenarios: mixed-modal retrieval, interactive image-grounded dialogue, and generative decision support leveraging both medical images and textual data (Xiao et al., 2024).

2. Core Architectural and Technical Principles

Med-MLLM architectures can be abstracted as follows:

• Input encoding:
  • $Z_x = V(I_x)$ (vision encoder transforms image $I_x$)
  • $T_x = E(x)$ (text encoder/tokenizer)
• Modality alignment: Modality-specific embeddings (e.g., patch features from a ViT) are projected, pooled, or attended to produce a joint context, typically via one of three strategies:
  1. Projection-based: Pooled visual tokens map to the LLM token space (linear/MLP).
  2. Query-based: Visual queries (e.g., Q-Former) extract task-relevant features.
  3. Cross-attention: Direct cross-modal attention subblocks integrate vision features at every transformer layer.

• Joint representation: $R = L(H_x, H_t)$, where $L$ denotes shared multimodal Transformer layers, yielding tasks such as report generation, VQA, or retrieval.

• Training objectives: Pretraining involves image–text matching (ITM), contrastive alignment (e.g., InfoNCE), and image–text generation (ITG). Fine-tuning applies instruction datasets for supervised and reinforced learning (Xiao et al., 2024, Ye et al., 29 Apr 2025).

3. Construction, Training Pipelines, and Data

Datasets: Pretraining and fine-tuning draw on diverse medical sources:

Modality	Principal Datasets	Task Types
Radiology	MIMIC-CXR, CheXpert, OpenI, BrainCT/MRI	Captioning, VQA, diagnosis
Pathology	PathVQA, PathCap, Quilt-1M, HE images	VQA, caption, retrieval
EHR/Text	MIMIC-III/IV, CPRD, USMLE, MedQA, PMC-OA	QA, summarization, retrieval
Multimodal	PMC-OA, PMC-15M, MultiMedQA, MedInstruct-52K, ChatDoctor	Multimodal QA, dialogue

Pretraining: Most Med-MLLMs use self-supervised objectives:

• ITM: Binary discrimination of real vs. shuffled image–text pairs.
• Contrastive (ITCL): Aligns visual and textual global embeddings.
• ITG: Multimodal context-injection for next-token prediction.

Instruction and alignment fine-tuning:

• Continuous joint image–text learning (e.g., on PMC-OA, Qilin-Med-VL).
• Supervised fine-tuning of report generation and VQA using curated benchmarks (e.g., VQA-RAD, PathVQA).
• Reinforcement Learning from Human/Ai Feedback (RLHF/RLAIF) for clinical alignment (Xiao et al., 2024).

Parameter-efficient fine-tuning (PEFT) strategies, such as LoRA, allow rapid adaptation of large pretrained Med-MLLMs using only a small proportion of weights (e.g., $<$1% in PeFoMed) and have demonstrated significant gains in medical imaging tasks (He et al., 2024).

4. Evaluation Frameworks and Clinical Benchmarks

Task Families and Representative Metrics:

Task	Metrics	Benchmark Datasets
Report Generation	BLEU, ROUGE, CIDEr, BERTScore	MIMIC-CXR, IU-Xray
Visual QA (VQA)	Accuracy	VQA-RAD, PathVQA
Diagnosis	USMLE-style accuracy	MultiMedQA
Retrieval	Recall@K	ROCO, PMC-15M
Safety/Hallucination	Med-Halt (human audit)	Med-Halt

Generalization & Robustness: Newer benchmarks such as Asclepius (Liu et al., 2024) and MedGaze-Bench (Liu et al., 11 Jan 2026) provide spectrum evaluation across specialties and capabilities—e.g., perception, diagnosis, planning, egocentric intent—using clinical images and simulated clinician workflows. Specialized metrics address spatial grounding (IoU, Dice), temporal intent, standard operating procedure adherence, hallucination and sycophancy rates.

Knowledge Editing: MedMKEB (Xu et al., 7 Aug 2025) introduces metrics for knowledge localization, reliability, generality, portability, and robustness under prompt attacks, revealing major challenges in keeping models clinically up to date without broad knowledge corruption.

5. Exemplar Clinical Applications and Advances

Radiology Interpretation: Systems such as R2GenGPT, Med-Flamingo, and ChatCAD generate structured radiology reports and support image-grounded QA, achieving human-comparable BLEU/ROUGE on large-scale benchmarks.

Digital Pathology: PathAsst and MedBLIP operate on high-resolution digital slides, automating tumor annotation and multi-turn diagnostic refinement.

Multimodal Triage and Decision Support: Qilin-Med-VL and AMIE integrate images, EHR, and vitals for automated triage and evidence-based clinical recommendation.

Advanced Modalities:

• 3D Imaging: Med-2E3 fuses 2D and 3D encoders with text-guided attention, setting new SOTA on volumetric CT tasks (Shi et al., 2024).
• Pixel-level Insight: MedPLIB employs mixture-of-experts to achieve pixel-level VQA and flexible region-based grounding, surpassing LISA and other segmentation-aware MLLMs (Huang et al., 2024).
• Unified Generation: MeDiM unifies text-to-image, image-to-text, and joint generation in a single discrete diffusion MLLM backbone (Mao et al., 7 Oct 2025).

Personalized & Interactive Consultation: Frameworks such as Med-PMC (Liu et al., 2024) and HeLM (Belyaeva et al., 2023) simulate clinician–patient information-gathering, enabling dynamic, patient-specific dialogue and risk estimation.

6. Ongoing Challenges and Open Directions

• Data Scarcity and Fragmentation: Many modalities suffer from limited, privacy-constrained annotation. Synthetic data augmentation, federated learning, and domain-specific pretraining are being developed to overcome these constraints (Xiao et al., 2024, Ye et al., 29 Apr 2025).
• Multimodal Alignment & Semantic Grounding: Weak cross-modal semantic consistency remains a prominent failure mode, often manifesting as hallucinations or poor paraphrase/generalization performance. Techniques such as graph alignment (EXGRA-MED (Nguyen et al., 2024)) and reinforcement with verifiable rewards (MedMO (Deria et al., 6 Feb 2026)) address this.
• Generalizability and Domain Shift: Performance may degrade outside the narrow training distribution (e.g., under new imaging devices or rare syndromes).
• Interpretability & Trust: Visualization of attention (e.g., grounding maps, region-anchored explanations) and modular explanation heads are proposed for greater clinical auditability.
• Hallucinations and Safety: Benchmarks such as MedGaze-Bench quantify perceptual and cognitive hallucination rates—current models fail to exceed 70% reliability, highlighting the need for mandatory trap detectors and safety-first inference routines (Liu et al., 11 Jan 2026).
• Regulatory, Ethical & Privacy Compliance: Compliance with HIPAA/GDPR and pathway toward device certification is an active area. Differential privacy, federated aggregation, and encrypted inference are deployed to address PHI leakage.

7. Controversies, Limitations, and Future Prospects

Hallucination and Knowledge Editing: Despite high measured accuracy on canonical tasks, Med-MLLMs are vulnerable to hallucination and brittle to external prompt attacks. Editing knowledge (e.g., for evolving guidelines) without catastrophic forgetting remains unresolved for vision–text models, as shown in MedMKEB (Xu et al., 7 Aug 2025).

Reward Model Evaluation: Med-RewardBench (Ding et al., 29 Aug 2025) demonstrates that optimizing only for diagnostic correctness misses other clinical critical dimensions (relevance, comprehensiveness, creativity, etc.), and even top models lag behind human experts in holistic evaluation.

Resource Efficiency and Small-Scale Deployment: Infi-Med (Liu et al., 29 May 2025) and InfiMed-Foundation (Zhu et al., 26 Sep 2025) demonstrate how data and compute-efficient strategies (sequence packing, multi-stage FT, PEFT) can yield small models (1.7–4B) that nearly match or outperform larger baselines, supporting broader healthcare accessibility.

Interactivity and Egocentric Understanding: Dynamic, interactive consultation and egocentric intent reasoning (e.g., via patient simulators or gaze tracking) expose new failure modes rarely captured in conventional benchmarks (Liu et al., 2024, Liu et al., 11 Jan 2026).

Bicultural and Cross-Lingual Generalization: Recent advances (e.g., MedRegA (Wang et al., 2024)) show state-of-the-art bilingual region-centric reasoning and report generation, but cultural and language-specific diagnostic nuances need further exploration.

A plausible implication is that as multimodal data, interactive evaluation, and continual learning modules are increasingly standardized, Med-MLLMs will transition from experimental systems to core infrastructure for real-world clinical decision support and patient-centric care, contingent on rigorous safety, adaptability, and explainability regimes.

---

References:

(Xiao et al., 2024, Ye et al., 29 Apr 2025, Nguyen et al., 2024, Xu et al., 7 Aug 2025, Mao et al., 7 Oct 2025, Shi et al., 2024, Huang et al., 2024, Liu et al., 29 May 2025, He et al., 2024, Liu et al., 11 Jan 2026, Liu et al., 2024, Belyaeva et al., 2023, Wang et al., 2024, Ding et al., 29 Aug 2025, Zhu et al., 26 Sep 2025)