Lingshu: Unified Multimodal Medical AI

• Lingshu is a unified multimodal medical AI model that leverages extensive curated datasets and a multi-stage, instruction-tuned architecture to support clinical diagnosis, reporting, and cross-modal Q&A.
• It employs a rigorous synthetic data generation pipeline and tailored reinforcement learning with verifiable rewards, achieving robust language clarity while facing challenges in domain-specific accuracy, especially in chest radiograph reporting.
• Benchmarking with the MedEvalKit framework demonstrates Lingshu’s state-of-the-art performance among open-source models on medical QA and report generation tasks despite noted clinical usability limitations.

Lingshu is an open-source, large-scale foundation model for unified multimodal medical understanding and reasoning, developed by Alibaba (China) and publicly released as Lingshu-7B on Hugging Face. Lingshu represents a technical synthesis of extensive medical and general-domain data curation, a multi-stage instruction-tuned architecture based on Qwen2.5-VL-Instruct, and tailored reinforcement learning for verifiable reward (RLVR). Its primary aim is to enhance visual-language AI models for clinical applications such as diagnosis, reporting, knowledge recall, and cross-modal question answering, with robust evaluation provided by the MedEvalKit framework. Empirically, Lingshu demonstrates state-of-the-art performance among open-source models on aggregate medical benchmarks, but exhibits limitations in domain-specific clinical end usability, especially in chest radiograph (CXR) reporting.

1. Data Curation and Synthetic Pipeline

Lingshu’s training corpus is assembled from open-source medical multimodal and unimodal datasets, large-scale general-domain sources, and systematically generated synthetic samples. Medical multimodal sources include PMC-OA, ROCO/ROCOv2, PubMedVision, LLaVA-Med, MedICaT, FairVLMed, MIMIC-CXR, and MedPix-2.0 for image-caption pairs, and a diverse array of VQA and instruction datasets. Unimodal medical sources span X-ray (including COVID19-Radiography, CheXpert), CT (DeepLesion), MRI (BraTS2024), ultrasound, dermoscopy, fundus, and histopathology. General-domain data is drawn from LLaVA-1.5 captions, PixMo, and OpenHermes-2.5.

Synthetic data generation remedies noisy or shallow captions and augments reasoning content:

• Long-form captions are created by metadata/ROI extraction (segmentation masks, bounding boxes), factual annotation by GPT-4o, and consolidated summarization integrating expert annotation.
• OCR-instruction pairs are produced using rendered biology/chemistry exam questions and Gemini-2.0–validated reasoning.
• VQA samples are generated using both template-based and self-instruct approaches, leveraging GPT-4o for few-shot QA sample synthesis and chain-of-thought traces.
• Consistency verification employs LLMs (GPT-4o) for filtering logically inconsistent or “hallucinated” instances.

Rigorous filtering and deduplication include image pixel size thresholds, perceptual hashing, token-length filtering, PII and legal-risk redaction, and exclusion of any samples overlapping with evaluation splits (MedEvalKit).

2. Model Architecture and Training Protocol

Lingshu utilizes Qwen2.5-VL-Instruct as its architectural backbone, comprising a shared visual encoder and a 7B-parameter instruction-tuned LLM. Images from any supported modality are embedded via a visual adapter (MLP projector) into the LLM embedding space. Unified input formatting and tokenization support up to 8,192 multimodal sequence length during supervised fine-tuning (SFT) stages.

Training occurs in four primary stages:

1. Medical Shallow Alignment: With the LLM frozen, only the vision encoder and projector are tuned against coarse medical captions using cross-entropy next-token loss:

$$\mathcal{L}_{\mathrm{CE}} = -\sum_{t=1}^T \log p_\theta(w_t \mid w_{<t}, I)$$

1. Medical Deep Alignment: Full-model end-to-end fine-tuning uses broader, longer captions across medical and general datasets.
2. Medical Instruction Tuning: Supervised fine-tuning (SFT) on multimodal and unimodal medical and general instructions, across QA formats, dialogues, and report generation with the same loss objective.
3. Reinforcement Learning with Verifiable Rewards (RLVR): Applied in Lingshu-RL, this uses Group Relative Policy Optimization (GRPO) on curated QA tasks. The reward $R(\tau)$ combines strict format ($R_{fmt}$) and answer accuracy ($R_{acc}$):

$$R(\tau) = 0.5\,R_{\mathrm{fmt}}(\tau) + 1.0\,R_{\mathrm{acc}}(\tau)$$

KL-regularized policy gradient is minimized:

$$\mathcal{L}_{\mathrm{RL}} = -\mathbb{E}_{\tau \sim \pi_{\theta}} [R(\tau)] + \beta D_{\mathrm{KL}}(\pi_{\theta}\|\pi_{\mathrm{ref}})$$

with $\beta=10^{-3}$.

Optimizer: AdamW with cosine scheduling, sequence length adjustments, and batch sizes are adapted to stage and hardware.

3. Evaluation Framework and Benchmarking

Lingshu’s evaluation is orchestrated via MedEvalKit, a unified platform aggregating 152,066 samples (121,622 images) across three core task categories:

• Multimodal QA: Datasets include VQA-RAD, SLAKE, PathVQA, PMC-VQA, OmniMedVQA, MMMU, MedXpertQA.
• Text-only QA: Benchmarks comprise MMLU-Med, PubMedQA, MedMCQA, MedQA-USMLE, MedBullets, MedXpertQA-Text, SuperGPQA.
• Report Generation: Key datasets are MIMIC-CXR, CheXpert Plus, IU-Xray.

Metrics used:

• QA: Closed-form accuracy (rule-based) and GPT-4.1–mediated judgments for open answers.
• Report Generation: ROUGE-L, CIDEr, RaTE, SembScore, and RadCliQ$^{-1}$.

Case studies highlight systematic chain-of-thought reasoning, structured report outputs, public health table extraction, and patient-doctor dialogue capabilities. Ablation studies demonstrate all four instruction-tuning data branches are crucial, with medical text contributing the most.

4. Clinical Performance in Chest Radiograph Reporting

A multicenter, reader-blinded paper (Lim et al., 29 Nov 2025) compared Lingshu (7B) to other leading medical VLMs and radiologist-written reports in emergency department (ED) CXR report generation, using real-world patient data (n=478) paired with same-day CT.

• RADPEER 3b error (clinically significant miss): 43.0% for Lingshu vs. 13.9% for radiologists ($P<.001$)
• Any clinically significant miss (RADPEER 2b or 3b): 57.1% vs. 30.3% ($P<.001$)
• Clinical acceptability (scores 3–4): 41.1% vs. 74.3% ($P<.05$)
• Hallucination rate: 11.0% vs. 0.1% ($P<.05$)
• Language clarity (scores 4–5): 88.0% vs. 78.1% ($P<.05$)

Finding-level analysis (CT as reference) reveals:

Finding	Sensitivity (%) [95% CI]	Specificity (%) [95% CI]
Lung opacity (CXR/CT)	16.3 [12.2–21.6]	89.1 [84.1–92.8]
Pleural effusion	21.3 [16.0–27.7]	90.2 [85.9–93.3]
Cardiomegaly	86.7 [76.4–93.1]	61.5 [56.6–66.3]
Emphysema	15.5 [8.8–25.4]	99.2 [97.8–99.8]

Other findings (aortic aneurysm, cavity, etc.) showed low or intermediate sensitivity; specificity remained generally high except for cardiomegaly.

Inferentially, Lingshu demonstrates marked inferiority to AIRead and radiologists across diagnostic-quality metrics except for language clarity. A plausible implication is that while broad, instruction-heavy training confers syntactic fluency, it does not guarantee clinical accuracy or hallucination resistance in acute CXR reporting, especially in the absence of tailored medical priors or clinical context.

5. Statistical and Analytical Methods

For report-level comparisons, a generalized linear mixed-effects model was used:

$$\begin{aligned} Y_{ijk} &\sim \mathrm{Bernoulli}(p_{ijk}) \ \mathrm{logit}(p_{ijk}) &= \beta_0 + \beta_1\,\mathbb{I}(\text{Model}_i\neq\text{Radiologist}) + u_j + v_k \ u_j &\sim N(0,\sigma_u^2) \ v_k &\sim N(0,\sigma_v^2) \end{aligned}$$

• Link function: logit.
• Fixed effect: report source.
• Random intercepts: reader ($u_j$) and case ($v_k$).
• Wald tests for hypothesis evaluation; $P<.05$ significant.
• Clopper–Pearson method for finding-level 95% CIs; no mixed model at finding level.
• Subgroup analyses by patient sex, age, and radiologist expertise.

6. Limitations and Future Directions

Identified limitations include:

• Open-source images often lack resolution; synthetic captions can increase hallucination risk.
• Clinical performance, especially for complex reasoning and diagnostic workflows, remains below top proprietary models and does not generalize outside defined benchmarks.
• Binary reward functions in RLVR provide only modest efficacy; reward granularity and curated reasoning data selection require refinement.

Future priorities articulated by the authors (Team et al., 8 Jun 2025):

• Expanding human-in-the-loop curation and developing granular quality assessment models.
• Collecting larger, expert-validated datasets encompassing whole-slide images (WSI), volumetric CT/MRI, and multi-modal genomic data.
• Simulating richer clinical workflows and longitudinal/multi-patient settings for benchmarking (e.g., proposed “HealthBench”).
• Architectural expansion for gigapixel/3D/graph-structured input and cross-modal retrieval.
• Incorporating continuous and semantic-reward reinforcement signals in RLVR.
• Integrating clinician-centric metrics and “agentic” capabilities, such as electronic health record (EHR) or PACS tool invocation and diagnostic/treatment planning.

7. Comparative Analysis and Domain Impact

Compared to other medical VLMs, Lingshu’s primary advantage is superior language clarity in generated text, with measured language clarity at 88.0% (scores 4–5), above both radiologists (78.1%) and all other VLMs in peer comparison. However, this is offset by the highest clinically significant error rates (RADPEER 3b: 43.0%), the lowest clinical acceptability (41.1%), and frequent hallucinations (11.0%). In aggregate medical QA/report generation benchmarks beyond CXR, Lingshu-7B achieves top open-source performance, while Lingshu-32B (proprietary tier) outperforms even GPT-4.1 on aggregate multimodal QA ($66.6\%$ vs. $63.4\%$) and text QA ($61.8\%$ vs $58.4\%$ best open-source). This suggests substantial potential for unified, cross-modality reasoning, though targeted clinical-constrained applications (e.g., ED chest imaging) remain challenging (Lim et al., 29 Nov 2025, Team et al., 8 Jun 2025).