Lingshu: A Generalist Foundation Model for Unified Multimodal Medical Understanding and Reasoning

Abstract: Multimodal LLMs (MLLMs) have demonstrated impressive capabilities in understanding common visual elements, largely due to their large-scale datasets and advanced training strategies. However, their effectiveness in medical applications remains limited due to the inherent discrepancies between data and tasks in medical scenarios and those in the general domain. Concretely, existing medical MLLMs face the following critical limitations: (1) limited coverage of medical knowledge beyond imaging, (2) heightened susceptibility to hallucinations due to suboptimal data curation processes, (3) lack of reasoning capabilities tailored for complex medical scenarios. To address these challenges, we first propose a comprehensive data curation procedure that (1) efficiently acquires rich medical knowledge data not only from medical imaging but also from extensive medical texts and general-domain data; and (2) synthesizes accurate medical captions, visual question answering (VQA), and reasoning samples. As a result, we build a multimodal dataset enriched with extensive medical knowledge. Building on the curated data, we introduce our medical-specialized MLLM: Lingshu. Lingshu undergoes multi-stage training to embed medical expertise and enhance its task-solving capabilities progressively. Besides, we preliminarily explore the potential of applying reinforcement learning with verifiable rewards paradigm to enhance Lingshu's medical reasoning ability. Additionally, we develop MedEvalKit, a unified evaluation framework that consolidates leading multimodal and textual medical benchmarks for standardized, fair, and efficient model assessment. We evaluate the performance of Lingshu on three fundamental medical tasks, multimodal QA, text-based QA, and medical report generation. The results show that Lingshu consistently outperforms the existing open-source multimodal models on most tasks ...

• The paper presents Lingshu as a foundation model that unifies multimodal medical data for enhanced understanding and reasoning.
• It employs a multi-stage training pipeline, including shallow and deep alignment, instruction tuning, and medical-oriented reinforcement learning.
• The model demonstrates superior benchmark performance and scalability, validated through the MedEvalKit evaluation framework in real-world case studies.

Lingshu: A Generalist Foundation Model for Unified Multimodal Medical Understanding and Reasoning

The paper presents Lingshu, a foundation model designed for comprehensive multimodal understanding and reasoning within the medical domain. This model addresses critical gaps in existing medical LLMs (MLLMs) through meticulous data curation, innovative training methodologies, and the introduction of a robust evaluation framework, MedEvalKit.

Data Curation Pipeline

Lingshu's development begins with a structured data curation process that gathers a diverse array of open-source medical data, general-domain data, and synthetic medical data (Figure 1). This multi-faceted approach aims to overcome limitations of previous models, which often suffer from restricted medical knowledge coverage and an increased risk of hallucinations due to suboptimal data curation.

Figure 1: The overall data curation pipeline of medical multimodal and textual data.

1. Data Collection: The model integrates multimodal medical datasets, including medical image captions and medical instruction data. Additionally, medical unimodal datasets comprising raw images and text are collected to enhance domain knowledge.
2. Data Synthesis: To address data scarcity, the paper proposes synthesizing long-form captions and visual question answering (VQA) samples. These efforts increase the diversity and depth of the training dataset, allowing the model to more accurately learn from detailed medical information.

Model Training

The architecture leverages the Qwen2.5-VL model, following a multi-stage training pipeline that adapates the model to the medical domain through a series of incremental learning stages (Figure 2).

Figure 2: The training pipeline of Lingshu, consisting of medical shallow alignment, medical deep alignment, medical instruction tuning, and medical-oriented reinforcement learning.

1. Medical Shallow Alignment: This initial stage focuses on harmonizing the model with medical imagery using coarsely annotated medical image-text pairs.
2. Medical Deep Alignment: Expanding beyond the foundational vision-language alignment, this stage integrates high-quality datasets and fine-grained textual information to embed deeper domain knowledge.
3. Medical Instruction Tuning: This phase enhances the model's instruction-following capacity, critical for applying the model in real-world medical use cases.
4. Medical-oriented Reinforcement Learning: To improve reasoning capabilities, the model uses reinforcement learning methodologies tailored specifically for medical scenarios.

Evaluation with MedEvalKit

MedEvalKit is introduced as a benchmark for evaluating the model's performance across various medical tasks, providing a standardized, scalable, and reproducible evaluation framework (Figure 3). This evaluation framework includes a diverse set of medical benchmarks encompassing multimodal QA, textual QA, and report generation tasks.

Figure 3: Medical Multimodal Benchmark overview.

• Performance Analysis: Lingshu demonstrates superior performance across multiple benchmarks, often outperforming existing open-source models and approaching the outputs of proprietary systems like GPT-4 in some tasks.
• Scalability: The performance is consistent across different scales of training data, demonstrating robust generalization capabilities (Figure 4).

Case Studies and Real-World Applications

The paper illustrates Lingshu's potential through five detailed case studies that underscore its applicability in practical medical contexts, such as medical diagnosis and report generation (Figures 10 to 15). These examples demonstrate Lingshu's ability to interpret complex medical information and generate accurate, contextually relevant outputs.

Figure 5: Case studies on Lingshu in visual question answering across various medical imaging modalities.

Conclusion

Lingshu represents a significant step toward developing robust, generalist MLLMs tailored to the complexities of medical data. By addressing existing limitations through innovative data curation and enrichment strategies, multi-stage training paradigms, and rigorous evaluation, this study suggests pathways for future development in the field of medical AI applications. The integration of reinforcement learning for enhanced reasoning and the proposed MedEvalKit offer a comprehensive approach to evaluating and deploying these advanced models in healthcare settings.