MedCLM: Unified Medical VQA Framework

• MedCLM is a unified framework for medical VQA that combines explicit lesion localization with chain-of-thought reasoning, mimicking clinical diagnostic workflows.
• It automatically transforms detection datasets into structured VQA–CoT corpora, enabling scalable model training and improved interpretability.
• A dynamic curriculum strategy transitions from explicit localization to implicit reasoning, achieving state-of-the-art performance on radiology benchmarks.

MedCLM refers to a unified framework for creating and training medical vision-LLMs that explicitly integrate clinical localization and step-wise diagnostic reasoning via a Chain-of-Thought (CoT) curriculum. It provides a scalable methodology for generating and leveraging large-scale visual question answering (VQA) datasets, as well as for aligning model training with the real-world interpretive workflow practiced by clinicians and radiologists. MedCLM advances the field by automatically converting detection-based imaging datasets into structured VQA–CoT corpora and employing a curriculum schedule that transitions from explicit localization to more abstract reasoning objectives, resulting in improved clinical alignment and state-of-the-art performance on medical VQA and radiology reporting benchmarks (Kim et al., 6 Oct 2025).

1. Automated Dataset Generation and Anatomical Contextualization

MedCLM introduces a pipeline that systematically converts lesion detection datasets, which include human-annotated bounding boxes and lesion class labels, into a comprehensive VQA–CoT training corpus. For each image:

• An anatomical segmenter (e.g., TotalSegmentator, CXAS) generates organ masks across the image.
• Each lesion box is linked to the most probable host organ, determined by maximizing the Intersection over Union (IoU) between the lesion bounding box $B_{ij}$ and organ mask $M_k$:

$O_{ij} = \arg\max_k \mathrm{IoU}(B_{ij}, M_k)$

• This mapping produces structured triplets: $$(\text{lesion class}, \text{bounding box}, \text{organ})$$.
• Each triplet is converted into a factual, structured seed sentence (e.g., “There is a [lesion type] in the [organ].”) that provides both anatomical and contextual metadata.
• Using the original image and the factual seed, a medical vision-LLM is then prompted to generate a sequence comprised of a localized medical VQA question, an answer, and a corresponding rationalization, thus enriching the dataset with both the factual and the interpretive clinical reasoning elements.

This process enables automated generation of large, clinically relevant VQA–CoT datasets from existing detection corpora without additional manual annotation, preserving explicit geometric and anatomical context.

2. Chain-of-Thought (CoT) Reasoning and Model Output

MedCLM’s training data is structured to encourage Chain-of-Thought reasoning, a sequential process that mimics how clinicians articulate their diagnostic steps. The question-answer pairs generated by the model are augmented with rationales—intermediate reasoning steps that are anchored in the factual seed derived from lesion and organ localization. During training:

• Model outputs are “teacher-forced” in early phases, where the expected rationale (CoT) is provided as supervision to guide the model towards generating anatomically correct and logically sound explanations.
• In later training phases, rationale supervision is relaxed, enabling the model to generate rationales independently, reinforcing its capacity for implicit step-wise reasoning.

This approach structurally reinforces interpretability and clinical trustworthiness, as the CoT output can be directly mapped to each stage of the clinical interpretive process.

3. Integrated CoT-Curriculum Strategy

MedCLM introduces an integrated curriculum learning schedule, explicitly staging training objectives to progressively increase task difficulty while decoupling spatial and reasoning components:

• Easy Stage (Explicit Localization):
  • The model receives images with rendered (explicit) lesion bounding boxes.
  • The learning objective includes answer and rationale likelihoods and an explicit alignment loss:

  $\mathcal{L}_\text{ground}$

  which enforces consistency between region-of-interest (ROI) features (obtained via ROIAlign) and textual lesion–organ anchors.

• Medium Stage (Implicit Localization):

  • Lesion boxes are not shown, but a soft attention mask loss, $\mathcal{L}_{\text{attn-mask}}$, is used. A soft mask $m_B$ is generated by Gaussian blurring the box mask, and the Kullback–Leibler divergence aligns this with attention maps:

  $$\mathcal{L}_{\mathrm{attn-mask}} = \mathrm{KL}(m_B \ \|\ a)$$

  where $a$ is the model's attention map. - Rationale supervision continues, but the model must now localize the lesion implicitly.

• Hard Stage (Answer-Only Reasoning):

  • Rationale supervision is removed.
  • The objective is to maximize answer correctness, with multiple rationales explored by the model but only those that maximize answer probability retained.

A dynamic curriculum scheduler monitors training domain-wise (by lesion type, imaging modality) using domain-specific loss EMAs and threshold margins to allocate samples among stages optimally. This staged progression builds robust localization and reasoning capabilities aligned with clinical workflows, and prevents premature generalization.

4. Empirical Performance and Comparative Evaluation

MedCLM has been empirically validated across standard medical VQA and radiology reporting tasks:

• On VQA benchmarks such as VQA-RAD, SLAKE, and PMC-VQA, MedCLM achieves state-of-the-art or near–state-of-the-art results, particularly for open-ended questions where step-wise anatomical and diagnostic reasoning is required.
• For radiology report generation (IU-Xray, MIMIC-CXR datasets), the framework consistently outperforms strong baselines across BLEU, ROUGE, and METEOR metrics. Improvements in these scores reflect advances in factual completeness, fluency, and clinical consistency.
• Qualitative evidence shows that MedCLM reliably identifies correct lesion locations, produces anatomically accurate descriptions, and offers coherent step-by-step rationales—features that are less prominent in models trained without explicit CoT-curriculum.

This suggests that curriculum-based integration of localization and rationale generation improves both answer accuracy and clinical interpretability in medical vision-LLMs (Kim et al., 6 Oct 2025).

5. Clinical Alignment, Scalability, and Implications

The MedCLM framework is designed to scale both to new data sources and across clinical imaging modalities:

• Clinical Alignment: By grounding reasoning processes in explicit lesion localization and anatomical context, MedCLM mirrors clinical diagnostic workflows, yielding outputs that are interpretable by practicing clinicians and that correspond to real medical reasoning steps.
• Scalability: Automated data conversion circumvents the need for manual expert annotation when building large VQA–CoT training sets and enables the rapid porting of the methodology to new imaging and lesion types.
• Computational Efficiency: Use of a 7B-parameter ViP-LLaVA backbone provides strong results even compared to significantly larger models, implying resource efficiency.
• Extensibility: The curriculum scheduler and data conversion pipeline are adaptable to various lesion types, organs, and imaging modalities, supporting future applications in other areas of medical imaging where step-wise reasoning and localization are required.

A plausible implication is that MedCLM’s principles could generalize to broader biomedical VQA and report generation domains where step-wise, anatomically grounded reasoning is necessary, and where expert annotation bottlenecks hamper progress.

6. Limitations and Future Directions

Several limitations are acknowledged:

• Dependency on Detection Dataset Quality: The effectiveness of MedCLM’s data generation pipeline is contingent on the quality and completeness of lesion detection/segmentation data. Incomplete or inaccurate annotations can propagate through to VQA–CoT outputs.
• Weakly Supervised Limits: As rationale supervision is reduced in harder curriculum stages, there is the potential for drift toward less interpretable or less clinically valid rationales. Monitoring for clinical soundness remains necessary.
• Curriculum Scheduling Complexity: The dynamic domain-specific scheduler requires careful calibration of loss thresholds and rates to prevent stagnation or premature escalation between training phases.

Future research directions include extending the framework to incorporate multi-modal fusion (e.g., laboratory, genomic, or longitudinal EHR data), further refinement of unsupervised rationale alignment, and application to other specialties where localization and step-wise reasoning are critical.

---

MedCLM represents a systematic advance in clinically-aligned medical vision-language modeling through its integration of anatomical localization, chain-of-thought rationale generation, and curriculum-based training, enabling scalable, interpretable, and high-performing models for a variety of clinically relevant medical AI tasks (Kim et al., 6 Oct 2025).