OpenMedReason: Scientific Reasoning Supervision for Medical Vision-Language Models

Abstract: High-stakes clinical use of large vision-LLMs (LVLMs) requires reasoning that is grounded in visual evidence and clinical knowledge, not just correct final answers. We introduce OpenMedReason, a large-scale, open multimodal medical reasoning corpus comprising approximately 450K image-question-answer instances whose reasoning traces are primarily derived from curated biomedical, human-authored scientific articles. OpenMedReason provides high-fidelity supervision beyond synthetic chains of thought, covering diverse medical domain vision modalities such as radiological scans, microscopic images, visible light photographs, charts, and others. We complement it with OpenMedReason-Bench, a held-out benchmark that allows fine-grained evaluation of LVLMs along three complementary axes of capability, including perception, medical knowledge, and rationale, enabling diagnostic evaluation beyond final-answer accuracy. OpenMedReason is a rich training resource that exhibits its effectiveness in both supervised fine-tuning (SFT) and reinforcement-based alignment. Training with OpenMedReason yields a 20% average improvement in VQA accuracy over the base model and achieves performance within 4.2% of the strongest comparable-scale medical LVLMs. Fine-grained performance analysis confirms that the gains are not concentrated in any single axis: OpenMedReason improves perception, medical knowledge, and rationale jointly, and its reasoning traces are preferred over those of the base model in 86.1% of pairwise comparisons. We release the code and dataset at huggingface.co/datasets/neginb/OpenMedReason.

• The paper presents a novel dataset (≈450K instances) for grounding LVLMs with source-anchored scientific reasoning in medical tasks.
• It details a rigorous multi-stage pipeline using quality filtering, clinical context extraction, and structured reasoning trace creation.
• Empirical results show 10–30% accuracy gains across modalities, enhancing perception, factual knowledge, and rationale.

OpenMedReason: Scientific Reasoning Supervision for Medical Vision–LLMs

Introduction and Motivation

The OpenMedReason project addresses critical limitations in current open medical vision-language datasets, specifically the lack of high-fidelity, source-grounded reasoning traces for large vision-LLMs (LVLMs) in clinical domains. Existing resources primarily provide image–question–answer (IQA) triples without intermediate reasoning chains, or rely on synthetic chains of thought generated by LLMs, which are often stylistically homogeneous and weakly connected to visual evidence. OpenMedReason constructs a large-scale (≈450K instances) open corpus derived primarily from human-authored scientific articles, yielding heterogeneous, clinically meaningful, and visually grounded reasoning supervision for training and evaluating LVLMs on complex medical tasks.

Dataset Curation and Pipeline

The curation of OpenMedReason involves a comprehensive multi-stage process targeting clinical fidelity, multimodal diversity, and supervision reliability. The pipeline includes rigorous quality filtering, clinical context extraction, constrained question generation, structured reasoning-trace creation, and multi-step verification, ensuring retention of only instances that support grounded clinical reasoning.

Figure 1: Schematic of the OpenMedReason pipeline, illustrating multi-level filtering, context-constrained question creation, reasoning trace generation, and verification, along with taxonomic coverage and imaging modality distribution.

Images are filtered using pixel-level heuristics—removing samples with low spatial resolution, heavy compression artifacts, nonstandard aspect ratios, or excessive white borders—to ensure all retained data supports robust perceptual grounding. Contextual textual data, including captions and in-text references, are subjected to LLM-based assessment for clinical relevance, image-text alignment, and reasoning readiness, resulting in a core set of high-quality (∼30K) “seed” samples. This is expanded into the full dataset by curated question and answer generation constrained to 19 clinically-motivated task categories (spanning perception, diagnosis, management, risk, and workup), capturing substantial clinical and modality diversity.

Figure 2: Sample images culled during the visual quality filtering stage, demonstrating the exclusion of blurry, artifact-ridden, undersized, or poorly formatted samples.

During question construction, an LLM-based prompt ensures each IQA triple is not trivially answerable from text alone and that reasoning traces are explicitly grounded in both visual evidence and article context. Reasoning chains are constructed using a draft-refine paradigm, with medical LVLMs organizing evidence and a secondary model enforcing source-consistency and inferential fidelity according to a standardized schema: modality/target → visual evidence → clinical interpretation → answer justification.

Benchmarking and Fine-grained Evaluation

OpenMedReason is accompanied by OpenMedReason-Bench—a held-out evaluation suite of 1,500 cases—designed to measure model competence decomposed into perception, medical knowledge, and rationale, rather than sole reliance on final-answer correctness. Each benchmark instance is annotated with a checklist of atomic unit claims, and model outputs are systematically evaluated along the axes of:

• Presence (is the required unit mentioned?)
• Correctness (is the claim stated accurately?)

Axis-level and joint trace scores provide diagnostic capability to pinpoint failure modes in grounding, factuality, or reasoning integrity.

Figure 3: Dataset statistics exhibit balanced coverage across question categories and imaging modalities, supporting robust generalization and transfer learning.

Empirical Results

Solid performance improvements are demonstrated on both in-distribution and out-of-distribution medical VQA and classification tasks. Training a Qwen2.5-VL-7B-Instruct backbone with OpenMedReason (SFT→GRPO) yields an average gain of 10–30% in benchmark accuracy and trace quality over baselines, with no single capability axis dominating the improvement signal; instead, the supervision enhances perception, factual knowledge, and rationale together. Notably, OpenMedReason-trained models outstrip comparable-scale open and domain-specific LVLMs on a broad suite of medical QA and classification tasks.

Figure 4: (Left) Head-to-head pairwise preference tests show OpenMedReason-trained models consistently preferred over four baselines in reasoning trace quality. (Right) Pass@k metrics for three medical VQA benchmarks demonstrate robust improvement over the non-finetuned backbone for both deterministic and stochastic decoding.

The model achieves 86.1% preference in direct reasoning trace comparison, and capability-decomposition reveals substantial improvements, particularly in the coverage of relevant perceptual and knowledge units. Gains are robust across radiology, pathology, photography, diagrams, and mixed-modal datasets. Fine-tuning and subsequent RL alignment with GRPO demonstrate cumulative and complementary benefits.

Qualitative Insights

Several representative cases illustrate the model’s clinical reasoning ability:

Figure 5: Transesophageal echocardiogram showing an atrial appendage mass, where proper trace construction supports immediate risk assessment for embolization events.

Figure 6: Coronal CT scan post-surgery, where reasoning distinguishes between surgical IVC transection versus other postoperative complications.

Figure 7: Retinal angiograms interpreted for evidence of macular neovascularization and appropriate therapy selection.

Implications and Future Directions

OpenMedReason sets a new standard for open, clinically grounded multimodal reasoning supervision. Its design enables the study of model failure at fine-grained cognitive levels, critical for development, regulatory analysis, and clinical translation of LVLMs. Because reasoning steps are explicitly anchored in concrete case evidence, error diagnosis is transparent, facilitating model interpretability, safety analysis, and robust alignment strategies.

Practically, OpenMedReason lowers the barrier for academic evaluation and reproducible research, as the released dataset, benchmark, and checkpoints all operate within a 7B parameter footprint and do not depend on closed or proprietary systems.

Theoretically, the results strongly support that source-grounded supervision—rather than synthetic rollouts—substantially increases reasoning depth, diversity, and reliability in high-stakes medical settings, and that outcome-level RL can add orthogonal gains on top of well-initialized supervised checkpoints.

Limitations

The dataset reflects the biases of the biomedical literature—such as over-sampling of rare or educationally interesting cases and modality imbalances. While rigorous multi-stage filtering is applied, reasoning traces are only as reliable as their sources, and accuracy on OpenMedReason-Bench must not be conflated with clinical readiness.

Conclusion

OpenMedReason delivers a clinically rigorous, open dataset for training and diagnosing medical LVLMs across perception, knowledge, and reasoning dimensions, demonstrating consistent performance gains over strong baselines and enabling granular failure analysis. Its open release provides a foundation for further research on visual reasoning, robust auditing, aligned model development, and (indirectly) more transparent clinical AI systems. Continued expansion toward prospectively acquired multimodal evidence and fine-grained benchmarking across underrepresented modalities and populations will further reduce gaps between research development and clinical application.