OpenMedReason-Bench: Grounded Medical Reasoning

• OpenMedReason-Bench is a benchmark for evaluating grounded medical reasoning in large vision-language models through a held-out 1.5K sample split.
• It employs an axis-decomposed scoring protocol across perception, medical knowledge, and rationale to clearly diagnose a model’s reasoning trace fidelity.
• The benchmark ensures non-overlap with training data and has demonstrated notable improvements, with trace scores rising significantly after supervised fine-tuning.

Searching arXiv for the primary paper and closely related benchmark papers to ground the article in current sources. {"query":"OpenMedReason scientific reasoning supervision for medical vision-LLMs arXiv (Baghbanzadeh et al., 10 Jun 2026)","max_results":5} OpenMedReason-Bench is the held-out evaluation companion to OpenMedReason, a large-scale multimodal medical reasoning resource introduced for large vision-LLMs (LVLMs) in high-stakes clinical settings. It is a separate 1.5K-sample split reserved exclusively for evaluation, and its central purpose is diagnostic rather than merely leaderboard-oriented: instead of only testing whether a model produces the correct final answer, it evaluates whether the model’s reasoning trace is grounded in the relevant visual evidence, invokes appropriate medical knowledge, and supplies a case-specific rationale that connects evidence to conclusion (Baghbanzadeh et al., 10 Jun 2026).

1. Definition and evaluation role

OpenMedReason-Bench is presented as a benchmark for grounded medical reasoning rather than a conventional visual question answering accuracy test. The broader OpenMedReason resource is an approximately 450K-example multimodal medical reasoning dataset built from biomedical publication figures and associated text, with source-grounded reasoning traces intended to be more faithful than purely synthetic chain-of-thought rollouts. OpenMedReason-Bench is the held-out slice of that pipeline and is reserved exclusively for evaluation (Baghbanzadeh et al., 10 Jun 2026).

The paper frames the benchmark as a mechanism for “capability-resolved diagnosis” of LVLM behavior. In that framing, a model may achieve a correct final answer while still omitting a relevant finding, failing to invoke the right clinical fact, or supplying a shallow explanation. OpenMedReason-Bench is intended to expose precisely those distinctions. It therefore operationalizes an evaluation target that is broader than final-answer accuracy: whether a model can produce a grounded reasoning trace rather than merely a correct label (Baghbanzadeh et al., 10 Jun 2026).

This positioning places OpenMedReason-Bench within a broader shift toward deeper medical-AI evaluation. Related work has argued that standard benchmarks can create an “evaluation illusion” in which strong label prediction or surface alignment obscures failures in clinical reasoning (Jing et al., 26 Sep 2025). In a different modality regime, DR.BENCH similarly reframed diagnostic reasoning as a generation-centered evaluation problem rather than a pure extraction or classification task (Gao et al., 2022).

2. Held-out construction and benchmark independence

The benchmark construction is tightly coupled to the OpenMedReason data curation pipeline. That pipeline filters biomedical publication figures for visual usability and textual adequacy, assigns clinical task categories, generates USMLE-style questions with anti-leakage constraints, and then creates source-grounded reasoning traces. OpenMedReason-Bench is a held-out slice of the same overall process, so its evaluation logic mirrors the training signal while remaining independent from it (Baghbanzadeh et al., 10 Jun 2026).

A central design feature is non-overlap with training sources. The paper emphasizes that OpenMedReason-Bench does not overlap with training sources at the source-article, image, or question level. Its role is therefore not to test memorization of figures or answers from the training corpus, but to assess whether a model trained on source-grounded supervision has learned generalizable reasoning behavior (Baghbanzadeh et al., 10 Jun 2026).

This suggests that the benchmark is intended to test grounded reasoning transfer under controlled source separation. A plausible implication is that reported gains on the benchmark are meant to be interpretable as improvements in reasoning behavior rather than artifact reuse from the training set.

3. Trace decomposition and scoring protocol

The distinctive methodological contribution of OpenMedReason-Bench is its trace-level, axis-decomposed evaluation. For each example, the reference reasoning trace is converted into a compact checklist of atomic unit claims, and every unit is assigned to exactly one of three complementary capability axes: perception, medical knowledge, or rationale. Candidate traces are then probed twice for each unit: once for presence, meaning whether the trace addresses that topic at all, and once for correctness, meaning whether it states the unit claim accurately. This permits the benchmark to separate omitted evidence from incorrect evidence (Baghbanzadeh et al., 10 Jun 2026).

The three axes are defined as follows.

Axis	Definition
Perception	The image-grounded part of reasoning: modality, anatomy, lesion appearance, spatial pattern, counts, or other visual facts
Medical knowledge	General clinical facts that should hold independently of the individual case
Rationale	The case-specific inferential bridge from the evidence to the answer

Formally, for axis $$a \in \{\text{perception}, \text{medical knowledge}, \text{rationale}\}$$, with $N_a$ checklist units and per-unit presence/correctness labels $p_i$ and $c_i$, the paper defines:

$$\mathrm{Presence}_a = \frac{1}{N_a} \sum_{i \in a} \frac{p_i}{2}.$$

This normalizes presence to $[0,1]$ and reflects how much of the required reasoning content the candidate attempted to cover. Correctness is measured only over units that were actually present:

$$\mathrm{Correctness}_a = \frac{ \left|\{i \in a : p_i \geq 1 \ \wedge\ c_i = +1\}\right| }{ \left|\{i \in a : p_i \geq 1\}\right| }.$$

The headline score is then

$$\mathrm{TraceScore} = \frac{1}{2} \sum_a \mathrm{Presence}_a \cdot \mathrm{Correctness}_a.$$

The paper explicitly reports six axis-level metrics—presence and correctness for each of the three axes—plus an aggregate trace score. The interpretation of this design is crucial: the benchmark is not just a final-answer test, but a diagnostic instrument that reveals whether a model fails by omitting relevant visual evidence, misstating medical facts, or failing to connect evidence to answer (Baghbanzadeh et al., 10 Jun 2026).

4. Modal coverage and reasoning targets

OpenMedReason-Bench covers mixed medical vision modalities rather than only conventional radiology. The paper states that the corpus and benchmark span more than eight imaging modalities overall, and the benchmark table and evaluation appendix describe coverage including radiology, pathology/microscopy, visible-light clinical photography, charts/plots, and other biomedical figures. The broader evaluation suite also includes radiology, pathology, ophthalmology, dermatology, endoscopy, mammography, ultrasound, and general biomedical figures (Baghbanzadeh et al., 10 Jun 2026).

This diversity matters because the benchmark is intended to test whether a reasoning model can handle the full range of visual evidence encountered in biomedical papers, not just one imaging domain. The target behavior is therefore multimodal scientific-medical reasoning grounded in heterogeneous figure types.

The benchmark’s diagnostic emphasis also distinguishes it from neighboring evaluation paradigms. R2MED addresses reasoning-driven medical retrieval, where relevance is mediated by an inferred diagnosis or reasoning path rather than lexical similarity (Li et al., 20 May 2025). Med-RewardBench, by contrast, evaluates whether a model can judge and rank medical responses in a clinician-aligned way, rather than whether it can solve the reasoning task itself (Ding et al., 29 Aug 2025). OpenMedReason-Bench belongs to the solver-oriented branch: it evaluates whether a model can perceive evidence, invoke medical knowledge, and generate a coherent rationale.

5. Quantitative results and axis-level behavior

The paper reports very large gains on OpenMedReason-Bench after training on OpenMedReason. In the ablation table, the Qwen2.5-VL-7B base model scores 47.11 on OpenMedReason-Bench. After supervised fine-tuning on OpenMedReason, this rises to 77.39, and after SFT plus GRPO it rises slightly further to 78.51. The authors use this result to argue that source-grounded reasoning supervision materially improves reasoning behavior, not just answer selection (Baghbanzadeh et al., 10 Jun 2026).

In the cross-model comparison table, the final OpenMedReason-trained Qwen2.5-VL-7B checkpoint achieves 78.51 on OpenMedReason-Bench, outperforming multiple comparable or larger reference systems.

Model	OpenMedReason-Bench
Qwen2.5-VL-7B base	47.11
Qwen2.5-VL-7B + SFT	77.39
Qwen2.5-VL-7B + SFT + GRPO	78.51
MedVL-Thinker 7B	52.03
QoQ-Med-VL 7B	47.86
OctoMed 7B	51.16
MedGemma 27B	48.22
Lingshu 7B	63.20

Across VQA benchmarks overall, the OpenMedReason-trained 7B model achieves an average VQA score of 60.04, described in the paper as the strongest among comparable-scale medical LVLMs, with an 11.7% relative gain over the next-best comparable model (Baghbanzadeh et al., 10 Jun 2026).

For reasoning-specific analysis, the paper reports that the OpenMedReason-trained model achieves the highest Reasoning Trace Score at 77.8, compared with Qwen2.5-VL-7B before post-training at 37.4. Its axis-level scores are as follows.

Axis	Presence / Correctness / Score
Perception	74.3 / 87.8 / 65.2
Medical knowledge	89.2 / 95.3 / 85.0
Rationale	87.9 / 94.5 / 83.1

The authors specifically highlight that the biggest relative improvement is in presence, especially for medical knowledge, and interpret this as the model learning to surface the clinically relevant evidence more reliably. The gains are therefore not concentrated in a single axis; the reported evidence indicates joint improvement in perception, medical knowledge, and rationale (Baghbanzadeh et al., 10 Jun 2026).

6. Preference evaluation, auditability, and significance

OpenMedReason-Bench is complemented by pairwise reasoning-trace preference evaluation on 1,500 shared instances. A judge model compares traces head-to-head, and the paper reports that the OpenMedReason checkpoint is preferred over the base model in 86.1% of pairwise comparisons. The trained checkpoint is favored across all four baseline matchups shown in the figure, with the largest margin against the pre-trained Qwen2.5-VL backbone (Baghbanzadeh et al., 10 Jun 2026).

This pairwise-preference result complements the benchmark’s trace score. The paper’s interpretation is that the trained traces are preferred because they surface better evidence and better reasoning structure, not merely because they terminate in the right answer. The benchmark and its trace-scoring protocol are also presented as more clinically auditable than a simple accuracy metric, because a reviewer can inspect whether success came from grounded reasoning (Baghbanzadeh et al., 10 Jun 2026).

In the broader benchmark landscape, this emphasis on auditable reasoning traces aligns with a wider movement toward deeper evaluation in medical AI. Neural-MedBench argues for a “Two-Axis Evaluation Framework” separating breadth-oriented statistical generalization from depth-oriented reasoning fidelity (Jing et al., 26 Sep 2025). DR.BENCH similarly emphasizes evidence integration, medical knowledge representation, and diagnosis generation in clinical NLP (Gao et al., 2022). OpenMedReason-Bench contributes to that trajectory by making multimodal medical reasoning traceable along three explicit axes—perception, medical knowledge, and rationale—and by tying benchmark success to grounded, inspectable evidence use rather than final-answer correctness alone (Baghbanzadeh et al., 10 Jun 2026).