OmniMedVQA: Medical Visual Q&A Benchmark

• OmniMedVQA is a comprehensive benchmark for medical visual question answering that integrates 127,995 QA pairs from 73 datasets, offering broad modality and anatomical coverage.
• The construction pipeline involves manual dataset curation, automated QA template conversion using ChatGPT-3.5, and inverse proportional sampling to ensure balanced and authentic evaluations.
• The benchmark supports multiple protocols—including closed-ended, open-ended, and confidence calibration tests—providing actionable insights into model robustness and performance variance.

OmniMedVQA is a medical visual question answering benchmark introduced as “a new large-scale comprehensive evaluation benchmark for medical LVLM,” designed to evaluate large vision-LLMs on authentic medical images rather than on narrowly scoped or synthetic surrogates (Hu et al., 2024). In its original form, it contains 118,010 images and 127,995 question-answer items collected from 73 medical datasets, spans 12 imaging modalities and more than 20 anatomical regions, and converts source classification or attribute annotations into closed-ended VQA instances (Hu et al., 2024). Since its release, the name has also come to denote several protocol variants built on top of the benchmark, including open-ended reformulations, robustness and calibration testbeds, and contamination audits of a public mirror, so reported “OmniMedVQA” results are not always tied to the same data split or answer format (Alsinglawi et al., 8 Apr 2025, Pramana et al., 22 Dec 2025, Xu et al., 8 Jun 2026).

1. Benchmark conception and medical scope

OmniMedVQA was proposed to address a structural limitation in earlier medical VQA resources: prior datasets such as VQA-RAD, SLAKE, Path-VQA, and VQA-Med were small and narrow in modality or anatomy coverage, and some used paper-derived figures rather than images from native medical datasets (Hu et al., 2024). The benchmark therefore emphasizes breadth and realism. Its 12 modalities are Colposcopy, CT, Digital Photography, Fundus Photography, Infrared Reflectance Imaging, MR, OCT, Dermoscopy, Endoscopy, Microscopy Images, X-Ray, and Ultrasound, and its anatomical coverage includes lung, mammary gland, hand, upper limb, eye, uterus, intestine, skin, shoulder, kidney, gallbladder, pancreas, spleen, liver, pelvic region, ovary, blood vessel, spine, urinary system, adipose tissue, muscle tissue, oral cavity, knee, foot, and lower limb (Hu et al., 2024).

The benchmark is assembled from 73 source medical classification datasets and is explicitly positioned as an evaluation resource rather than a model paper. Of these source datasets, 42 are fully open access, 31 are restricted access, and one is partially open; for restricted datasets, only evaluation QA items plus mapping instructions are released so that the benchmark can be reconstructed after obtaining the underlying data (Hu et al., 2024). The emphasis on authentic medical scenarios is central to its intended use: OmniMedVQA is meant to test foundational medical image understanding across heterogeneous specialties rather than only radiology-style recognition (Hu et al., 2024).

2. Construction pipeline and task design

The construction pipeline has four stages. First, the authors manually collected the 73 source datasets. Second, they converted native labels and metadata into QA templates. Disease labels became disease-diagnosis questions; anatomy and modality labels became anatomy-identification or modality-recognition questions; severity annotations became lesion-grading questions; and other medically meaningful attributes such as cell shape, cancer status, imaging direction, laterality, or eye state became “other biological attributes” questions (Hu et al., 2024). Third, ChatGPT-3.5 was used to diversify wording and generate incorrect answer choices. Fourth, human double checking and an inverse proportional sampling strategy were used to control template imbalance and reduplication (Hu et al., 2024).

The resulting benchmark contains five question families: Modality Recognition with 19,427 items, Anatomy Identification with 20,330 items, Disease Diagnosis with 73,455 items, Lesion Grading with 2,621 items, and Other Biological Attributes with 12,162 items (Hu et al., 2024). By modality, MR and Microscopy Images are the largest contributors, with 32,705 and 21,743 QA items respectively, followed by CT with 15,836 and Ultrasound with 10,991 (Hu et al., 2024).

Although the underlying questions are medically open in content, the released benchmark is transformed into a closed-ended multiple-choice task with 2–4 candidate options and exactly one correct answer (Hu et al., 2024). The original paper evaluates models in two ways. “Question-answering Score” feeds the image, question, and options to the LVLM with a prompt instructing it to select one option, then maps the free-form response to the most similar candidate option. “Prefix-based Score” instead scores each candidate option by conditional likelihood, using the image as a visual prefix to the textual option, to reduce the effect of imperfect multiple-choice instruction following (Hu et al., 2024). This dual evaluation protocol is one of the benchmark’s defining methodological features.

3. Original zero-shot evaluation and early empirical conclusions

The benchmark paper evaluates 12 LVLMs in a zero-shot setting: eight general-domain systems and four medical-specialized systems (Hu et al., 2024). On the full benchmark, random-guess accuracy is 28.28% because the number of options varies across items. Under Question-answering Score / Prefix-based Score, the best overall model is BLIP-2 at 50.69 / 33.43, followed by InstructBLIP at 42.49 / 28.71; among medical-specialized models, MedVInT reaches 41.50 / 25.81, Med-Flamingo 36.17 / 26.54, RadFM 26.82 / 29.00, and LLaVA-Med 28.78 / 24.06 (Hu et al., 2024).

The type-level breakdown is uneven. Under Question-answering Score, InstructBLIP is best on Modality Recognition at 70.62 and Lesion Grading at 54.60, while BLIP-2 is best on Anatomy Identification at 49.19, Disease Diagnosis at 46.24, and Other Biological Attributes at 73.52 (Hu et al., 2024). Under Prefix-based Score, RadFM leads Modality Recognition at 38.57, BLIP-2 leads Anatomy Identification at 64.75, and mPLUG-Owl leads Disease Diagnosis at 29.32, Lesion Grading at 76.96, and Other Biological Attributes at 43.70 (Hu et al., 2024). Modality-level results likewise show that general-domain models dominate many cells, while medical-specialized models retain pockets of strength on selected modalities such as MR, Ultrasound, CT, or X-Ray (Hu et al., 2024).

The benchmark’s main empirical conclusion is therefore not that medical specialization is useless, but that existing medical LVLM specialization was uneven and often too narrow relative to the benchmark’s modality breadth (Hu et al., 2024). The paper attributes this to weak medical image-text alignment, limited diversity in medical training corpora, and overconcentration on a few modalities. In that sense, OmniMedVQA functions as a stress test for breadth rather than as a single-domain leaderboard (Hu et al., 2024).

4. Protocol variants and derivative evaluation regimes

Later work has reused OmniMedVQA under substantially different protocols. A large-scale benchmarking study evaluated it alongside MedXpert, PMC-VQA, PathVQA, MMMU, SLAKE, and VQA-RAD, using the prompt “Please reason step by step, and put the final answer in $\backslash\boxed\{\}$.” The boxed answer was extracted and scored by accuracy against the ground-truth option. That study also decomposed non-MedXpert datasets into predicted “understanding” and “reasoning” subsets using a classifier trained on MedXpert embeddings. On OmniMedVQA, Lingshu-32B was best overall at 0.7662, best on reasoning at 0.7526, and best on understanding at 0.7120; notably, every model in the table had slightly higher reasoning than understanding accuracy on this benchmark (Liu et al., 15 Jul 2025).

Another line of work explicitly reformulated OmniMedVQA into an open-ended generation task. In that setting, the original multiple-choice options were removed and answer IDs were replaced by answer text, yielding a “Revised OmniMedVQA.” The usable public subset contained 82,405 images and 88,996 QA pairs, was split with a 70:30 train/test ratio, and was evaluated on 7,930 images and 8,832 question-answer pairs. The reported outcome was 73.4% overall accuracy, with 70.7% on open-end questions and 76.9% on yes/no questions (Alsinglawi et al., 8 Apr 2025). This was not an evaluation of the released closed-ended benchmark as such, but a deliberate task reformulation.

OmniMedVQA has also been used in confidence-calibration work under yet another protocol. A recent calibration study constructed a 20,000-sample open-access pool and used 4,000 training and 3,000 test samples, instructing models to emit a short rationale, an option answer, and a confidence score on a 1–10 scale. In that setup, the benchmark served less as a raw accuracy test and more as a multimodal uncertainty-calibration substrate (Senoglu et al., 25 Jun 2026). This suggests that “OmniMedVQA performance” is protocol-dependent in a strong sense: reported numbers can refer to original multiple-choice zero-shot evaluation, open-ended reformulation, confidence-conditioned answering, or capability decomposition, and those numbers are not directly commensurate.

5. Role in later model development

OmniMedVQA rapidly became a standard external benchmark for medical multimodal model development. In “MMedExpert-R1,” it is part of a MedEvalKit suite alongside PMC-VQA, GMAI-MMBench, and MedXpertQA-MM, and the main reported OmniMedVQA result is 83.03 from the 7B MMedExpert-R1 model, narrowly above Lingshu’s 82.85. The same paper also reports an appendix-only data-efficient setting in which MMedExpert-R1 was trained on 1,000 OmniMedVQA samples and reached 77.93; that appendix experiment is explicitly distinguished from the main 83.03 benchmark result (Ding et al., 16 Jan 2026).

Other model papers use OmniMedVQA as a broad capability readout rather than as a benchmark-specific architectural target. “MedXIAOHE” reports 83.40 on OmniMedVQA, above GPT-5.2 Thinking at 73.95, Gemini 2.5 Pro at 71.86, and Gemini 3.0 Pro at 81.96, and treats the benchmark as part of a unified medical imaging evaluation framework (Shi et al., 13 Feb 2026). An entity-centric data-engineering framework based on Medical Entity Trees reports 83.36 on OmniMedVQA, slightly above Lingshu’s 82.90, and uses the benchmark to argue that ontology-guided data curation improves broad medical VQA transfer (Lin et al., 28 Apr 2026).

OmniMedVQA has also been used to study failure modes beyond nominal accuracy. “SafeMed-R1” instantiates it as 88,995 image-question pairs spanning eight modalities, split into 71,333 training and 17,662 test examples, and shows that clean fine-tuned Qwen3-VL-4B models can reach 95.43% accuracy on clean inputs yet collapse to 26.07% or 25.14% under PGD attack. The adversarially trained SafeMed-R1 “Think” model retains 84.45% attacked accuracy under the same evaluation (Pramana et al., 22 Dec 2025). In that usage, OmniMedVQA is less a leaderboard benchmark than a test of whether high clean accuracy is compatible with robust multimodal reasoning.

6. Contamination, public mirrors, and interpretive caveats

A major interpretive complication emerged when a contamination audit examined not the original benchmark release, but “the public Hugging Face mirror of OmniMedVQA.” That mirror exposes only a train split; after dropping one corrupt image and capping the run to the first 5,000 rows, the effective sample is 4,999 examples. The audit therefore does not treat OmniMedVQA “as a held-out benchmark in the same sense as SLAKE-En, PathVQA, or VQA-RAD,” but as “an auxiliary text-side contamination stress test” (Xu et al., 8 Jun 2026).

On that 4,999-example public mirror, canonical-order exchangeability is significant for InternVL3-8B, Qwen2.5-VL-7B-Instruct, CheXagent-8b, LLaVA-OneVision-7B, and MedGemma-4B-IT, with reported $p$-values of $\le 10^{-4}$, $\le 10^{-4}$, $\le 10^{-4}$, $8\times10^{-4}$, and $2.8\times10^{-3}$, while the external non-medical baseline BLIP-2 remains clean at $p=1.000$. The release-order signal disappears under hash ordering for the five firing models, and the authors recommend treating the public OmniMedVQA mirror as contaminated for text-side evaluation (Xu et al., 8 Jun 2026). The same paper is careful not to equate this with proof of exact memorization: it reports no image-side audit for OmniMedVQA, no per-example membership proof, and frames the result as statistical evidence of likely leakage or source overlap rather than verified training-set inclusion (Xu et al., 8 Jun 2026).

Other papers expose related leaderboard caveats from a different angle. “MMedExpert-R1” explicitly states that “MedVLM-R1 and Med-R1 is trained on part of the OmniMedVQA test set, making its results on OmniMedVQA meaningless,” and therefore excludes those values from clean comparison in its main table (Ding et al., 16 Jan 2026). The open-ended reformulation paper, meanwhile, warns of an “accuracy paradox,” namely that repetitive dataset structure may allow models to achieve high measured accuracy without robust multimodal generalization (Alsinglawi et al., 8 Apr 2025). Taken together, these results suggest that OmniMedVQA should be read as a family of related but non-identical evaluation artifacts: the original closed-ended benchmark, task-specific public subsets, open-ended derivatives, calibration pools, and a public train-split mirror used for contamination stress testing. Direct cross-paper comparison therefore requires explicit attention to split provenance, answer format, and contamination assumptions.