RadImageNet-VQA Radiologic Visual Q&A Dataset

• The dataset RadImageNet-VQA is engineered through expert curation phases including image sourcing, radiologic caption synthesis, and tailored Q&A pair generation.
• It supports comprehensive evaluation with diverse question types—open-ended, closed-ended, and multiple-choice—across anatomy, abnormality, and pathology tasks.
• Advanced distractor curation and fine-tuning protocols enhance model performance in fine-grained pathology identification despite persistent diagnostic challenges.

RadImageNet-VQA is a large-scale visual question answering (VQA) dataset designed for radiologic reasoning on CT and MRI imagery. It responds to the limitations of existing medical VQA resources, which are typically dominated by X-ray images, lack sufficient data breadth, and are prone to text-based shortcut exploitation. Built with expert curation, RadImageNet-VQA contains 750,000 expert-annotated images paired with 7.5 million question-answer samples, supporting comprehensive evaluation of both general-purpose and medical-domain vision–LLMs on fine-grained clinical tasks (Butsanets et al., 19 Dec 2025).

1. Dataset Composition and Construction

RadImageNet-VQA is derived from the CT/MRI subset of the broader RadImageNet repository. The construction pipeline comprises three critical phases:

1. Image Sourcing: 750,000 CT and MRI slices, labeled by radiologists, serve as the data backbone. Each image is annotated with its imaging modality, anatomical region, and associated pathologies.
2. Radiologic Caption Synthesis: Each image receives a structured caption, automatically verbalized from metadata using template-based natural language prompts (e.g., “A contrast-enhanced CT of the abdomen/pelvis shows a hepatic lesion”).
3. VQA Pair Generation: Diverse question-answer pairs are generated through templates, spanning three tasks (anatomy recognition, abnormality detection, pathology identification) and three formats (open-ended, closed-ended, multiple-choice).

The multiple-choice format features distractors drawn from plausible alternative pathologies within the same anatomical region; a “no pathology seen” option is always provided to mitigate bias. Data stratification is as follows:

Imaging Modality	Images	Percent of Total
CT	450,000	60%
MRI	300,000	40%
Total	750,000	100%

Anatomical Region	Images	% of Dataset
Abdomen/Pelvis	150,000	20%
Ankle/Foot	75,000	10%
Brain	112,500	15%
Hip	37,500	5%
Knee	112,500	15%
Chest	150,000	20%
Shoulder	75,000	10%
Spine	37,500	5%
Total	750,000	100%

The dataset covers 97 fine-grained pathology categories across the eight anatomical regions.

2. Formal Task Definitions

RadImageNet-VQA centers on three supervised radiologic VQA tasks over 2D slices, each implemented with its own objective:

• Anatomy Recognition: Given an image $I$, infer its region $r \in \mathcal{R}$, with $|\mathcal{R}|=8$. Trained via multi-class cross-entropy loss:

$$\mathcal{L}_{\mathrm{anat}} = -\sum_{r' \in \mathcal{R}} \mathbb{1}[r' = r^{*}] \log p(r'|I)$$

• Abnormality Detection: Assign a label $y \in \{0\,(\text{normal}), 1\,(\text{abnormal})\}$. Uses binary cross-entropy:

$$\mathcal{L}_{\mathrm{abn}} = -[y \log p(1|I) + (1 - y) \log p(0|I)]$$

• Pathology Identification: Predict the fine-grained pathology $p \in \mathcal{P}_{r^{*}}$ for given region $r^*$, over region-specific pathology classes:

$$\mathcal{L}_{\mathrm{path}} = -\sum_{p' \in \mathcal{P}_{r^{*}}} \mathbb{1}[p' = p^{*}] \log p(p'|I, r^{*})$$

Question types per task are:

• Open-ended: Free-text response (e.g., "What anatomical region is this scan showing?").
• Closed-ended: Binary answers (e.g., "Is there any abnormality present in the image?"—yes/no).
• Multiple-choice (MC): Forced choice among four options, carefully curated with realistic distractors and a “no pathology seen” alternative.

3. Benchmark Construction and Experimental Protocol

A rigorously stratified benchmark subset comprises 1,000 images and 9,000 QA pairs, balanced across tasks, question types, and normal/abnormal cases. This benchmark supports both zero-shot and fine-tuned testing and includes:

• Evaluation of Vision–LLMs: Both general-purpose (LLaVA-OneVision-Qwen2-7B, InternVL3.5 variants, GPT-5, Gemini 2.5 Pro) and medical-specific models (LLaVA-Med, HuatuoGPT-Vision, MedGemma, Lingshu) are assessed.
• Fine-tuning Regimen: Two-phase protocol involves visual alignment (freeze LLM, update vision) and instruction tuning (all modules), with training on RadImageNet-VQA, KiTS22, AbdomenAtlas, VQA-RAD, and SLAKE. Technical hyperparameters include batch size 4, AdamW optimizer, learning rates of $1 \times 10^{-5}$ (LLM) and $r \in \mathcal{R}$0 (vision), 4× H100 GPUs, 3% warm-up, and cosine decay scheduling.
• Evaluation Metrics:
  • Accuracy: $r \in \mathcal{R}$1
  • F1 score: $r \in \mathcal{R}$2

4. Quantitative Results and Performance Analysis

Zero-shot evaluations on the benchmarked models produce the following observed accuracy ranges (task × question format):

Task (Question Type)	Open-ended (%)	Closed-ended (%)	Multiple-choice (%)
Anatomy	48–66	72–98	80–94
Abnormality	27–75	49–74	59–75
Pathology	9–31	47–89	26–47
Average	48–64	—	—

InternVL3.5-14B achieves the top accuracy among general-purpose models (63.6%). MedGemma-4B attains the highest for open-ended pathology (30.6%). Fine-grained pathology, especially in open-ended form, presents the lowest performance across all architectures, with most models achieving less than 20% accuracy.

Ablation Studies: In text-only (image-ablated) settings, VLMs reach 11–33% accuracy on VQA-RAD/SLAKE, but drop to near-random (2–10%) for open-ended and 25% for MC on RadImageNet-VQA, verifying effective minimization of linguistic shortcuts.

Effect of Fine-Tuning:

• Multiple-choice anatomy: >98%
• Closed-ended abnormality: 85–88% (improvement +16 to +40 percentage points)
• Open-ended pathology: ~40% (improvement +25 to +30 points)
• Mean overall accuracy lift: +19.5 to +22.5 points.

Despite these gains, pathology identification persists as the dominant bottleneck, remaining ~30 percentage points behind the other tasks after model adaptation.

5. Methodological Advances and Validation Strategies

RadImageNet-VQA introduces several methodological innovations and validation criteria to ensure benchmark rigor:

• Distractor Curation: Multiple-choice distractors are not arbitrary but sampled from regionally and clinically plausible alternative findings, with a default "no pathology seen" answer to suppress prior bias.
• Template Diversification: Broader template-based question synthesis minimizes patterns that could be leveraged as cues.
• Ablation-based Validation: Text-only ablation robustly demonstrates the absence of shortcut exploitation, a limitation of legacy datasets.
• Balanced Task and Format Distribution: Careful stratification guards against task and data distributional artifacts.

6. Implications, Open Challenges, and Future Directions

Empirical results highlight open problems and inform future research trajectories:

• Persistent Difficulty in Fine-Grained Pathology: Fine-grained disease recognition continues to challenge VLMs, even following large-scale, multimodal instruction tuning. This suggests an urgent need for models with enriched visual reasoning and pathology-specialized features.
• Model Scaling Over Domain Specialization: Results indicate that large general-purpose VLMs, especially InternVL variants, may surpass smaller medical-specific models, pointing to the disproportionate impact of scale and pretraining diversity relative to current domain-specific architectural choices.
• Benchmark Validity and Utility: By eliminating linguistic shortcuts, RadImageNet-VQA establishes a robust substrate for the assessment of genuine visual reasoning in radiologic VQA.

Recommendations for advancing the field include:

• 3D/integrated volumetric context for cross-slice VQA.
• Hybrid models combining vision transformers and explicit spatial reasoning.
• Use of contrastive/self-supervised pretraining for improved pathology semantics.
• Prospective clinical validation of VQA-augmented radiologist workflow.

RadImageNet-VQA is publicly accessible at https://huggingface.co/datasets/raidium/RadImageNet-VQA and provides both unprecedented scale and methodological rigor for developing and benchmarking vision–LLMs capable of clinically meaningful radiologic reasoning (Butsanets et al., 19 Dec 2025).