MediConfusion Benchmark Evaluation

• MediConfusion Benchmark is a specialized evaluation protocol for multimodal medical imaging that tests the ability to differentiate visually confusing yet clinically distinct cases.
• It constructs 176 confusing image pairs from the ROCO dataset using stringent similarity criteria to force reliance on true visual reasoning rather than language context.
• Empirical results reveal that state-of-the-art models perform below random guessing on pairwise tasks, highlighting critical reliability issues in safety-critical healthcare environments.

MediConfusion Benchmark is a specialized evaluation protocol for multimodal LLMs (MLLMs) in medical imaging, explicitly designed to probe fine-grained failure modes involving visually confusable but clinically distinct cases. Unlike prior medical VQA datasets solvable by text priors, MediConfusion targets tasks unsolvable by language context alone, requiring true visual reasoning and discrimination between "look-alike" conditions. Empirical findings reveal that both medical and proprietary foundation models perform below random guessing on challenging pairwise tasks, highlighting urgent reliability gaps for deployment in safety-critical healthcare settings (Sepehri et al., 2024, Patel et al., 2 Feb 2026).

1. Dataset Construction and Selection Criteria

MediConfusion draws upon the ROCO radiology dataset (≈80,000 medical images with captions) to construct a set of 176 confusing image pairs, each pair forming the basis for two visual question answering (VQA) instances. The benchmark encompasses nine clinical/anatomical categories: cerebral, spinal, cardiac, gastrointestinal, musculoskeletal, vascular, pulmonary, head & neck, and nuclear medicine.

Pair Selection Procedure

Pairs are algorithmically selected to maximize confusion for medical-domain encoders but not for general vision encoders:

• Medical similarity ($$\mathrm{sim}_{\mathrm{med}}(i,j) = \cos(e_{\mathrm{BiomedCLIP}}(i),\,e_{\mathrm{BiomedCLIP}}(j))$$) is required to satisfy $\mathrm{sim}_{\mathrm{med}}(i,j)\geq0.9$.
• General-domain dissimilarity ($$\mathrm{sim}_{\mathrm{gen}}(i,j) = \cos(e_{\mathrm{DINOv2}}(i),\,e_{\mathrm{DINOv2}}(j))$$) must satisfy $\mathrm{sim}_{\mathrm{gen}}(i,j)\leq0.75$.

This ensures the selection of pairs that are feature-ambiguous in the medical domain but visually distinct for nonspecialized vision models (Sepehri et al., 2024).

VQA Generation and Annotation

Each pair is presented to GPT-4 with both captions to generate a multiple-choice question whose answer differs between the two images, resulting in two VQA items per pair. Radiologists subsequently score each instance on correctness, clinical relevance, and medical language precision (1–10 scale), with only question pairs scoring ≥5 on both examples retained (Sepehri et al., 2024).

2. Task Formalism and Evaluation Metrics

The core MediConfusion task formalizes paired VQA for binary (A/B) image-level decision making:

• Input: An image pair $I=\{i_1,i_2\}$, a question $q\in Q$, and answer set $\mathcal A=\{A,B\}$
• Output: A model $f$ maps $(I,q)\to\hat a\in\mathcal A$

Primary evaluation metrics are:

Metric	Definition
Ind. Accuracy	$$\mathrm{Acc}_{\rm ind} = \frac{1}{N_Q}\sum_{k=1}^{N_Q} \mathbf{1}[f(I_k,q_k)=a_k]$$
Set Accuracy	$$\mathrm{Acc}_{\rm set} = \frac{1}{N_{\rm pair}} \sum_{p=1}^{N_{\rm pair}} \mathbf{1}[f(I_{2p-1},q_{2p-1})=a_{2p-1}\wedge f(I_{2p},\,q_{2p})=a_{2p}]$$
Confusion Score	Proportion of pairs with the same predicted answer $$\mathrm{Confusion} = \frac{1}{N_{\rm pair}}\sum_{p=1}^{N_{\rm pair}}\mathbf{1}[\hat a_{2p-1}=\hat a_{2p}]$$

Random guessing yields $\mathrm{Acc}_{\rm ind}^{\rm rand}=0.5$ and $\mathrm{Acc}_{\rm set}^{\rm rand}=0.25$. Model performance is thus judged not just by image-level accuracy, but stringently by set accuracy and confusion collapse (Sepehri et al., 2024, Patel et al., 2 Feb 2026).

3. Model Performance and Comparative Baselines

MediConfusion exposes that state-of-the-art MLLMs—including LLaVA-7B, BLIP-2, InstructBLIP, LLaVA-Med, RadFM, Med-Flamingo, as well as commercial models such as GPT-4o, Claude 3 Opus, and Gemini 1.5 Pro—achieve set accuracies significantly below the random baseline. Confusion scores usually exceed 75%, establishing a pattern of models defaulting to the same answer for visually confusable pairs and neglecting fine-grained distinction.

Model	Acc_set (%)	Acc_ind (%)	Confusion (%)
LLaVA-7B	9.1	51.7	85.8
BLIP-2	6.8	51.4	94.9
InstructBLIP	12.5	53.1	90.3
LLaVA-Med	1.1	49.7	99.4
RadFM	5.7	50.3	98.3
Med-Flamingo	4.6	52.0	94.9
GPT-4o	18.8	56.3	75.0
Claude 3 Opus	8.5	50.9	84.1
Gemini 1.5 Pro	19.9	51.1	58.5
Random Guess	25.0	50.0	–

This benchmark is fundamentally distinct from prior VQA datasets (VQA-Med, Path-VQA, VQA-Rad), which are solvable via language priors and do not probe the visual ambiguity targeted by MediConfusion (Sepehri et al., 2024).

4. Failure Mode Analysis and Confusion Taxonomy

LLM-based prompting and radiologist review categorize the 176 image pairs into four clinical confusion patterns:

1. Normal/variant anatomy vs. pathology: e.g., normal C1–C2 alignment vs. atlantoaxial subluxation.
2. Lesion signal characteristics: e.g., solid vs. cystic lesions on T2-weighted MRI.
3. Vascular conditions: e.g., aneurysm vs. normal vessel bifurcation.
4. Medical devices: e.g., presence or absence of stents.

All models display high confusion scores (>75%) across categories, with set accuracy rarely surpassing 25%. GPT-4o achieves relatively higher set accuracy (34.25%) and individual accuracy (67.12%) on vascular cases, but confusion remains pronounced overall (Sepehri et al., 2024).

5. Advances via Contrastive Reasoning: DoubleTake and CCI

The DoubleTake system (Patel et al., 2 Feb 2026) demonstrates that standard nearest-neighbor retrieval (e.g., Top-k by similarity) reinforces dominant hypotheses and fails to support discriminative reasoning for confusing cases. In contrast, a contrastive, document-aware reference selection strategy—specifically, a triad comprising an anchor (most relevant), hard negative (differently embedded), and boundary probe (distinct provenance/lexical overlap)—enables more faithful decision-making.

• Counterfactual-Contrastive Inference (CCI): For each query, CCI executes structured pairwise comparisons with reference images, aggregates votes via confidence-weighted margins, and employs abstention/secondary adjudication for ambiguous or same-answer pairs.
• Method ablations show that removing either hard negative or boundary probe drops set accuracy by ≈12 points, while disabling pair-level adjudication causes confusion to surge over 63%.

Method	Set Acc (%)	Img Acc (%)	Conf (%)	Abstain (%)
Direct (single-shot)	29.6	61.9	67.1	–
Top-k retrieval (k=3)	30.7	51.4	30.1	3.1
DoubleTake CCI (full)	43.8	64.2	36.9	3.7
Random guessing	25.0	50.0	50.0	–

An absolute 14.2 point gain in set-level accuracy is attained over direct inference, marking a 48% relative boost. The confusion rate falls nearly in half, demonstrating the efficacy of explicit contrastive reasoning protocols for resolving ambiguous medical image pairs (Patel et al., 2 Feb 2026).

6. Recommendations for Model and Benchmark Development

Several priorities emerge for future research and deployment:

• Model improvements: Pretraining specialized medical image encoders, integrating explicit spatial-reasoning and laterality tasks, training for better recognition of devices and visual cues, and employing adversarial/contrastive image perturbations to increase robustness (Sepehri et al., 2024).
• Benchmark expansion: Incorporating additional imaging modalities (ultrasound, PET), per-pattern annotations, bounding boxes for failure localization, and direct clinical expert feedback during model refinement.
• Deployment best practices: Employing MediConfusion as an adversarial safety check in clinical AI pipelines to surface and address systematic model vulnerabilities before real-world integration (Sepehri et al., 2024).

7. Scientific and Clinical Significance

The MediConfusion Benchmark establishes a critical evaluation axis for the discriminative competence of multimodal medical foundation models. By focusing on settings that frustrate language priors and demand fine-grained visual differentiation, MediConfusion re-centers evaluation on clinically meaningful ambiguities. Initial results underscore that currently available models—both open and proprietary—are susceptible to high rates of confusion in precisely the types of subtle, expert-level discriminations where reliability is non-negotiable.

The DoubleTake/CCI framework points toward a new paradigm in medical AI: contrastive, provenance-aware evidence construction coupled with faithful abstention and adjudication. This suggests that progress on MediConfusion may serve as a barometer for readiness of AI radiologists in safety-critical workflows, and that similar benchmarks are necessary to stress-test future models for robust deployment in medical imaging (Sepehri et al., 2024, Patel et al., 2 Feb 2026).