AdversarialAnatomyBench Evaluation

• AdversarialAnatomyBench is a benchmark that exposes VLM bias by testing on rare, clinically significant anatomical variants using paired imaging cases.
• The dataset comprises 200 multi-modal samples equally split between typical and adversarial images to rigorously evaluate anatomical bias.
• Empirical results reveal a drastic accuracy drop of around 45 percentage points, underscoring the need for debiasing strategies in clinical AI.

AdversarialAnatomyBench is a benchmark designed to systematically expose and quantify the failure modes of state-of-the-art vision–LLMs (VLMs) on naturally occurring rare anatomical variants, termed "natural adversarial anatomy." These are clinically real imaging cases where human anatomy strongly deviates from typical textbook presentations (e.g., polydactyly, horseshoe kidney, situs inversus), yet remains visually unambiguous to trained clinicians. Existing VLM benchmarks only capture common anatomy and thereby obscure critical limitations—namely, that such models overwhelmingly rely on strong learned priors about "typical" anatomy, producing high-confidence errors on rare but clinically important variants even when these are visually clear. AdversarialAnatomyBench fills this gap by providing the first systematically curated, multi-modal test set explicitly aimed at the measurement of anatomical bias in medical AI (Mayer et al., 3 Dec 2025).

1. Motivation and Formalization

"Natural adversarial anatomy" comprises real-world medical images where the ground truth labels are visually identifiable, but where population priors violate observed features. The motivation for AdversarialAnatomyBench is that VLMs have exhibited catastrophic generalization failure on such rare presentations, even as they are increasingly considered for clinical workflows. Traditional benchmarks (e.g., CheXpert, MURA) measure performance purely on common cases, missing critical blind spots.

The data is formally split into:

• Typical anatomy set: $$\mathcal{D}_{typ} = \{(x_i,\,y_i)\}_{i=1}^{N_{typ}}$$
• Adversarial anatomy set: $$\mathcal{D}_{adv} = \{(x_j,\,y_j)\}_{j=1}^{N_{adv}}$$
• Full benchmark: $$\mathcal{D} = \mathcal{D}_{typ} \cup \mathcal{D}_{adv}$$

For AdversarialAnatomyBench, $N_{typ} = N_{adv} = 100$.

Each VLM $f$ is evaluated by comparing predictions $\hat y = f(x)$ against ground truth $y$ separately on typical and atypical cases.

2. Dataset Composition and Diversity

The benchmark comprises 200 imaging samples (100 typical, 100 atypical) spanning seven imaging modalities and 20 anatomical regions:

Modality	Typical ($N_{typ,m}$)	Atypical ($N_{adv,m}$)
X-ray	15	15
CT	15	15
MRI	14	14
MRA	14	14
Ultrasound	14	14
Fluoroscopy	14	14
Photography	14	14
Totals	100	100

These span anatomical regions such as bones/joints (e.g., digits, limbs), thorax (heart apex laterality), abdomen (kidney/ureter variants), craniofacial (teeth), vasculature, and soft tissue. Each typical image is paired with an atypical variant, encompassing conditions like extra or missing digits (polydactyly/syndactyly), fused or absent organs (horseshoe kidney, renal agenesis), and mirrored anatomical structures (situs inversus).

3. Task Types and Evaluation Strategy

VLMs are assessed using perception-level medical tasks:

• Anatomical structure identification (e.g., finger/teeth counting)
• Laterality judgments (e.g., determining the side of heart apex)
• Anomaly detection (e.g., detecting fused kidneys vs. two separate kidneys)

Evaluation metrics:

1. Accuracy on typical cases:

$$\mathrm{Acc}_{typ} = \frac{1}{N_{typ}}\sum_{(x,y)\in \mathcal{D}_{typ}} \mathbf{1}(f(x)=y)$$

2. Accuracy on adversarial (atypical) cases:

$$\mathcal{D}_{adv} = \{(x_j,\,y_j)\}_{j=1}^{N_{adv}}$$0

3. Performance drop:

$$\mathcal{D}_{adv} = \{(x_j,\,y_j)\}_{j=1}^{N_{adv}}$$1

4. Bias rate on atypical cases: the fraction of responses that default to the prior (typical) anatomy:

$$\mathcal{D}_{adv} = \{(x_j,\,y_j)\}_{j=1}^{N_{adv}}$$2

where $$\mathcal{D}_{adv} = \{(x_j,\,y_j)\}_{j=1}^{N_{adv}}$$3 indicates the answer associated with the typical case.

Confidence intervals are computed through stratified bootstrapping (1,000 replicates).

4. Empirical Results and Metric Outcomes

When benchmarking 22 modern VLMs (including GPT-5, Gemini 2.5 Pro, Llama 4 Maverick), three critical outcomes were observed:

• Sharp accuracy drop: Mean $$\mathcal{D}_{adv} = \{(x_j,\,y_j)\}_{j=1}^{N_{adv}}$$4 whereas $$\mathcal{D}_{adv} = \{(x_j,\,y_j)\}_{j=1}^{N_{adv}}$$5, yielding $$\mathcal{D}_{adv} = \{(x_j,\,y_j)\}_{j=1}^{N_{adv}}$$6 percentage points.
• Consistent model-wide bias: Each model exhibited catastrophic failure to handle rare variants, regardless of scale or prompting.
• Top model summary:

Model	$$\mathcal{D}_{adv} = \{(x_j,\,y_j)\}_{j=1}^{N_{adv}}$$7	$$\mathcal{D}_{adv} = \{(x_j,\,y_j)\}_{j=1}^{N_{adv}}$$8	$$\mathcal{D}_{adv} = \{(x_j,\,y_j)\}_{j=1}^{N_{adv}}$$9
GPT-5 (R)	79.9%	43.0%	36.9 pp
Gemini 2.5 Pro (R)	81.0%	40.0%	41.0 pp
Llama 4 Maverick	67.1%	39.8%	27.3 pp
Mean (22 VLMs)	74%	29%	45 pp

95% confidence intervals confirmed that these observed drops are robust across question types.

5. Failure Mode Analysis and Bias Quantification

Model errors on atypical cases overwhelmingly aligned with typical-anatomy priors:

• In polydactyly ($$\mathcal{D} = \mathcal{D}_{typ} \cup \mathcal{D}_{adv}$$0 fingers), 80% of responses defaulted to "5 fingers."
• For horseshoe kidney, 95% of responses asserted "two separate kidneys."
• Ureteral duplication: 82% of responses matched "two ureters."

Measured bias rates ranged from 65% to 95% across models, with no model achieving a bias rate below 41%. This demonstrates a predominant reliance on population priors over visually observed rare features, effectively overriding clear evidence to produce high-confidence but incorrect outputs.

6. Mitigation Attempts and Recommendations

Various mitigation strategies were tested:

• Architectural scaling: Increasing parameters in Qwen3-VL from 2B to 235B led to modest increases in typical accuracy (from ~71% to ~78%) but left atypical accuracy flat (~25%).
• Bias-aware prompting: Phrasing prompts to suggest rare cases produced $$\mathcal{D} = \mathcal{D}_{typ} \cup \mathcal{D}_{adv}$$1 gains of +5–13 pp for select models, but did not close the performance gap.
• Test-time reasoning: Modulating token budgets caused inconsistent changes (±1–4 pp), with no substantial reduction in the accuracy drop.
• "Unsure" response allowance: Elicited in $$\mathcal{D} = \mathcal{D}_{typ} \cup \mathcal{D}_{adv}$$2 of typical and $$\mathcal{D} = \mathcal{D}_{typ} \cup \mathcal{D}_{adv}$$3 of atypical cases, indicating overconfidence persists even in rare presentations.

Recommended directions include architectural changes to penalize over-reliance on priors, explicit inclusion of rare variants in training, and the establishment of regulatory standards mandating performance and bias reporting on rare-case benchmarks.

A plausible implication is that untested deployment of VLMs risks misdiagnosis in patients with rare anatomical variants and necessitates radiologist-in-the-loop workflows or specialized debiasing protocols (Mayer et al., 3 Dec 2025).

7. Significance and Future Impact

AdversarialAnatomyBench establishes the first rigorous and systematic standard for evaluating anatomical bias in multimodal medical AI. It exposes a previously unmeasured and substantial limitation in generalization for state-of-the-art VLMs, regardless of model size, prompting strategies, or auxiliary reasoning.

By making performance on such rare and clinically critical presentations measurable and comparable, this benchmark creates a foundation for:

• Systematic mitigation of bias in model development and deployment.
• Regulatory oversight of VLM-based diagnostic tools.
• A template for rare-variant evaluation in other high-stakes domains.

In sum, AdversarialAnatomyBench is central to the assessment and improvement of VLM robustness in real-world clinical environments, addressing a critical gap in the current paradigm for medical AI validation (Mayer et al., 3 Dec 2025).