Radiology’s Last Exam (RadLE) Benchmark
- The paper introduces RadLE, a rigorously designed benchmark that evaluates AI models against board-certified radiologists using 50 expert-level imaging cases.
- It details a blinded, multi-run evaluation protocol with precise scoring and reproducibility metrics (Cohen’s κ and ICC) to assess diagnostic accuracy.
- Findings reveal that even advanced multimodal AI systems underperform compared to experts, highlighting the need for specialized, domain-tuned models in radiology.
Radiology’s Last Exam (RadLE) is a rigorously designed benchmark for evaluating the performance of frontier multimodal AI models against human radiology experts in challenging image-based diagnostic scenarios. Developed as a pilot benchmark of 50 expert-level “spot diagnosis” cases sampled from multiple imaging modalities and clinical systems, RadLE quantifies the current state of large language and vision–LLMs (LLMs, VLMs) when tasked with true expert-level radiology image interpretation. Its blinded, multi-run protocol and taxonomy of visual reasoning errors establish a comprehensive framework for understanding AI limitations and guiding future model development in medical imaging (Datta et al., 29 Sep 2025).
1. Benchmark Design and Dataset Construction
Fifty cases were curated by two board-certified radiologists, each with over five years of clinical experience, from a crowdsourced pool of de-identified multi-institutional cases. Inclusion criteria demanded scenarios of high diagnostic complexity commonly encountered in clinical practice or board examinations within the preceding five years, requiring that a single, unambiguous reference diagnosis be derivable solely from the presented image. Exclusion criteria eliminated cases with broad differentials or those necessitating ancillary laboratory or multimodality data for definitive diagnosis.
Imaging Modalities and Clinical Systems
| Clinical System | Number of Cases (n) | Percentage |
|---|---|---|
| Cardiothoracic | 7 | 14% |
| Gastrointestinal | 8 | 16% |
| Genitourinary | 7 | 14% |
| Musculoskeletal | 9 | 18% |
| Head/Neck/Neuro | 9 | 18% |
| Paediatric | 10 | 20% |
Difficulty and real-world relevance were ensured by spectrum bias toward complex and subtle findings, as well as confirmation that images did not overlap with public datasets such as CheXpert or MIMIC-CXR through reverse image search.
2. Evaluation Protocol
The evaluation included three reader groups: board-certified radiologists (experts, ), radiology trainees (residents, ), and five AI models, each tested in three independent runs with distinct users. The AI systems evaluated were OpenAI o3, OpenAI GPT-5 (across both web and API “low/medium/high effort” reasoning modes), Google Gemini 2.5 Pro, xAI Grok-4, and Anthropic Claude Opus 4.1. All models were tested via their native web interfaces except GPT-5, which also underwent API-based stress testing.
Each test run used a highly constrained, standardized prompt explicitly requiring a one-line diagnosis as if by a board-certified radiologist: “You are a board-certified diagnostic radiologist… Output only the diagnosis as a single line.” All outputs—human and AI—were scored under blinded conditions by expert adjudicators. AI outputs were evaluated for reproducibility via three independent runs per model with identical input conditions.
3. Scoring Criteria and Reliability Metrics
Performance was scored per case using a tiered rubric:
- Exact match: $1.0$
- Partial match: $0.5$
- Incorrect: $0.0$
Overall diagnostic accuracy was calculated as:
Reproducibility across independent runs was evaluated using two established reliability metrics:
- Quadratic-weighted Cohen’s :
where is observed agreement, and is chance agreement.
- Intraclass correlation ICC(2,1): two-way random-effects model.
No automated formula was employed for diagnostic error taxonomy; errors were annotated manually by consensus between radiologists and cognitive psychologists according to structured definitions.
4. Quantitative Performance Results
Board-certified radiologists demonstrated the highest mean diagnostic accuracy (0.83, Wilson 95% CI: 0.75–0.90). Radiology trainees achieved substantially lower accuracy (0.45, 0.39–0.52). Among AI models, GPT-5 performed best (0.30, 0.20–0.42), followed by Gemini 2.5 Pro (0.29, 0.19–0.39), OpenAI o3 (0.23, 0.14–0.33), Grok-4 (0.12, 0.06–0.19), and Claude Opus 4.1 (0.01, 0.00–0.03).
Excerpt: Modality-Specific Results
- MRI cases: Experts 0.98, Trainees 0.58, GPT-5 0.45, Gemini 0.35, o3 0.33
- CT cases: Experts 0.79, Trainees 0.57, GPT-5 0.22
Model Reproducibility Across Three Runs
| Model | 0 Range | Mean 1 | Interpretation | ICC(2,1) (95% CI) |
|---|---|---|---|---|
| GPT-5 | 0.59–0.73 | 0.64 | Substantial | 0.64 (0.50–0.76) |
| OpenAI o3 | 0.54–0.75 | 0.61 | Substantial | 0.61 (0.46–0.74) |
| Gemini 2.5 Pro | 0.47–0.61 | 0.53 | Moderate | 0.54 (0.37–0.68) |
| Grok-4 | 0.27–0.62 | 0.41 | Moderate | 0.41 (0.24–0.59) |
| Claude Opus 4.1 | –0.02–0.00 | –0.01 | Poor | ≈0 |
A negligible improvement (≈1 percentage point) was observed with extended “high-effort” reasoning, at a sixfold increase in latency (65 s vs 10 s).
5. Taxonomy of Visual Reasoning Errors
A structured, consensus-driven taxonomy of AI diagnostic errors was developed:
| Category | Subtypes / Definitions |
|---|---|
| Perceptual errors | Under-detection (missed findings), over-detection (hallucinated findings), mislocalization |
| Interpretive errors | Misinterpretation/misattribution (finding correctly seen but misattributed), premature closure |
| Communication errors | Findings–summary discordance (internal contradiction in reasoning vs. output) |
| Cognitive bias mods | Confirmation/Anchoring bias, availability bias, inattentional bias, framing effects, classification rule errors |
For example, GPT-5 under-detected a ureterocele (perceptual error), hallucinated kidney cysts not present in a normal kidney (over-detection), and mislocated mediastinal cysts. Cognitive biases such as anchoring and availability further modulated error types. Premature closure, exemplified by a diagnosis of central pontine myelinolysis without full analysis, highlighted reasoning incompleteness.
6. Implications and Recommendations
RadLE demonstrates that even advanced generalist multimodal AI underperforms experts by a large margin on challenging, high-complexity diagnostic cases. Primary limitations observed in AI models include under-detection of subtle pathology, frequent hallucinations, mislocalization phenomena, and reasoning failures amplified by cognitive biases.
The findings underscore critical recommendations:
- Prioritize benchmarks with spectrum bias and high case complexity to better expose and track AI failure modes.
- Favor medical domain-specialized, fine-tuned vision–language architectures over generalist models for clinical deployment.
- Standardize rigorous multi-run, blinded evaluation protocols featuring partial credit, reproducibility (κ, ICC), and detailed error taxonomy.
- Mandate human oversight and transparent model limitation reporting in clinical and consumer applications.
- Advocate for regulatory frameworks incorporating high-complexity diagnostic performance, reproducibility, and error taxonomies prior to AI model deployment.
By integrating a robust benchmarking approach, formal reproducibility metrics, and qualitative error analysis, RadLE contributes a template for evaluation and iterative advancement in AI for medical imaging, highlighting the persistent gap between human expertise and current generative visual LLMs (Datta et al., 29 Sep 2025).
