--- title: Radiology’s Last Exam (RadLE) Benchmark url: https://www.emergentmind.com/topics/radiology-s-last-exam-radle-76e37a23-cec8-4fb2-a6f7-e12a902c8247 type: topic --- # Radiology’s Last Exam (RadLE) Benchmark Radiology’s Last Exam (RadLE) is a rigorously designed benchmark for evaluating the performance of frontier multimodal AI models against human radiology experts in challenging image-based diagnostic scenarios. Developed as a pilot benchmark of 50 expert-level “spot diagnosis” cases sampled from multiple imaging modalities and clinical systems, RadLE quantifies the current state of large language and vision–language models (LLMs, VLMs) when tasked with true expert-level radiology image interpretation. Its blinded, multi-run protocol and taxonomy of visual reasoning errors establish a comprehensive framework for understanding AI limitations and guiding future model development in medical imaging [2509.25559]. ## 1. Benchmark Design and Dataset Construction Fifty cases were curated by two board-certified radiologists, each with over five years of clinical experience, from a crowdsourced pool of de-identified multi-institutional cases. Inclusion criteria demanded scenarios of high diagnostic complexity commonly encountered in clinical practice or board examinations within the preceding five years, requiring that a single, unambiguous reference diagnosis be derivable solely from the presented image. Exclusion criteria eliminated cases with broad differentials or those necessitating ancillary laboratory or multimodality data for definitive diagnosis. ### Imaging Modalities and Clinical Systems | Modality | Number of Cases (n) | Percentage | |-------------------|--------------------|------------| | Radiograph (X-ray)| 13 | 26% | | CT | 24 | 48% | | MRI | 13 | 26% | | Clinical System | Number of Cases (n) | Percentage | |--------------------------|--------------------|------------| | Cardiothoracic | 7 | 14% | | Gastrointestinal | 8 | 16% | | Genitourinary | 7 | 14% | | Musculoskeletal | 9 | 18% | | Head/Neck/Neuro | 9 | 18% | | Paediatric | 10 | 20% | Difficulty and real-world relevance were ensured by spectrum bias toward complex and subtle findings, as well as confirmation that images did not overlap with public datasets such as CheXpert or MIMIC-CXR through reverse image search. ## 2. Evaluation Protocol The evaluation included three reader groups: board-certified radiologists (experts, $n=4$), radiology trainees (residents, $n=4$), and five AI models, each tested in three independent runs with distinct users. The AI systems evaluated were OpenAI o3, OpenAI GPT-5 (across both web and API “low/medium/high effort” reasoning modes), Google Gemini 2.5 Pro, xAI Grok-4, and Anthropic Claude Opus 4.1. All models were tested via their native web interfaces except GPT-5, which also underwent API-based stress testing. Each test run used a highly constrained, standardized prompt explicitly requiring a one-line diagnosis as if by a board-certified radiologist: “You are a board-certified diagnostic radiologist… Output only the diagnosis as a single line.” All outputs—human and AI—were scored under blinded conditions by expert adjudicators. AI outputs were evaluated for reproducibility via three independent runs per model with identical input conditions. ## 3. Scoring Criteria and Reliability Metrics Performance was scored per case using a tiered rubric: - Exact match: $1.0$ - Partial match: $0.5$ - Incorrect: $0.0$ Overall diagnostic accuracy was calculated as: \[ \text{Accuracy} = \frac{\sum_i \text{Score}_i}{\text{Total Cases}} \] Reproducibility across independent runs was evaluated using two established reliability metrics: - Quadratic-weighted Cohen’s $\kappa$: \[ \kappa = \frac{P_o - P_e}{1 - P_e} \] where $P_o$ is observed agreement, and $P_e$ is chance agreement. - Intraclass correlation ICC(2,1): two-way random-effects model. No automated formula was employed for diagnostic error taxonomy; errors were annotated manually by consensus between radiologists and cognitive psychologists according to structured definitions. ## 4. Quantitative Performance Results Board-certified radiologists demonstrated the highest mean diagnostic accuracy (0.83, Wilson 95% CI: 0.75–0.90). Radiology trainees achieved substantially lower accuracy (0.45, 0.39–0.52). Among AI models, GPT-5 performed best (0.30, 0.20–0.42), followed by Gemini 2.5 Pro (0.29, 0.19–0.39), OpenAI o3 (0.23, 0.14–0.33), Grok-4 (0.12, 0.06–0.19), and Claude Opus 4.1 (0.01, 0.00–0.03). ### Excerpt: Modality-Specific Results - MRI cases: Experts 0.98, Trainees 0.58, GPT-5 0.45, Gemini 0.35, o3 0.33 - CT cases: Experts 0.79, Trainees 0.57, GPT-5 0.22 ### Model Reproducibility Across Three Runs | Model | $\kappa$ Range | Mean $\kappa$ | Interpretation | ICC(2,1) (95% CI) | |-----------------|----------------|--------------|---------------|-----------------------| | GPT-5 | 0.59–0.73 | 0.64 | Substantial | 0.64 (0.50–0.76) | | OpenAI o3 | 0.54–0.75 | 0.61 | Substantial | 0.61 (0.46–0.74) | | Gemini 2.5 Pro | 0.47–0.61 | 0.53 | Moderate | 0.54 (0.37–0.68) | | Grok-4 | 0.27–0.62 | 0.41 | Moderate | 0.41 (0.24–0.59) | | Claude Opus 4.1 | –0.02–0.00 | –0.01 | Poor | ≈0 | A negligible improvement (≈1 percentage point) was observed with extended “high-effort” reasoning, at a sixfold increase in latency (65 s vs 10 s). ## 5. Taxonomy of Visual Reasoning Errors A structured, consensus-driven taxonomy of AI diagnostic errors was developed: | Category | Subtypes / Definitions | |-----------------------|--------------------------------------------------------------------------------------------------------| | Perceptual errors | Under-detection (missed findings), over-detection (hallucinated findings), mislocalization | | Interpretive errors | Misinterpretation/misattribution (finding correctly seen but misattributed), premature closure | | Communication errors | Findings–summary discordance (internal contradiction in reasoning vs. output) | | Cognitive bias mods | Confirmation/Anchoring bias, availability bias, inattentional bias, framing effects, classification rule errors | For example, GPT-5 under-detected a ureterocele (perceptual error), hallucinated kidney cysts not present in a normal kidney (over-detection), and mislocated mediastinal cysts. Cognitive biases such as anchoring and availability further modulated error types. Premature closure, exemplified by a diagnosis of central pontine myelinolysis without full analysis, highlighted reasoning incompleteness. ## 6. Implications and Recommendations RadLE demonstrates that even advanced generalist multimodal AI underperforms experts by a large margin on challenging, high-complexity diagnostic cases. Primary limitations observed in AI models include under-detection of subtle pathology, frequent hallucinations, mislocalization phenomena, and reasoning failures amplified by cognitive biases. The findings underscore critical recommendations: - Prioritize benchmarks with spectrum bias and high case complexity to better expose and track AI failure modes. - Favor medical domain-specialized, fine-tuned vision–language architectures over generalist models for clinical deployment. - Standardize rigorous multi-run, blinded evaluation protocols featuring partial credit, reproducibility (κ, ICC), and detailed error taxonomy. - Mandate human oversight and transparent model limitation reporting in clinical and consumer applications. - Advocate for regulatory frameworks incorporating high-complexity diagnostic performance, reproducibility, and error taxonomies prior to AI model deployment. By integrating a robust benchmarking approach, formal reproducibility metrics, and qualitative error analysis, RadLE contributes a template for evaluation and iterative advancement in AI for medical imaging, highlighting the persistent gap between human expertise and current generative visual language models [2509.25559]. Source: https://www.emergentmind.com/topics/radiology-s-last-exam-radle-76e37a23-cec8-4fb2-a6f7-e12a902c8247