RadLE Radiology Benchmark

• RadLE Radiology Benchmark is a curated, expert-level dataset designed to assess AI systems' diagnostic accuracy on nuanced radiology cases.
• It evaluates model performance using rigorous metrics such as accuracy, reproducibility, and inter-rater reliability in a controlled multi-run setup.
• The benchmark also identifies specific reasoning errors across perceptual, interpretive, and communication domains, emphasizing the need for domain-specific AI improvements.

Radiology’s Last Exam (RadLE) is a spectrum-biased expert-level radiology benchmark specifically constructed to evaluate frontier multimodal AI systems and compare their performance with human experts on challenging diagnostic scenarios. Developed to address limitations in existing public datasets—which largely feature common pathologies and non-complex findings—RadLE rigorously assesses AI’s spot-diagnostic capabilities via a set of expertly curated cases, purposefully taxing advanced models in domains where subtle visual reasoning and domain-specific expertise are critical (Datta et al., 29 Sep 2025).

1. Benchmark Construction and Dataset Characteristics

RadLE v1 comprises 50 single-image “spot diagnosis” studies sampled across three principal imaging modalities: radiography (n=13), computed tomography (CT, n=24), and magnetic resonance imaging (MRI, n=13). Case selection was stringent, requiring (1) recent, expert-level scenarios (all from within the preceding five years), (2) a single, unambiguous reference diagnosis derivable from the image alone, and (3) exclusion of cases reliant on ancillary data (laboratory findings, prior studies, or multi-modality correlates). This results in a dataset enriched for non-obvious, diagnostically challenging cases spanning cardiothoracic, neuro/head-neck, pediatric, gastrointestinal, and musculoskeletal systems.

Each instance features a de-identified medical image and a standardized prompt: "You are a board-certified diagnostic radiologist. Given a medical image…provide the single most specific final diagnosis…Output only the diagnosis as a single line." No additional clinical history is provided. The reference diagnosis is curated and maintained by the CRASH Lab for blinded performance assessment (Datta et al., 29 Sep 2025).

2. Evaluation Methodology and Metrics

Performance on RadLE is measured in a strictly controlled, multi-reader design involving both human and AI participants. Human cohorts include four board-certified radiologists and four radiology trainees. AI systems are evaluated through three independent runs per case, probing their “reasoning” or “thinking” modes (as available through native web interfaces). The assessment suite includes OpenAI o3 (Aug 2025), Gemini 2.5 Pro, Grok-4, Claude Opus 4.1, and OpenAI GPT-5; additional evaluations of GPT-5 span low-/medium-/high-effort reasoning via API.

Key quantitative metrics, precisely as defined in the RadLE paper (Datta et al., 29 Sep 2025), comprise:

• Accuracy

$$\text{Accuracy} \;=\; \frac{1}{N} \sum_{i=1}^N \mathbf{1}\{\hat{y}_i = y_i\}$$

where $\hat{y}_i$ is the model or reader diagnosis, $y_i$ is the reference standard, and $N=50$.

• Reproducibility (percent agreement across independent runs for each AI system):

$$\text{Percent Agreement} = \frac{\text{Identical outputs across runs}}{N} \times 100\%$$

• Inter-rater reliability: quadratic-weighted Cohen’s $\kappa$ and intraclass correlation coefficient (ICC(2,1)) across runs, supporting assessment of determinism and systematic output consistency.

3. Comparative Performance: Human and AI Cohorts

Quantitative results underscore a persistent and substantial performance gap between frontier AI and clinical experts on RadLE’s expert-level cases. Mean group-level diagnostic accuracies and confidence intervals are as follows (Datta et al., 29 Sep 2025):

Group	Mean Accuracy	95% CI
Board-certified radiologists (n=4)	0.83	0.75–0.90
Radiology trainees (n=4)	0.45	0.39–0.52
GPT-5	0.30	0.20–0.42
Gemini 2.5 Pro	0.29	0.19–0.39
OpenAI o3	0.23	0.14–0.33
Grok-4	0.12	0.06–0.19
Claude Opus 4.1	0.01	0.00–0.03

Reliability, as captured by mean Cohen’s $\kappa$ and ICC(2,1), attained highest levels for GPT-5 and OpenAI o3 (mean $\kappa = 0.64$, $\text{ICC} = 0.64$ for GPT-5), moderate levels for Gemini 2.5 Pro and Grok-4, and poor determinism for Claude Opus 4.1 (Datta et al., 29 Sep 2025).

4. Taxonomy of Visual Reasoning Errors

RadLE introduces a systematic taxonomy of AI model reasoning failures, distilled into three principal categories (Datta et al., 29 Sep 2025):

• Perceptual errors: under-detection (missed visible findings), over-detection (hallucinated findings), and mislocalization (incorrect anatomical placement of an otherwise correct feature).
  • Example: GPT-5 omits distal ureter prolapse (under-detection) or erroneously reports nonexistent spider-like renal cysts (over-detection).
• Interpretive errors: misattribution of detected features and incomplete reasoning/premature closure.
  • Example: Clavicle elevation described but labeled as posterior shoulder dislocation (misattribution); resolving to central pontine myelinolysis with incomplete reasoning in a Joubert’s syndrome case (premature closure).
• Communication errors: discordance between enumerated findings and final diagnostic label.
  • Example: Listing hyperinflation and depressed diaphragms but charting “normal chest radiograph.”

Further, RadLE identifies cognitive-bias modifiers: confirmation/anchoring bias (defaulting to prototypical diagnoses), availability bias, inattentional bias, and framing effects—all of which influence error patterns in both human and AI interpretations.

5. Failure Mode Analysis and Qualitative Findings

Failure analysis of reasoning traces revealed that even the most advanced models (e.g., GPT-5) consistently demonstrated:

• Early anchoring on incorrect hypotheses, persisting despite subsequent contradictory observations.
• Hallucination of findings with high confidence (linguistic over-certainty).
• Premature cessation of reasoning, often generating concise but incorrect final diagnoses.
• Internal inconsistencies, for example, providing logically incompatible summary statements and differentiating poorly between closely related disease entities.

These patterns were consistent across cases designed for subtlety, such as neuroimaging findings or chest radiographs with minimal overt pathology, highlighting the limitations of current AI reasoning backbones (Datta et al., 29 Sep 2025).

6. Implications for Model Development, Deployment, and Regulation

The RadLE benchmark demonstrates that current generalist multimodal AI systems, when evaluated at their reasoning limits, underperform not only board-certified radiologists but also trainees, with mean accuracy differences up to 53 percentage points. This suggests a continued necessity for specialized, domain-fine-tuned models over unmodified consumer-facing AI for critical diagnostic tasks.

RadLE meets requirements for high-complexity, expert-level benchmarks suitable for regulatory evaluation and exposes important limitations in model determinism, susceptibility to cognitive errors, and the inability to match clinical performance without domain-specific adaptation. The authors advise against unsupervised deployment of generalist AI and recommend future work in domain adaptation, improvement of perceptual submodules, and standardization of evaluation protocols on spectrum-biased collections (Datta et al., 29 Sep 2025).

Limitations of the benchmark itself include its modest size (n=50), restricted group sizes, spectrum bias that may limit generalizability to routine clinical practice, and non-public image data (to prevent dataset contamination). Even so, RadLE stands as a rigorous reference standard for evaluating the clinical readiness and diagnostic robustness of next-generation AI systems in radiology.