DiagBench: Clinical LLM Evaluation Benchmark

• DiagBench is a diagnostic benchmarking framework that assesses LLMs' clinical reasoning using dynamic, multi-step evaluation and physician-validated cases.
• It curates 750 detailed cases from the MIMIC-IV dataset, covering 863 ICD-coded diseases with comprehensive patient and examination data.
• The framework employs rigorous evaluation protocols—including single-turn, end-to-end, and rubric-based scoring—to measure diagnostic accuracy and decision-making quality.

DiagBench is a diagnostic benchmarking framework for evaluating LLMs on clinical reasoning, diagnosis process management, and examination recommendation. Developed as part of a virtual clinical environment, it is designed to measure dynamic, multi-step diagnostic skills using physician-validated cases, outcome-based feedback, and expert-written process rubrics. Distinct from previous static benchmarking protocols, DiagBench targets realistic, interactive diagnosis policy learning and systematically annotates not only final outcomes but also the intermediate reasoning and decision-making processes (Qiu et al., 28 Oct 2025).

1. Composition and Data Curation

DiagBench comprises 750 physician-validated diagnostic trajectories, each representing an admission episode from the MIMIC-IV dataset and covering 863 distinct ICD-coded diseases. Each case is extensively structured, including items such as chief complaint, history of present illness, social/family history, allergies, and the ground-truth final diagnosis. Examination modalities per case are distributed as follows: an average of 8.77 physical exams, 28.37 laboratory events, 2.04 microbiology events, and 2.01 radiology events.

Data construction utilizes MIMIC-IV discharge notes and event tables. Patient profiles are extracted with a hybrid approach combining heuristic heading matching and LLM-assisted reformatting. Only cases for which the final diagnosis does not appear in the past history and those retaining plausible pre-admission events are included. Test names are standardized via MIMIC-CDM mapping, and the final case pool is curated and validated for clinical plausibility by expert review.

A subset of 99 cases is further annotated with 973 physician-authored process rubrics to enable fine-grained analysis of diagnostic reasoning beyond outcome accuracy (Qiu et al., 28 Oct 2025).

2. Process Rubrics: Structure and Annotation

For rubric-rich process evaluation, 99 cases are randomly sampled from the main pool. Two practicing physicians independently author rubrics for each, targeting key decision points (history collection, hypothesis formation, examination selection, and closure points). A third physician reconciles rubric coverage and clarity, excising insufficiently annotated cases. Finally, a fourth physician assigns an importance weight $w_r \in [0,10]$ to each rubric criterion.

Each rubric is a concise decision/process criterion (e.g., “Ensure CBC is ordered before imaging when infection is suspected”), categorized by diagnostic phase such as Initial Hypothesis, Test Selection, or Interpretation. These rubrics quantitatively encode process quality and allow for nuanced evaluation across entire diagnostic trajectories, not just at the point of diagnosis.

3. Evaluation Protocols and Tasks

DiagBench supports two primary task protocols: single-turn and end-to-end evaluation.

• Single-Turn Evaluation: At each step in a multi-turn clinical trajectory, the agent is presented with ground-truth history up to that point and must either recommend the next examination or make a final diagnosis. The metric is the Hit Ratio for examination recommendations (whether the agent's recommended exam occurs in the subsequent gold-standard chain) and accuracy for the diagnosis turn.
• End-to-End Evaluation: The agent interacts with the DiagGym world model, simulating each examination's outcome and iteratively selecting further actions until it elects to make a final diagnosis. Two metric variants are employed: (a) automatic scoring over all 750 cases using exam-list precision, recall, F1, and diagnostic accuracy, and (b) rubric-based scoring for the 99 annotated cases.

Agent guidelines explicitly require rationale generation for examination selection and final diagnosis.

4. Evaluation Metrics

DiagBench implements a comprehensive and formally specified metric suite:

• Diagnostic Accuracy:

$$\text{Accuracy} = \frac{\#\{\text{cases with correct diagnosis}\}}{\#\{\text{evaluated cases}\}}$$

Validated to account for synonym variation with an LLM judge.

• Examination Recommendation Hit Ratio (Single-Turn):

$$\text{HitRatio} = \frac{\#\{\text{recommended exams that appear in reference chain}\}}{\#\{\text{exam turns}\}}$$

• End-to-End Exam Recommendation Metrics (agent predictions $\hat{E}$, reference set $E$):

$$\text{Precision} = \frac{|\hat{E} \cap E|}{|\hat{E}|}, \qquad \text{Recall} = \frac{|\hat{E} \cap E|}{|E|}, \qquad F_1 = 2 \cdot \frac{\text{Precision} \times \text{Recall}}{\text{Precision} + \text{Recall}}$$

• Weighted Rubric Score (rubric set $R_c$ for case $c$, weights $w_r$):

$$s_c = \frac{\sum_{r \in A_c} w_r}{\sum_{r \in R_c} w_r}, \qquad \bar{s} = \frac{1}{99}\sum_{c=1}^{99} s_c$$

Where $$\text{Accuracy} = \frac{\#\{\text{cases with correct diagnosis}\}}{\#\{\text{evaluated cases}\}}$$0 is the set of satisfied rubrics in case $$\text{Accuracy} = \frac{\#\{\text{cases with correct diagnosis}\}}{\#\{\text{evaluated cases}\}}$$1, judged by a GPT-4o-based system.

5. Baseline Architectures and Performance Benchmarks

Multiple state-of-the-art LLMs and agentic paradigms are evaluated:

• General-purpose LLMs: GPT-4o, Claude-4-sonnet, Llama3.3-70B, DeepSeek-v3-671B, Qwen2.5-72B, Qwen3-235B, GPT-OSS-120B, OpenBioLLM-70B, Baichuan-M1-14B.
• Medical-specialized models: MedGemma-27B
• Agentic systems: MedAgents (multidisciplinary LLM cooperation) and MDAgents (adaptive collaborative agents).

Table of key performance deltas (DiagAgent-14B vs. best baseline, by setting):

Setting	Exam Hit Ratio	Diagnostic Accuracy	F1 Exam Recommendation	Rubric Score (Weighted)
Single-Turn	+44.03 pp	+9.34 pp	—	—
End-to-End	—	+15.12 pp	+23.09	—
Rubric Eval	—	—	—	+7.1 pp

For example, in the end-to-end setting, DiagAgent-14B achieves an F1 of 42.89 on exam recommendations (vs. 19.80 for DeepSeek-v3), and 62.20% diagnostic accuracy (vs. 47.08%). In weighted rubric evaluation, DiagAgent-14B achieves 32.86% (vs. 25.76% for Qwen3-235B) (Qiu et al., 28 Oct 2025).

6. Usage, Accessibility, and Limits

DiagBench is strictly an evaluation dataset. The labeled split includes 750 cases for automatic metrics and a 99-case analytical subset for rubric-based evaluation. Complementary training sets—for agent policy development—are described separately (over 130k simulated EHR trajectories).

Benchmark construction, evaluation pipelines, the DiagGym world model, and DiagAgent implementations are available at https://github.com/MAGIC-AI4Med/DiagGym. MIMIC-IV data access requires PhysioNet credentialing and compliance.

The focus on realism, rubric granularity, and policy-centric benchmarks distinguishes DiagBench among clinical diagnostic evaluation suites. Benchmark limitations are primarily rooted in the properties of MIMIC-IV and challenge the generalization of diagnostic skills to other datasets or specialties, but case selection criteria and rubric annotation procedures aim to maximize both clinical fidelity and evaluation rigor (Qiu et al., 28 Oct 2025).