DR.BENCH Diagnostic Reasoning Benchmark

• DR.BENCH is a generative AI benchmarking suite for clinical NLP that evaluates evidence gathering, knowledge inference, and diagnostic synthesis.
• It reformulates diverse clinical tasks into a sequence-to-sequence framework to enable standardized, multi-task evaluation of language models.
• In-domain pretraining on clinical notes significantly boosts performance, setting a new state-of-the-art in diagnostic reasoning benchmarks.

DR.BENCH denotes the Diagnostic Reasoning Benchmark, a generative artificial intelligence benchmarking suite designed for clinical natural language processing and diagnostic reasoning tasks. Initiated to address limitations in existing clinical NLP evaluation, DR.BENCH explicitly models core cognitive workflows in patient care, moving beyond traditional entity extraction and classification to assess evidence comprehension, medical knowledge reasoning, and diagnosis generation. Its unified framework enables standardized evaluation of pretrained LLMs within a clinically relevant context.

1. Conceptual Overview and Rationale

DR.BENCH was developed in response to the increasing information overload encountered by clinicians due to the expansion of electronic health records (EHRs). Diagnostic error, amplified by fragmented EHR content, remains a leading cause of medical mistakes. While previous clinical NLP tasks emphasized classification and extraction, DR.BENCH introduces multi-component generative benchmarks designed to mirror genuine forward diagnostic reasoning: evidence gathering, knowledge-based inference, and diagnostic synthesis (Gao et al., 2022, Sharma et al., 2023). By recasting each evaluation as a sequence-to-sequence task, DR.BENCH facilitates robust assessment of decision-oriented LLMs.

2. Suite Structure and Task Design

DR.BENCH comprises six core tasks organized into three principal categories. Each task is derived from public datasets and reformulated as a seq2seq generation problem:

Category	Task Name	Description
Medical Knowledge Representation	MedNLI	Clinical NLI with premise–hypothesis pairs (14,049 samples)
	AP Reasoning	Labeling causal Assessment–Plan relations (5,897 samples)
Evidence Understanding & Integration	emrQA	QA over de-identified discharge summaries (53,199 QA pairs)
	SOAP Labeling	Classify sentences into SOAP note sections (134,089 samples)
Diagnosis Generation & Summarization	MedQA	USMLE-style board exam QA, open-book format (12,725 pairs)
	ProbSumm	Generate active problem/diagnoses lists from notes (2,783 samples)

The Problem Summarization (ProbSumm) task is recognized as the most challenging and actionable, requiring abstractive synthesis of patient issues from multi-section progress notes.

3. Model Architectures and Pretraining Paradigms

Experiments are conducted using the T5 text-to-text transformer architecture (12–24 layers, 220M–770M parameters). To evaluate clinical reasoning capacity, models are pretrained either:

• Out-of-domain: C4-T5 (general web text)
• Biomedical: SciFive (PubMed, PMC)
• Clinical: Clinical-T5 (MIMIC-III EHR notes)

Domain-adaptive pretraining, i.e., further pretraining on unlabeled clinical notes, is critical for aligning vocabulary and token distributions with EHR-style texts (Sharma et al., 2023). Prefix-based prompts are utilized to designate task context to the model (e.g., “summarize: <SOAP> …”).

4. Multi-task Training Objectives and Protocols

All six tasks are formulated via cross-entropy, either at sequence or token level, enabling a unified multi-task learning approach:

$L_{total} = \sum_{i=1}^6 \lambda_i L_i$

where $L_i$ is standard negative log-likelihood for generation/classification. Uniform task sampling ($\lambda_i=1 \forall i$) is employed; epochs are balanced to present each task equally, and prefix tokens signal requisite output format. This structure allows joint optimization across medical reasoning, evidence integration, and clinical reporting.

5. Benchmarking Results and Comparative Analysis

Key experiments investigate (a) out-of-domain vs. in-domain pretraining and (b) single-task vs. multi-task finetuning, particularly on the Problem Summarization (ProbSumm) challenge, scored by ROUGE-L:

Model	Pretraining	Training	ROUGE-L
Best prior DR.BENCH	—	Single	24.00 (19.86–28.13)
T5-220M (C4)	Web	Multi	24.84 (20.28–29.40)
SciFive-770M	PubMed, PMC	Multi	25.31 (21.45–29.17)
Clinical-T5-770M	MIMIC-III	Single	28.28 (24.10–32.46)
Clinical-T5-770M	MIMIC-III	Multi	28.55 (24.29–32.80)

In-domain pretraining on clinical notes results in a substantial performance increase, with Clinical-T5 (multi-task) establishing a new state-of-the-art at 28.55 ROUGE-L (+4.55 over prior best). Multi-task finetuning confers smaller gains; the principal factor is domain-adaptive pretraining.

6. Error Analysis, Insights, and Limitations

Expert review reveals that multi-task Clinical-T5 is sometimes prone to over-extraction (listing detailed physiologic states rather than abstract diagnoses), whereas single-task Clinicial-T5 better abstracts diagnoses in some cases. Multi-task training enhances recall for clinical entities, but may dilute task-specific abstraction. A limitation is reliance on ROUGE-L, which may inadequately reflect semantic and clinical correctness. Further, transductive leakage may affect generalizability, as Clinical-T5 is trained on the same corpus (MIMIC-III) from which DR.BENCH annotations are derived.

Carbon footprint analyses indicate multi-task training incurs substantially higher emissions (∼35.5 kg CO₂ vs. ∼4.5 kg CO₂ for single-task), highlighting resource considerations. The evaluation framework does not yet incorporate semantically informed metrics (e.g., BERTScore) or support for decoder-only architectures capable of handling longer-context clinical documents.

7. Implications for Clinical AI and Research Directions

DR.BENCH underscores the necessity of in-domain data for optimization of diagnostic reasoning capabilities in clinical AI systems. Multi-task generative training fosters shared inductive biases, supporting unified reasoning models for EHR tasks in place of specialized models per task. Proposed future directions include implementation of semantically-aware evaluation, extension to external datasets for improved generalization, adoption of prompt-tuning, architectural innovation toward decoder-only models, and reduction of computational expense.

The systematic organization of evidence comprehension, medical knowledge representation, and diagnosis generation in DR.BENCH offers a robust paradigm for advancing clinical decision support and for benchmarking LLMs against complex clinical reasoning targets. DR.BENCH serves as a foundation for harmonized evaluation and progressive improvement of clinical AI methodologies (Gao et al., 2022, Sharma et al., 2023).