DoctorFLAN-test: LLM Clinical Evaluation Benchmark

• DoctorFLAN-test is a rigorously constructed evaluation benchmark that measures large language models' performance on doctor-facing tasks within authentic clinical workflows.
• It comprises 550 expert-curated Chinese Q&A items across 22 clinical tasks and 27 specialties, ensuring diverse and realistic assessments.
• The benchmark utilizes both automated LLM-as-judge and human expert evaluations to highlight workflow alignment and fine-tuning improvements in model performance.

DoctorFLAN-test is a rigorously constructed, single-turn evaluation benchmark designed to assess the capacity of LLMs to perform doctor-facing tasks aligned with real-world clinical workflows. Released as part of the DoctorFLAN corpus, it addresses the need for trustworthy, workflow-aligned evaluation in medical LLM research—particularly for models that act as clinical assistants rather than direct patient advisors. DoctorFLAN-test comprises 550 expert-curated Chinese Q&A items partitioned across 22 clinically relevant tasks and 27 medical specialties. Each sample emulates realistic doctor queries drawn from authentic clinical education and medical practice, with reference answers generated and verified through a multistep expert review pipeline. The benchmark enables granular, phase-specific evaluation and has become a central resource for measuring and improving LLM alignment with physician needs (Xie et al., 2024, Xie et al., 13 Oct 2025).

1. Construction and Dataset Structure

DoctorFLAN-test is sampled from the full DoctorFLAN dataset (≈92,000 items), which itself covers the end-to-end clinical workflow as defined by a panel of clinical experts. The 550 DoctorFLAN-test items are constructed by randomly selecting 25 representative examples from each of 22 workflow-specific task categories. The source pool comprises expert-vetted MCQs (Medtiku), medical encyclopedia questions (120Ask), and prompt datasets (PromptCBLUE). All questions are reformulated to natural instructions and normalized by 27 board-certified specialists, thereby ensuring authentic, role-consistent prompts and diverse clinical representation (Xie et al., 2024).

Each benchmark item consists of three components:

1. Instruction: A concise, role-aligned system prompt for the task (e.g., "Based on the patient's history and symptoms, list differential diagnoses and describe confirmatory steps").
2. Free-form question: Typically a patient scenario or clinical vignette.
3. Reference answer: Expert-written, expanded, and grounded in authoritative medical sources, averaging 150–300 Chinese characters with a focus on professional depth and utility.

Table: Task and Specialty Breakdown

Phase	Tasks	Sample Specialties (from 27 total)
Pre-Diagnosis	Triage (1)	Cardiology, Pediatrics, Neurology, etc.
Diagnosis	11 tasks (Inquiry, Symptom, Case, etc.)	Oncology, Gastroenterology, Ophthalmology
Treatment	9 tasks (Plan, Complications, Surgery)	Endocrinology, Orthopedics, Emergency Med
Post-Treatment	Health Guidance, Follow-up	Multidisciplinary, General Surgery, ENT

2. Data Generation and Validation Methodology

Question sourcing applies multi-layered deduplication (Jaccard similarity ≥ 0.8) and task-specific regular expressions, iteratively revised with senior physician input to ensure task-to-query fidelity. Each candidate Q&A is mapped to a clinical task, and standardized instructions are drafted to reflect authentic workflow needs.

Reference answers are generated in two stages. First, answers are expanded by prompting GPT-4 to rewrite draft content for professional granularity and explicit source grounding. Second, a subset (50 per task, 1,050 total) is evaluated by three independent board-certified medical reviewers for:

• Correctness: No factual inaccuracies (requirement: 100% correctness).
• Practicality: Clinical utility at or above original reference (requirement: ≥99.9%). Items failing these criteria are revised in situ, and all reference material is subject to further senior expert oversight (Xie et al., 2024, Xie et al., 13 Oct 2025).

3. Benchmark Task Spectrum

The 22 tasks are designed to cover the entire clinical workflow, supporting both phase-specific and generalist evaluations. For each task, 25 distinct single-turn Q&As ensure statistical robustness. The workflow phases are:

• Pre-Diagnosis: Triage.
• Diagnosis: Includes Inquiry Prompt, Symptom Inquiry, Case Summary, Differential Diagnosis, Next Examinations, Test Results Interpretation, Definitive Diagnosis, Disease Grading.
• Treatment: Includes Emergency Advice, Treatment Plan, Medication Inquiry, Medication Advice, Complications Analysis, Treatment Adjustment, Surgery Necessity, Surgical Plan, Preoperative Education.
• Post-Treatment: Health Guidance, Follow-up Plan.

All 27 specialties—spanning disciplines such as Hematology, Stomatology, Rheumatology, and Vascular Surgery—are represented, with precise mapping ensuring diversity and eliminating specialty bias.

4. Evaluation Methodologies and Metrics

Evaluation proceeds via two complementary axes:

A. Automatic (LLM-as-Judge) Evaluation

• GPT-4 is instructed to “act as an impartial judge” (system prompt), considering accuracy, completeness, coherence, and relevance, and issuing a scalar rating on a 1–10 scale per response.
• Each of the 550 predicted answers is evaluated against the reference using a standardized template with explicit [Question], [Reference], [Assistant’s Answer] segmentation.
• The aggregate benchmark score is computed as the arithmetic mean:

$S = \frac{1}{550}\sum_{i=1}^{550}\mathrm{Score}_i$

Traditional n-gram metrics (BLEU, ROUGE) are judged inadequate for such open-ended, knowledge-rich evaluation; instead, the LLM-as-judge rubric is employed for its ability to capture semantic completeness and instruction following.

B. Human Expert Evaluation

• Six licensed medical experts evaluate six selected models, each annotating ~92 items, for a total of 3,300 rated responses. Ratings again use the 1–10 rubric. Inter-method agreement between average human and GPT-4 scores at task level is quantified by Pearson correlation ($r = 0.84$).
• For multi-turn (dialog) testing, three additional experts are used for DotaBench (Xie et al., 2024).

5. Experimental Results and Comparative Analysis

DoctorFLAN-test enables precise, head-to-head comparison across model classes and reveals systematic strengths and weaknesses.

A. Model Performance:

• Generalist open-source LLMs: Qwen-1.8B-Chat (~4.48), Baichuan2-7B-Chat (~6.59), Yi-6B-Chat (~6.98), Yi-34B-Chat (~7.80).
• Baseline domain LLMs: BianQue-2 (6B: ~3.72), HuatuoGPT (7B: ~4.29), HuatuoGPT-II (7B: ~7.03).
• Proprietary LLMs: GPT-3.5 (~6.64), Claude-3 (~8.38), GPT-4 (~8.42).
• Fine-tuned (DotaGPT on DoctorFLAN) models: DotaGPT$_{\mathrm{Yi-6B}}$ (~7.81, +11.9% over base), DotaGPT$_{\mathrm{Baichuan2-7B}}$ (~8.25, +25.2%).

B. Task-Level Findings:

• All model types show marked underperformance in "Diagnosis" and "Treatment" tasks—especially in Definitive Diagnosis, Medication Advice, and Disease Grading (untuned model averages <6).
• Triage, Preoperative Education, Case Summary, and Medication Inquiry—tasks flagged as high-utility by surveyed doctors—realize the largest performance gains in DotaGPT variants.
• Fine-tuning on workflow-aligned data yields near-state-of-the-art open-model performance, closely matching Claude-3 and GPT-4 on DoctorFLAN-test (Xie et al., 2024, Xie et al., 13 Oct 2025).
• Human and automated scores are strongly correlated, confirming LLM-as-judge frameworks as reliable proxies for expert annotation.

6. Illustrative Examples and Practical Utility

Several benchmark items highlight model strengths and deficiencies:

• Differential Diagnosis Example: For an 11-month-old with leukocoria, DotaGPT output matches reference (congenital cataract, retinoblastoma, imaging), while baseline HuatuoGPT proposes incorrect (adult/inflammatory) etiologies and generic workups.
• Medication Inquiry Example: For renal dosing of Drug X, non-fine-tuned models omit critical adjustments; DotaGPT supplies accurate contraindications, interaction listing, and eGFR-based dosing.

DoctorFLAN-test outputs demonstrate that workflow-trained models can support clerical tasks (triage, exam planning, follow-up) and reduce cognitive load; however, for high-stakes diagnostic or management calls, responses require physician verification. The dataset’s breadth allows guided curriculum learning and targeted re-tuning, especially for low-performing tasks.

7. Significance and Research Directions

DoctorFLAN-test establishes a high-fidelity, workflow-aligned evaluation paradigm that diverges from conventional, patient-oriented, or multiple-choice benchmarks. Its clinical realism—anchored in expert-in-the-loop annotation, multi-specialty breadth, and phase coverage—enables systematic analysis of LLM deficits and progress in medical reasoning, factuality, and instruction following (Xie et al., 13 Oct 2025).

Future directions include:

• Extending benchmarks to multi-turn, longitudinal workflows (cf. DotaBench).
• Enhancing domain-specific performance in diagnostic and therapeutic reasoning.
• Continuous integration of doctor feedback in both benchmark design and model iteration.

By aligning model evaluation with actual physician workflows and priorities, DoctorFLAN-test informs both benchmark development and model-training strategies capable of supporting safe and effective AI-assisted medicine (Xie et al., 2024, Xie et al., 13 Oct 2025).