DoctorFLAN: Chinese Medical Dataset for Clinicians
- DoctorFLAN is a workflow-aligned Chinese medical dataset that comprehensively covers pre-diagnosis through discharge with high-fidelity question–answer pairs and multi-turn dialogues.
- It aggregates 92,000 single-turn QAs and 74 multi-turn dialogues from diverse clinical sources, validated through rigorous expert review and high inter-rater agreement.
- The dataset enables fine-tuning of large language models for doctor-facing tasks, yielding significant performance gains in diagnosis and treatment benchmarks.
DoctorFLAN is a large-scale, workflow-aligned Chinese medical dataset designed specifically to facilitate the development and evaluation of LLMs as clinical assistants for physicians rather than as direct patient-facing agents. DoctorFLAN targets coverage of the full clinical workflow—spanning pre-diagnosis, diagnosis, treatment, and discharge—by providing high-fidelity question–answer (QA) pairs and multi-turn dialogues aligned with real-world physician practices. Distinct from prior patient-centric medical LLM corpora, DoctorFLAN emphasizes tasks and benchmarks that reflect actual doctor needs, supporting the integration and safe deployment of LLMs into clinical settings under physician supervision (Xie et al., 2024, Xie et al., 13 Oct 2025).
1. Motivation and Background
Traditional medical LLM datasets are predominantly built from patient–doctor dialogs or multiple-choice questions, focusing on the emulation of patient-doctor interactions or factual medical recall. However, this approach exhibits multiple deficiencies: direct patient-facing LLM deployments risk propagating hallucinated or low-accuracy content, which non-expert users cannot reliably identify, thereby constituting significant safety hazards. Benchmarks such as MedQA, PubMedQA, and CMExam primarily measure closed-ended knowledge rather than the open-ended, multifaceted tasks physicians routinely perform.
To systematically address these limitations, DoctorFLAN advances the "LLMs for Doctors" paradigm, wherein LLMs act as assistants that collaborate with expert clinicians, allowing for output vetting and correction. A two-stage inspiration–feedback survey underpins the design of DoctorFLAN: first, an expert symposium of 16 specialists identified 22 representative doctor-facing tasks across all clinical workflow phases; subsequently, 71 physicians from top-tier (tertiary) hospitals rated the expected efficiency gain for each task (1–5 scale), with universal high demand for LLM augmentation (all tasks scoring ≥4) (Xie et al., 2024, Xie et al., 13 Oct 2025).
2. Dataset Composition and Construction
DoctorFLAN comprises 92,000 single-turn QA instances and supports 22 clinically-relevant tasks drawn from diverse data sources:
- Professional MCQ bank from Medtiku, ensuring broad coverage of real-world medical conditions and scenarios.
- Symptom, disease, and medication queries from 120ask.com, an online Chinese medical encyclopedia.
- High-quality instruction–response samples from PromptCBLUE, with an emphasis on accurate and realistic case summaries.
Data are mapped to 27 specialties, reflecting incidence distributions found in actual clinical practice. Annotation follows a multi-step protocol: clinical experts normalize prompts to match specific doctor tasks, GPT-4 generates detailed explanatory answers using MCQ reference responses, and three medical reviewers verify 1050 samples (50 per task) for correctness (100% pass rate) and practicality (99.9% pass rate). The full dataset is de-duplicated via Jaccard threshold filtering and instruction–response pairs are iteratively refined until inter-rater agreement exceeds 95% (Cohen’s κ > 0.95) (Xie et al., 2024, Xie et al., 13 Oct 2025).
3. Dataset Schema and Benchmark Components
DoctorFLAN’s structure is engineered for workflow fidelity:
- Question/Instruction: Standardized, scenario-driven prompts (e.g., "Suggest differential diagnoses given the patient history").
- Input Field: Clinical case descriptions, MCQ stems, or patient data.
- Answer Field: Free-form, stepwise rationale and possible next steps.
Metadata includes task and specialty identifiers, workflow phase (pre-diagnosis, diagnosis, treatment, discharge), and data provenance (MCQA, encyclopedia, or PromptCBLUE). The dataset provides specific held-out test sets:
- DoctorFLAN-test: 550 single-turn questions (25 per task), used to measure generalization.
- DotaBench: 74 three-turn multi-turn dialogues based on real CMB-Clin clinical records, with physicians reconstructing realistic stepwise inquiries, to measure multi-turn consistency and logical progression.
| Benchmark | Content | Instances |
|---|---|---|
| DoctorFLAN-test | Single-turn, QA tasks | 550 |
| DotaBench | Multi-turn dialogues | 74 |
4. Evaluation Methodology and Metrics
Evaluation leverages both automatic and expert review. For automatic metrics, GPT-4 ("gpt-4-0125-preview") serves as a judge, scoring model outputs (1–10) on accuracy, coherence, relevance, and thoroughness. Human evaluations are conducted by six physicians for single-turn QA (DoctorFLAN-test) and three for multi-turn dialogues (DotaBench), with calibrated cross-review for consistency.
The principal summary metric is the average score:
where is the sample score. Automated and human rating agreement is high (Pearson’s –$0.84$). Traditional n-gram metrics (F1, BLEU) are eschewed as insufficiently sensitive to semantic validity in free-form medical reasoning (Xie et al., 2024, Xie et al., 13 Oct 2025).
5. Experimental Results
Open-source LLM baselines (e.g., Baichuan2-7B-Chat, Yi-6B-Chat, HuatuoGPT, DISC-MedLLM) often underperform in doctor-facing settings, especially on complex diagnosis and treatment tasks. Fine-tuning on DoctorFLAN (producing models labeled DotaGPT) yields substantial improvements. For DoctorFLAN-test (automatic evaluation), DotaGPT (Baichuan2-7B-Base) achieves a score of 8.25 versus its baseline's 6.59 (+25.2%), approaching GPT-4 (8.42) and Claude-3 (8.38). Similarly, on DotaBench, DotaGPT (Baichuan2-7B-Base) scores 9.00 per turn, surpassing baseline and even larger generalist models.
Performance gains are marked in diagnosis and treatment phases (up to +29.8%). Human evaluations corroborate these trends: on DoctorFLAN-test, DotaGPT (Baichuan2-7B-Base) reaches 7.83 versus Baichuan2-7B-Chat's 6.69 and HuatuoGPT's 4.97; on DotaBench, DotaGPT scores 8.54 compared to HuatuoGPT-II's 7.98. On external benchmarks (CMMLU (Med.), CMExam, MMLU (Med.), CMB-Exam), DoctorFLAN-fine-tuned models outperform their chat-trained counterparts in 3 of 4 tasks (Xie et al., 2024, Xie et al., 13 Oct 2025).
6. Insights, Limitations, and Extension Prospects
Despite the improvements, diagnosis and treatment tasks remain especially challenging and demand further model advancement. Excessive specialization (as in DISC-MedLLM) can degrade general clinical reasoning, and accurate medication inquiry requires continual pharmacological updates. Fine-tuning with DoctorFLAN imparts transferable gains in both in-domain and out-of-domain scenarios; tasks rated as low-risk and high-volume (Triage, Case Summary, Medication Inquiry, Preoperative Education) are prioritized for early clinical integration.
Current limitations include the Chinese-only language scope, restricted multi-turn dialogue length, and the absence of multi-modal (e.g., imaging) or dynamic retrieval-augmented features. Planned extensions include dataset expansion to additional languages, deeper multi-turn and real clinical transcript coverage, task-specific RLHF, and integration of safety filters and structured knowledge retrieval (Xie et al., 2024, Xie et al., 13 Oct 2025).
7. Accessibility and Research Impact
DoctorFLAN and supporting benchmarks are open-access under responsible use terms, with code, model weights, and datasets available at https://github.com/FreedomIntelligence/DotaGPT and on the Hugging Face Hub. The resources enable reproducible research in workflow-aligned medical LLM development, and can be adapted for language translation or new specialties. The explicit alignment with physician workflows bridges the gap between generic medical QA and complex, real-world clinical practice, supporting the safe, effective integration of LLMs as AI assistants in healthcare (Xie et al., 2024, Xie et al., 13 Oct 2025).
