CMtMedQA: Chinese Medical Dialogue Dataset

• CMtMedQA is a comprehensive, multi-turn Chinese medical dialogue corpus capturing real doctor–patient interactions across 14 clinical specialties.
• It addresses limitations of single-turn datasets by emphasizing doctor-led symptom probing and proactive diagnostic reasoning to train advanced language models.
• The dataset underpins supervised fine-tuning, reinforcement learning from human feedback, and rigorous benchmarking for safety and professionalism in medical LLMs.

CMtMedQA is a large-scale, de-identified, multi-turn Chinese medical dialogue corpus collected from real doctor–patient consultations and curated specifically to facilitate research on multi-turn diagnostic reasoning, proactive inquiry, and safety-critical natural language generation in the biomedical domain. The dataset contains 70,000 authentic dialogues spanning 14 clinical specialties, and serves as a benchmark for supervised fine-tuning (SFT), reinforcement learning from human feedback (RLHF), and safety/professionalism evaluation of LLMs in Chinese medical settings (Yang et al., 2023).

1. Dataset Motivation and Scope

CMtMedQA was introduced to address deficits in existing Chinese medical dialogue corpora, which are predominantly single-turn and do not capture the complex, doctor-led, multi-turn reasoning observed in authentic consultations. Single-turn datasets fail to reflect doctors' initiative in symptom probing, limiting a model's ability to learn proactive diagnostic strategies. CMtMedQA was constructed to provide multi-turn dialogues that emphasize physician agency, enabling the development and evaluation of LLMs with enhanced coherence, initiative, and safety under real-world consultation conditions.

Primary downstream tasks enabled by CMtMedQA include:

• SFT of Chinese medical LLMs for improved dialogue coherence and proactivity
• RLHF reward-model training grounded in real-world interactions
• Benchmarking LLMs for safety, professionalism, and fluency on medical dialogue tasks
• Enabling patient-facing diagnostic chatbot research and educational simulators for medical training

2. Data Collection, Privacy, and Departmental Coverage

The dataset was harvested from doctor–patient records on Chinese public hospital online platforms, app-based telemedicine interfaces, and EHR (Electronic Health Records) systems. No audio or visual materials are included; only text transcripts are used. All data is fully de-identified: names, dates of birth, contact information, and any personally identifying elements are removed or masked prior to release. Data collection and handling protocols comply with institutional IRB requirements and applicable patient privacy regulations.

CMtMedQA encompasses dialogue data from 14 clinical specialties, with the following proportional distribution:

Department	Approx. Percentage
Internal Medicine	20%
Dermatology	15%
Pediatrics	12%
Surgery	10%
Cardiology	8%
Obstetrics/Gynecology	7%
Neurology	7%
Endocrinology	6%
Oncology	5%
Orthopedics	5%
ENT	4%
Ophthalmology	4%
Gastroenterology	3%
Others	4%

A plausible implication is that the predominance of internal medicine and dermatology is reflective of patient case distribution in outpatient digital health systems.

3. Dataset Structure and Annotation

Scale and Dialogue Structure

• Number of dialogues: $N = 70,000$
• Total conversational turns: $T = 400,000$
• Average turns per dialogue: $\mu_{\mathrm{turns}} \approx 5.7$ ($\sigma_{\mathrm{turns}} \approx 2.3$)
• Distribution: 48% of dialogues have 3–5 turns, 37% have 6–10, and 15% have more than 10 turns.

Each dialogue is a JSON object, storing the clinical department, consultation scenario (e.g., “Disease_Diagnosis”), and an array of turns, where each turn records the speaker (doctor or patient) and the utterance. Example schema:

{ "dialogue_id": "000123", "department": "Dermatology", "scenario": "Disease_Diagnosis", "turns": [ {"speaker":"patient","utterance":"我手脚出现红斑并脱皮五个月..."}, {"speaker":"doctor","utterance":"您之前有没有类似过敏史？"} ] }

Annotation and Filtering

• KG-Instruction Filtering: Medical entities are extracted with CMeKG (Chinese Medical Entity Knowledge Graph) and injected into prompts for validation; dialogues lacking relevant entities are filtered out as low-quality or off-topic.
• RLHF Scores: While raw CMtMedQA turns are not annotated with RLHF scores, model outputs evaluated on CMtMedQA are scored along nine axes, grouped under three prioritized dimensions: Safety (accuracy, safety, ethics), Professionalism (comprehension, clarity, initiative), and Fluency (coherence, consistency, tone). Clinical annotators rank model responses; adjudication is by a third expert as needed.

Example (editorially reconstructed for clarity):

{ "dialogue_id": "000123", "turn_id": 5, "speaker": "doctor", "utterance": "您最近有无发热或皮疹扩散？", "tags": ["Symptom_Inquiry"], "rlhf_scores": { "Accuracy": 5, "Safety": 5, "Ethics": 5, "Comprehension": 5, "Clarity": 4, "Initiative": 5, "Coherence": 5, "Consistency": 5, "Warm_Tone": 4 } }

4. Data Partitioning, Format, and Licensing

CMtMedQA is split as follows:

Data Split	Number of Dialogues
Training	≈62,100
Validation	≈6,900
Test	1,000 (held-out)

The test set consists of dialogues entirely unobserved during collection. All files use JSON Lines format, with one dialogue per line. The dataset and all supporting code are released under the CC BY-NC 4.0 license, restricting use to non-commercial research purposes. Redistribution of raw, de-identified dialogues is explicitly permitted for research, but not for incorporation into commercial products.

5. Evaluation, Benchmarks, and Applications

Metrics and Model Benchmarking

CMtMedQA defines win/tie/lose rates for each of the three evaluation dimensions (Safety, Professionalism+Fluency), aggregated at the dialogue level. Model assessment involves clinical raters comparing LLM responses side-by-side. Selected results for the CMtMedQA test set:

Model	Prof/Flu Win	Tie	Lose	Safety Win	Tie	Lose
ChatGPT	45%	25%	30%	30%	20%	50%
Ziya-LLaMA	85%	10%	5%	90%	8%	2%
BenTsao	88%	7%	5%	92%	6%	2%
DoctorGLM	86%	9%	5%	88%	10%	2%
HuatuoGPT	80%	11%	9%	85%	10%	5%

Zhongjing outperforms all open-source baselines in multi-turn professionalism and fluency, and matches or exceeds ChatGPT for several abilities on the test set.

Use Cases

• Fine-tuning LLMs for patient-facing clinical chatbots and diagnosis assistants
• RLHF reward model training aligned with clinical-judgment ground truth
• Medical student educational simulators for patient interview training
• Benchmarking Chinese medical dialogue generation along axes of safety, professionalism, and fluency

6. Limitations and Practical Considerations

CMtMedQA is limited to text-only dialogue; there is no support for imaging, laboratory results, or other non-text modalities. The dataset likely overrepresents departments such as internal medicine and dermatology, which may affect generalization performance for underrepresented specialties. Institutional biases in language and consultation style may exist, impacting cross-institution or regional transfer. Despite filtering measures (e.g., KG-based validation), some annotation noise remains. Models trained on this dataset, though improved in safety and initiative, may still hallucinate or fall short of safety guidelines when addressing rare or ambiguous conditions (Yang et al., 2023).

7. Access and Related Resources

All code, models, and the CMtMedQA dataset are available at https://github.com/SupritYoung/Zhongjing, supporting reproducibility and further research in Chinese medical dialogue systems. The license stipulates CC BY-NC 4.0, permitting research and academic use only, with redistribution rights for de-identified dialogues.