ChiMed 2.0: Chinese Medical LLM Dataset

• ChiMed 2.0 is a comprehensive Chinese medical dataset that integrates raw documents, QA pairs, and preference tuples from both traditional Chinese medicine and modern general medicine.
• It expands the original ChiMed corpus by increasing scale and task diversity, supporting a full LLM training pipeline from continued pre-training to RLHF.
• Empirical results show that models trained on ChiMed 2.0 achieve improved diagnostic accuracy and more refined medical reasoning across varied scales.

ChiMed 2.0 is a large-scale Chinese-language medical dataset designed to support all three stages of LLM training in the medical domain: domain continued pre-training, supervised fine-tuning, and reinforcement learning from human feedback through pairwise preference data. It extends the earlier ChiMed dataset from a single-task question-answering corpus to an integrated resource spanning raw documents, question–answer pairs, and preference tuples, and it combines traditional Chinese medicine material with modern general medical content. The published resource contains 204.4M Chinese characters, including 164.8K pre-training documents, 351.6K supervised fine-tuning question–answer pairs, and 41.7K RLHF preference tuples (Tian et al., 21 Jul 2025).

1. Definition, scope, and design rationale

ChiMed 2.0 was introduced to address several limitations of existing Chinese medical datasets: limited scale, narrow domain coverage, and a predominant focus on fine-tuning-only supervision. The dataset is explicitly organized around a full LLM training pipeline comprising raw documents for continued pre-training, instruction-style question–answer pairs for supervised fine-tuning, and pairwise preference data for RLHF. In the authors’ formulation, this makes ChiMed 2.0 an integrated training resource rather than a single-task benchmark (Tian et al., 21 Jul 2025).

The expansion relative to the earlier ChiMed corpus is defined along four axes. First, scale increases from 52.21M to 204.40M Chinese characters. Second, task coverage expands from QA only to documents, QA, and preference data. Third, training support expands from fine-tuning only to pre-training, SFT, and RLHF. Fourth, domain coverage broadens from online general medical consultation data to a mixture of traditional Chinese medicine classics and resources, along with modern and general medicine, including online consultations and encyclopedic content (Tian et al., 21 Jul 2025).

The intended application space includes Chinese medical LLMs for diagnostic assistance, clinical decision support, and medical question answering. The dataset’s internal organization also reflects a bilinguality of medical paradigms specific to the Chinese context: traditional Chinese medicine material is retained rather than treated as a marginal specialty, while modern general medicine dominates the conversational and QA-heavy portions of the corpus. This suggests that ChiMed 2.0 is structured not merely as a data repository but as a domain-adaptation substrate for heterogeneous Chinese medical knowledge.

2. Corpus composition and internal structure

ChiMed 2.0 is partitioned into three main components—pre-training corpus, supervised fine-tuning data, and preference data for RLHF—and each component is further divided into traditional Chinese medicine (“Trad”) and general or modern medicine (“Gen”) subsets (Tian et al., 21 Jul 2025).

Component	Trad / Gen breakdown	Total
Pre-training	55.7M + 2.1M characters; 160.8K + 4.0K documents	57.8M characters; 164.8K documents
SFT	11.2M + 127.1M characters; 9.8K + 341.8K QA pairs	138.3M characters; 351.6K QA pairs
RLHF	2.3M + 6.0M characters; 11.7K + 30.0K tuples	8.3M characters; 41.7K tuples

Across the full dataset, the traditional Chinese medicine portion contains 69.2M characters, approximately 33.9% of the total, while the general medicine portion contains 135.2M characters, approximately 66.1% (Tian et al., 21 Jul 2025). The distribution is uneven across components. Pre-training is TCM-heavy in document count, with 160.8K traditional documents versus 4.0K general documents, whereas SFT and RLHF are dominated by modern medical QA and dialogue data, with 341.8K general QA pairs versus 9.8K TCM QA pairs, and 30.0K general preference tuples versus 11.7K TCM tuples. The authors attribute this to source availability: TCM offers abundant textual classics, while modern medicine is more prevalent in online consultation dialogue (Tian et al., 21 Jul 2025).

The pre-training corpus is document-style raw text used for continued pre-training. Its TCM segment includes classical medical texts, classical prescriptions and formulas, TCM theory such as meridians, organs, and syndrome differentiation, clinical case records, and research articles on TCM. Its general-medicine segment includes clinical guidelines, research reviews, online consultation records, patient–doctor dialogues, FAQ-style health information, and encyclopedic articles (Tian et al., 21 Jul 2025).

The supervised fine-tuning component consists entirely of question–answer pairs. Some are real patient–doctor QAs from online platforms such as 39 Ask Doctors and 120 Ask Health QA; others are generated from documents via LLM-based extraction under constraints that the question must be based on document content, the answer must be directly derivable from the document without additional information, both must be in Chinese, and output must follow a fixed Question: / Answer: format (Tian et al., 21 Jul 2025).

The RLHF component is organized as pairwise preference tuples of the form

$$(\text{prompt}, \text{chosen answer}, \text{rejected answer}),$$

or equivalently

$(\mathbf{x}, \mathbf{y}^{+}, \mathbf{y}^{-}).$

The prompt is a patient question or instruction; the chosen answer is the higher-quality response; the rejected answer is the lower-quality response. The preference is pairwise rather than multi-way. The paper does not provide explicit PPO or DPO loss formulas, but it states that the data is intended for RLHF training and is structurally compatible with standard pairwise-preference methods (Tian et al., 21 Jul 2025).

3. Data sources and processing pipeline

The raw corpus was collected from public Chinese medical websites selected for authority and volume, with a raw size before cleaning of approximately 221M characters. The source platforms are grouped into three categories: TCM-oriented platforms, mixed Chinese medicine and modern health portals, and online consultation platforms (Tian et al., 21 Jul 2025).

The TCM-oriented sources include the Chinese Medicine Dictionary (www.zydcd.com), described as a nationally authoritative digital TCM knowledge platform containing classical medical texts, clinical case records, and TCM research articles, and the Traditional Chinese Medicine Database (www.zhongyoo.com), which focuses on medicinal herbs and processing, formulas and patent medicines, acupoints and folk remedies, and herb–formula–disease relationships. The mixed source China Health Network (www.cnkang.com) contributed classical formula collections, herbal medicine knowledge, and clinical applications and guidelines. Modern consultation and dialogue data were drawn primarily from 39 Ask Doctors (ask.39.net) and 120 Ask Health QA (www.120ask.com) (Tian et al., 21 Jul 2025).

Processing proceeds in two layers. The basic layer performs deduplication by URL, regex-based text cleaning, perplexity-based quality filtering, and length normalization. Regex cleaning removes HTML tags, CSS styles, URLs, non-text characters, and irrelevant boilerplate. Perplexity filtering concatenates the question and answer for each item, computes perplexity using the HuggingFace Chinese GPT-2 model uer/gpt2-chinese-cluecorpussmall, and removes samples in the top 1% of perplexity. Texts shorter than 30 tokens are discarded (Tian et al., 21 Jul 2025). The standard perplexity expression implicitly used for filtering is

$$\text{PPL}(x_{1:n}) = \exp\left( -\frac{1}{n} \sum_{t=1}^{n} \log p(x_t \mid x_{<t}) \right).$$

The advanced layer uses LLM prompting for four tasks. The first is sensitive content cleaning and privacy anonymization. Samples containing personal identifiers, toxic content, or other inappropriate or unsafe medical information are detected by prompting an LLM as a content-safety classifier that returns “是” or “否”; flagged samples are removed. The authors report periodic manual sampling and review to estimate false positives and false negatives and to iteratively refine prompts and thresholds (Tian et al., 21 Jul 2025).

The second advanced step is classical-to-modern Chinese translation for TCM material. Classical passages are translated into accessible modern Chinese while retaining professional TCM terminology and medical concepts. The stated purpose is to improve readability and model learning of TCM knowledge (Tian et al., 21 Jul 2025).

The third step is QA pair generation from documents. An LLM, prompted as a “professional medical QA generation assistant,” extracts core information from a document and generates one question and one answer grounded directly in the source text. The output of this stage is approximately 220K high-quality QA pairs used for SFT and RLHF (Tian et al., 21 Jul 2025).

The fourth step is preference data construction. For each patient question, one LLM generates two distinct candidate answers under constraints including conciseness and factuality, and another LLM ranks the pair using four criteria: accuracy, usefulness, completeness, and conciseness. The better answer becomes the chosen response, the worse answer becomes the rejected response, and together with the prompt they form a preference tuple (Tian et al., 21 Jul 2025).

4. Training pipeline and empirical validation

To validate the dataset, the authors train representative general-domain LLMs through the full three-stage sequence of domain continued pre-training, supervised fine-tuning, and RLHF. The base models are Qwen-3 (1.7B), with 28 transformer layers and hidden size 2048, and Qwen-3 (14B), with 40 transformer layers and hidden size 5120. The training schedule uses 2 epochs for pre-training, 2 epochs for SFT, and 1 epoch for RLHF, with LoRA applied at all stages (Tian et al., 21 Jul 2025).

The reported stage-wise hyperparameters are: batch size 64, learning rate $1\times 10^{-5}$, and warmup ratio 0.1 for pre-training; batch size 32, learning rate $5\times 10^{-5}$, and warmup ratio 0.1 for fine-tuning; and batch size 16, learning rate $1\times 10^{-5}$, and warmup ratio 0.1 for RLHF. All experiments were run on 8 × NVIDIA A40 GPUs with 48GB memory each (Tian et al., 21 Jul 2025).

Evaluation uses medical subsets of CMMLU and CEval, specifically CMMLU Genetics, CMMLU College Medicine, and CEval Physician, under the LM Evaluation Harness protocol. These are multiple-choice QA tasks scored with accuracy,

$$\text{Accuracy} = \frac{\#\text{correct predictions}}{\#\text{total questions}}.$$

For Qwen-3 (1.7B), the original model scores 48.30 on CMMLU Genetics, 60.07 on CMMLU College Medicine, and 59.18 on CEval Physician; after training on ChiMed 2.0, the corresponding scores are 51.14, 61.17, and 63.27. For Qwen-3 (14B), the original scores are 87.55, 76.70, and 85.71, and the ChiMed 2.0-trained scores are 89.01, 76.70, and 87.76 (Tian et al., 21 Jul 2025).

The improvements amount to +2.84, +1.10, and +4.09 points for Qwen-3 (1.7B), and +1.46, 0, and +2.05 points for Qwen-3 (14B) on the three benchmarks respectively (Tian et al., 21 Jul 2025). The paper does not include an ablation isolating the contributions of pre-training, SFT, and RLHF, but it reports that the full pipeline yields consistent gains across model scales. The relatively larger gains for the 1.7B model suggest that domain-specific corpora can be especially valuable under resource-constrained deployment settings.

Qualitative analysis reported in the paper states that post-ChiMed 2.0 responses exhibit more accurate medical reasoning, clearer guidance on when to seek urgent care, and more professional TCM terminology and syndrome reasoning in TCM cases. Rejected responses tend to be vague, incomplete, or insufficiently aligned with domain standards (Tian et al., 21 Jul 2025).

5. Limitations, safety considerations, and misuse risks

The paper identifies several limitations. Coverage remains constrained by the scope of the scraped websites, which may underrepresent certain specialties or rare diseases. The dataset is also imbalanced: modern medical QA content is much larger than TCM QA content, while TCM is disproportionately represented in document count. Distribution across departments is not uniform (Tian et al., 21 Jul 2025).

Quality limitations are also explicit. Because some sources are user-generated websites, residual noise, bias, or outdated practice may remain even after filtering. In addition, LLM-generated components—classical-to-modern translations, synthetic QA pairs, and preference labels—can inherit biases or errors from the underlying models. A large portion of QA data and all preference labels are synthetic, which may bias downstream model behavior toward the style of the generative LLMs used in construction (Tian et al., 21 Jul 2025).

The safety discussion focuses on medical hallucinations, over-reliance, and privacy. Models trained on ChiMed 2.0 may still produce incorrect medical advice, and non-professional users may use them for self-diagnosis or treatment without clinician oversight. Although sensitive content is filtered, residual identifiers may remain. The mitigation strategies proposed in the paper are robust privacy detection and removal at the data level, deployment as assistive tools for clinicians rather than independent decision-makers, human-in-the-loop validation in real clinical usage, further RLHF on safety, and continuous curation with expanded specialties and more up-to-date guidelines (Tian et al., 21 Jul 2025).

These cautions are consistent with adjacent Chinese medical LLM work. “MedChatZH” emphasizes that medical advice is sensitive and that hallucinations remain possible in Chinese medical dialogue models (Tan et al., 2023). The Qilin-Med paper similarly restricts its ChiMed-based system to research and academic purposes and states that it should not be used for direct patient diagnosis or as a standalone tool for medical decision-making (Ye et al., 2023).

6. Position in the Chinese medical LLM ecosystem

Within the comparison table reported by the authors, ChiMed 2.0 is the largest Chinese medical dataset listed at 204.40M characters. The datasets contrasted with it include the original ChiMed at 52.21M characters, CMB at 12.03M, CMExam at 12.72M, MedQA at 6.76M, and cMtMedQA / ZhongJing at 6.57M. The distinguishing property emphasized by the paper is not only scale but also heterogeneity: most prior Chinese medical datasets are QA-only or otherwise task-specific, whereas ChiMed 2.0 combines documents, QA, and preference data, directly aligned with pre-training, SFT, and RLHF (Tian et al., 21 Jul 2025).

A potential source of terminological confusion is that “ChiMed” was also used in the Qilin-Med literature to denote a stage-partitioned corpus comprising ChiMed-CPT, ChiMed-SFT, and ChiMed-DPO. That paper does not define an explicit version called “ChiMed 2.0” (Ye et al., 2023). By contrast, the 2025 ChiMed 2.0 paper defines a specific dataset that extends an earlier single-task QA corpus into a three-stage training resource (Tian et al., 21 Jul 2025).

ChiMed 2.0 also sits within a broader methodological trend in Chinese medical language modeling. “MedChatZH” does not mention ChiMed or ChiMed 2.0 by name, but it independently demonstrates that combining domain pre-training with carefully curated medical instruction data materially improves Chinese medical QA performance (Tan et al., 2023). A plausible implication is that ChiMed 2.0’s pre-training-plus-SFT design belongs to a convergent data-engineering pattern rather than an isolated artifact.

A second plausible implication concerns future extensions. “ShizhenGPT,” published shortly after ChiMed 2.0, argues that TCM remains underexplored because of both data scarcity and the multimodal nature of diagnosis, and it introduces a TCM-focused benchmark spanning text, vision, audio, and physiological signals (Chen et al., 20 Aug 2025). Relative to that direction, ChiMed 2.0 remains a text-only resource; its integration of TCM classics, modern medical QA, and RLHF-ready preference data can therefore be understood as a substantial textual foundation, but not a multimodal endpoint.

The project resources for ChiMed 2.0 are reported at https://github.com/synlp/ChiMed-2.0, where data source information and associated materials are provided. The paper notes that licensing details are not fully specified in the body text and that practitioners should consult the repository and original website constraints, especially because parts of the corpus are scraped from public online platforms (Tian et al., 21 Jul 2025).