DentalBench: Bilingual Dental LLM Benchmark
- DentalBench is a bilingual benchmark suite that measures LLM competency in dentistry using specialized English and Chinese datasets.
- It integrates DentalQA for diverse question formats and DentalCorpus for domain adaptation across 16 dental subfields.
- The benchmark highlights language-sensitive performance gaps and demonstrates improvements through supervised fine-tuning and retrieval augmentation.
Searching arXiv for DentalBench and closely related benchmark papers to ground the article and citations. DentalBench is a bilingual benchmark-and-resource suite introduced to measure and improve LLMs’ ability to understand dentistry. It is presented as the first comprehensive English–Chinese benchmark specifically for dental-domain LLM evaluation and adaptation, and is built around two complementary components: DentalQA, a bilingual question-answering benchmark, and DentalCorpus, a curated bilingual dental corpus for domain adaptation. Its central premise is that strong performance on broad medical benchmarks does not adequately characterize competence in dentistry, which the benchmark treats as a highly specialized branch of medicine with its own subfields, terminology, procedures, and knowledge structure (Zhu et al., 28 Aug 2025).
1. Rationale and benchmark identity
DentalBench is motivated by a gap between general medical LLM evaluation and dentistry-specific competence. The benchmark is framed around the claim that general medical corpora and benchmarks may support strong average medical performance, but do not provide the fine-grained, dental-specific coverage needed to evaluate whether a model can answer oral-health questions reliably or support real-world dental applications. In that sense, DentalBench is designed not only as an evaluation set but also as an adaptation platform, so that researchers can test current models and study how in-domain data changes performance (Zhu et al., 28 Aug 2025).
The suite has two major components. DentalQA contains 36,597 questions in English and Chinese. DentalCorpus contains 337.35 million tokens of bilingual dental text curated for domain adaptation, including supervised fine-tuning and retrieval-augmented generation. This dual structure is methodologically important because it separates measurement from adaptation: DentalQA operationalizes dental understanding, while DentalCorpus provides the in-domain resource needed to investigate how that understanding can be improved (Zhu et al., 28 Aug 2025).
The bilingual design is central rather than decorative. The benchmark explicitly evaluates Chinese and English performance separately, and the paper treats cross-lingual asymmetry as part of the object of study. A plausible implication is that DentalBench is intended to expose not just whether models know dentistry, but whether that knowledge is stable across linguistic and educational contexts.
2. Core structure of DentalQA
DentalQA spans 4 task formats across 16 dental subfields, and it mixes closed-form factual answering with open-ended clinical-style generation and terminology understanding. The 16 subfields follow standard textbook classifications from the 8th round of the National Higher Education Curriculum for Five-Year Undergraduate Dental Medicine Programs: dental materials (DM), endodontics (ENDO), occlusion (OCL), oral anatomy (OA), oral biology (OB), oral implantology (OI), oral mucosal diseases (OMD), oral pathology (OP), oral pharmacology (OPH), oral and maxillofacial radiology (OMFR), oral and maxillofacial surgery (OMFS), orthodontics (ORTHO), pediatric dentistry (PED), periodontics (PER), preventive dentistry (PREV), and prosthodontics (PRO) (Zhu et al., 28 Aug 2025).
The task formats are heterogeneous by design.
| Task | Availability and purpose | Evaluation |
|---|---|---|
| MCQ | English and Chinese; factual recall | Accuracy |
| MAQ | Chinese only; comprehensive diagnostic knowledge | Accuracy, precision, recall, F1 |
| OEQ | English and Chinese; clinical or theoretical scenarios | BERTScore F1 |
| DEF | English and Chinese; terminology definition | BERTScore F1 |
MCQ denotes single-answer multiple-choice questions, with 4 options in English and 5 options in Chinese. MAQ denotes multi-answer multiple-choice questions and is available only in Chinese. OEQ denotes open-ended questions that simulate clinical or theoretical scenarios. DEF denotes terminology definition questions intended to test whether the model understands specialized dental terms and can ground them in correct definitions. The paper names the scoring metrics but does not provide explicit formulas for accuracy, precision, recall, F1, or BERTScore, and it does not give a formal symbolic objective for multi-answer scoring or generation scoring in the main text (Zhu et al., 28 Aug 2025).
DentalQA is split into train and test in a 4:1 ratio while preserving each subfield’s proportions. The paper does not provide exact per-task or per-language counts adding up to 36,597 in the main text, and it does not specify exact train/test counts beyond this ratio. That omission matters because it makes the benchmark more useful as an evaluation resource than as a fully specified reproducibility artifact.
3. DentalQA data sources and curation pipeline
The English and Chinese sides of DentalQA are constructed from different source ecosystems. The English side is curated from seven public medical QA sources: MMLU, MedQA, MedMCQA, MedQuAD, PubMedQA, iCliniq, and the Medical Meadow Flashcards and Medical Meadow Wikidoc resources. Because these are broad medical datasets rather than dental datasets, they are filtered using a keyword list derived from DentalCorpus construction. English DEF questions are created by taking dental terms from a bilingual glossary compiled during corpus construction and retrieving their definitions from UMLS (Zhu et al., 28 Aug 2025).
The Chinese side is more directly dentistry-native. It is built from the China National Dental Licensing Examination from 1999 to 2021, 34 dental textbooks and auxiliary materials, and 181 open-ended questions derived from real orthodontist–patient interactions. This gives the Chinese portion a closer alignment with dental education and practice as represented in the source description (Zhu et al., 28 Aug 2025).
The construction pipeline is unified across languages as much as possible. MCQ and MAQ items are normalized to fixed option counts. DEF items are generated by filling 50 predefined templates per language with extracted dental terms and their definitions. OEQ items are kept in original form. Domain relevance is enforced using GPT-4o to classify every question as oral-related, non-oral, or insufficient, where “insufficient” denotes incomplete or corrupted items. Only oral-related questions are retained. Manual validation of this filtering step on 300 representative samples, with 50 examples for each language-category combination, yielded agreement rates of 100%, 96%, and 94% for English and 96%, 92%, and 92% for Chinese across the three categories (Zhu et al., 28 Aug 2025).
This curation strategy makes DentalQA a filtered, structured benchmark rather than a raw aggregate of pre-existing sources. At the same time, the paper does not report inter-annotator agreement beyond those validation percentages, and it does not define an exact rule for converting model generations into answer options in the main text.
4. DentalCorpus and domain adaptation infrastructure
DentalCorpus is the adaptation half of DentalBench. The final corpus contains 337.35 million tokens in total, comprising 1.06 million English chunks totaling 319.08 million tokens and 66.3 thousand Chinese chunks totaling 18.27 million tokens. The corpus is therefore heavily English-dominant, and the paper explicitly notes that this asymmetry becomes important when interpreting later bilingual results (Zhu et al., 28 Aug 2025).
The corpus is built from three source types. First, textbooks: the authors collect 40 Chinese dental textbooks and auxiliary materials, remove non-content sections, and apply OCR, yielding 4.1 million characters of clean text. They also extract a bilingual glossary of 1,971 dental terms from textbook glossaries. Second, PubMed articles: using 28 MeSH terms related to dentistry, they retrieve 54,651 freely accessible full-text articles from 2000 to 2024, producing 983.3 million English characters and 5.4 million Chinese characters before filtering. Third, open medical datasets: they filter MMedC, whose raw scale is reported as 10.56B English and 4.35B Chinese tokens, and MedRAG, which contains 23.9M PubMed snippets, to retain only dental-relevant content (Zhu et al., 28 Aug 2025).
The filtering pipeline is relatively detailed. It uses a rule-based keyword filtering method driven by keyword lists derived from TF-IDF analysis on dental versus general medical corpora. The final keyword lists contain 440 English keywords and 235 Chinese keywords. Texts are retained if keyword density is greater than 1% and if there are at least two distinct keyword matches per sentence. English matching uses space tokenization, while Chinese uses direct string matching. After filtering, the corpus is deduplicated with MD5 hashes, embedded using bge-m3, and segmented into chunks of up to 512 tokens. Human validation on 100 random samples per language found domain relevance rates of 99% for English and 96% for Chinese (Zhu et al., 28 Aug 2025).
DentalCorpus supports two adaptation regimes in the paper: supervised fine-tuning and retrieval-augmented generation. For retrieval, the authors use FAISS over bge-m3 embeddings with top-5 passage retrieval. The paper does not provide a mathematical retrieval equation, and it does not specify a formal supervised loss beyond standard fine-tuning practice.
5. Benchmark evaluation of current LLMs
The paper evaluates 14 LLMs in a zero-shot setting with task-specific prompts on the held-out test set, using eight NVIDIA RTX 3090 GPUs. The evaluated models span three groups. The proprietary models are GPT-4o, GPT-4o-mini, DeepSeek-V3, and DeepSeek-R1. The open-source general models are LLaMA-3.2-3B-Instruct, LLaMA-3.1-8B-Instruct, and Qwen-2.5-1.5B/3B/7B/14B/32B-Instruct. The medical-specific models are BioMistral-7B, HuatuoGPT2-7B, and LLaMA-3-8B-UltraMedical (Zhu et al., 28 Aug 2025).
The results establish DentalBench as difficult for current systems. On Chinese MCQ, DeepSeek-R1 is best with 76.06 accuracy, followed by Qwen2.5-32B at 70.86 and DeepSeek-V3 at 69.28. On Chinese MAQ, DeepSeek-R1 is again strongest, with 43.51 exact accuracy, 88.64 precision, 86.68 recall, and 87.65 F1. On English MCQ, GPT-4o leads with 73.98 accuracy, with DeepSeek-V3 second at 68.28. For English OEQ, GPT-4o-mini achieves the highest BERTScore at 34.55, narrowly above BioMistral-7B at 34.50 and Qwen2.5-3B at 34.48. For English DEF, Qwen2.5-3B is highest at 29.62. In Chinese OEQ, DeepSeek-V3 reaches 27.79 BERTScore, while GPT-4o leads Chinese DEF at 21.60, with HuatuoGPT2-7B close at 21.04 (Zhu et al., 28 Aug 2025).
Several patterns are emphasized. First, there is a strong task-type gap: MCQ and MAQ scores are much higher than OEQ and DEF scores across nearly all models and both languages. Second, there is a language gap: Chinese and English do not have the same winners, and adaptation effects differ by language. Third, medical-specific models are not automatically better. The paper explicitly notes that medical LLMs do relatively well on OEQ and DEF but underperform general-purpose models on MCQ and MAQ; for example, LLaMA-3.1-8B beats LLaMA-3-8B-UltraMedical on the multiple-choice tasks. Fourth, within the Qwen-2.5 family, increasing scale clearly improves MCQ and MAQ, but yields limited gains on OEQ and DEF, which the paper interprets as evidence that larger parameter counts benefit factual recall more than dental generative reasoning (Zhu et al., 28 Aug 2025).
6. Adaptation experiments with Qwen2.5-3B
The domain adaptation experiments are organized around Qwen2.5-3B-Instruct. The baseline is zero-shot Qwen2.5-3B, and the paper compares supervised fine-tuning (SFT), retrieval-augmented generation (RAG), and SFT+RAG. For SFT, the model is fully fine-tuned on the DentalQA training split for four epochs with learning rate , batch size 16, and bfloat16 precision. For RAG, the system retrieves the top-5 most relevant passages from DentalCorpus using FAISS over bge-m3 embeddings and prepends them to the prompt. SFT+RAG applies both procedures (Zhu et al., 28 Aug 2025).
SFT produces clear gains on factual and multi-answer tasks. Chinese MCQ rises from 48.63 to 54.58, and English MCQ from 41.77 to 47.90. Chinese MAQ exact accuracy rises from 19.19 to 25.60, and recall increases from 80.37 to 93.24, although precision drops from 77.70 to 75.57. On generative tasks, gains are more modest: Chinese OEQ rises from 20.89 to 22.42, Chinese DEF from 11.16 to 15.29, English OEQ from 34.48 to 37.74, and English DEF from 29.62 to 30.79 (Zhu et al., 28 Aug 2025).
RAG changes the pattern. It also improves multiple-choice performance, but its largest gains are on knowledge-intensive open-ended tasks. Chinese OEQ rises from 20.89 to 30.18, which the paper explicitly contrasts with the smaller SFT gain of 1.53 points on the same task. Chinese DEF improves from 11.16 to 22.13. English OEQ rises to 36.47 and English DEF to 30.04. These results are used to argue that retrieval particularly helps tasks requiring access to specialized domain knowledge and terminology rather than only internal parametric recall (Zhu et al., 28 Aug 2025).
The combined SFT+RAG setting yields the strongest or near-strongest results overall for the adapted Qwen2.5-3B. Chinese MCQ reaches 60.06 and Chinese MAQ exact accuracy reaches 29.07, with 93.46 recall and 84.62 F1. The paper notes additive gains on Chinese MCQ and MAQ of +11.43 and +9.88 relative to the base model. On Chinese OEQ, SFT+RAG reaches 30.06, slightly below RAG alone’s 30.18. On Chinese DEF it reaches 20.85, below RAG alone’s 22.13 but still well above baseline. In English, SFT+RAG gives 52.15 on MCQ and 37.68 on OEQ, while DEF is 29.65, which is not an improvement over SFT alone and is only slightly above baseline. The paper therefore concludes that combining SFT and RAG is broadly beneficial, especially in Chinese, but that retrieval effectiveness is language-sensitive and less consistent in English (Zhu et al., 28 Aug 2025).
7. Position within dental AI benchmarking and acknowledged limitations
DentalBench occupies a specific niche within the broader dental AI benchmark landscape. Other benchmark efforts described in the literature focus on different modalities and task families: CTooth and CTooth+ target 3D CBCT tooth volume segmentation (Cui et al., 2022, Cui et al., 2022); Teeth3DS targets intraoral 3D scan segmentation and labeling (Ben-Hamadou et al., 2022); DENTEX and PerioXrays/PerioDet focus on panoramic radiograph abnormal-tooth or apical periodontitis detection (Hamamci et al., 2023, Fang et al., 25 Jul 2025); PRAD-10K targets periapical radiograph segmentation (Zhou et al., 10 Apr 2025); MMOral-Bench targets panoramic multimodal VQA (Hao et al., 11 Sep 2025); OralMLLM-Bench evaluates multimodal dental radiographic cognition (Wang et al., 2 May 2026); Dental-TriageBench evaluates hierarchical multimodal triage from complaint and OPG (He et al., 18 Mar 2026); and GlobalDentBench evaluates multinational dental LLM reasoning and safety (Zhao et al., 23 May 2026). Against that background, DentalBench is distinctive in being a bilingual, text-centered dentistry benchmark and adaptation suite for LLMs rather than a radiograph-first or geometry-first benchmark.
The paper also identifies several limitations. The most prominent is bilingual asymmetry: although both DentalQA and DentalCorpus are English–Chinese, their distributions, source diversity, and coverage depth are not fully aligned. The corpus itself is far larger in English than in Chinese, while the QA sources differ substantially by language. A second limitation is that MAQ exists only in Chinese, so English multi-answer reasoning is not evaluated. The paper also does not discuss clinical deployment validation, safety auditing, or direct real-world clinical outcome testing in depth. In addition, the main text does not provide exact per-task counts, exact train/test counts, explicit metric formulas, or explicit objective equations for SFT or retrieval (Zhu et al., 28 Aug 2025).
Taken together, DentalBench turns dentistry into a measurable and improvable LLM domain. It shows that current models possess nontrivial dental knowledge but exhibit uneven capabilities across task types and languages, that broad medical competence does not imply dentistry competence, and that explicitly dental adaptation resources can substantially improve performance. This suggests that, within dental AI, benchmark design must track not only modality and task format but also specialty structure, terminology density, and cross-lingual variation.