---
title: 'DentalBench: Bilingual Dental LLM Benchmark'
url: https://www.emergentmind.com/topics/dentalbench
type: topic
---

# DentalBench: Bilingual Dental LLM Benchmark

Searching arXiv for DentalBench and closely related benchmark papers to ground the article and citations.
DentalBench is a bilingual benchmark-and-resource suite introduced to measure and improve large language models’ ability to understand dentistry. It is presented as the first comprehensive English–Chinese benchmark specifically for dental-domain LLM evaluation and adaptation, and is built around two complementary components: **DentalQA**, a bilingual question-answering benchmark, and **DentalCorpus**, a curated bilingual dental corpus for domain adaptation. Its central premise is that strong performance on broad medical benchmarks does not adequately characterize competence in dentistry, which the benchmark treats as a highly specialized branch of medicine with its own subfields, terminology, procedures, and knowledge structure [2508.20416].

## 1. Rationale and benchmark identity

DentalBench is motivated by a gap between general medical LLM evaluation and dentistry-specific competence. The benchmark is framed around the claim that general medical corpora and benchmarks may support strong average medical performance, but do not provide the fine-grained, dental-specific coverage needed to evaluate whether a model can answer oral-health questions reliably or support real-world dental applications. In that sense, DentalBench is designed not only as an evaluation set but also as an adaptation platform, so that researchers can test current models and study how in-domain data changes performance [2508.20416].

The suite has two major components. **DentalQA** contains **36,597 questions** in English and Chinese. **DentalCorpus** contains **337.35 million tokens** of bilingual dental text curated for domain adaptation, including supervised fine-tuning and retrieval-augmented generation. This dual structure is methodologically important because it separates measurement from adaptation: DentalQA operationalizes dental understanding, while DentalCorpus provides the in-domain resource needed to investigate how that understanding can be improved [2508.20416].

The bilingual design is central rather than decorative. The benchmark explicitly evaluates Chinese and English performance separately, and the paper treats cross-lingual asymmetry as part of the object of study. A plausible implication is that DentalBench is intended to expose not just whether models know dentistry, but whether that knowledge is stable across linguistic and educational contexts.

## 2. Core structure of DentalQA

DentalQA spans **4 task formats** across **16 dental subfields**, and it mixes closed-form factual answering with open-ended clinical-style generation and terminology understanding. The 16 subfields follow standard textbook classifications from the 8th round of the National Higher Education Curriculum for Five-Year Undergraduate Dental Medicine Programs: dental materials (DM), endodontics (ENDO), occlusion (OCL), oral anatomy (OA), oral biology (OB), oral implantology (OI), oral mucosal diseases (OMD), oral pathology (OP), oral pharmacology (OPH), oral and maxillofacial radiology (OMFR), oral and maxillofacial surgery (OMFS), orthodontics (ORTHO), pediatric dentistry (PED), periodontics (PER), preventive dentistry (PREV), and prosthodontics (PRO) [2508.20416].

The task formats are heterogeneous by design.

| Task | Availability and purpose | Evaluation |
|---|---|---|
| MCQ | English and Chinese; factual recall | Accuracy |
| MAQ | Chinese only; comprehensive diagnostic knowledge | Accuracy, precision, recall, F1 |
| OEQ | English and Chinese; clinical or theoretical scenarios | BERTScore F1 |
| DEF | English and Chinese; terminology definition | BERTScore F1 |

MCQ denotes single-answer multiple-choice questions, with **4 options in English** and **5 options in Chinese**. MAQ denotes multi-answer multiple-choice questions and is available **only in Chinese**. OEQ denotes open-ended questions that simulate clinical or theoretical scenarios. DEF denotes terminology definition questions intended to test whether the model understands specialized dental terms and can ground them in correct definitions. The paper names the scoring metrics but does not provide explicit formulas for accuracy, precision, recall, F1, or BERTScore, and it does not give a formal symbolic objective for multi-answer scoring or generation scoring in the main text [2508.20416].

DentalQA is split into train and test in a **4:1 ratio** while preserving each subfield’s proportions. The paper does not provide exact per-task or per-language counts adding up to 36,597 in the main text, and it does not specify exact train/test counts beyond this ratio. That omission matters because it makes the benchmark more useful as an evaluation resource than as a fully specified reproducibility artifact.

## 3. DentalQA data sources and curation pipeline

The English and Chinese sides of DentalQA are constructed from different source ecosystems. The **English** side is curated from seven public medical QA sources: **MMLU, MedQA, MedMCQA, MedQuAD, PubMedQA, iCliniq, and the Medical Meadow Flashcards and Medical Meadow Wikidoc resources**. Because these are broad medical datasets rather than dental datasets, they are filtered using a keyword list derived from DentalCorpus construction. English DEF questions are created by taking dental terms from a bilingual glossary compiled during corpus construction and retrieving their definitions from **UMLS** [2508.20416].

The **Chinese** side is more directly dentistry-native. It is built from the **China National Dental Licensing Examination from 1999 to 2021**, **34 dental textbooks and auxiliary materials**, and **181 open-ended questions derived from real orthodontist–patient interactions**. This gives the Chinese portion a closer alignment with dental education and practice as represented in the source description [2508.20416].

The construction pipeline is unified across languages as much as possible. MCQ and MAQ items are normalized to fixed option counts. DEF items are generated by filling **50 predefined templates per language** with extracted dental terms and their definitions. OEQ items are kept in original form. Domain relevance is enforced using **GPT-4o** to classify every question as **oral-related**, **non-oral**, or **insufficient**, where “insufficient” denotes incomplete or corrupted items. Only oral-related questions are retained. Manual validation of this filtering step on **300 representative samples**, with **50 examples for each language-category combination**, yielded agreement rates of **100%, 96%, and 94%** for English and **96%, 92%, and 92%** for Chinese across the three categories [2508.20416].

This curation strategy makes DentalQA a filtered, structured benchmark rather than a raw aggregate of pre-existing sources. At the same time, the paper does not report inter-annotator agreement beyond those validation percentages, and it does not define an exact rule for converting model generations into answer options in the main text.

## 4. DentalCorpus and domain adaptation infrastructure

DentalCorpus is the adaptation half of DentalBench. The final corpus contains **337.35 million tokens** in total, comprising **1.06 million English chunks totaling 319.08 million tokens** and **66.3 thousand Chinese chunks totaling 18.27 million tokens**. The corpus is therefore heavily English-dominant, and the paper explicitly notes that this asymmetry becomes important when interpreting later bilingual results [2508.20416].

The corpus is built from three source types. First, **textbooks**: the authors collect **40 Chinese dental textbooks and auxiliary materials**, remove non-content sections, and apply OCR, yielding **4.1 million characters** of clean text. They also extract a bilingual glossary of **1,971 dental terms** from textbook glossaries. Second, **PubMed articles**: using **28 MeSH terms related to dentistry**, they retrieve **54,651 freely accessible full-text articles from 2000 to 2024**, producing **983.3 million English characters** and **5.4 million Chinese characters** before filtering. Third, **open medical datasets**: they filter **MMedC**, whose raw scale is reported as **10.56B English and 4.35B Chinese tokens**, and **MedRAG**, which contains **23.9M PubMed snippets**, to retain only dental-relevant content [2508.20416].

The filtering pipeline is relatively detailed. It uses a rule-based keyword filtering method driven by keyword lists derived from TF-IDF analysis on dental versus general medical corpora. The final keyword lists contain **440 English keywords** and **235 Chinese keywords**. Texts are retained if keyword density is **greater than 1%** and if there are **at least two distinct keyword matches per sentence**. English matching uses space tokenization, while Chinese uses direct string matching. After filtering, the corpus is deduplicated with **MD5 hashes**, embedded using **bge-m3**, and segmented into chunks of up to **512 tokens**. Human validation on **100 random samples per language** found domain relevance rates of **99% for English** and **96% for Chinese** [2508.20416].

DentalCorpus supports two adaptation regimes in the paper: **supervised fine-tuning** and **retrieval-augmented generation**. For retrieval, the authors use **FAISS** over **bge-m3 embeddings** with **top-5 passage retrieval**. The paper does not provide a mathematical retrieval equation, and it does not specify a formal supervised loss beyond standard fine-tuning practice.

## 5. Benchmark evaluation of current LLMs

The paper evaluates **14 LLMs** in a zero-shot setting with task-specific prompts on the held-out test set, using **eight NVIDIA RTX 3090 GPUs**. The evaluated models span three groups. The proprietary models are **GPT-4o, GPT-4o-mini, DeepSeek-V3, and DeepSeek-R1**. The open-source general models are **LLaMA-3.2-3B-Instruct, LLaMA-3.1-8B-Instruct, and Qwen-2.5-1.5B/3B/7B/14B/32B-Instruct**. The medical-specific models are **BioMistral-7B, HuatuoGPT2-7B, and LLaMA-3-8B-UltraMedical** [2508.20416].

The results establish DentalBench as difficult for current systems. On **Chinese MCQ**, **DeepSeek-R1** is best with **76.06** accuracy, followed by **Qwen2.5-32B** at **70.86** and **DeepSeek-V3** at **69.28**. On **Chinese MAQ**, **DeepSeek-R1** is again strongest, with **43.51** exact accuracy, **88.64** precision, **86.68** recall, and **87.65** F1. On **English MCQ**, **GPT-4o** leads with **73.98** accuracy, with **DeepSeek-V3** second at **68.28**. For **English OEQ**, **GPT-4o-mini** achieves the highest BERTScore at **34.55**, narrowly above **BioMistral-7B** at **34.50** and **Qwen2.5-3B** at **34.48**. For **English DEF**, **Qwen2.5-3B** is highest at **29.62**. In **Chinese OEQ**, **DeepSeek-V3** reaches **27.79** BERTScore, while **GPT-4o** leads **Chinese DEF** at **21.60**, with **HuatuoGPT2-7B** close at **21.04** [2508.20416].

Several patterns are emphasized. First, there is a strong **task-type gap**: MCQ and MAQ scores are much higher than OEQ and DEF scores across nearly all models and both languages. Second, there is a **language gap**: Chinese and English do not have the same winners, and adaptation effects differ by language. Third, **medical-specific models are not automatically better**. The paper explicitly notes that medical LLMs do relatively well on OEQ and DEF but underperform general-purpose models on MCQ and MAQ; for example, **LLaMA-3.1-8B** beats **LLaMA-3-8B-UltraMedical** on the multiple-choice tasks. Fourth, within the **Qwen-2.5** family, increasing scale clearly improves MCQ and MAQ, but yields limited gains on OEQ and DEF, which the paper interprets as evidence that larger parameter counts benefit factual recall more than dental generative reasoning [2508.20416].

## 6. Adaptation experiments with Qwen2.5-3B

The domain adaptation experiments are organized around **Qwen2.5-3B-Instruct**. The baseline is zero-shot Qwen2.5-3B, and the paper compares **supervised fine-tuning (SFT)**, **retrieval-augmented generation (RAG)**, and **SFT+RAG**. For SFT, the model is fully fine-tuned on the DentalQA training split for **four epochs** with learning rate **\(1 \times 10^{-6}\)**, **batch size 16**, and **bfloat16** precision. For RAG, the system retrieves the **top-5 most relevant passages** from DentalCorpus using **FAISS** over **bge-m3 embeddings** and prepends them to the prompt. SFT+RAG applies both procedures [2508.20416].

SFT produces clear gains on factual and multi-answer tasks. **Chinese MCQ** rises from **48.63** to **54.58**, and **English MCQ** from **41.77** to **47.90**. **Chinese MAQ** exact accuracy rises from **19.19** to **25.60**, and recall increases from **80.37** to **93.24**, although precision drops from **77.70** to **75.57**. On generative tasks, gains are more modest: **Chinese OEQ** rises from **20.89** to **22.42**, **Chinese DEF** from **11.16** to **15.29**, **English OEQ** from **34.48** to **37.74**, and **English DEF** from **29.62** to **30.79** [2508.20416].

RAG changes the pattern. It also improves multiple-choice performance, but its largest gains are on knowledge-intensive open-ended tasks. **Chinese OEQ** rises from **20.89** to **30.18**, which the paper explicitly contrasts with the smaller SFT gain of **1.53** points on the same task. **Chinese DEF** improves from **11.16** to **22.13**. **English OEQ** rises to **36.47** and **English DEF** to **30.04**. These results are used to argue that retrieval particularly helps tasks requiring access to specialized domain knowledge and terminology rather than only internal parametric recall [2508.20416].

The combined **SFT+RAG** setting yields the strongest or near-strongest results overall for the adapted Qwen2.5-3B. **Chinese MCQ** reaches **60.06** and **Chinese MAQ** exact accuracy reaches **29.07**, with **93.46** recall and **84.62** F1. The paper notes additive gains on Chinese MCQ and MAQ of **+11.43** and **+9.88** relative to the base model. On **Chinese OEQ**, SFT+RAG reaches **30.06**, slightly below RAG alone’s **30.18**. On **Chinese DEF** it reaches **20.85**, below RAG alone’s **22.13** but still well above baseline. In English, SFT+RAG gives **52.15** on MCQ and **37.68** on OEQ, while **DEF** is **29.65**, which is not an improvement over SFT alone and is only slightly above baseline. The paper therefore concludes that combining SFT and RAG is broadly beneficial, especially in Chinese, but that retrieval effectiveness is language-sensitive and less consistent in English [2508.20416].

## 7. Position within dental AI benchmarking and acknowledged limitations

DentalBench occupies a specific niche within the broader dental AI benchmark landscape. Other benchmark efforts described in the literature focus on different modalities and task families: **CTooth** and **CTooth+** target **3D CBCT tooth volume segmentation** [2206.08778], [2208.01643]; **Teeth3DS** targets **intraoral 3D scan segmentation and labeling** [2210.06094]; **DENTEX** and **PerioXrays/PerioDet** focus on **panoramic radiograph abnormal-tooth or apical periodontitis detection** [2305.19112], [2507.18958]; **PRAD-10K** targets **periapical radiograph segmentation** [2504.07760]; **MMOral-Bench** targets **panoramic multimodal VQA** [2509.09254]; **OralMLLM-Bench** evaluates **multimodal dental radiographic cognition** [2605.01333]; **Dental-TriageBench** evaluates **hierarchical multimodal triage from complaint and OPG** [2604.13060]; and **GlobalDentBench** evaluates **multinational dental LLM reasoning and safety** [2605.24636]. Against that background, DentalBench is distinctive in being a **bilingual, text-centered dentistry benchmark and adaptation suite** for LLMs rather than a radiograph-first or geometry-first benchmark.

The paper also identifies several limitations. The most prominent is **bilingual asymmetry**: although both DentalQA and DentalCorpus are English–Chinese, their distributions, source diversity, and coverage depth are not fully aligned. The corpus itself is far larger in English than in Chinese, while the QA sources differ substantially by language. A second limitation is that **MAQ exists only in Chinese**, so English multi-answer reasoning is not evaluated. The paper also does not discuss **clinical deployment validation**, **safety auditing**, or **direct real-world clinical outcome testing** in depth. In addition, the main text does not provide exact per-task counts, exact train/test counts, explicit metric formulas, or explicit objective equations for SFT or retrieval [2508.20416].

Taken together, DentalBench turns dentistry into a measurable and improvable LLM domain. It shows that current models possess nontrivial dental knowledge but exhibit uneven capabilities across task types and languages, that broad medical competence does not imply dentistry competence, and that explicitly dental adaptation resources can substantially improve performance. This suggests that, within dental AI, benchmark design must track not only modality and task format but also specialty structure, terminology density, and cross-lingual variation.

Source: https://www.emergentmind.com/topics/dentalbench