Caries DETR: Tooth Structure-aware Prior and Lesion-aware Dynamic Loss Refinement for DETR Based Caries Detection

Abstract: As dental caries appear as subtle, low-contrast lesions in intraoral imaging, existing deep learning models face significant challenges in the early detection of caries. While recent Transformer-based detectors have shown promising results in natural images, they often fail to capture the domain-specific anatomical priors crucial for dental caries detection. In this paper, we propose Caries-DETR, a specialized Transformer framework for caries detection in intraoral images. A Tooth Structure-aware Query Initialization (TSQI) is designed, leveraging large-scale intraoral photograph pre-training and a structure perception branch (SPB) to integrate high-frequency structural priors, guiding the model to focus on anatomically significant lesion areas. Furthermore, we design a Lesion-aware Dynamic Loss Refinement (LDLR) to implement quality-driven hard mining through adaptive loss reweighting based on lesion size, anatomical relevance, and prediction quality, optimizing detection for subtle lesions. Extensive experiments on two public datasets (i.e., AlphaDent and DentalAI) demonstrate that Caries-DETR achieves a state-of-the-art performance compared to existing methods and exhibits good generalization and robustness. Code and data at https://github.com/XuefenLiu-SZU/Caries-DETR}{https://github.com/XuefenLiu-SZU/Caries-DETR.

• The paper introduces a novel DETR-based framework that integrates tooth structure-aware query initialization and lesion-aware dynamic loss refinement to enhance caries detection.
• It combines large-scale unsupervised pre-training with adaptive loss weighting to address class imbalance and subtle lesions in intraoral photography.
• The proposed method outperforms conventional CNN and Transformer detectors, achieving significant mAP improvements on AlphaDent and DentalAI datasets.

Caries-DETR: Tooth Structure-aware Prior and Lesion-aware Dynamic Loss Refinement for DETR-based Caries Detection

Introduction

The accurate detection of dental caries in intraoral photography is a critical yet challenging computer vision task, due to the low contrast, high class imbalance, and complex anatomical structure of lesions in dental images. Standard object detection architectures, whether CNN-based or Transformer-based, exhibit substantial limitations for this domain: CNNs struggle with morphological variability, while canonical DETR/Transformer frameworks lack domain-specific anatomical priors and demonstrate sub-optimal convergence on small, subtle lesions. The Caries-DETR framework directly tackles these issues by integrating large-scale unsupervised pre-training, a novel tooth structure-aware query initialization (TSQI), and a lesion-aware dynamic loss refinement (LDLR) into Transformer-based detection.

Methodology

Framework Overview

Caries-DETR follows a two-phase training regimen, coupling unsupervised anatomical pre-training on intraoral photographs with supervised lesion detection. Its architecture enhances existing DETR frameworks with explicit dental structural priors and adaptive loss reweighting, both designed specifically for the dental imaging context.

Figure 1: Overall framework of the proposed Caries-DETR.

Tooth Structure-aware Query Initialization (TSQI)

Unlike the appearance-driven and randomly-initialized queries in canonical DETR, TSQI leverages explicit tooth anatomy. A self-supervised pretext task is utilized on more than 30,000 unlabeled intraoral images, with a Scharr-operator-based gradient map serving as the high-frequency structural supervision target. A Structure Perception Branch (SPB) learns to extract robust anatomical features from the backbone's feature maps. During fine-tuning, the SPB produces structural saliency maps, which modulate semantic features to yield hybrid attention scores. These scores drive both the selection of anchor points and the content of queries, ensuring the Transformer's decoder is biased—by design—toward anatomically meaningful, lesion-prone regions rather than generic foreground proposals.

Lesion-aware Dynamic Loss Refinement (LDLR)

Class imbalance and hard sample mining remain persistent challenges, especially for small, low-contrast or ambiguous carious lesions. LDLR introduces a dynamic weighting scheme conditioned on the prediction quality across class confidence, box localization, and IoU alignment. By adaptively amplifying gradients on weakly localized or low-confidence lesions through linear penalty factors, LDLR induces an implicit curriculum: early training emphasizes the most challenging positives, while later stages drive fine-grained refinement. This scheme is critical for optimizing model sensitivity/precision balance on subtle pathologies.

Experimental Evaluation

Datasets and Protocol

Caries-DETR is comprehensively evaluated on AlphaDent and DentalAI datasets, both of which feature high variability in patient cohort, imaging conditions, and caries morphology. The framework employs ResNet-50 and Swin-Large backbones to assess scalability and generalization.

Quantitative Performance

Caries-DETR decisively outperforms mainstream CNN and Transformer-based detectors across all tested metrics. On AlphaDent (R-50), Caries-DETR achieves 29.9% mAP, surpassing the top DETR baselines (e.g., DINO, Co-DETR, DEIMv2) by 2–2.9 points, and yields notable AP improvements for Caries 4 and Caries 6—some of the most elusive classes from a boundary/localization perspective. Its performance advantage extends to stronger Swin-L backbones and generalizes on DentalAI, confirming the benefits of anatomy-centric priors and hard-sample optimized loss even for out-of-distribution samples.

Figure 2: Visual comparison of caries detection results on challenging cases across YOLOv12, RF-DETR, and Caries-DETR.

Qualitative Analysis

Visual evidence underscores Caries-DETR's superior sensitivity to subtle and ambiguous lesions. In challenging test cases, YOLOv12 and RF-DETR fail to localize small, low-contrast caries or capture fine-grained structural anomalies; Caries-DETR, by contrast, succeeds owing to its explicit spatial and structural inductive biases derived from TSQI and its robust gradients on difficult samples induced by LDLR.

Ablation Study

Ablation demonstrates that each architectural component contributes significant incremental value. Integrated, the three innovations yield an mAP gain of 2.7 points over the DINO baseline, with LDLR and TSQI especially impactful on hard classes and small/early-stage lesions.

Theoretical and Practical Implications

The explicit use of domain priors—via self-supervised learning of dental structure—and prediction-quality-adaptive loss scaling augments standard DETR dynamics in two key ways: improving data efficiency and convergence for domain-shifted imagery, and yielding superior robustness to class and quality imbalance endemic to medical image detection. Practically, these advances may enable more reliable, high-throughput caries screening and remote diagnostic support in tele-dentistry, particularly in low-resource settings or clinical contexts with scarce expert annotations. Theoretically, this approach evidences the utility of hybrid attention schemes and curriculum-based loss modulations in anatomical object detection, suggesting fruitful directions for other fine-grained medical imaging domains where small, morphologically variable, and ambiguous structures are prevalent.

Future research may further exploit joint multi-modal or multi-view learning, exploit richer forms of structural priors (e.g., hierarchical or geometric), or fuse temporal information for longitudinal monitoring in clinical workflows.

Conclusion

Caries-DETR substantively advances the state of DETR-style models for medical object detection by integrating large-scale self-supervised anatomical pre-training, a domain-specific query initialization regime, and adaptive loss reweighting targeting clinical detection challenges (2604.23718). Its demonstrated improvements across datasets, backbones, and lesion classes support both its methodological contributions and its translational potential in real-world dental computer-aided diagnosis.