DermaBench: AI Benchmarks for Dermatology

• DermaBench is a comprehensive suite of standardized benchmarks and protocols designed to evaluate dermatology AI models across diverse tasks including classification, reasoning, and explainability.
• It offers rigorous methodologies such as chain-of-thought evaluation, pixel-level explanation assessment, and hierarchical classification to ensure reproducibility and clinical relevance.
• The benchmarks facilitate improved model calibration, fairness assessment, and transparency by comparing AI outputs against expert-verified clinical references.

DermaBench is a collective term for a suite of standardized dermatology benchmarks and protocols that evaluate classification, reasoning, explainability, and multimodal understanding in medical AI systems. DermaBench benchmarks play a central role across distinct research streams: (1) vision-LLMs (VLMs) for diagnostic chain-of-thought (CoT) reasoning, (2) explainability-centric evaluation of convolutional neural networks (CNNs), (3) high-integrity image classification, (4) hierarchical and taxonomy-aware model benchmarking, and (5) dermatology visual question answering (VQA) and multimodal reasoning. Distinct benchmarks released under the name DermaBench cover different technical requirements and modalities, but all establish rigorous standards for model assessment, reproducibility, and clinical relevance in computational dermatology.

1. Chain-of-Thought Reasoning Benchmarks

DermaBench defines a protocol for evaluating the narrative quality of dermatologic CoT reasoning generated by VLMs. It is grounded on 3,000 certified image–narrative pairs from DermNet, each rigorously reviewed by board-certified dermatologists. The evaluation is six-dimensional, with each generated CoT narrative assigned an integer score (1–5) on the following axes:

• Accuracy: Concordance of findings and diagnosis with expert gold reference.
• Safety and Harmfulness: Potential risk from following model advice.
• Medical Groundedness: Consistency with evidence-based dermatology knowledge.
• Clinical Coverage: Coverage of morphology, differentials, and management.
• Reasoning Coherence: Logical, structured justification progression.
• Description Precision: Technical clarity and specificity in lesion description.

The scoring protocol enforces reproducibility via a fixed prompt, inference seed, and a single multimodal evaluator per case. For fourteen reference VLMs spanning general, medical, and reasoning-specialized systems, DermaBench provides standardized scores as the mean per-dimension and an overall average. The diagnostic reasoning VLM SkinGPT-R1 achieves the highest average score (4.031/5), outpacing contemporaries across accuracy and reasoning, demonstrating the value of chain-of-thought supervision and dermatologist-aligned evaluation (Shen et al., 19 Nov 2025).

A related framework uses similar clinically grounded axes, but with LLM-based judges (e.g., GPT-4) that compare candidate outputs to expert reference narratives, generating quantitative metrics with mean deviation from physician scoring (Δ̄ = 0.251). Applications include model fine-tuning, regulatory calibration, and robust assessment of narrative “explainability” (Shen et al., 12 Nov 2025).

2. Explainability and Localization Assessment

Explainability benchmarking within DermaBench is exemplified by the DermaXDB protocol, which evaluates the quality of CNN-generated heatmaps for dermatology diagnosis using pixel-level comparisons to expert annotation maps. DermaXDB consists of:

• 524 clinical images spanning six disease categories,
• Dense, fine-grained characteristic masks generated independently by eight expert dermatologists,
• Ground-truth explanation maps per characteristic and image: $E^c(x, y)$ and $E_i(x, y)$,
• Benchmark tasks quantifying fuzzy overlap via “explainability F1,” sensitivity, and specificity:

$$F1 = 2 \cdot \frac{\sum_p \min(G_p, E_p)}{\sum_p G_p + \sum_p E_p}$$

A diverse range of architectures (DenseNet121, EfficientNet-B0, InceptionV3, MobileNet variants, NASNetMobile, ResNet50, VGG16, Xception, etc.) are assessed for both class-level diagnosis and the spatial alignment of their Grad-CAM heatmaps with expert masks. Results demonstrate a trade-off: Xception maximizes explainability F1 (0.46) and diagnostic F1 (~0.81 on vitiligo), while NASNetMobile achieves superior sensitivity on small feature localization despite average classification accuracy. No CNN achieves expert-level explainability across all features, motivating the need for both robust localization metrics and domain-specific model selection (Jalaboi et al., 2023).

3. High-Integrity Image Classification and Data Protocols

DermaBench protocols for image classification emphasize dataset curation, stratification, and leakage avoidance, especially per lesion/patient. Key datasets include DermaMNIST and Fitzpatrick17k:

• DermaMNIST: 10,015 dermoscopic images, 7-class diagnosis, with critical corrections for duplicate lesions and partition leakage (e.g., 26.2% have ≥2 views). The corrected DermaMNIST-C version enforces strict lesion-level splitting, resulting in more robust assessments ($\Delta$AUC = +0.031; $\Delta$ACC = +0.090 vs. original).
• Fitzpatrick17k: 16,577 clinical images, 114-label taxonomy with rigorous duplicate/mislabel detection and partitioning protocols (embedding similarity thresholds, union-find clustering, patient/atlas stratification).

DermaBench mandates transparent train/val/test splits, per-class and group-wise reporting (e.g., by Fitzpatrick Skin Type), and public code for comparisons and reproducibility (Abhishek et al., 2024).

A parallel technical specification details full-stack model evaluation, including dataset preparation, augmentation (split-before-augment), macro-F1 and balanced accuracy reporting, attention map extraction, and standardized training pipelines (e.g., DINOv2-Large with AdamW, fixed seeds, 80/10/10 splits). Error analysis tools characterize model vulnerabilities, such as “typical” vs. “atypical” prediction behaviors and attention map fragmentation in composite images (Miętkiewicz et al., 4 Feb 2025).

4. Hierarchical and Taxonomy-Aware Benchmarking

The hierarchical DermaBench framework addresses the granularity gap in dermatologic diagnosis: models may excel at top-level (binary or coarse) distinctions but struggle with fine-grained subclass discrimination. Key elements:

• DERM12345 Dataset: 12,345 dermatoscopic images annotated with a four-level diagnostic hierarchy: 40 subclasses, 15 main classes, four superclasses, and binary malignancy.
• Benchmark Pipeline: Frozen feature extraction with ten general, medical, or domain-specific foundation models; adapters (KNN, LR, SVM, RF, MLP, XGBoost) trained on embeddings; five-fold cross-validation.
• Multi-level Evaluation: For each image, coarse-level predictions are probability sums over subclass indices. Weighted F1-Score is primary, with supplemental balanced accuracy and per-class metrics. Illustration: | Model | Binary F1 | 40-way F1 (Subclass) | |------------------|-----------|----------------------| | MedImageInsights | 97.52% | 65.50% | | MedSigLip | 96.43% | 69.79% | | Derm Foundation | 96.04% | 69.50% |

Findings highlight steep performance drop-offs from binary to fine-grained classification (“granularity gap”) and demonstrate that domain-targeted models (MONET, Derm Foundation) or adapters (MLP/XGBoost) can partially close this gap. The pipeline is fully reproducible and designed for extensibility to new foundation models (Yuceyalcin et al., 18 Jan 2026).

5. Multimodal Reasoning and Visual Question Answering Benchmarks

Recent DermaBench developments encompass clinician-annotated, VQA-format benchmarks that move beyond classification to probe model performance on image understanding, language grounding, and reasoning:

• Clinician-Annotated VQA: Built atop Diverse Dermatology Images (DDI), this metadata-only benchmark comprises 656 images of 570 unique patients (skin types I–VI), with ~14,474 structured question–answer pairs.
• Annotation Schema: 22 main hierarchical questions (diagnosis, anatomic site, lesion morphology, distribution, color, artifacts, image quality, etc.), multi-choice and open-ended clinical summaries.
• Downstream Tasks: Disease classification, morphology recognition, multi-label reasoning, and free-text narrative synthesis. Single-label tasks are evaluated via accuracy; multi-label by precision, recall, and F1; narratives by BLEU, ROUGE-L, and BERTScore (Yilmaz et al., 20 Jan 2026).

DermoBench (distinct from the above but using a similar nomenclature) also exists as a comprehensive multimodal benchmark (33,999 samples; 3,600 expert-verified cases) covering open-ended and MCQA tasks across four axes: morphology, diagnosis, reasoning, and fairness. Each open-ended output is scored using LLM-as-Judge metrics that decompose textual outputs into atomic claims and apply fidelity (recall-like) and precision-penalty adjustments. Core subtasks include fine-grained and OOD diagnosis, structured JSON-based lesion descriptions, clinical attribute detection, reasoning-inference consistency, and skin-type group fairness. Human baselines and detailed usage pseudocode are supplied (Ru et al., 5 Jan 2026).

6. Key Limitations, Recommendations, and Future Directions

DermaBench benchmarks collectively represent the state-of-the-art in dermatology AI evaluation, but several limitations are acknowledged:

• Skin-Type and Geographical Bias: Most datasets over-represent lighter phototypes or are geographically narrow (e.g., Turkey, North America). Expansion to underrepresented groups is ongoing (Abhishek et al., 2024, Shen et al., 12 Nov 2025, Yuceyalcin et al., 18 Jan 2026).
• Data Quality: Mislabeling, duplicates, partition leakage, and improper augmentation have previously inflated results. DermaBench correction protocols, embedding-based duplicate removal, and expert consensus labeling are essential for trustworthy measurement (Abhishek et al., 2024).
• Scope: Some modalities (histopathology, clinical photos, video) and rare lesion types remain underrepresented. Furthermore, VQA/CoT benchmarks have relatively modest case counts compared to pure classification datasets (Yilmaz et al., 20 Jan 2026).
• Inter-Annotator Agreement: While consensus review is standard, explicit agreement metrics (Cohen’s κ, Krippendorff’s α) are often missing.
• Model Calibration and Explainability: No single method achieves optimal diagnosis and interpretability across all tasks; trade-offs are model- and application-specific (Jalaboi et al., 2023, Yuceyalcin et al., 18 Jan 2026).
• Evaluation Automation: LLM judges closely track expert scores, but rare edge cases require human adjudication (Shen et al., 12 Nov 2025).

Best practices and recommendations include strict patient/lesion stratification, open releases of code and metadata, clear documentation of all processing steps, and reporting both macro- and group-stratified performance metrics. The modularity of DermaBench benchmarks enables extension to larger, international cohorts and novel modalities, supporting continuous updating as the AI dermatology landscape evolves.

7. Comparative Summary Table

Benchmark Variant	Modality / Focus	Primary Tasks	Evaluation Axes	Notable Features
CoT Narrative (SkinGPT-R1)	Image+Text CoT reasoning	Diagnostic narrative generation	6 clinician-defined dimensions	3K gold DermNet cases, evaluator-aligned scoring
Explainability (DermaXDB)	CNN image classification	Heatmap localization, diagnosis	F1, sensitivity, specificity (pixel-level)	524 images, 33 lesion features, expert heatmaps
High-Integrity Classification	Dermoscopic, clinical images	Multiclass, fairness assessment	AUC, ACC, per-class/group metrics	Data-cleaning protocols, lesion-level splits
Hierarchical (DERM12345)	Dermatoscopic	40/15/4/2-class diagnosis	Weighted F1, balanced accuracy, macro F1	Foundation model, frozen embeddings, adapters
VQA/Reasoning (DDI)	Clinical/demographic diversity	VQA, multi-label, open narrative	Accuracy, F1, BLEU, ROUGE-L, BERTScore	656 images, 14K QA, 22 hierarchical questions
Multitask MLLM (DermoBench)	Comprehensive reasoning	MCQA, open-ended, fairness	LLM-judge fidelity, group fairness, accuracy	33K instances, core set expert-verified, human baselines

Each benchmark is tailored for specific research questions but, taken together, cover the full spectrum of computational challenges in dermatological AI: from diagnostic accuracy and explainability to narrative reasoning and fairness. The proliferation and adoption of DermaBench standards have led to increased methodological rigor, transparency, and alignment with clinical practice in dermatology AI research.