DermEval: Multimodal Dermatology Evaluation

• DermEval is a multimodal evaluation framework that systematically scores dermatology diagnostic narratives based on six key clinical dimensions such as accuracy, safety, and reasoning coherence.
• It fuses processed dermatology images and text narratives using specialized encoders and linear projections to provide a structured critique aligned with board-certified criteria.
• Quantitative validation shows strong alignment with expert ratings, making it essential for benchmarking, data curation, and comparative analysis in dermatologic reasoning.

DermEval is a multimodal evaluation framework for systematically scoring and analyzing dermatology diagnostic narratives produced by vision-LLMs. It operates as an automatic evaluator, ingesting both clinical dermatology images and generated diagnostic text, and outputs a structured critique—six scalar ratings aligned to board-certified dermatologist criteria—along with an aggregate overall score. DermEval has emerged as a central tool for benchmarking, data curation, and comparative analysis of dermatologic reasoning in large multimodal LLMs, underpinning benchmarks such as DermBench and datasets like DermCoT (Shen et al., 12 Nov 2025, Shen et al., 19 Nov 2025).

1. System Overview and Architecture

DermEval is structured as a reference-free, learned multimodal scoring system. For each input, a dermatology image is resized to $224\times224$ pixels and processed through a frozen CLIP-ViT-B/32 image encoder, producing a 512-dimensional image embedding $\mathbf{v}$. The generated diagnostic narrative, capped at 512 tokens, is tokenized and fed through a fine-tuned ClinicalBERT text encoder, yielding a 768-dimensional text embedding $\mathbf{t}$.

Both modalities are projected to 512-dimensional subspaces via independent linear layers:

• $\mathbf{v}' = W_v\mathbf{v} + \mathbf{b}_v$
• $\mathbf{t}' = W_t\mathbf{t} + \mathbf{b}_t$

These projected embeddings are concatenated to form a 1,024-dimensional vector $[\mathbf{v}';\mathbf{t}']$, which is processed through a two-layer MLP with ReLU activations and dropout. The output layer emits six scalar ratings $\mathbf{r} = (r_1, ..., r_6)$, each corresponding to a clinically relevant dimension and constrained to $[1,5]$. An overall score $S$ is computed as the unweighted mean:

$S = \frac{1}{6} \sum_{i=1}^{6} r_i$

DermEval provides both per-dimension analyses and aggregate model competence measurement (Shen et al., 12 Nov 2025, Shen et al., 19 Nov 2025).

2. Clinically Grounded Evaluation Dimensions

DermEval’s rating scale mirrors the six axes used by expert dermatologists to judge written consultations. Each dimension $\mathbf{v}$0 is regressed to minimize deviation from expert average scores, with anchors defined as follows:

Dimension	Clinical Focus	Scoring Target (1–5)
Accuracy ($\mathbf{v}$1)	Correctness of diagnosis	1=wrong; 5=perfect
Safety ($\mathbf{v}$2)	Harm avoidance, red flag recognition	1=unsafe/misleading; 5=fully safe
Medical Groundedness ($\mathbf{v}$3)	Adherence to dermatologic principles	1=no rationale; 5=textbook correctness
Clinical Coverage ($\mathbf{v}$4)	Feature/differential/next-step completeness	1=incomplete; 5=comprehensive
Reasoning Coherence ($\mathbf{v}$5)	Logical, stepwise justification	1=jumps/contradicts; 5=sequential clarity
Description Precision ($\mathbf{v}$6)	Specificity, objectivity of lesion description	1=vague; 5=measure-specific

These scoring axes are defined with detailed rubrics. For example, “Accuracy” requires matching the expert reference on both diagnosis and descriptive findings (5: fully consistent, 1: misses essentials); “Safety” penalizes recommendations that could introduce iatrogenic harm or false reassurance (Shen et al., 12 Nov 2025, Shen et al., 19 Nov 2025).

3. Training Data Curation and Optimization

For supervised training, DermEval leverages 1,200 image–narrative pairs from 200 real dermatology cases. Each case is sourced from state-of-the-art vision-LLMs (e.g., GPT-4o-mini, Gemini 2.5 Flash, GPT-4+Visual), with five board-certified dermatologists independently rating each candidate narrative across all six dimensions. The dimension-wise means of these scores establish ground truth regression targets $\mathbf{v}$7.

Preprocessing includes center-cropping and ImageNet-based normalization of images and ClinicalBERT tokenization/lowercasing/truncation for narratives. Training employs AdamW ($\mathbf{v}$8 learning rate, $\mathbf{v}$9 weight decay, batch size 16), dropout ($\mathbf{t}$0), and early stopping on a 20% held-out fold. The objective function is mean squared error:

$\mathbf{t}$1

Further, in broader pipelines such as SkinGPT-R1, DermEval undergoes staged learning: initial supervised emission of parseable critiques, then REINFORCE-based alignment to expert scores (Shen et al., 12 Nov 2025, Shen et al., 19 Nov 2025).

4. Validation, Quantitative Performance, and Benchmark Role

Validation on a held-out test split of 240 pairs demonstrates that DermEval’s automated ratings closely align with the five-physician consensus. For each dimension, Pearson correlation coefficients ($\mathbf{t}$2) between DermEval and expert ratings range from $\mathbf{t}$3 to $\mathbf{t}$4, with mean absolute error (MAE) under $\mathbf{t}$5 on the five-point scale. The aggregate score reaches $\mathbf{t}$6, MAE = $\mathbf{t}$7.

Dimension	Pearson $\mathbf{t}$8	MAE
Accuracy	0.82	0.34
Safety	0.75	0.40
Groundedness	0.78	0.37
Coverage	0.74	0.42
Coherence	0.76	0.39
Precision	0.79	0.35
Overall	0.87	0.31

Key analyses in (Shen et al., 12 Nov 2025) demonstrate that multimodal fusion is critical to performance, with ablations revealing decreases in Pearson $\mathbf{t}$9 of $\mathbf{v}' = W_v\mathbf{v} + \mathbf{b}_v$0–$\mathbf{v}' = W_v\mathbf{v} + \mathbf{b}_v$1 when either modality is omitted. Dropout regularization modestly curbs overfitting ($\mathbf{v}' = W_v\mathbf{v} + \mathbf{b}_v$2 increase in MAE if removed). Notably, the evaluation highlights a mild performance gap (drop of $\mathbf{v}' = W_v\mathbf{v} + \mathbf{b}_v$3 in $\mathbf{v}' = W_v\mathbf{v} + \mathbf{b}_v$4, $\mathbf{v}' = W_v\mathbf{v} + \mathbf{b}_v$5 MAE) on Fitzpatrick IV–VI (darker skin types), suggesting avenue for further bias mitigation research.

Within benchmarking infrastructures such as DermBench and training data pipelines like DermCoT, DermEval acts both as gatekeeper—filtering high-quality data pairs (score $\mathbf{v}' = W_v\mathbf{v} + \mathbf{b}_v$6)—and as an independent judge for system-level evaluation (Shen et al., 12 Nov 2025, Shen et al., 19 Nov 2025).

5. Role in Benchmark Construction and Comparative Evaluation

DermEval’s six-dimensional scoring underlies the DermBench benchmark, which consists of 3,000 certified test cases (paired images and expert narratives). In the benchmarking workflow, candidate and reference narratives are jointly fed, with all decoding parameters locked across comparisons, to a fixed LLaVA-based vision-language evaluator implementing the DermEval protocol. Scores are averaged per dimension, then across all six, yielding aggregate metrics to rank and compare competing multimodal LLMs.

In the context of model development (e.g., SkinGPT-R1), DermEval gates data curation for instruction-tuning corpora (DermCoT), selects narrative quality for training, and provides a uniform axis for comparison. SkinGPT-R1, for example, achieves an overall score of $\mathbf{v}' = W_v\mathbf{v} + \mathbf{b}_v$7 (out of 5) on DermBench, ranking first across 14 models and outperforming general vision-language baselines by substantial margins, particularly on reasoning coherence and description precision (Shen et al., 19 Nov 2025).

6. Data Cleaning Protocol and Benchmark Integrity

A separate but related “DermEval” protocol is employed for stringent data cleaning and preparation of evaluation benchmarks (Gröger et al., 2023). This protocol algorithmically ranks images for irrelevance, near-duplication, and label-error likelihood, using a ViT-Tiny trained via self-supervised DINO. Candidates are presented in descending risk order to dermatology experts, with manual review halted upon 58 consecutive negatives (calculated to bound missed positives under 5%). Only unanimous triplet agreements prompt removal/flagging, and label errors are flagged for prevalence estimation, not automatic correction.

Empirical application of this cleaning protocol to six ISIC benchmarks yields removal rates of $\mathbf{v}' = W_v\mathbf{v} + \mathbf{b}_v$8 for most error types, yet even such small fractions impact downstream model metrics (up to –1.0% AUROC, –3.6% AUPRG). The cleaned benchmarks provide more trustworthy estimates of true model generalization (Gröger et al., 2023).

7. Impact, Known Limitations, and Use Cases

DermEval provides an automated yet clinically meaningful reference for assessing dermatology-focused multimodal LLMs. Per-case and aggregate score alignment with expert consensus is high, enabling fine-grained model diagnostics, ablation studies, and bias detection. Most errors arise from models producing overly terse or non-informative narratives. Multimodal fusion, robust regularization, and careful preprocessing are all integral to stable, high-fidelity evaluation.

Limitations include a mild performance gap on Fitzpatrick IV–VI skin types and dependence on the quality and diversity of reference training cases. In practice, DermEval is central to curation of narrative datasets, benchmarking leaderboards, and comparative assessment of new dermatology reasoning models (Shen et al., 12 Nov 2025, Shen et al., 19 Nov 2025, Gröger et al., 2023).