DermaMNIST: Compact Dermatology Image Benchmark

Updated 6 July 2026

DermaMNIST is a 7-class dermatoscopic benchmark that downscales high-resolution HAM10000 images to 28×28, preserving key diagnostic challenges.
It supports diverse tasks such as multi-class classification, binary risk screening, open-set recognition, and robustness testing under label noise.
Variants DermaMNIST-C and DermaMNIST-E address lesion-level leakage and preprocessing artifacts to improve evaluation reliability and performance comparability.

DermaMNIST is the dermatology subset of MedMNIST, a seven-class dermatoscopic image benchmark derived from HAM10000 and distributed in a compact 28×28 format for lightweight experimentation. In the literature it functions both as a conventional multi-class lesion-classification task and as a stress test for class imbalance, open-set recognition, label-noise robustness, ambiguity-aware calibration, self-supervised learning, and adversarial defense. Its research importance is inseparable from its design constraints: aggressive downsampling, highly skewed class frequencies, and later-documented lesion-level leakage in the original split all materially affect how published results should be interpreted (Abhishek et al., 2024, Vishal et al., 1 May 2026).

1. Origin, source data, and label space

DermaMNIST is described as the dermatology subset of MedMNIST and is derived from HAM10000, a large multi-source dermatoscopic repository of common pigmented skin lesions (Abhishek et al., 2024). The source collection contains 10,015 images, with original image resolution reported as 600×450 in the quality-analysis study and as dermatoscopic images with dimensions 450 × 600 pixels in the residual-LLM study; DermaMNIST converts these data into a lightweight benchmark by resizing them to 28×28 for the standard MedMNIST setting (Abhishek et al., 2024, Lai et al., 2024).

The task is seven-way lesion classification. Across the papers, the label set is described using HAM10000-style diagnostic groupings. One study names the classes as Melanocytic Nevi, Melanoma, Benign Keratosis-like lesions, Basal Cell Carcinoma, Actinic Keratoses, Vascular Lesions, and Dermatofibroma; another gives the corresponding HAM10000 abbreviations akiec, bcc, bkl, df, mel, nv, and vasc (Gonzalez et al., 1 Apr 2025, Kadric et al., 17 Jul 2025). The HAM10000 source labels are reported to cover 95% of pigmented lesions encountered in clinical practice, with ground truth established via histopathology, confocal microscopy, clinical follow-up, or expert consensus (Abhishek et al., 2024).

Several studies emphasize that DermaMNIST preserves clinically meaningful difficulty despite its compressed format. At the same time, the 28×28 representation removes much of the fine dermoscopic structure that would ordinarily support lesion discrimination, and later work explicitly treats this downsampling as a source of ambiguity, calibration error, and reduced separability among visually similar classes (Kadric et al., 17 Jul 2025, Tao et al., 24 Mar 2026).

2. Splits, resizing, and corrected variants

The original DermaMNIST release uses fixed train, validation, and test partitions. A later audit reconstructed the split manifests and showed that the benchmark was released as pre-processed NumPy arrays without filenames, which initially obscured leakage analysis (Abhishek et al., 2024).

Variant	Split sizes	Defining property
Original DermaMNIST	7,007 / 1,003 / 2,005	Official MedMNIST split
DermaMNIST-C	8,208 / 575 / 1,232	Lesion-level leakage corrected
DermaMNIST-E	10,015 / 193 / 1,512	HAM10000 train with ISIC 2018 validation/test

The key curation issue is lesion-level overlap across partitions. HAM10000 contains 10,015 images but only 7,470 unique lesions, and 1,956 lesions have two or more images. Because lesion-level grouping was not enforced in the original split, the same lesion can appear in multiple partitions. The audit reports 886 leaked images between train and test, 440 between train and validation, 128 between validation and test, and 51 in train–validation–test triple overlap; in aggregate, 1,006 of the 7,470 unique lesions appear in more than one partition (Abhishek et al., 2024).

The same study identifies a second benchmark artifact in MedMNIST’s 224×224 setting: the official code upsamples already downsampled 28×28 images to 224×224 with nearest-neighbor interpolation, producing visibly pixelated images and misrepresenting a high-resolution regime. To address both problems, DermaMNIST-C resizes directly from original high-resolution images using bicubic interpolation and enforces lesion-level non-overlap by moving all images of any lesion present in train back into the training split. DermaMNIST-E goes further by using HAM10000 for training and ISIC 2018 validation and test partitions for external held-out evaluation (Abhishek et al., 2024).

These corrected variants change both the partition structure and the effective evaluation problem. The quality-analysis paper therefore cautions that direct numerical comparison between the original DermaMNIST and DermaMNIST-C is not straightforward, because leakage correction also enlarges the training set and shrinks the validation and test sets (Abhishek et al., 2024).

3. Imbalance structure and task formulations

Class imbalance is one of the defining properties of DermaMNIST. In a study centered on lightweight dermatological classification, the training split histogram is reported as 4,693 Melanocytic Nevi, 779 Melanoma, 769 Benign Keratosis-like lesions, 359 Basal Cell Carcinoma, 228 Actinic Keratoses, 99 Vascular Lesions, and 80 Dermatofibroma, yielding a reported “balance index” of 0.02 (Gonzalez et al., 1 Apr 2025). A separate self-supervised learning study quantifies the inherent MedMNIST-v2 imbalance of DermaMNIST with Imbalance Ratio $58.66$, Coefficient of Variation $1.65$, Normalized Entropy $0.58$, Gini Index $0.64$, and Rare Class Ratio $1.14$ (Sharma et al., 2 Apr 2026).

This skew supports several distinct experimental formulations. In the standard closed-set setting, DermaMNIST is used as a seven-class classification benchmark. In binary risk-aware screening, one study binarizes the task into malignant versus benign, defining the positive class as melanoma, basal cell carcinoma, and actinic keratosis, and the negative class as melanocytic nevus, benign keratosis, vascular lesion, and dermatofibroma. Under that binarization, the reported class counts are 5,641 benign and 1,366 malignant in train, 807 benign and 196 malignant in validation, and 1,613 benign and 392 malignant in test, corresponding to malignant prevalence of about 19.5% across splits (Pereira et al., 26 Apr 2026).

DermaMNIST is also used for open-set recognition. In the DMDSC study, the dataset is partitioned into known and unknown classes over $t=4$ trials, with unknown classes held out during training and with “300k Random Images” from a subset of Tiny ImageNet used as auxiliary background data. The paper states that the known/unknown classes are chosen randomly per trial, but it does not specify which of the seven classes are held out in each case, nor the exact number of known versus unknown classes per trial (Vishal et al., 1 May 2026).

A plausible implication is that DermaMNIST is no longer a single task in the literature. It is a compact dermatology benchmark whose statistical imbalance is reused across substantially different objectives: seven-way diagnosis, malignant-screening binarization, open-set rejection, noisy-label stress testing, and calibration under ambiguous supervision.

4. Closed-set benchmark behavior across model families

The literature uses DermaMNIST to compare CNNs, ViTs, frozen-language-model hybrids, lightweight CPU-oriented models, and distilled interpretable surrogates. Reported numbers vary substantially with preprocessing, resolution, and split choice.

Setting	Metric summary	Source
ViT-S + residual LLM block	ACC 79.50, AUC 94.50	(Lai et al., 2024)
ResNet-18 vs ViT family at 224×224	ResNet-18 80.26%; ViT-Small (P16) 81.56%; ViT-Tiny (P16) ER 11.23	(Amangeldi et al., 13 May 2025)
Lightweight CNN with instance selection	Best mean ACC 69.38 ± 1.55; best single run 71.57; $\sim 472$ K parameters	(Gonzalez et al., 1 Apr 2025)
CNN-to-decision-tree distillation	CNN 78.5%; decision tree 68.1%; tree depth 4	(Srinivas et al., 2024)
EfficientNetV2L on DermaMNIST-C	ACC 0.8490, Precision 0.8935, Recall 0.8239, AUC 0.9768	(Kadric et al., 17 Jul 2025)

In the residual-based language-model study, DermaMNIST is one of the MedMNIST v2 2D benchmarks used to test a frozen LLaMA-7B transformer block inserted into a ViT-S encoder. The reported DermaMNIST gain is modest but positive: ACC improves from 78.95 to 79.50 and AUC from 94.27 to 94.50 (Lai et al., 2024). In the CNN–ViT efficiency study, a 224×224 pipeline finds ViT-Small with patch size 16 to be the most accurate model at 81.56%, while ViT-Tiny with patch size 16 attains the highest reported efficiency ratio at 11.23 and uses 5.53M parameters versus 11.18M for ResNet-18 (Amangeldi et al., 13 May 2025).

Two other lines of work use DermaMNIST to probe low-resource modeling. A CPU-only study applies k-means instance selection to the majority Melanocytic Nevi class, reducing that class from 4,693 to 1,000 representatives and shrinking the training set from 7,007 to 3,314 images before augmentation; its best mean accuracy is 69.38 ± 1.55 with ELU and RGB inputs, and its best single run reaches 71.57%, close to the cited MedMNIST ResNet-18 and ResNet-50 baselines of 73.5% (Gonzalez et al., 1 Apr 2025). A separate interpretability study compresses final CNN feature maps to a four-dimensional bottleneck and trains a shallow decision tree on those features; on DermaMNIST the teacher CNN reaches 78.5% and the depth-4, 10-node tree reaches 68.1% (Srinivas et al., 2024).

Resolution and curation strongly alter the reported operating regime. In a transfer-learning study, frozen ResNet-50 and EfficientNetV2L models on the original 28×28 DermaMNIST reach ACC 0.6663 and 0.7017, respectively, with AUC 0.9178 and 0.9334, while the same paper reports much stronger results on the curated 224×224 DermaMNIST-C, where a frozen EfficientNetV2L with a minimal 64-unit head attains ACC 0.8490 and AUC 0.9768 (Kadric et al., 17 Jul 2025). This supports the broader observation from the quality-analysis paper that the 28×28 representation is convenient for benchmarking but materially constrains clinically relevant visual detail (Abhishek et al., 2024).

5. DermaMNIST as a testbed for open-set learning, robustness, and uncertainty

DermaMNIST is frequently used not merely for closed-set classification but for evaluating geometric, probabilistic, and robustness-oriented learning paradigms. In open-set recognition, the DMDSC framework fixes class prototypes at simplex equiangular tight frame vertices on a hypersphere and replaces a uniform margin with a class-specific dynamic margin,

$m_{c} = m_{\min} + \bigl(m_{\max} - m_{\min}\bigr)\left(1-\frac{n_c}{\sum_{j=1}^{C} n_j}\right),$

so that minority classes receive larger margins. On DermaMNIST, with $R=100$ , $m_{\min}=35$ , and $1.65$0, DMDSC reports ACC 86.82, AUROC 81.44, and OSCR 73.64 over $1.65$1 open-set trials. It achieves the highest AUROC among the compared methods, although DIAS attains slightly higher OSCR at 74.56 (Vishal et al., 1 May 2026).

In self-supervised learning under real imbalance, AMIMV constructs asymmetric multi-image, multi-view positives rather than relying on two strong augmentations of the same image. Using ResNet-50 pretraining at 64×64 and linear evaluation, the study reports DermaMNIST top-1 accuracy of 77.85 and AUC of 93.60, improving over the best baseline ReSSL at 74.75 and 89.87, respectively (Sharma et al., 2 Apr 2026). The same paper provides a confusion matrix and per-class accuracy visualization for DermaMNIST, though it does not tabulate numeric per-class values.

Under label noise, two complementary directions appear. A geometry-aware reliability framework uses frozen DINOv2-base embeddings and NNK neighborhoods; on clean DermaMNIST, NNK diameter ratio with weighted inference reaches 0.759 accuracy versus 0.712 for standard k-NN, while under 60% symmetric noise unweighted methods become more robust, with NNK ensemble (UW) at 0.699 versus 0.689 for k-NN (Bozkurt et al., 31 Jul 2025). In a separate binarized screening study, noise-robust training methods are evaluated through both balanced accuracy and a cost-sensitive Global Risk,

$1.65$2

with $1.65$3 and $1.65$4 in the clinically asymmetric setting. On DermaMNIST, UNICON+CS yields the lowest average Risk II across 0%, 20%, and 40% noise at approximately 0.50, while Co-teaching+CS provides a stronger BAC–risk compromise at 20% noise, with BAC 0.760 and Risk II 1.059 (Pereira et al., 26 Apr 2026).

DermaMNIST is also used to study calibration under ambiguous labels. Because the dataset lacks real multi-annotator distributions, one calibration paper simulates $1.65$5 clinically informed annotators per image and reports a mean annotator agreement of 64.7%, the lowest among its benchmarks. On ResNet-18, standard Temperature Scaling yields true-label ECE of about 22.3%, whereas SLTS reduces it to 5.13%, MCTS with $1.65$6 to 5.07%, Dirichlet-Soft to 3.94%, IR-Soft to 2.20%, and the annotation-free LS-TS to 5.05%. On ViT-S/16, the corresponding Temperature Scaling error is about 24.8%, while IR-Soft reaches about 1.9–2.1% and Dirichlet-Soft about 3.0–3.2% (Tao et al., 24 Mar 2026).

Finally, DermaMNIST serves as an adversarial-robustness benchmark. MedRDF is an inference-time defense that perturbs each test image with isotropic noise, denoises the perturbed copies, and uses majority voting with an abstention test. For ResNet-50 on DermaMNIST, clean accuracy without defense is 73.0%; under PGD-20 it drops to 1.0% and under CW to 0.1%. With MedRDF using Poisson noise and a median filter, the same model reports 68.1% under PGD-20 and 70.0% under CW, with $1.65$7 noisy copies and approximately 1.1 seconds per image in the reported latency ablation (Xu et al., 2021).

6. Methodological cautions, reproducibility issues, and continuing debates

The central caution in the DermaMNIST literature is that benchmark scores are highly protocol-dependent. The quality-analysis paper shows that the original split violates lesion-level independence and that the official 224×224 setting may be produced by nearest-neighbor upsampling from 28×28 rather than resizing from the original images. It explicitly recommends DermaMNIST-C for leakage-free lightweight benchmarking and DermaMNIST-E for stricter external validation (Abhishek et al., 2024).

A second issue is preprocessing heterogeneity. Different studies use 28×28, 64×64, and 224×224 inputs; some resize directly from the original images, one uses 64×64 views for self-supervised learning, and one efficiency study describes the inputs as 28×28 grayscale replicated to three channels for its ViT pipeline (Kadric et al., 17 Jul 2025, Sharma et al., 2 Apr 2026, Amangeldi et al., 13 May 2025). This suggests that “DermaMNIST performance” is often inseparable from the chosen image formation protocol, not only from the backbone or loss.

A third issue is incomplete reporting. The open-set study does not disclose which DermaMNIST classes are known or unknown in each trial, nor any fixed rejection threshold or calibration strategy; the label-noise and efficiency papers often omit seeds, parameter sweeps, or full training hyperparameters; the lightweight CPU study does not specify the optimizer or epoch count; and the calibration study uses synthetic, class-conditional annotator distributions rather than real instance-specific reader disagreement (Vishal et al., 1 May 2026, Amangeldi et al., 13 May 2025, Gonzalez et al., 1 Apr 2025, Tao et al., 24 Mar 2026). A plausible implication is that cross-paper comparison is frequently governed as much by benchmark instantiation and reporting granularity as by algorithmic novelty.

Despite these caveats, several patterns recur consistently. DermaMNIST remains useful precisely because its compact size makes it easy to repurpose across methodological questions; its imbalance exposes minority-class and calibration failures quickly; and its downsampled, ambiguity-prone representation is severe enough to differentiate between protocols that exploit geometry, uncertainty, or curation. The literature therefore treats DermaMNIST less as a definitive clinical dataset than as a compact dermatology benchmark whose value lies in controlled comparison—provided that split integrity, resizing provenance, and evaluation protocol are made explicit.