MedMNIST: Biomedical Benchmark Suite

• MedMNIST is a standardized biomedical image classification benchmark that consolidates diverse 2D and 3D datasets into a common format.
• The suite supports binary, multi-class, ordinal regression, and multi-label tasks with official train/validation/test splits for reproducibility.
• Extensions like MedMNIST+ and MedMNIST-C enhance the benchmark by enabling higher resolution evaluation and robustness testing under realistic corruptions.

MedMNIST is a standardized benchmark suite for biomedical image classification that was introduced in an MNIST-like format to support education, rapid prototyping, multi-modal machine learning, and AutoML in medical image analysis (Yang et al., 2020). The original release comprised 10 pre-processed medical image datasets standardized to $28\times 28$ images (Yang et al., 2020). MedMNIST v2 expanded this design into a larger-scale benchmark with 12 2D datasets and 6 3D datasets, all preprocessed to $28\times 28$ for 2D or $28\times 28\times 28$ for 3D, and covering binary classification, multi-class classification, ordinal regression, and multi-label classification (Yang et al., 2021). Its central purpose is not to approximate full clinical pipelines, but to provide a lightweight, diverse, and reproducible benchmark for comparing learning algorithms across biomedical modalities under controlled conditions (Yang et al., 2020, Yang et al., 2021).

1. Origins and benchmark rationale

MedMNIST was created in response to a practical difficulty in medical image analysis: datasets are often heterogeneous, nonstandard, modality-specific, and burdensome to preprocess, which makes experimentation slow and complicates method comparison (Yang et al., 2020). The benchmark therefore standardized diverse open medical datasets into a common small-image classification format that requires no specialized biomedical preprocessing knowledge from the user (Yang et al., 2020). In its original formulation, this standardization was also tied to the MedMNIST Classification Decathlon, a challenge-style evaluation over all constituent datasets without manual per-dataset tuning (Yang et al., 2020).

MedMNIST v2 generalized this idea from a 2D-only benchmark to a combined 2D and 3D biomedical classification suite (Yang et al., 2021). The v2 paper explicitly framed the collection as a lightweight benchmark intended to isolate the machine-learning component of biomedical classification, in contrast to benchmarks where performance depends strongly on complex preprocessing, postprocessing, augmentation, or challenge-specific engineering (Yang et al., 2021). This design choice made MedMNIST particularly suitable for comparative studies of model families, transfer-learning schemes, AutoML systems, robustness methods, and representation-learning pipelines across heterogeneous medical domains (Yang et al., 2021, Wu et al., 24 Jan 2025).

2. Dataset collection and task structure

MedMNIST v2 contains 18 datasets: 12 in 2D and 6 in 3D (Yang et al., 2021). The 2D collection spans pathology, chest X-ray, dermatoscopy, retinal OCT, fundus photography, breast ultrasound, blood-cell microscopy, kidney cortex microscopy, and abdominal CT; the 3D collection spans abdominal CT, chest CT, adrenal-shape masks, vessel shapes derived from brain MRA, fracture-centered CT crops, and electron microscopy (Yang et al., 2021).

Subset	Format and modality	Task
PathMNIST	2D colon pathology	Multi-class (9)
ChestMNIST	2D chest X-ray	Multi-label binary (14 labels)
DermaMNIST	2D dermatoscope	Multi-class (7)
OCTMNIST	2D retinal OCT	Multi-class (4)
PneumoniaMNIST	2D chest X-ray	Binary (2)
RetinaMNIST	2D fundus camera	Ordinal regression (5 levels)
BreastMNIST	2D breast ultrasound	Binary (2)
BloodMNIST	2D blood cell microscope	Multi-class (8)
TissueMNIST	2D kidney cortex microscope	Multi-class (8)
OrganAMNIST	2D abdominal CT	Multi-class (11)
OrganCMNIST	2D abdominal CT	Multi-class (11)
OrganSMNIST	2D abdominal CT	Multi-class (11)
OrganMNIST3D	3D abdominal CT	Multi-class (11)
NoduleMNIST3D	3D chest CT	Binary (2)
AdrenalMNIST3D	3D adrenal shape from abdominal CT	Binary (2)
FractureMNIST3D	3D chest CT	Multi-class (3)
VesselMNIST3D	3D vessel shape from brain MRA	Binary (2)
SynapseMNIST3D	3D electron microscope	Binary (2)

The benchmark was explicitly designed to span major axes of biomedical-image variation: modality, task type, and scale (Yang et al., 2020, Yang et al., 2021). Sample sizes range from around $10^2$ to $10^5$, and the suite includes both small datasets such as BreastMNIST and RetinaMNIST and large datasets such as TissueMNIST, PathMNIST, OCTMNIST, and ChestMNIST (Yang et al., 2020, Yang et al., 2021). This heterogeneity is one reason MedMNIST became attractive for comparative studies that aim to distinguish method-specific behavior from dataset-specific effects (Wu et al., 24 Jan 2025, Doerrich et al., 2024).

3. Standardization, preprocessing, and benchmark protocol

A defining feature of MedMNIST is aggressive standardization. In MedMNIST v2, all 2D images are preprocessed to $28\times28$, all 3D volumes to $28\times28\times28$, and the datasets are distributed with official train/validation/test splits in NumPy .npz files (Yang et al., 2021). When official source splits exist, they are retained; otherwise, the benchmark uses either a $9{:}1$ split of source training data when a source validation set is available as test, or a patient-level $7{:}1{:}2$ split when no official split exists (Yang et al., 2021). Labels are packaged as shape $N\times1$ for binary, multi-class, and ordinal tasks, and shape $28\times 28$0 for multi-label tasks such as ChestMNIST (Yang et al., 2021).

This preprocessing confers two important properties. First, experiments are cheap and reproducible: the small inputs make it easy to benchmark conventional CNNs, ViTs, AutoML systems, and newer representation-learning pipelines on a single GPU or modest compute budget (Yang et al., 2020, Yang et al., 2021). Second, the benchmark is intentionally simplified. The v1 and v2 papers both stress that MedMNIST is a benchmarking and educational resource rather than a clinically deployable dataset, because aggressive resizing can remove disease-relevant detail (Yang et al., 2020, Yang et al., 2021).

The benchmark protocol in the original and v2 papers uses AUC and ACC as the core evaluation metrics (Yang et al., 2020, Yang et al., 2021). The rationale is that AUC is threshold-free and less sensitive to class imbalance, while ACC remains easy to interpret (Yang et al., 2020). MedMNIST v2 further provided strong standardized baselines: 2D ResNet-18 and ResNet-50 at $28\times 28$1 and $28\times 28$2 input settings, 3D ResNet variants using 2.5D, 3D, and ACS convolutions, and several AutoML systems including auto-sklearn, AutoKeras, and Google AutoML Vision for 2D data (Yang et al., 2021).

4. Evolution of the ecosystem: v2, MedMNIST+, and MedMNIST-C

The original MedMNIST design made low-resolution benchmarking easy, but later work argued that the fixed $28\times 28$3 setting is too restrictive for modern pretrained architectures and for systematic analysis of resolution effects (Doerrich et al., 2024). MedMNIST+ addressed this limitation by keeping the same underlying datasets and official splits while extending the image resolutions to $28\times 28$4, $28\times 28$5, $28\times 28$6, and $28\times 28$7 (Doerrich et al., 2024). The MedMNIST+ benchmark re-evaluated common CNNs and ViT architectures across resolutions, training methodologies, and dataset regimes, and concluded that higher resolutions do not consistently improve performance beyond a certain threshold, that computationally efficient training schemes can be viable alternatives to costly end-to-end training, and that CNNs remain competitive with ViTs in this ecosystem (Doerrich et al., 2024).

MedMNIST-C extended the collection in a different direction: robustness evaluation under realistic corruptions (Salvo et al., 2024). Built on MedMNIST+ at $28\times 28$8 resolution, it provides corrupted test sets spanning 12 datasets, 9 imaging modalities, 5 corruption categories, and 5 severity levels per corruption (Salvo et al., 2024). Its corruption design is explicitly modality-aware rather than globally shared, with categories Digital, Noise, Blur, Color, and Task-specific (Salvo et al., 2024). To account for class imbalance, MedMNIST-C introduced normalized balanced error and relative balanced error: $28\times 28$9 and

$28\times 28\times 28$0

These metrics turn MedMNIST from a clean-image benchmark into a broader ecosystem for corruption robustness, corruption-aware augmentation, and domain-generalization studies (Salvo et al., 2024).

5. MedMNIST as a research testbed

Because MedMNIST standardizes diverse biomedical tasks, it has become a common platform for research on transfer learning, architecture selection, robustness, representation learning, and OOD detection. A large benchmark study of foundation models on all 12 2D datasets found that end-to-end fine-tuning consistently outperforms linear probing across all datasets, except for a single small PathMNIST AUC exception, and that ViT-based models generally outperform CNNs in that study, with DINO, DINOv2, and ViT-B/16 emerging as particularly strong choices (Wu et al., 24 Jan 2025). By contrast, the MedMNIST+ benchmark emphasized that CNNs remain competitive, especially when resolution, training scheme, and computational budget are taken into account (Doerrich et al., 2024). Taken together, these results suggest that MedMNIST is especially useful for testing claims about architectural superiority across many medical domains rather than on a single narrow dataset.

Robustness studies have used MedMNIST to challenge the adequacy of clean-image evaluation. MedMNIST-C showed that established architectures are not robust under clinically motivated corruptions; across the 12-dataset benchmark, ViT-B/16 was the most robust overall, and corruption-informed targeted augmentation achieved an average corrupted-test AUC gain of 9.2, compared with 3.9 for RandAugment, 2.7 for MixUp, and 1.0 for CutMix (Salvo et al., 2024). A probabilistic-robustness study on PathMNIST further found that high clean performance does not imply trustworthy behavior under realistic corruptions: clean ACC was 90.68% for ResNet-18 and 92.05% for ResNet-50, but under Motion corruption the reported probabilistic robustness dropped to 40.20% and 40.74%, respectively (Zhang et al., 4 Jul 2026).

The benchmark has also supported studies of representation structure and feature reliance. A backpropagation-free, frozen-foundation-model pipeline called A-ROM was evaluated on 11 of the 12 2D MedMNIST datasets and reported 83.7% average accuracy and 0.940 average AUC, while using a human-readable concept dictionary and a Mahalanobis-distance $28\times 28\times 28$1-NN classifier in place of fine-tuned decision layers (Karnes et al., 7 Apr 2026). A separate study on five MedMNIST tasks concluded that nonrobust-only models achieve well-above-chance balanced accuracy, while robust models sacrifice in-distribution accuracy but yield markedly better performance on MedMNIST-C, thereby exposing a robustness–accuracy trade-off in biomedical image classification (Mouton et al., 24 Apr 2026). For post-hoc OOD detection, feature-space density modeling with a normalizing flow on a frozen OrganAMNIST ResNet18 achieved AUROC 93.8 on a MedMNIST benchmark of near- and far-OOD datasets (Lotfi et al., 17 Feb 2025).

6. Limitations, criticisms, and disputed issues

The principal limitation of MedMNIST is also its defining simplification: the benchmark reduces medical images to very small fixed sizes (Yang et al., 2020, Yang et al., 2021). This enables fast experimentation, but it also removes fine-grained medical detail and can force task simplifications. The v1 paper explicitly merged normal and benign ultrasound findings into a single negative class in BreastMNIST because the original 3-class distinction was judged unsuitable for the low-resolution setting (Yang et al., 2020). The v2 paper similarly warns that MedMNIST is not intended for clinical use because the reduction to $28\times 28\times 28$2 or $28\times 28\times 28$3 may be insufficient to capture pathology (Yang et al., 2021). The creation of MedMNIST+ can be read as a direct response to this concern, since it preserves the same benchmark logic while allowing higher resolutions (Doerrich et al., 2024).

Critiques have also arisen at the subset level. A DermaMNIST case study argued that standard DermaMNIST is inadequate for reliable skin-lesion classification and attributed this to very low resolution (28×28), class imbalance, and poor organization; that study reported much better results on an external high-resolution derivative called DermaMNIST-C, but also noted that this derivative is not the same benchmark as official MedMNIST DermaMNIST (Kadric et al., 17 Jul 2025). More broadly, robustness studies indicate that clean MedMNIST performance can overstate trustworthiness, since models that score well on standard test splits may fail under corruption, distribution shift, or OOD conditions (Salvo et al., 2024, Zhang et al., 4 Jul 2026, Lotfi et al., 17 Feb 2025).

A more disputed line of criticism concerns label consistency. One paper on “error-free training” claimed that ChestMNIST, OCTMNIST, and TissueMNIST contain a “double-labeling problem,” defined as duplicated inputs with conflicting labels, and argued that this prevents perfect training-set fitting unless the conflicting examples are removed (Deng, 20 Apr 2026). This should be understood as a reported claim rather than a settled property of the benchmark. Even if accepted at face value, it concerns deterministic memorization on training data rather than the standard validation/test protocol used in MedMNIST benchmarking (Deng, 20 Apr 2026).

In practice, MedMNIST is best interpreted as a compact and highly influential benchmark ecosystem rather than as a proxy for clinical deployment. Its importance lies in providing a common substrate on which clean accuracy, transferability, robustness, OOD detection, representation quality, and computational efficiency can all be studied across many biomedical domains under a uniform interface (Yang et al., 2020, Yang et al., 2021, Salvo et al., 2024).