---
title: 'MedMNIST: Biomedical Benchmark Suite'
url: https://www.emergentmind.com/topics/medmnist
type: topic
---

# MedMNIST: Biomedical Benchmark Suite

MedMNIST is a standardized benchmark suite for biomedical image classification that was introduced in an MNIST-like format to support education, rapid prototyping, multi-modal machine learning, and AutoML in medical image analysis [2010.14925]. The original release comprised 10 pre-processed medical image datasets standardized to \(28\times 28\) images [2010.14925]. MedMNIST v2 expanded this design into a larger-scale benchmark with **12 2D datasets** and **6 3D datasets**, all preprocessed to **\(28\times 28\)** for 2D or **\(28\times 28\times 28\)** for 3D, and covering **binary classification, multi-class classification, ordinal regression, and multi-label classification** [2110.14795]. Its central purpose is not to approximate full clinical pipelines, but to provide a lightweight, diverse, and reproducible benchmark for comparing learning algorithms across biomedical modalities under controlled conditions [2010.14925][2110.14795].

## 1. Origins and benchmark rationale

MedMNIST was created in response to a practical difficulty in medical image analysis: datasets are often heterogeneous, nonstandard, modality-specific, and burdensome to preprocess, which makes experimentation slow and complicates method comparison [2010.14925]. The benchmark therefore standardized diverse open medical datasets into a common small-image classification format that requires no specialized biomedical preprocessing knowledge from the user [2010.14925]. In its original formulation, this standardization was also tied to the **MedMNIST Classification Decathlon**, a challenge-style evaluation over all constituent datasets without manual per-dataset tuning [2010.14925].

MedMNIST v2 generalized this idea from a 2D-only benchmark to a combined **2D and 3D biomedical classification** suite [2110.14795]. The v2 paper explicitly framed the collection as a lightweight benchmark intended to isolate the machine-learning component of biomedical classification, in contrast to benchmarks where performance depends strongly on complex preprocessing, postprocessing, augmentation, or challenge-specific engineering [2110.14795]. This design choice made MedMNIST particularly suitable for comparative studies of model families, transfer-learning schemes, AutoML systems, robustness methods, and representation-learning pipelines across heterogeneous medical domains [2110.14795][2501.14685].

## 2. Dataset collection and task structure

MedMNIST v2 contains 18 datasets: 12 in 2D and 6 in 3D [2110.14795]. The 2D collection spans pathology, chest X-ray, dermatoscopy, retinal OCT, fundus photography, breast ultrasound, blood-cell microscopy, kidney cortex microscopy, and abdominal CT; the 3D collection spans abdominal CT, chest CT, adrenal-shape masks, vessel shapes derived from brain MRA, fracture-centered CT crops, and electron microscopy [2110.14795].

| Subset | Format and modality | Task |
|---|---|---|
| PathMNIST | 2D colon pathology | Multi-class (9) |
| ChestMNIST | 2D chest X-ray | Multi-label binary (14 labels) |
| DermaMNIST | 2D dermatoscope | Multi-class (7) |
| OCTMNIST | 2D retinal OCT | Multi-class (4) |
| PneumoniaMNIST | 2D chest X-ray | Binary (2) |
| RetinaMNIST | 2D fundus camera | Ordinal regression (5 levels) |
| BreastMNIST | 2D breast ultrasound | Binary (2) |
| BloodMNIST | 2D blood cell microscope | Multi-class (8) |
| TissueMNIST | 2D kidney cortex microscope | Multi-class (8) |
| OrganAMNIST | 2D abdominal CT | Multi-class (11) |
| OrganCMNIST | 2D abdominal CT | Multi-class (11) |
| OrganSMNIST | 2D abdominal CT | Multi-class (11) |
| OrganMNIST3D | 3D abdominal CT | Multi-class (11) |
| NoduleMNIST3D | 3D chest CT | Binary (2) |
| AdrenalMNIST3D | 3D adrenal shape from abdominal CT | Binary (2) |
| FractureMNIST3D | 3D chest CT | Multi-class (3) |
| VesselMNIST3D | 3D vessel shape from brain MRA | Binary (2) |
| SynapseMNIST3D | 3D electron microscope | Binary (2) |

The benchmark was explicitly designed to span major axes of biomedical-image variation: modality, task type, and scale [2010.14925][2110.14795]. Sample sizes range from around \(10^2\) to \(10^5\), and the suite includes both small datasets such as BreastMNIST and RetinaMNIST and large datasets such as TissueMNIST, PathMNIST, OCTMNIST, and ChestMNIST [2010.14925][2110.14795]. This heterogeneity is one reason MedMNIST became attractive for comparative studies that aim to distinguish method-specific behavior from dataset-specific effects [2501.14685][2404.15786].

## 3. Standardization, preprocessing, and benchmark protocol

A defining feature of MedMNIST is aggressive standardization. In MedMNIST v2, all 2D images are preprocessed to \(28\times28\), all 3D volumes to \(28\times28\times28\), and the datasets are distributed with official train/validation/test splits in NumPy `.npz` files [2110.14795]. When official source splits exist, they are retained; otherwise, the benchmark uses either a \(9{:}1\) split of source training data when a source validation set is available as test, or a patient-level \(7{:}1{:}2\) split when no official split exists [2110.14795]. Labels are packaged as shape \(N\times1\) for binary, multi-class, and ordinal tasks, and shape \(N\times L\) for multi-label tasks such as ChestMNIST [2110.14795].

This preprocessing confers two important properties. First, experiments are cheap and reproducible: the small inputs make it easy to benchmark conventional CNNs, ViTs, AutoML systems, and newer representation-learning pipelines on a single GPU or modest compute budget [2010.14925][2110.14795]. Second, the benchmark is intentionally simplified. The v1 and v2 papers both stress that MedMNIST is a benchmarking and educational resource rather than a clinically deployable dataset, because aggressive resizing can remove disease-relevant detail [2010.14925][2110.14795].

The benchmark protocol in the original and v2 papers uses **AUC** and **ACC** as the core evaluation metrics [2010.14925][2110.14795]. The rationale is that AUC is threshold-free and less sensitive to class imbalance, while ACC remains easy to interpret [2010.14925]. MedMNIST v2 further provided strong standardized baselines: 2D ResNet-18 and ResNet-50 at \(28\) and \(224\) input settings, 3D ResNet variants using 2.5D, 3D, and ACS convolutions, and several AutoML systems including auto-sklearn, AutoKeras, and Google AutoML Vision for 2D data [2110.14795].

## 4. Evolution of the ecosystem: v2, MedMNIST+, and MedMNIST-C

The original MedMNIST design made low-resolution benchmarking easy, but later work argued that the fixed \(28\times28\) setting is too restrictive for modern pretrained architectures and for systematic analysis of resolution effects [2404.15786]. MedMNIST+ addressed this limitation by keeping the same underlying datasets and official splits while extending the image resolutions to **\(28\times28\), \(64\times64\), \(128\times128\), and \(224\times224\)** [2404.15786]. The MedMNIST+ benchmark re-evaluated common CNNs and ViT architectures across resolutions, training methodologies, and dataset regimes, and concluded that higher resolutions do not consistently improve performance beyond a certain threshold, that computationally efficient training schemes can be viable alternatives to costly end-to-end training, and that CNNs remain competitive with ViTs in this ecosystem [2404.15786].

MedMNIST-C extended the collection in a different direction: robustness evaluation under realistic corruptions [2406.17536]. Built on MedMNIST+ at **\(224\times224\)** resolution, it provides corrupted test sets spanning **12 datasets**, **9 imaging modalities**, **5 corruption categories**, and **5 severity levels per corruption** [2406.17536]. Its corruption design is explicitly modality-aware rather than globally shared, with categories **Digital**, **Noise**, **Blur**, **Color**, and **Task-specific** [2406.17536]. To account for class imbalance, MedMNIST-C introduced normalized **balanced error** and **relative balanced error**:
\[
BE_c^f = \frac{\sum_{s=1}^5 BE_{s,c}^f}{\sum_{s=1}^5 BE_{s,c}^{AlexNet}}
\]
and
\[
rBE_c^f = \frac{\sum_{s=1}^5 (BE_{s,c}^f - BE_{clean}^f)}{\sum_{s=1}^5 (BE_{s,c}^{AlexNet} - BE_{clean}^{AlexNet})}.
\]
These metrics turn MedMNIST from a clean-image benchmark into a broader ecosystem for corruption robustness, corruption-aware augmentation, and domain-generalization studies [2406.17536].

## 5. MedMNIST as a research testbed

Because MedMNIST standardizes diverse biomedical tasks, it has become a common platform for research on transfer learning, architecture selection, robustness, representation learning, and OOD detection. A large benchmark study of foundation models on all 12 2D datasets found that **end-to-end fine-tuning consistently outperforms linear probing across all datasets**, except for a single small PathMNIST AUC exception, and that ViT-based models generally outperform CNNs in that study, with DINO, DINOv2, and ViT-B/16 emerging as particularly strong choices [2501.14685]. By contrast, the MedMNIST+ benchmark emphasized that CNNs remain competitive, especially when resolution, training scheme, and computational budget are taken into account [2404.15786]. Taken together, these results suggest that MedMNIST is especially useful for testing claims about architectural superiority across many medical domains rather than on a single narrow dataset.

Robustness studies have used MedMNIST to challenge the adequacy of clean-image evaluation. MedMNIST-C showed that established architectures are not robust under clinically motivated corruptions; across the 12-dataset benchmark, **ViT-B/16** was the most robust overall, and corruption-informed targeted augmentation achieved an average corrupted-test **AUC gain of 9.2**, compared with **3.9** for RandAugment, **2.7** for MixUp, and **1.0** for CutMix [2406.17536]. A probabilistic-robustness study on **PathMNIST** further found that high clean performance does not imply trustworthy behavior under realistic corruptions: clean ACC was **90.68%** for ResNet-18 and **92.05%** for ResNet-50, but under **Motion** corruption the reported probabilistic robustness dropped to **40.20%** and **40.74%**, respectively [2607.03797].

The benchmark has also supported studies of representation structure and feature reliance. A backpropagation-free, frozen-foundation-model pipeline called A-ROM was evaluated on **11 of the 12 2D MedMNIST datasets** and reported **83.7% average accuracy** and **0.940 average AUC**, while using a human-readable concept dictionary and a Mahalanobis-distance \(k\)-NN classifier in place of fine-tuned decision layers [2604.06017]. A separate study on five MedMNIST tasks concluded that **nonrobust-only models achieve well-above-chance balanced accuracy**, while robust models sacrifice in-distribution accuracy but yield markedly better performance on MedMNIST-C, thereby exposing a robustness–accuracy trade-off in biomedical image classification [2604.22579]. For post-hoc OOD detection, feature-space density modeling with a normalizing flow on a frozen OrganAMNIST ResNet18 achieved **AUROC 93.8** on a MedMNIST benchmark of near- and far-OOD datasets [2502.11638].

## 6. Limitations, criticisms, and disputed issues

The principal limitation of MedMNIST is also its defining simplification: the benchmark reduces medical images to very small fixed sizes [2010.14925][2110.14795]. This enables fast experimentation, but it also removes fine-grained medical detail and can force task simplifications. The v1 paper explicitly merged normal and benign ultrasound findings into a single negative class in BreastMNIST because the original 3-class distinction was judged unsuitable for the low-resolution setting [2010.14925]. The v2 paper similarly warns that MedMNIST is **not intended for clinical use** because the reduction to \(28\times28\) or \(28\times28\times28\) may be insufficient to capture pathology [2110.14795]. The creation of MedMNIST+ can be read as a direct response to this concern, since it preserves the same benchmark logic while allowing higher resolutions [2404.15786].

Critiques have also arisen at the subset level. A DermaMNIST case study argued that standard DermaMNIST is inadequate for reliable skin-lesion classification and attributed this to **very low resolution (28×28)**, **class imbalance**, and **poor organization**; that study reported much better results on an external high-resolution derivative called **DermaMNIST-C**, but also noted that this derivative is not the same benchmark as official MedMNIST DermaMNIST [2507.12961]. More broadly, robustness studies indicate that clean MedMNIST performance can overstate trustworthiness, since models that score well on standard test splits may fail under corruption, distribution shift, or OOD conditions [2406.17536][2607.03797][2502.11638].

A more disputed line of criticism concerns label consistency. One paper on “error-free training” claimed that **ChestMNIST**, **OCTMNIST**, and **TissueMNIST** contain a “double-labeling problem,” defined as duplicated inputs with conflicting labels, and argued that this prevents perfect training-set fitting unless the conflicting examples are removed [2604.18916]. This should be understood as a reported claim rather than a settled property of the benchmark. Even if accepted at face value, it concerns deterministic memorization on training data rather than the standard validation/test protocol used in MedMNIST benchmarking [2604.18916].

In practice, MedMNIST is best interpreted as a compact and highly influential benchmark ecosystem rather than as a proxy for clinical deployment. Its importance lies in providing a common substrate on which clean accuracy, transferability, robustness, OOD detection, representation quality, and computational efficiency can all be studied across many biomedical domains under a uniform interface [2010.14925][2110.14795][2406.17536].

Source: https://www.emergentmind.com/topics/medmnist