---
title: 'MedMNIST v2: A Benchmark for Biomedical Image Classification'
url: https://www.emergentmind.com/topics/medmnist-v2
type: topic
---

# MedMNIST v2: A Benchmark for Biomedical Image Classification

MedMNIST v2 is a large-scale lightweight benchmark for biomedical image classification that standardizes 12 two-dimensional datasets and 6 three-dimensional datasets into MNIST-like tensors of \(28\times 28\) or \(28\times 28\times 28\) with corresponding classification labels, spanning binary classification, multi-class classification, ordinal regression, and multi-label prediction across major biomedical imaging modalities [2110.14795]. Its organizing premise is deliberate simplification: fixed small inputs, predefined splits, and uniform packaging isolate model design and evaluation from dataset-specific preprocessing, licensing, and clinical-format engineering, making the benchmark suitable for rapid prototyping, AutoML, and education as well as comparative methodology research [2010.14925].

## 1. Design philosophy and benchmark scope

MedMNIST v2 extends the original MedMNIST “classification decathlon” from 10 pre-processed datasets to 12 2D datasets and 6 3D datasets while preserving the same core abstraction: standardized biomedical classification tasks at very small spatial resolution with fixed train/validation/test splits and publicly distributed data files [2010.14925][2110.14795]. The benchmark was explicitly framed as a large-scale lightweight collection that covers primary biomedical imaging modalities, diverse task types, and dataset scales ranging from small to very large, while requiring no background knowledge from users beyond ordinary image classification workflows [2110.14795].

This decathlon framing is methodologically important. Rather than treating any single subset as definitive, MedMNIST v2 was designed to expose whether an architecture, optimizer, or AutoML system can generalize across radiography, CT, ultrasound, OCT, histopathology, dermatoscopy, fundus photography, microscopy, and electron microscopy under a common interface [2010.14925]. Later work continued to use that framing to study cross-task robustness, label uncertainty, low-resolution learning, and architectural transfer, including both 2D and 3D settings [2304.10054].

## 2. Dataset collection and preprocessing contract

In MedMNIST v2, 2D images are standardized to \(28\times 28\) and 3D volumes to \(28\times 28\times 28\), with data distributed in NumPy `.npz` files containing `train_images`, `val_images`, `test_images`, and the corresponding label arrays [2110.14795]. The task abbreviations below follow the benchmark convention: **BC** for binary classification, **MC** for multi-class classification, **ML** for multi-label classification, and **OR** for ordinal regression. Split figures are reported as train / val / test.

| 2D subset | Modality | Task and split |
|---|---|---|
| PathMNIST | Colon histology | MC(9); 89,996 / 10,004 / 7,180 |
| ChestMNIST | Chest X-ray | ML(14), BC(2); 78,468 / 11,219 / 22,433 |
| DermaMNIST | Dermatoscopy | MC(7); 7,007 / 1,003 / 2,005 |
| OCTMNIST | Retinal OCT | MC(4); 97,477 / 10,832 / 1,000 |
| PneumoniaMNIST | Chest X-ray | BC(2); 4,708 / 524 / 624 |
| RetinaMNIST | Fundus photography | OR(5); 1,080 / 120 / 400 |
| BreastMNIST | Breast ultrasound | BC(2); 546 / 78 / 156 |
| BloodMNIST | Blood cell microscopy | MC(8); 11,959 / 1,712 / 3,421 |
| TissueMNIST | Fluorescence microscopy | MC(8); 165,466 / 23,640 / 47,280 |
| OrganAMNIST | Abdominal CT, axial | MC(11); 34,581 / 6,491 / 17,778 |
| OrganCMNIST | Abdominal CT, coronal | MC(11); 13,000 / 2,392 / 8,268 |
| OrganSMNIST | Abdominal CT, sagittal | MC(11); 13,940 / 2,452 / 8,829 |

| 3D subset | Modality | Task and split |
|---|---|---|
| OrganMNIST3D | Abdominal CT | MC(11); 972 / 161 / 610 |
| NoduleMNIST3D | Thoracic CT | BC(2); 1,158 / 165 / 310 |
| AdrenalMNIST3D | Adrenal-gland 3D shapes | BC(2); 1,188 / 98 / 298 |
| FractureMNIST3D | Rib fracture CT patches | MC(3); 1,027 / 103 / 240 |
| VesselMNIST3D | Brain MRA vessel shapes | BC(2); 1,335 / 192 / 382 |
| SynapseMNIST3D | Electron microscopy volumes | BC(2); 1,230 / 177 / 352 |

Standardization is aggressive and modality-specific. The original benchmark uses cubic spline interpolation for resizing; center cropping is applied in several 2D datasets; CT subsets use Hounsfield-unit windowing with an abdominal window before grayscale conversion; TissueMNIST reduces \(32\times 32\times 7\) microscopy stacks to 2D via maximum-intensity projection; VesselMNIST3D voxelizes mesh representations; and several 3D subsets crop fixed physical regions of interest before resizing [2110.14795]. This contract removes most dataset-specific engineering, but it also creates an intrinsic information bottleneck. Later multi-resolution work identified the strict \(28\times 28\) regime as a key limitation because it can obscure texture and local detail that modern architectures and clinically meaningful tasks rely on [2404.15786].

## 3. Tasks, metrics, and original baseline evaluation

MedMNIST v2 evaluates models with two metrics: **AUC** and **ACC** [2110.14795]. The original MedMNIST work emphasized that AUC is threshold-free and less sensitive to class imbalance than ACC, while ACC remains the most direct discrete-decision summary [2010.14925]. The benchmark therefore reports both per dataset and averages them across the 12 2D or 6 3D subsets to obtain decathlon-style summaries.

The initial baseline suite combined hand-designed CNNs and AutoML systems. For MedMNIST2D, the paper benchmarked ResNet-18 and ResNet-50 at both \(28\times 28\) and \(224\times 224\), alongside auto-sklearn, AutoKeras, and Google AutoML Vision; for MedMNIST3D, it benchmarked ResNet-18 and ResNet-50 with 2.5D, 3D, and ACS convolutions, plus auto-sklearn and AutoKeras [2110.14795]. The aggregate results established several reference points. On MedMNIST2D, Google AutoML Vision achieved the highest average AUC, \(0.927\), whereas ResNet-18 and ResNet-50 at \(224\times 224\) each reached the highest average ACC, \(0.821\) [2110.14795]. On MedMNIST3D, ResNet-50 with full 3D convolutions achieved the strongest average performance, with AUC \(0.863\) and ACC \(0.780\) [2110.14795].

These baseline results also exposed heterogeneity across tasks. Histopathology and organ subsets are relatively easy for standard deep models; RetinaMNIST and several 3D subsets are materially harder; ChestMNIST is structurally different because it is multi-label rather than single-label [2110.14795]. A notable auxiliary analysis showed that, for organ classification, carefully chosen 2D central slices could be competitive with full 3D input, indicating that MedMNIST’s design sometimes compresses volumetric structure into surprisingly strong 2D surrogates [2110.14795].

## 4. Multi-resolution and robustness extensions

The most consequential extension of MedMNIST v2 is **MedMNIST+**, which retains the same 12 2D datasets as MedMNIST v2 and adds \(64\times 64\), \(128\times 128\), and \(224\times 224\) versions on top of the original \(28\times 28\), with official fixed train/val/test splits aligned one-to-one with MedMNIST v2 [2404.15786]. In that sense, MedMNIST+ is a multi-resolution superset of the 2D portion of MedMNIST v2: using only the \(28\times 28\) images and official splits corresponds to the original regime, while any larger resolution enters the extended benchmark [2404.15786].

The robustness counterpart is **MedMNIST-C**, a corruption benchmark built on top of MedMNIST+ test sets and covering 12 datasets and 9 imaging modalities [2406.17536]. It defines five corruption categories—digital, noise, blur, color/intensity, and task-specific—each with five severity levels, and pairs the pre-generated corrupted test sets with a corruption API that can be reused as a training-time augmentation library [2406.17536]. The benchmark showed that robustness rankings on medical images do not simply mirror ImageNet-C: averaged across 12 datasets, ViT-B/16 was the most robust of the tested models, with \(BE=76.3\) and \(rBE=59.9\), while targeted domain-aware augmentation improved corrupted-test AUC by \(+9.2\) points over a no-augmentation ResNet18 baseline, compared with \(+3.9\) for RandAugment [2406.17536].

Together, MedMNIST+, MedMNIST-C, and derivative higher-resolution subsets such as DermaMNIST-C shift MedMNIST from a pure low-resolution clean-data benchmark toward a broader evaluation ecosystem: resolution scaling, transfer-learning regimes, and robustness to acquisition artifacts become first-class experimental variables rather than external concerns.

## 5. Architectural findings and benchmark-sensitive conclusions

Later benchmark studies reached non-identical conclusions about which architectures are strongest on MedMNIST, and the disagreement is itself instructive. A large MedMNIST+ study in 2024 compared ImageNet-pretrained CNNs and ViT variants under end-to-end fine-tuning, linear probing, and k-NN on frozen embeddings. It found that CNNs were “at least as strong, often stronger than ViTs” in end-to-end training, with DenseNet-121 and VGG16 frequently appearing among the top-5 models, while DINO ViT-B/16 was especially strong in linear probing and k-NN regimes. The same study reported that the main resolution gains occur from \(28\to 64\) and \(64\to 128\), whereas \(128\to 224\) gives only marginal improvement, sometimes none or slight regression; end-to-end fine-tuning had median ACC \(82.77\%\), compared with \(76.65\%\) for linear probing and \(\sim72.1\%\) for k-NN [2404.15786].

A 2025 foundation-model benchmark on MedMNIST v2 reached a different aggregate conclusion. Using 12 pretrained CNN and ViT backbones, it reported that end-to-end fine-tuning “almost always outperforms linear probing” and that ViT-based models generally outperform CNNs when properly fine-tuned, especially on more challenging datasets such as DermaMNIST, OCTMNIST, OrganCMNIST, OrganSMNIST, and TissueMNIST. It also argued that optimization details were decisive: encoder learning rates around \(10^{-4}\) worked best for CNNs, \(10^{-5}\) for ViTs, and interpolative scaling to \(224\times 224\) outperformed zero-padding at all examined native resolutions on DermaMNIST [2501.14685].

A plausible implication is that MedMNIST model rankings are highly sensitive to protocol choices, especially input handling, encoder learning-rate schedules, and whether comparison is made in low-resolution native space or in a higher-resolution transfer-learning setup. MedMNIST therefore functions not only as a scoreboard but also as a controlled environment for studying benchmark dependence itself.

## 6. Criticisms, derivative datasets, and continuing research uses

Criticism of MedMNIST v2 has focused most sharply on resolution and data curation. A DermaMNIST case study described the original \(28\times 28\) subset as suffering from “poor organization and class imbalance” and used a higher-resolution, better curated derivative, **DermaMNIST-C**, at \(224\times 224\). In that study, EfficientNetV2L on original DermaMNIST reached ACC \(0.7017\), whereas the best EfficientNetV2L configuration on DermaMNIST-C reached ACC \(0.8490\), with higher recall and more balanced confusion matrices [2507.12961]. That finding is consistent with broader MedMNIST+ results: for visually fine-grained tasks, the \(28\times 28\) abstraction can be too severe for serious diagnostic modeling.

At the same time, MedMNIST v2 has become a general-purpose methodological testbed far beyond ordinary supervised CNN benchmarking. PathMNIST was used as the biomedical benchmark in a hybrid quantum–classical diffusion model study, where the quantum model improved FID from \(95.72\) to \(84.40\) but degraded SSIM from \(0.4107\) to \(0.0931\), indicating better global distributional similarity but poorer structural fidelity on color pathology images [2504.00034]. A separate study benchmarked eight MedMNIST v2 2D datasets on real IBM quantum hardware and showed that, on OCTMNIST, adding dynamical decoupling, gate twirling, and matrix-free measurement mitigation improved hardware ACC from \(0.286\) to \(0.397\) and AUC from \(0.545\) to \(0.622\) [2502.13056]. Interpretability-oriented work also used MedMNIST v2 as a proving ground: A-ROM, a backpropagation-free pipeline built on frozen DINOv2 features, PCA, K-means, LDA, and k-NN, reported the highest average accuracy, \(83.7\%\), and an average AUC of \(0.940\) across 11 of the 12 2D datasets, excluding ChestMNIST because of its multi-label structure [2604.06017].

Other later studies further broadened the benchmark’s role. A Fisher-vector-encoded hybrid CNN+ViT model reported improvements over original MedMNIST v2 benchmark results on all 18 datasets [2605.01667]. An “error-free training” study claimed that, among the 18 MedMNIST biomedical datasets, all but three could be trained to \(100\%\) accuracy as provided, while ChestMNIST, OCTMNIST, and TissueMNIST were limited by a “double-labeling problem” until conflicting duplicates were removed [2604.18916]. These uses underscore MedMNIST v2’s peculiar combination of strengths and limits: it is lightweight, standardized, and broad enough to support very different research programs, but it remains a simplified abstraction rather than a clinical deployment benchmark. The original benchmark paper already emphasized the central caveats—small image size, strong preprocessing, task simplification, limited task types, and the fact that good performance on MedMNIST does not imply clinical readiness [2110.14795].

Source: https://www.emergentmind.com/topics/medmnist-v2