MedMNIST v2: A Benchmark for Biomedical Image Classification

• MedMNIST v2 is a benchmark for biomedical image classification that standardizes 2D and 3D datasets into MNIST-like tensors, enabling diverse task evaluations.
• It simplifies model testing by fixing image sizes, splits, and preprocessing, allowing researchers to focus on architecture design and cross-task robustness.
• The benchmark supports varying tasks such as binary, multi-class, ordinal regression and multi-label prediction, promoting rapid prototyping and comparative studies.

MedMNIST v2 is a large-scale lightweight benchmark for biomedical image classification that standardizes 12 two-dimensional datasets and 6 three-dimensional datasets into MNIST-like tensors of $28\times 28$ or $28\times 28\times 28$ with corresponding classification labels, spanning binary classification, multi-class classification, ordinal regression, and multi-label prediction across major biomedical imaging modalities (Yang et al., 2021). Its organizing premise is deliberate simplification: fixed small inputs, predefined splits, and uniform packaging isolate model design and evaluation from dataset-specific preprocessing, licensing, and clinical-format engineering, making the benchmark suitable for rapid prototyping, AutoML, and education as well as comparative methodology research (Yang et al., 2020).

1. Design philosophy and benchmark scope

MedMNIST v2 extends the original MedMNIST “classification decathlon” from 10 pre-processed datasets to 12 2D datasets and 6 3D datasets while preserving the same core abstraction: standardized biomedical classification tasks at very small spatial resolution with fixed train/validation/test splits and publicly distributed data files (Yang et al., 2020, Yang et al., 2021). The benchmark was explicitly framed as a large-scale lightweight collection that covers primary biomedical imaging modalities, diverse task types, and dataset scales ranging from small to very large, while requiring no background knowledge from users beyond ordinary image classification workflows (Yang et al., 2021).

This decathlon framing is methodologically important. Rather than treating any single subset as definitive, MedMNIST v2 was designed to expose whether an architecture, optimizer, or AutoML system can generalize across radiography, CT, ultrasound, OCT, histopathology, dermatoscopy, fundus photography, microscopy, and electron microscopy under a common interface (Yang et al., 2020). Later work continued to use that framing to study cross-task robustness, label uncertainty, low-resolution learning, and architectural transfer, including both 2D and 3D settings (Zheng et al., 2023).

2. Dataset collection and preprocessing contract

In MedMNIST v2, 2D images are standardized to $28\times 28$ and 3D volumes to $28\times 28\times 28$, with data distributed in NumPy .npz files containing train_images, val_images, test_images, and the corresponding label arrays (Yang et al., 2021). The task abbreviations below follow the benchmark convention: BC for binary classification, MC for multi-class classification, ML for multi-label classification, and OR for ordinal regression. Split figures are reported as train / val / test.

2D subset	Modality	Task and split
PathMNIST	Colon histology	MC(9); 89,996 / 10,004 / 7,180
ChestMNIST	Chest X-ray	ML(14), BC(2); 78,468 / 11,219 / 22,433
DermaMNIST	Dermatoscopy	MC(7); 7,007 / 1,003 / 2,005
OCTMNIST	Retinal OCT	MC(4); 97,477 / 10,832 / 1,000
PneumoniaMNIST	Chest X-ray	BC(2); 4,708 / 524 / 624
RetinaMNIST	Fundus photography	OR(5); 1,080 / 120 / 400
BreastMNIST	Breast ultrasound	BC(2); 546 / 78 / 156
BloodMNIST	Blood cell microscopy	MC(8); 11,959 / 1,712 / 3,421
TissueMNIST	Fluorescence microscopy	MC(8); 165,466 / 23,640 / 47,280
OrganAMNIST	Abdominal CT, axial	MC(11); 34,581 / 6,491 / 17,778
OrganCMNIST	Abdominal CT, coronal	MC(11); 13,000 / 2,392 / 8,268
OrganSMNIST	Abdominal CT, sagittal	MC(11); 13,940 / 2,452 / 8,829

3D subset	Modality	Task and split
OrganMNIST3D	Abdominal CT	MC(11); 972 / 161 / 610
NoduleMNIST3D	Thoracic CT	BC(2); 1,158 / 165 / 310
AdrenalMNIST3D	Adrenal-gland 3D shapes	BC(2); 1,188 / 98 / 298
FractureMNIST3D	Rib fracture CT patches	MC(3); 1,027 / 103 / 240
VesselMNIST3D	Brain MRA vessel shapes	BC(2); 1,335 / 192 / 382
SynapseMNIST3D	Electron microscopy volumes	BC(2); 1,230 / 177 / 352

Standardization is aggressive and modality-specific. The original benchmark uses cubic spline interpolation for resizing; center cropping is applied in several 2D datasets; CT subsets use Hounsfield-unit windowing with an abdominal window before grayscale conversion; TissueMNIST reduces $32\times 32\times 7$ microscopy stacks to 2D via maximum-intensity projection; VesselMNIST3D voxelizes mesh representations; and several 3D subsets crop fixed physical regions of interest before resizing (Yang et al., 2021). This contract removes most dataset-specific engineering, but it also creates an intrinsic information bottleneck. Later multi-resolution work identified the strict $28\times 28$ regime as a key limitation because it can obscure texture and local detail that modern architectures and clinically meaningful tasks rely on (Doerrich et al., 2024).

3. Tasks, metrics, and original baseline evaluation

MedMNIST v2 evaluates models with two metrics: AUC and ACC (Yang et al., 2021). The original MedMNIST work emphasized that AUC is threshold-free and less sensitive to class imbalance than ACC, while ACC remains the most direct discrete-decision summary (Yang et al., 2020). The benchmark therefore reports both per dataset and averages them across the 12 2D or 6 3D subsets to obtain decathlon-style summaries.

The initial baseline suite combined hand-designed CNNs and AutoML systems. For MedMNIST2D, the paper benchmarked ResNet-18 and ResNet-50 at both $28\times 28$ and $224\times 224$, alongside auto-sklearn, AutoKeras, and Google AutoML Vision; for MedMNIST3D, it benchmarked ResNet-18 and ResNet-50 with 2.5D, 3D, and ACS convolutions, plus auto-sklearn and AutoKeras (Yang et al., 2021). The aggregate results established several reference points. On MedMNIST2D, Google AutoML Vision achieved the highest average AUC, $0.927$, whereas ResNet-18 and ResNet-50 at $224\times 224$ each reached the highest average ACC, $28\times 28\times 28$0 (Yang et al., 2021). On MedMNIST3D, ResNet-50 with full 3D convolutions achieved the strongest average performance, with AUC $28\times 28\times 28$1 and ACC $28\times 28\times 28$2 (Yang et al., 2021).

These baseline results also exposed heterogeneity across tasks. Histopathology and organ subsets are relatively easy for standard deep models; RetinaMNIST and several 3D subsets are materially harder; ChestMNIST is structurally different because it is multi-label rather than single-label (Yang et al., 2021). A notable auxiliary analysis showed that, for organ classification, carefully chosen 2D central slices could be competitive with full 3D input, indicating that MedMNIST’s design sometimes compresses volumetric structure into surprisingly strong 2D surrogates (Yang et al., 2021).

4. Multi-resolution and robustness extensions

The most consequential extension of MedMNIST v2 is MedMNIST+, which retains the same 12 2D datasets as MedMNIST v2 and adds $28\times 28\times 28$3, $28\times 28\times 28$4, and $28\times 28\times 28$5 versions on top of the original $28\times 28\times 28$6, with official fixed train/val/test splits aligned one-to-one with MedMNIST v2 (Doerrich et al., 2024). In that sense, MedMNIST+ is a multi-resolution superset of the 2D portion of MedMNIST v2: using only the $28\times 28\times 28$7 images and official splits corresponds to the original regime, while any larger resolution enters the extended benchmark (Doerrich et al., 2024).

The robustness counterpart is MedMNIST-C, a corruption benchmark built on top of MedMNIST+ test sets and covering 12 datasets and 9 imaging modalities (Salvo et al., 2024). It defines five corruption categories—digital, noise, blur, color/intensity, and task-specific—each with five severity levels, and pairs the pre-generated corrupted test sets with a corruption API that can be reused as a training-time augmentation library (Salvo et al., 2024). The benchmark showed that robustness rankings on medical images do not simply mirror ImageNet-C: averaged across 12 datasets, ViT-B/16 was the most robust of the tested models, with $28\times 28\times 28$8 and $28\times 28\times 28$9, while targeted domain-aware augmentation improved corrupted-test AUC by $28\times 28$0 points over a no-augmentation ResNet18 baseline, compared with $28\times 28$1 for RandAugment (Salvo et al., 2024).

Together, MedMNIST+, MedMNIST-C, and derivative higher-resolution subsets such as DermaMNIST-C shift MedMNIST from a pure low-resolution clean-data benchmark toward a broader evaluation ecosystem: resolution scaling, transfer-learning regimes, and robustness to acquisition artifacts become first-class experimental variables rather than external concerns.

5. Architectural findings and benchmark-sensitive conclusions

Later benchmark studies reached non-identical conclusions about which architectures are strongest on MedMNIST, and the disagreement is itself instructive. A large MedMNIST+ study in 2024 compared ImageNet-pretrained CNNs and ViT variants under end-to-end fine-tuning, linear probing, and k-NN on frozen embeddings. It found that CNNs were “at least as strong, often stronger than ViTs” in end-to-end training, with DenseNet-121 and VGG16 frequently appearing among the top-5 models, while DINO ViT-B/16 was especially strong in linear probing and k-NN regimes. The same study reported that the main resolution gains occur from $28\times 28$2 and $28\times 28$3, whereas $28\times 28$4 gives only marginal improvement, sometimes none or slight regression; end-to-end fine-tuning had median ACC $28\times 28$5, compared with $28\times 28$6 for linear probing and $28\times 28$7 for k-NN (Doerrich et al., 2024).

A 2025 foundation-model benchmark on MedMNIST v2 reached a different aggregate conclusion. Using 12 pretrained CNN and ViT backbones, it reported that end-to-end fine-tuning “almost always outperforms linear probing” and that ViT-based models generally outperform CNNs when properly fine-tuned, especially on more challenging datasets such as DermaMNIST, OCTMNIST, OrganCMNIST, OrganSMNIST, and TissueMNIST. It also argued that optimization details were decisive: encoder learning rates around $28\times 28$8 worked best for CNNs, $28\times 28$9 for ViTs, and interpolative scaling to $28\times 28\times 28$0 outperformed zero-padding at all examined native resolutions on DermaMNIST (Wu et al., 24 Jan 2025).

A plausible implication is that MedMNIST model rankings are highly sensitive to protocol choices, especially input handling, encoder learning-rate schedules, and whether comparison is made in low-resolution native space or in a higher-resolution transfer-learning setup. MedMNIST therefore functions not only as a scoreboard but also as a controlled environment for studying benchmark dependence itself.

6. Criticisms, derivative datasets, and continuing research uses

Criticism of MedMNIST v2 has focused most sharply on resolution and data curation. A DermaMNIST case study described the original $28\times 28\times 28$1 subset as suffering from “poor organization and class imbalance” and used a higher-resolution, better curated derivative, DermaMNIST-C, at $28\times 28\times 28$2. In that study, EfficientNetV2L on original DermaMNIST reached ACC $28\times 28\times 28$3, whereas the best EfficientNetV2L configuration on DermaMNIST-C reached ACC $28\times 28\times 28$4, with higher recall and more balanced confusion matrices (Kadric et al., 17 Jul 2025). That finding is consistent with broader MedMNIST+ results: for visually fine-grained tasks, the $28\times 28\times 28$5 abstraction can be too severe for serious diagnostic modeling.

At the same time, MedMNIST v2 has become a general-purpose methodological testbed far beyond ordinary supervised CNN benchmarking. PathMNIST was used as the biomedical benchmark in a hybrid quantum–classical diffusion model study, where the quantum model improved FID from $28\times 28\times 28$6 to $28\times 28\times 28$7 but degraded SSIM from $28\times 28\times 28$8 to $28\times 28\times 28$9, indicating better global distributional similarity but poorer structural fidelity on color pathology images (Chen et al., 30 Mar 2025). A separate study benchmarked eight MedMNIST v2 2D datasets on real IBM quantum hardware and showed that, on OCTMNIST, adding dynamical decoupling, gate twirling, and matrix-free measurement mitigation improved hardware ACC from $32\times 32\times 7$0 to $32\times 32\times 7$1 and AUC from $32\times 32\times 7$2 to $32\times 32\times 7$3 (Singh et al., 18 Feb 2025). Interpretability-oriented work also used MedMNIST v2 as a proving ground: A-ROM, a backpropagation-free pipeline built on frozen DINOv2 features, PCA, K-means, LDA, and k-NN, reported the highest average accuracy, $32\times 32\times 7$4, and an average AUC of $32\times 32\times 7$5 across 11 of the 12 2D datasets, excluding ChestMNIST because of its multi-label structure (Karnes et al., 7 Apr 2026).

Other later studies further broadened the benchmark’s role. A Fisher-vector-encoded hybrid CNN+ViT model reported improvements over original MedMNIST v2 benchmark results on all 18 datasets (Lyra et al., 3 May 2026). An “error-free training” study claimed that, among the 18 MedMNIST biomedical datasets, all but three could be trained to $32\times 32\times 7$6 accuracy as provided, while ChestMNIST, OCTMNIST, and TissueMNIST were limited by a “double-labeling problem” until conflicting duplicates were removed (Deng, 20 Apr 2026). These uses underscore MedMNIST v2’s peculiar combination of strengths and limits: it is lightweight, standardized, and broad enough to support very different research programs, but it remains a simplified abstraction rather than a clinical deployment benchmark. The original benchmark paper already emphasized the central caveats—small image size, strong preprocessing, task simplification, limited task types, and the fact that good performance on MedMNIST does not imply clinical readiness (Yang et al., 2021).