Ranked MNISTs: A Benchmarking Ecosystem

• The paper introduces Ranked MNISTs as a multifaceted benchmarking framework that ranks MNIST and MNIST-style datasets based on test errors, linear separability, and cross-domain performance.
• It details diverse methodologies including classical leaderboards, pairwise digit separability, and dataset generation techniques, highlighting metrics like accuracy, AUC, and test error percentages.
• The work informs model selection and training strategies by comparing architectural patterns and evaluation protocols, while encouraging reproducible, layered benchmarks across different domains.

Ranked MNISTs denotes a family of benchmarking practices in which MNIST and MNIST-style datasets are ordered by measured difficulty or model performance. In this literature, ranking may refer to the classical MNIST leaderboard by test error, the ordering of digit pairs by linear separability, the comparison of heterogeneous architectures on MNIST-like variants such as Overhead-MNIST and MNIST-1D, or the generation of new MNIST-style datasets whose complexity can itself be compared. The resulting picture is not a single leaderboard but a benchmark ecosystem spanning handwritten digits, overhead imagery, one-dimensional digit signals, and domain-specific synthetic MNIST analogues (0710.2231, Hajnal, 13 Mar 2026, Larsen et al., 2021, Beebe et al., 12 Feb 2026, Shaeri et al., 16 Jul 2025).

1. Benchmark concept and measurement regimes

In the original MNIST setting, ranking is usually defined by test error on the standard split of 60,000 training images and 10,000 test images, with performance reported as the percentage of misclassified digits. This convention underlies the historical record tables in the handwritten-digit literature, where small changes such as $0.40\%$, $0.35\%$, or $0.31\%$ mark distinct positions in the benchmark hierarchy (0710.2231, Ciresan et al., 2010, Cireşan et al., 2011).

The same ranking logic is generalized in later work. On Overhead-MNIST, 23 algorithms are compared using 10-fold stratified cross-validation and ranked primarily by accuracy, while AUC and F1 are also recorded; in the same benchmark family, a tuned CNN is then scored on unseen test data by categorical accuracy (Larsen et al., 2021). On the linear-separability study of MNIST, ranking is not based on classification accuracy at all, but on whether an LP feasibility problem is OPTIMAL or INFEASIBLE, yielding yes/no judgments for pairwise and one-vs-rest separability (Hajnal, 13 Mar 2026). On MNIST-1D, the relevant ranking is the ordering of architectures such as logistic regression, MLP, CNN, GRU, ResNet, TCN, and DCNN by test accuracy on a 40-point signal benchmark (Beebe et al., 12 Feb 2026).

This suggests that “Ranked MNISTs” is best understood as a benchmarking grammar rather than a single dataset: the same MNIST-shaped format supports rankings of methods, rankings of tasks, and rankings of datasets.

2. Historical leaderboard on the original handwritten-digit benchmark

The original MNIST literature records a long sequence of leaderboard improvements. Among the milestones explicitly documented in the cited papers are a single hidden-layer MLP with 800 units at $0.70\%$, a CNN with elastic distortions at $0.40\%$, a CNN with unsupervised layer-wise pretraining at $0.39\%$, a hypothetical combination of four state-of-the-art systems at $0.35\%$, a deep big simple MLP at $0.35\%$ validation-selected and $0.32\%$ best test, a committee of deep GPU-trained MLPs at $0.31\%$, and a later massively deep RBM-based fully connected network at $0.35\%$0 validation-selected with a reported $0.35\%$1 “maximum capacity” best test point (0710.2231, Ciresan et al., 2010, Cireşan et al., 2011, O'Shea, 2015).

Milestone	Reported result	Benchmark position in its paper
Single hidden-layer MLP with 800 units	$0.35\%$2	Strong pre-deep baseline
CNN with elastic distortions	$0.35\%$3	Long-standing reference point
CNN with unsupervised pretraining	$0.35\%$4	State-of-the-art before deep GPU MLPs
Combined classifier of four systems	$0.35\%$5	Best published result in 2007
Deep big simple MLP	$0.35\%$6 / $0.35\%$7	Best single-network result in 2010
Committee of deep neural nets	$0.35\%$8	Best reported result in 2011
Massively deep RBM network	$0.35\%$9 / $0.31\%$0	Non-convolutional sub-1% result in 2015

The low-error regime also exposed evaluation subtleties. The 2007 combined classifier is explicitly described as hypothetical and as an upper bound because the combination used correctness information rather than full class labels (0710.2231). The 2010 and 2015 papers distinguish validation-selected performance from the best test-set point reached during training, so the same architecture can occupy different positions depending on the selection protocol (Ciresan et al., 2010, O'Shea, 2015). A ranked view of MNIST therefore depends not only on architecture but also on whether the comparison is based on proper validation selection, committee averaging, or retrospective test-set minima.

3. Architectural patterns behind top-ranked non-convolutional systems

A recurrent finding in the ranked-MNIST literature is that plain fully connected models become competitive when they are made deep, wide, aggressively trained, and GPU-accelerated. In 2010, “good old on-line back-propagation” on a standard feed-forward MLP with 784 inputs, scaled hyperbolic tangent activations

$0.31\%$1

uniform initialization in $0.31\%$2, and continual deformation of all 60,000 training images at the beginning of every epoch reduced test error from $0.31\%$3 for $0.31\%$4 hidden units to $0.31\%$5 for $0.31\%$6, while a nine-layer $0.31\%$7 network reached $0.31\%$8 (Ciresan et al., 2010). The empirical pattern was that increasing depth and width helped, but uniform depth alone was not sufficient.

The 2011 committee result preserved the same basic philosophy but added ensemble diversity. Each net had architecture $0.31\%$9, and seven nets were trained on differently width-normalized data: WN 10, 12, 14, 16, 18, 20, plus one on the original preprocessing. Committee outputs were averaged, reducing test error to $0.70\%$0. The paper attributes feasibility to GPU acceleration, reporting $0.70\%$1 speedup for forward propagation and back-propagation and $0.70\%$2 speedup for elastic deformations (Cireşan et al., 2011).

The 2015 RBM-based system places stacked Restricted Boltzmann Machines in the same leaderboard frame. MNIST images are mapped to $0.70\%$3, hidden layers range from 2 to 9, rectified hidden units are used, the forward nonlinearity is a scaled hyperbolic tangent

$0.70\%$4

weights are initialized from $0.70\%$5, weight decay is $0.70\%$6, and on-line backpropagation is used without momentum or DropOut. Performance improves from $0.70\%$7 for $0.70\%$8 to $0.70\%$9 for $0.40\%$0, whereas $0.40\%$1 yields $0.40\%$2, again indicating that layer-size scheduling matters (O'Shea, 2015).

This suggests that rank position on original MNIST was shaped by a common recipe: many hidden layers, many neurons, continual augmentation or pretraining, and enough GPU throughput to make large-scale forward propagation and back-propagation practical.

4. Internal rankings within MNIST: linear separability of digits and digit pairs

A different meaning of Ranked MNISTs emerges when the objects being ordered are not models but digit classes. The linear-separability study formulates the question as LP feasibility: given labels $0.40\%$3 and vectors $0.40\%$4, a subset is linearly separable iff there exist $0.40\%$5 and $0.40\%$6 such that

$0.40\%$7

The paper solves this feasibility problem in CVXPY with Clarabel and evaluates pairwise and one-vs-rest constructions on the training, test, and combined sets (Hajnal, 13 Mar 2026).

The main findings are highly structured. On the 60,000-image training set, the non-separable digit pairs are exactly $0.40\%$8, $0.40\%$9, $0.39\%$0, $0.39\%$1, $0.39\%$2, $0.39\%$3, and $0.39\%$4; all other pairs are linearly separable. The same seven pairs remain non-separable on the 70,000-image combined set. Digits 0, 1, and 6 are pairwise separable from every other digit, while digits 3 and 8 each have three conflict partners and are identified as the most challenging digits for linear pairwise classification. On the 10,000-image test set, by contrast, all 45 digit pairs are linearly separable (Hajnal, 13 Mar 2026).

The one-vs-rest picture is stricter. On both the training set and the combined set, every digit $0.39\%$5 is non-separable from the union of the remaining digits. On the test set only, digits 0, 1, 2, 3, 4, 6, and 7 are separable from the rest, whereas 5, 8, and 9 are not. The paper interprets the full pairwise separability of the test set as a finite-sample effect in $0.39\%$6-dimensional space rather than evidence that the underlying full dataset is linearly separable (Hajnal, 13 Mar 2026).

This yields an internal rank order of digit difficulty. From easiest to hardest in the pairwise-linear sense, the paper supports 0, 1, and 6 as the easiest digits; 4 and 7 as moderately difficult; 2, 5, and 9 as more difficult; and 3 and 8 as the most difficult. In this formulation, Ranked MNISTs is a ranking of class geometry rather than classifier engineering.

5. Cross-domain rankings on MNIST-like variants

MNIST-style ranking now extends well beyond handwritten digits. Overhead MNIST is designed as a 28×28 grayscale, 10-class satellite benchmark with classes such as car, harbor, helicopter, oil gas field, parking lot, plane, runway mark, ship, stadium, and storage tank. In its benchmark-dataset form, it contains approximately 9,584–10,000 thumbnails overall, uses a 90:10 train/test split, and a MobileNetV2 transfer-learning model achieves $0.39\%$7 average accuracy, exceeding a reported human accuracy of $0.39\%$8 on 1042 thumbnails (Noever et al., 2021). In the larger Overhead-MNIST baseline study, the dataset contains 76,690 grayscale images of shape $0.39\%$9, 23 machine-learning algorithms are trained, and CatBoost, LightGBM, and XGBoost rank highest in a PyCaret general comparison; CatBoost records accuracy $0.35\%$0, AUC $0.35\%$1, and F1 $0.35\%$2, while a tuned CNN reaches $0.35\%$3 categorical accuracy on unseen test data (Larsen et al., 2021).

MNIST-1D changes the geometry rather than the domain. Each sample is a one-dimensional sequence of length 40 generated from digit templates through padding, circular shift, Gaussian noise, Gaussian smoothing with $0.35\%$4, and downsampling. On the original benchmark, logistic regression reaches $0.35\%$5 test accuracy, an MLP $0.35\%$6, a CNN $0.35\%$7, a GRU $0.35\%$8, and the human benchmark is $0.35\%$9. The extended comparison then places TCN and DCNN at the top, ResNet next, CNN and GRU in a strong middle tier, MLP below them, and logistic regression last (Beebe et al., 12 Feb 2026).

Variant	Ranking criterion	Representative outcome
Overhead MNIST / Overhead-MNIST	Accuracy, AUC, F1, categorical accuracy	CatBoost $0.35\%$0 accuracy; tuned CNN $0.35\%$1; MobileNetV2 $0.35\%$2
MNIST-1D	Test accuracy across model families	TCN/DCNN $0.35\%$3 ResNet $0.35\%$4 CNN/GRU $0.35\%$5 MLP $0.35\%$6 logistic regression
Tree-MNIST / Food-MNIST	Reference-model accuracy and dataset difficulty	CNN $0.35\%$7 on Tree-MNIST; CNN $0.35\%$8 on Food-MNIST

These variants demonstrate that rank order is domain-dependent. Overhead-MNIST models “exhibit weaker performance than with their digital counterparts” because of larger effective feature space, background clutter, and very limited augmentations, whereas MNIST-1D was introduced precisely because classic MNIST often saturates and fails to separate advanced architectures (Larsen et al., 2021, Beebe et al., 12 Feb 2026). A plausible implication is that Ranked MNISTs has become a way to measure inductive bias transfer across benchmark families rather than merely to optimize a single digit-recognition leaderboard.

6. Generation, curation, and methodological interpretation

MNIST-Gen turns ranking into a dataset-construction problem. It defines a hierarchical semantic structure

$0.35\%$9

combines CLIP-based semantic analysis with reinforcement learning and human feedback, and models the full processing pipeline as a composition of morphisms

$0.32\%$0

Its processing modes—Individual Review, Smart Batch Processing, and Fast Batch Processing—trade accuracy against human time. On Food-MNIST with 5,000 images, manual labeling takes 25.0 hours at $0.32\%$1 accuracy, Individual Review 18.5 hours at $0.32\%$2, Smart Batch 5.2 hours at $0.32\%$3, and Fast Batch 3.1 hours at $0.32\%$4. The generated Tree-MNIST contains 1500 labeled images across 4 classes and yields $0.32\%$5 CNN accuracy; Food-MNIST uses 5,000 images across 10 main categories and 30 subcategories and yields $0.32\%$6 CNN accuracy (Shaeri et al., 16 Jul 2025).

This constructional view complements the interpretive cautions already present in the original MNIST leaderboard papers. The 2007 combination result explicitly warns that its $0.32\%$7 is an upper bound because the ensemble used correctness information rather than class labels, and its improvement over the best CNN has a significance level of $0.32\%$8, not below the commonly used $0.32\%$9 threshold (0710.2231). The 2010 and 2015 deep-network papers distinguish between validation-selected performance and a lower “maximum capacity” or best-test-point figure, which changes how a method should be ranked (Ciresan et al., 2010, O'Shea, 2015). The 2011 committee paper, meanwhile, notes that in 29 out of 31 cases the network’s second most likely prediction is correct, reinforcing the view that MNIST is close to saturation at very low error rates (Cireşan et al., 2011).

This suggests that Ranked MNISTs is now a layered benchmarking regime. At one layer it is a leaderboard of test errors on handwritten digits; at another it is a map of which digits and digit pairs are easy or hard for linear decision boundaries; at another it is a cross-domain comparison among MNIST-shaped datasets such as Overhead-MNIST and MNIST-1D; and at still another it is a framework for generating new MNIST-style corpora whose class count, semantic overlap, annotation effort, and reference-model accuracy can themselves be ordered.