RotatedMNIST Benchmark

• RotatedMNIST is a benchmark suite derived from MNIST that incorporates systematic rotations to analyze rotation invariance and robust feature learning.
• It includes various protocols such as random continuous rotations, fixed-angle domains, and sequential tasks to evaluate different architectural and augmentation strategies.
• Empirical results reveal significant performance variations, highlighting the trade-offs between rotation equivariance, canonicalization methods, and domain transfer in neural networks.

Searching arXiv for recent and foundational papers on RotatedMNIST to ground the article in cited sources. RotatedMNIST is a family of benchmark settings derived from MNIST by introducing controlled planar rotations, and it is used to study rotation invariance, rotation equivariance, domain generalization, continual learning, canonicalization, and transformation-aware representation learning. Across the literature, the name can denote several related but non-identical protocols: a fully rotated classification benchmark in which digits are rotated by random angles over $[0^\circ,360^\circ)$; a domain generalization benchmark in which domains are fixed rotation angles such as $$\{0^\circ,15^\circ,30^\circ,45^\circ,60^\circ,75^\circ\}$$; and continual-learning benchmarks in which tasks are sequentially defined by fixed rotations such as $0^\circ,10^\circ,\dots,90^\circ$ or $0^\circ,10^\circ,\dots,350^\circ$ (Li et al., 2017, Zhang et al., 2021, Loke et al., 14 Jul 2025, Wortsman et al., 2020). The benchmark’s importance follows from its controlled geometry: the label space is unchanged while the input distribution shifts in a precisely specified way, making RotatedMNIST a compact testbed for evaluating whether a model genuinely learns rotation-robust features rather than merely memorizing pose-specific statistics (Mo et al., 2022, Zhu et al., 2022).

1. Dataset definitions and benchmark variants

RotatedMNIST is not a single universally fixed dataset specification. Multiple papers use the term for closely related constructions, with different rotation protocols, splits, and evaluation goals.

A common version, often traced to Larochelle et al. (2007), rotates MNIST digits by random angles uniformly drawn from $[0,2\pi]$ or $[0^\circ,360^\circ)$, producing grayscale digit images under continuous orientation variability (Li et al., 2017, Shen et al., 2017, Tripathi et al., 2019). In one widely used split, the dataset contains 62,000 grayscale images divided into 10,000 training, 2,000 validation, and 50,000 test examples, with random rotations applied across train, validation, and test and no additional augmentation reported (Li et al., 2017). Another standard protocol, denoted MNIST-rot or MNIST-rot-12k, uses 12,000 training instances and 50,000 test instances with rotations over the full $0^\circ$–$360^\circ$ range (Tripathi et al., 2019, Shen et al., 2017).

Other works adopt the MNIST-rot split of 10,000 training images, 2,000 validation images, and 50,000 test images, explicitly stating that samples were rotated by random angles and retaining the original split without cross-validation (Giuffrida et al., 2016). In that formulation, pixel values near zero were thresholded at $\tau = 0.3$ to reduce the effect of small spurious non-zero entries, and no further normalization or rotation augmentation was applied because rotations were intrinsic to the dataset (Giuffrida et al., 2016).

A distinct usage appears in domain generalization. There, domains are defined by fixed rotation angles, specifically $$\{0^\circ,15^\circ,30^\circ,45^\circ,60^\circ,75^\circ\}$$, with per-domain counts $$\{0^\circ,15^\circ,30^\circ,45^\circ,60^\circ,75^\circ\}$$0 for a total of 70,000 grayscale $$\{0^\circ,15^\circ,30^\circ,45^\circ,60^\circ,75^\circ\}$$1 images; the $$\{0^\circ,15^\circ,30^\circ,45^\circ,60^\circ,75^\circ\}$$2 domain is held out as target and the remaining five rotations serve as source domains (Zhang et al., 2021). This version isolates covariate shift without label shift and is therefore used to study transferability rather than pure rotation-invariant classification (Zhang et al., 2021).

A further set of variants arises in continual learning. One study defines a three-task protocol using $$\{0^\circ,15^\circ,30^\circ,45^\circ,60^\circ,75^\circ\}$$3, $$\{0^\circ,15^\circ,30^\circ,45^\circ,60^\circ,75^\circ\}$$4, and $$\{0^\circ,15^\circ,30^\circ,45^\circ,60^\circ,75^\circ\}$$5, a small-angle variant using $$\{0^\circ,15^\circ,30^\circ,45^\circ,60^\circ,75^\circ\}$$6, $$\{0^\circ,15^\circ,30^\circ,45^\circ,60^\circ,75^\circ\}$$7, and $$\{0^\circ,15^\circ,30^\circ,45^\circ,60^\circ,75^\circ\}$$8, and a ten-task benchmark using rotations incremented by $$\{0^\circ,15^\circ,30^\circ,45^\circ,60^\circ,75^\circ\}$$9 from $0^\circ,10^\circ,\dots,90^\circ$0 through $0^\circ,10^\circ,\dots,90^\circ$1 (Loke et al., 14 Jul 2025). Another treats RotatedMNIST as 36 tasks at $0^\circ,10^\circ,\dots,90^\circ$2 increments from $0^\circ,10^\circ,\dots,90^\circ$3 to $0^\circ,10^\circ,\dots,90^\circ$4, all sharing the same ten digit labels (Wortsman et al., 2020).

A more recent protocol distinguishes in-distribution and distribution-shift settings. In the in-distribution case, every train and test image is rotated by a random angle in $0^\circ,10^\circ,\dots,90^\circ$5 degrees. In the out-of-distribution case, training images are rotated within $0^\circ,10^\circ,\dots,90^\circ$6 around zero, while test images are rotated in $0^\circ,10^\circ,\dots,90^\circ$7 or $0^\circ,10^\circ,\dots,90^\circ$8, thereby testing generalization to unseen rotation ranges (Santos-Escriche et al., 4 Jun 2025).

These variants are related but not interchangeable. This suggests that reported numbers on “RotatedMNIST” are meaningful only relative to the exact construction, split, and task definition used in a given paper.

2. Why RotatedMNIST matters

RotatedMNIST is used because it exposes a basic limitation of standard convolutional architectures: translation equivariance does not imply rotation invariance. A standard CNN trained on unrotated MNIST can achieve very high accuracy on the original test set yet collapse under rotated evaluation; for example, one study reports 99.55% on MNIST and 45.42% on RotMNIST for a baseline CNN trained without rotation augmentation (Valero-Abundio et al., 24 Feb 2026). A closely related result reports 99.55% on the original $0^\circ,10^\circ,\dots,90^\circ$9 test set and 45.42% on a rotated test set generated by rotating the 10,000 original test images by angles from $0^\circ,10^\circ,\dots,350^\circ$0 to $0^\circ,10^\circ,\dots,350^\circ$1 in $0^\circ,10^\circ,\dots,350^\circ$2 increments, yielding 360,000 rotated test images (Mo et al., 2022).

Because the label semantics are unchanged under rotation, the benchmark separates geometric nuisance variation from semantic content. In the domain generalization setting, this makes it possible to ask whether features are transferable across domains that differ only by a clean covariate shift (Zhang et al., 2021). In probing analyses, rotation is treated as an environment variable whose encoding can be localized layer by layer; one study finds that information about rotation is most visible in lower layers, unlike style information in PACS or VLCS, which is most visible in middle layers (Zhu et al., 2022).

The benchmark is also useful for contrasting invariance and equivariance. Some methods target invariance directly, requiring the output to remain unchanged under rotation. Others seek equivariance, so that internal feature maps transform predictably under rotation and invariance emerges only after pooling or classifier design (Li et al., 2017, Santos-Escriche et al., 4 Jun 2025). RotatedMNIST therefore functions as a controlled setting in which these design choices can be compared quantitatively.

Its simplicity is equally important for continual learning. When sequential tasks differ only by rotation angle, forgetting cannot be attributed to class changes or major semantic shifts; it reflects instability under domain drift across otherwise identical digit classes (Loke et al., 14 Jul 2025, Wortsman et al., 2020).

3. Methodological families evaluated on RotatedMNIST

Research on RotatedMNIST spans several methodological families rather than a single dominant paradigm.

One family uses explicit rotation-aware weight sharing in shallow generative models. The Explicit Rotation-Invariant Restricted Boltzmann Machine maintains orientation-specific weight matrices indexed by a discrete angle set $0^\circ,10^\circ,\dots,350^\circ$3 and couples them by rotating contrastive-divergence updates across bins (Giuffrida et al., 2016). The related $0^\circ,10^\circ,\dots,350^\circ$4-RBM is an unfactored gated Boltzmann machine with a third-order tensor $0^\circ,10^\circ,\dots,350^\circ$5 and a one-hot rotation code, again enforcing rotational consistency through slice-wise rotated gradients (Giuffrida et al., 2016). In both cases, dominant orientation is estimated externally, typically through Sobel filters or histogram-of-oriented-gradients procedures (Giuffrida et al., 2016, Giuffrida et al., 2016).

A second family builds rotation equivariance into convolutional architectures. Deep Rotation Equivariant Networks introduce cycle layers, isotonic layers, and decycle layers acting on filters rather than feature maps, targeting exact equivariance to the cyclic group $0^\circ,10^\circ,\dots,350^\circ$6 corresponding to rotations by multiples of $0^\circ,10^\circ,\dots,350^\circ$7 (Li et al., 2017). RIC-CNN replaces standard convolutions with Rotation-Invariant Coordinate Convolutions defined on a rotation-invariant coordinate system centered at the image center, implemented efficiently through fixed offsets in deform_conv2d (Mo et al., 2022). CyCNN instead maps the image to polar or log-polar coordinates, converting rotation into translation along the angular axis, and uses cylindrical convolution with periodic padding along $0^\circ,10^\circ,\dots,350^\circ$8 (Kim et al., 2020).

A third family canonicalizes the input before standard processing. General Intensity Direction estimates a single global orientation

$0^\circ,10^\circ,\dots,350^\circ$9

from intensity-weighted pixel directions relative to the image center, rotates each image into a canonical orientation, and feeds the aligned image to an otherwise standard CNN (Valero-Abundio et al., 24 Feb 2026). Deformable Classifiers perform class-conditional latent optimization over transformations, maximizing each class score over a rotation variable $[0,2\pi]$0 and using alternating optimization between latent transformations and classifier parameters (Shen et al., 2017). A variational autoencoder approach uses class-specific fixed reference digits as reconstruction targets, forcing rotated variants of the same digit to collapse into compact latent regions associated with canonical unrotated templates (Yevick, 2022).

A fourth family uses augmentation or soft-equivariance rather than hard architectural constraints. SEMoLA learns a Lie algebra basis inside a 6D affine search space, discovers the generator corresponding to in-plane rotation, and applies an equivariance loss

$[0,2\pi]$1

with $[0,2\pi]$2 on RotatedMNIST (Santos-Escriche et al., 4 Jun 2025). The base model in that study is a simple four-layer CNN with two max-pooling layers and no specialized steerable filters or spatial transformers (Santos-Escriche et al., 4 Jun 2025).

A fifth family studies RotatedMNIST as a non-stationary task stream. Elastic Weight Consolidation applies diagonal Fisher-weighted quadratic penalties across previously learned rotation tasks (Loke et al., 14 Jul 2025). Supermasks in Superposition keeps a fixed random MLP backbone and learns a binary mask per rotation task, inferring the correct mask at test time by minimizing output entropy under a superposition of masks (Wortsman et al., 2020). A meta-learned transformer optimizer has also been evaluated in a ten-task RotatedMNIST stream, using support and query splits per task and explicitly optimizing forward and backward transfer (Vettoruzzo et al., 2024).

These families correspond to different views of the problem: symmetry-aware parameter sharing, canonicalization, augmentation-driven approximate equivariance, and sequential retention under domain drift.

4. Representative empirical results

Reported performance on RotatedMNIST varies sharply by protocol. On the 10k/2k/50k continuously rotated benchmark, DREN attains 1.78% test error with 22k parameters, while DRENMaxPooling reaches 1.56% error with 25k parameters (Li et al., 2017). The same comparison table lists P4CNN at 2.28%, H-Net at 1.69%, Dieleman et al. at 1.78%, Z2CNN at 5.03%, Transformation RBM at 4.2%, and Conv-RBM at 3.98% (Li et al., 2017).

On the same general benchmark family, the context-aware convolution filter framework reports 1.13% error on MNIST-rot-12k, improving on TI-POOLING and MINTIN at all reported transformation-set sizes; for example, with 24 channels the proposed method yields 1.13% versus 1.61% for TI-POOLING and 1.57% for MINTIN (Tripathi et al., 2019). Deformable Classifiers report 1.25% error for DC-ESGD, 2.31% for DC-ES, 2.64% for CSTN, 5.71% for STN, and 4.1% for the original CNN baseline, with TI-Pooling at 1.2% in the cited comparison (Shen et al., 2017).

On the unrotated-train, rotated-test evaluation used to isolate architectural invariance, RIC-CNN achieves 95.52% on the rotated test set while preserving 99.02% on the original test set (Mo et al., 2022). In the same table, H-Net obtains 92.44%, GA-CNN 93.29%, B-CNN 88.29%, E(2)-CNN 94.37%, CNN 45.42%, and DEF-CNN 46.97% (Mo et al., 2022). A closely related study reports Conv32+GID at 96.32% on RotMNIST, compared with RIC-CNN at 95.52%, E(2)-CNN at 94.37%, H-Net at 92.44%, GA-CNN at 93.29%, and the baseline CNN at 45.42% (Valero-Abundio et al., 24 Feb 2026).

On MNIST-rot with RBM-family models and 100 hidden units, ERI-RBM reaches 92.08% test accuracy with RBF SVM when $[0,2\pi]$3, compared with 87.37% for the classic RBM, 87.37% for O-RBM, 80.63% for TI-RBM, and substantially lower scores for D-RBM as $[0,2\pi]$4 increases (Giuffrida et al., 2016). The $[0,2\pi]$5-RBM emphasizes invariance scores as well as classification, reaching train/test $[0,2\pi]$6 values around 0.90 and test error near 10% with an RBF-SVM classifier, outperforming TI-RBM, O-RBM, and the standard RBM under that protocol (Giuffrida et al., 2016).

On the distribution-shift benchmark with train rotations restricted to $[0,2\pi]$7 and test rotations in the complementary quadrants, SEMoLA reports 90.82 ± 6.49 accuracy and equivariance error 3.26 ± 0.62, compared with 55.24 ± 1.14 and 7.19 ± 0.54 for a plain CNN, 57.68 ± 1.45 and 3.97 ± 0.51 for Augerino+, 91.59 ± 1.07 and 5.82 ± 0.47 for CNN + LieGAN augmentations, and 99.09 ± 0.08 with ground-truth augmentations (Santos-Escriche et al., 4 Jun 2025). In the in-distribution setting with full $[0,2\pi]$8 rotations in train and test, SEMoLA reaches 99.08 ± 0.06 accuracy and equivariance error 3.10 ± 0.32, essentially matching CNN + GT augmentations at 99.06 ± 0.00 while outperforming plain CNN, LieGAN augmentation, Augerino+, and a $[0,2\pi]$9 GCNN (Santos-Escriche et al., 4 Jun 2025).

The following table summarizes several representative results from different protocols.

Setting	Method	Reported result
10k/2k/50k rotated benchmark	DRENMaxPooling	1.56% test error (Li et al., 2017)
MNIST-rot-12k	Visual Context-aware Filters	1.13% error (Tripathi et al., 2019)
mnist-rot	DC-ESGD	1.25% error (Shen et al., 2017)
Unrotated train, rotated test	Conv32+GID	96.32% accuracy (Valero-Abundio et al., 24 Feb 2026)
Unrotated train, rotated test	RIC-CNN	95.52% accuracy (Mo et al., 2022)
MNIST-rot with RBM features	ERI-RBM + RBF SVM	92.08% accuracy (Giuffrida et al., 2016)
OOD quadrants shift	SEMoLA	90.82 ± 6.49 accuracy (Santos-Escriche et al., 4 Jun 2025)

Because these results come from distinct protocols, they are not directly rank-comparable. A plausible implication is that RotatedMNIST is better understood as a benchmark family than as a single leaderboard.

5. RotatedMNIST in domain generalization and probing studies

In domain generalization, RotatedMNIST is valuable precisely because the domain variable is explicit and low-dimensional. One study defines transferability through excess-risk stability and uses the fixed-angle domains $[0^\circ,360^\circ)$0 to quantify whether near-optimal source hypotheses remain near-optimal on target (Zhang et al., 2021). Its transfer measures

$[0^\circ,360^\circ)$1

compare excess risks rather than merely marginal discrepancies, and the resulting target bound

$[0^\circ,360^\circ)$2

is used to motivate a minimax training objective over perturbations of the classifier (Zhang et al., 2021). Empirically, many algorithms that perform well on source rotations become brittle under small classifier perturbations, whereas CORAL, Spectral Decoupling, and the proposed Transfer algorithm are more robust (Zhang et al., 2021).

RotatedMNIST has also been used to interpret out-of-domain generalization internally. OOD-Probe attaches a linear domain classifier to each hidden layer of a four-layer CNN and asks how linearly decodable the rotation domain is from intermediate representations (Zhu et al., 2022). The main observation is a decrease-with-depth pattern: lower layers encode rotation most linearly, and probe accuracy drops upward through the network (Zhu et al., 2022). Aggregated Pearson correlations between probe accuracy and OOD generalization accuracy are strongest in the lowest layers, specifically 0.7955 at Probe_0, 0.8982 at Probe_1, and 0.7451 at Probe_2, all significant at $[0^\circ,360^\circ)$3, while higher probes are weak or non-significant (Zhu et al., 2022). Even when high-level feature maps appear visually abstract, domain prediction can remain strong; Probe 3 in the ERM example reaches .75 accuracy on predicting the rotation domain (Zhu et al., 2022).

These results complicate a common misconception that good OOD generalization must correspond to purging domain information from the representation. On RotatedMNIST, high-performing models do not eliminate rotation information; rather, early-layer linear decodability of rotation correlates positively with leave-one-domain-out performance (Zhu et al., 2022). This suggests that encoding the nuisance factor in a stable, organized way may be more useful than suppressing it entirely.

6. RotatedMNIST in continual learning

RotatedMNIST has become a canonical continual-learning benchmark because it defines a sequence of tasks with identical labels but systematically changing input distributions.

A replicated and extended study of Elastic Weight Consolidation uses three-task sequences such as $[0^\circ,360^\circ)$4 and ten-task sequences from $[0^\circ,360^\circ)$5 to $[0^\circ,360^\circ)$6 in steps of $[0^\circ,360^\circ)$7 (Loke et al., 14 Jul 2025). The model is a fully connected network with two hidden layers of 400 ReLU units, optional batch normalization, and a single 10-class output layer, trained with mini-batch SGD with momentum (Loke et al., 14 Jul 2025). EWC consistently reduces forgetting relative to SGD and naïve L2 penalties, especially when angle jumps are large (Loke et al., 14 Jul 2025). Exact tabulated RotatedMNIST accuracies are not provided in that study, but the qualitative conclusion is that EWC preserves early tasks better than SGD or L2, while exhibiting the expected stability–plasticity trade-off (Loke et al., 14 Jul 2025).

Supermasks in Superposition defines 36 tasks at $[0^\circ,360^\circ)$8 increments from $[0^\circ,360^\circ)$9 to $0^\circ$0, trains a binary mask per task on a fixed random MLP with two hidden layers of size 1024, and infers task identity at test time by minimizing output entropy over a linear superposition of masks (Wortsman et al., 2020). On this benchmark, SupSup in the harder task-identity-unknown setting achieves 96.13% average accuracy, compared with 95.87% for PSP and 92.40% for BatchEnsemble in the task-identity-given setting; the upper bound of separate models is 98.01%, while a forgetting-prone lower bound yields 48.71% (Wortsman et al., 2020). Because neighboring rotation tasks are highly similar, the study reports that one-shot task inference is insufficient and that a binary elimination procedure using the full batch is necessary (Wortsman et al., 2020).

A meta-learned transformer optimizer has also been evaluated on a ten-task RotatedMNIST stream with a single convolutional layer and support/query episodic adaptation (Vettoruzzo et al., 2024). In that setting, the proposed method attains average accuracy 62.09 ± 1.78, backward transfer $0^\circ$1, and forward transfer 66.07 ± 0.61, whereas DER++ reaches higher average accuracy at 73.67 ± 0.75 but much worse backward transfer at $0^\circ$2 (Vettoruzzo et al., 2024). This result foregrounds a characteristic tension in continual RotatedMNIST benchmarks: strong forward adaptation to new rotations often comes at the cost of retaining earlier ones.

The continual-learning literature therefore uses RotatedMNIST not merely as a toy benchmark but as a controlled test of how well an algorithm can separate reusable digit structure from rotation-specific adaptation.

7. Limitations, ambiguities, and recurring misconceptions

Several recurring issues surround the interpretation of RotatedMNIST results.

First, “RotatedMNIST” often refers to different tasks. Some works rotate both train and test images by random continuous angles (Li et al., 2017, Tripathi et al., 2019). Others train on unrotated MNIST and test on rotated versions to isolate architectural invariance (Mo et al., 2022, Valero-Abundio et al., 24 Feb 2026). Domain generalization papers use fixed-angle domains (Zhang et al., 2021), whereas continual-learning papers define sequential rotation tasks (Loke et al., 14 Jul 2025, Wortsman et al., 2020). Directly comparing numbers across these protocols can therefore be misleading.

Second, exact invariance is rarely identical to empirical invariance. DREN proves exact equivariance only for rotations by multiples of $0^\circ$3 in the group

$0^\circ$4

yet still performs strongly on continuously rotated data (Li et al., 2017). RIC-CNN is invariant to arbitrary rotations around the input center in theory, but pooling and interpolation introduce small periodic errors, producing accuracy dips toward $0^\circ$5 and recovery toward $0^\circ$6 (Mo et al., 2022). CyCNN likewise relies on centered objects and approximate translation invariance after polar remapping; off-center digits or distortions near the origin can weaken the intended rotation-to-translation equivalence (Kim et al., 2020).

Third, canonicalization methods can fail when orientation is ambiguous. GID estimates a global orientation from the intensity-weighted resultant of pixel directions; if the intensity distribution is nearly isotropic, the vector sum can be near zero and the orientation becomes unstable (Valero-Abundio et al., 24 Feb 2026). ERI-RBM and $0^\circ$7-RBM rely on dominant orientation estimation via gradient histograms or HOG, and coarse orientation binning or noisy assignments can degrade performance (Giuffrida et al., 2016, Giuffrida et al., 2016). Digits such as 0, 6, 8, and 9 also create genuine orientation-class ambiguities, a point explicitly noted in the per-class analysis of RIC-CNN (Mo et al., 2022).

Fourth, strong performance on RotatedMNIST does not imply broad rotation robustness in natural images. Several methods exploit assumptions specific to MNIST: centered foreground objects, simple black backgrounds, limited texture, and stable global orientation statistics (Valero-Abundio et al., 24 Feb 2026, Kim et al., 2020). This suggests that RotatedMNIST is best treated as a controlled geometry benchmark rather than a complete proxy for real-world rotational variability.

Finally, the benchmark can reward different inductive biases depending on protocol. When train and test both contain full random rotations, augmentation-based methods can already approach ceiling performance; for example, conventional CNNs trained with rotation augmentation reach about 99.5% on MNIST-r in one CyCNN study, leaving little room for architectural gains (Kim et al., 2020). By contrast, the unrotated-train, rotated-test setting sharply separates architecture-driven invariance from augmentation effects (Mo et al., 2022).

RotatedMNIST remains influential precisely because these distinctions are visible. It exposes whether a method is exploiting data augmentation, hard-coded symmetry, canonicalization, latent-variable optimization, or memory mechanisms, and it does so under a transformation that is mathematically simple but algorithmically revealing.