---
title: RotatedMNIST Benchmark
url: https://www.emergentmind.com/topics/rotatedmnist
type: topic
---

# RotatedMNIST Benchmark

Searching arXiv for recent and foundational papers on RotatedMNIST to ground the article in cited sources.
RotatedMNIST is a family of benchmark settings derived from MNIST by introducing controlled planar rotations, and it is used to study rotation invariance, rotation equivariance, domain generalization, continual learning, canonicalization, and transformation-aware representation learning. Across the literature, the name can denote several related but non-identical protocols: a fully rotated classification benchmark in which digits are rotated by random angles over $[0^\circ,360^\circ)$; a domain generalization benchmark in which domains are fixed rotation angles such as $\{0^\circ,15^\circ,30^\circ,45^\circ,60^\circ,75^\circ\}$; and continual-learning benchmarks in which tasks are sequentially defined by fixed rotations such as $0^\circ,10^\circ,\dots,90^\circ$ or $0^\circ,10^\circ,\dots,350^\circ$ [1705.08623] [2106.03632] [2507.10485] [2006.14769]. The benchmark’s importance follows from its controlled geometry: the label space is unchanged while the input distribution shifts in a precisely specified way, making RotatedMNIST a compact testbed for evaluating whether a model genuinely learns rotation-robust features rather than merely memorizing pose-specific statistics [2211.11812] [2208.12352].

## 1. Dataset definitions and benchmark variants

RotatedMNIST is not a single universally fixed dataset specification. Multiple papers use the term for closely related constructions, with different rotation protocols, splits, and evaluation goals.

A common version, often traced to Larochelle et al. (2007), rotates MNIST digits by random angles uniformly drawn from $[0,2\pi]$ or $[0^\circ,360^\circ)$, producing grayscale digit images under continuous orientation variability [1705.08623] [1712.06715] [1906.09986]. In one widely used split, the dataset contains 62,000 grayscale images divided into 10,000 training, 2,000 validation, and 50,000 test examples, with random rotations applied across train, validation, and test and no additional augmentation reported [1705.08623]. Another standard protocol, denoted MNIST-rot or MNIST-rot-12k, uses 12,000 training instances and 50,000 test instances with rotations over the full $0^\circ$–$360^\circ$ range [1906.09986] [1712.06715].

Other works adopt the MNIST-rot split of 10,000 training images, 2,000 validation images, and 50,000 test images, explicitly stating that samples were rotated by random angles and retaining the original split without cross-validation [1604.07045]. In that formulation, pixel values near zero were thresholded at $\tau = 0.3$ to reduce the effect of small spurious non-zero entries, and no further normalization or rotation augmentation was applied because rotations were intrinsic to the dataset [1604.07045].

A distinct usage appears in domain generalization. There, domains are defined by fixed rotation angles, specifically $\{0^\circ,15^\circ,30^\circ,45^\circ,60^\circ,75^\circ\}$, with per-domain counts $11{,}667, 11{,}667, 11{,}667, 11{,}667, 11{,}666, 11{,}666$ for a total of 70,000 grayscale $28\times 28$ images; the $0^\circ$ domain is held out as target and the remaining five rotations serve as source domains [2106.03632]. This version isolates covariate shift without label shift and is therefore used to study transferability rather than pure rotation-invariant classification [2106.03632].

A further set of variants arises in continual learning. One study defines a three-task protocol using $0^\circ$, $40^\circ$, and $90^\circ$, a small-angle variant using $0^\circ$, $10^\circ$, and $20^\circ$, and a ten-task benchmark using rotations incremented by $10^\circ$ from $0^\circ$ through $90^\circ$ [2507.10485]. Another treats RotatedMNIST as 36 tasks at $10^\circ$ increments from $0^\circ$ to $350^\circ$, all sharing the same ten digit labels [2006.14769].

A more recent protocol distinguishes in-distribution and distribution-shift settings. In the in-distribution case, every train and test image is rotated by a random angle in $[0,360)$ degrees. In the out-of-distribution case, training images are rotated within $\pm 90^\circ$ around zero, while test images are rotated in $[90^\circ,180^\circ)$ or $[-90^\circ,-180^\circ)$, thereby testing generalization to unseen rotation ranges [2506.03914].

These variants are related but not interchangeable. This suggests that reported numbers on “RotatedMNIST” are meaningful only relative to the exact construction, split, and task definition used in a given paper.

## 2. Why RotatedMNIST matters

RotatedMNIST is used because it exposes a basic limitation of standard convolutional architectures: translation equivariance does not imply rotation invariance. A standard CNN trained on unrotated MNIST can achieve very high accuracy on the original test set yet collapse under rotated evaluation; for example, one study reports 99.55% on MNIST and 45.42% on RotMNIST for a baseline CNN trained without rotation augmentation [2602.20930]. A closely related result reports 99.55% on the original $32\times 32$ test set and 45.42% on a rotated test set generated by rotating the 10,000 original test images by angles from $0^\circ$ to $360^\circ$ in $10^\circ$ increments, yielding 360,000 rotated test images [2211.11812].

Because the label semantics are unchanged under rotation, the benchmark separates geometric nuisance variation from semantic content. In the domain generalization setting, this makes it possible to ask whether features are transferable across domains that differ only by a clean covariate shift [2106.03632]. In probing analyses, rotation is treated as an environment variable whose encoding can be localized layer by layer; one study finds that information about rotation is most visible in lower layers, unlike style information in PACS or VLCS, which is most visible in middle layers [2208.12352].

The benchmark is also useful for contrasting invariance and equivariance. Some methods target invariance directly, requiring the output to remain unchanged under rotation. Others seek equivariance, so that internal feature maps transform predictably under rotation and invariance emerges only after pooling or classifier design [1705.08623] [2506.03914]. RotatedMNIST therefore functions as a controlled setting in which these design choices can be compared quantitatively.

Its simplicity is equally important for continual learning. When sequential tasks differ only by rotation angle, forgetting cannot be attributed to class changes or major semantic shifts; it reflects instability under domain drift across otherwise identical digit classes [2507.10485] [2006.14769].

## 3. Methodological families evaluated on RotatedMNIST

Research on RotatedMNIST spans several methodological families rather than a single dominant paradigm.

One family uses explicit rotation-aware weight sharing in shallow generative models. The Explicit Rotation-Invariant Restricted Boltzmann Machine maintains orientation-specific weight matrices indexed by a discrete angle set $\Phi=\{\phi_1,\dots,\phi_S\}$ and couples them by rotating contrastive-divergence updates across bins [1604.07045]. The related $\theta$-RBM is an unfactored gated Boltzmann machine with a third-order tensor $W\in\mathbb{R}^{H\times V\times S}$ and a one-hot rotation code, again enforcing rotational consistency through slice-wise rotated gradients [1606.08805]. In both cases, dominant orientation is estimated externally, typically through Sobel filters or histogram-of-oriented-gradients procedures [1604.07045] [1606.08805].

A second family builds rotation equivariance into convolutional architectures. Deep Rotation Equivariant Networks introduce cycle layers, isotonic layers, and decycle layers acting on filters rather than feature maps, targeting exact equivariance to the cyclic group $C_4$ corresponding to rotations by multiples of $\pi/2$ [1705.08623]. RIC-CNN replaces standard convolutions with Rotation-Invariant Coordinate Convolutions defined on a rotation-invariant coordinate system centered at the image center, implemented efficiently through fixed offsets in `deform_conv2d` [2211.11812]. CyCNN instead maps the image to polar or log-polar coordinates, converting rotation into translation along the angular axis, and uses cylindrical convolution with periodic padding along $\theta$ [2007.10588].

A third family canonicalizes the input before standard processing. General Intensity Direction estimates a single global orientation
\[
\alpha=\operatorname{atan2}(S_{\sin},S_{\cos})
\]
from intensity-weighted pixel directions relative to the image center, rotates each image into a canonical orientation, and feeds the aligned image to an otherwise standard CNN [2602.20930]. Deformable Classifiers perform class-conditional latent optimization over transformations, maximizing each class score over a rotation variable $\phi_c\in[0,2\pi)$ and using alternating optimization between latent transformations and classifier parameters [1712.06715]. A variational autoencoder approach uses class-specific fixed reference digits as reconstruction targets, forcing rotated variants of the same digit to collapse into compact latent regions associated with canonical unrotated templates [2206.13388].

A fourth family uses augmentation or soft-equivariance rather than hard architectural constraints. SEMoLA learns a Lie algebra basis inside a 6D affine search space, discovers the generator corresponding to in-plane rotation, and applies an equivariance loss
\[
L_{\text{total}}=\alpha L_{\text{task}}+\beta L_{\text{eq}}+\lambda l_{\text{areg}}+\eta l_{\text{bcreg}}+\nu l_{\text{bsreg}}
\]
with $C=1$ on RotatedMNIST [2506.03914]. The base model in that study is a simple four-layer CNN with two max-pooling layers and no specialized steerable filters or spatial transformers [2506.03914].

A fifth family studies RotatedMNIST as a non-stationary task stream. Elastic Weight Consolidation applies diagonal Fisher-weighted quadratic penalties across previously learned rotation tasks [2507.10485]. Supermasks in Superposition keeps a fixed random MLP backbone and learns a binary mask per rotation task, inferring the correct mask at test time by minimizing output entropy under a superposition of masks [2006.14769]. A meta-learned transformer optimizer has also been evaluated in a ten-task RotatedMNIST stream, using support and query splits per task and explicitly optimizing forward and backward transfer [2408.03219].

These families correspond to different views of the problem: symmetry-aware parameter sharing, canonicalization, augmentation-driven approximate equivariance, and sequential retention under domain drift.

## 4. Representative empirical results

Reported performance on RotatedMNIST varies sharply by protocol. On the 10k/2k/50k continuously rotated benchmark, DREN attains 1.78% test error with 22k parameters, while DRENMaxPooling reaches 1.56% error with 25k parameters [1705.08623]. The same comparison table lists P4CNN at 2.28%, H-Net at 1.69%, Dieleman et al. at 1.78%, Z2CNN at 5.03%, Transformation RBM at 4.2%, and Conv-RBM at 3.98% [1705.08623].

On the same general benchmark family, the context-aware convolution filter framework reports 1.13% error on MNIST-rot-12k, improving on TI-POOLING and MINTIN at all reported transformation-set sizes; for example, with 24 channels the proposed method yields 1.13% versus 1.61% for TI-POOLING and 1.57% for MINTIN [1906.09986]. Deformable Classifiers report 1.25% error for DC-ESGD, 2.31% for DC-ES, 2.64% for CSTN, 5.71% for STN, and 4.1% for the original CNN baseline, with TI-Pooling at 1.2% in the cited comparison [1712.06715].

On the unrotated-train, rotated-test evaluation used to isolate architectural invariance, RIC-CNN achieves 95.52% on the rotated test set while preserving 99.02% on the original test set [2211.11812]. In the same table, H-Net obtains 92.44%, GA-CNN 93.29%, B-CNN 88.29%, E(2)-CNN 94.37%, CNN 45.42%, and DEF-CNN 46.97% [2211.11812]. A closely related study reports Conv32+GID at 96.32% on RotMNIST, compared with RIC-CNN at 95.52%, E(2)-CNN at 94.37%, H-Net at 92.44%, GA-CNN at 93.29%, and the baseline CNN at 45.42% [2602.20930].

On MNIST-rot with RBM-family models and 100 hidden units, ERI-RBM reaches 92.08% test accuracy with RBF SVM when $S=18$, compared with 87.37% for the classic RBM, 87.37% for O-RBM, 80.63% for TI-RBM, and substantially lower scores for D-RBM as $S$ increases [1604.07045]. The $\theta$-RBM emphasizes invariance scores as well as classification, reaching train/test $\gamma$ values around 0.90 and test error near 10% with an RBF-SVM classifier, outperforming TI-RBM, O-RBM, and the standard RBM under that protocol [1606.08805].

On the distribution-shift benchmark with train rotations restricted to $\pm 90^\circ$ and test rotations in the complementary quadrants, SEMoLA reports 90.82 ± 6.49 accuracy and equivariance error 3.26 ± 0.62, compared with 55.24 ± 1.14 and 7.19 ± 0.54 for a plain CNN, 57.68 ± 1.45 and 3.97 ± 0.51 for Augerino+, 91.59 ± 1.07 and 5.82 ± 0.47 for CNN + LieGAN augmentations, and 99.09 ± 0.08 with ground-truth augmentations [2506.03914]. In the in-distribution setting with full $[0,360)$ rotations in train and test, SEMoLA reaches 99.08 ± 0.06 accuracy and equivariance error 3.10 ± 0.32, essentially matching CNN + GT augmentations at 99.06 ± 0.00 while outperforming plain CNN, LieGAN augmentation, Augerino+, and a $p4$ GCNN [2506.03914].

The following table summarizes several representative results from different protocols.

| Setting | Method | Reported result |
|---|---|---|
| 10k/2k/50k rotated benchmark | DRENMaxPooling | 1.56% test error [1705.08623] |
| MNIST-rot-12k | Visual Context-aware Filters | 1.13% error [1906.09986] |
| mnist-rot | DC-ESGD | 1.25% error [1712.06715] |
| Unrotated train, rotated test | Conv32+GID | 96.32% accuracy [2602.20930] |
| Unrotated train, rotated test | RIC-CNN | 95.52% accuracy [2211.11812] |
| MNIST-rot with RBM features | ERI-RBM + RBF SVM | 92.08% accuracy [1604.07045] |
| OOD quadrants shift | SEMoLA | 90.82 ± 6.49 accuracy [2506.03914] |

Because these results come from distinct protocols, they are not directly rank-comparable. A plausible implication is that RotatedMNIST is better understood as a benchmark family than as a single leaderboard.

## 5. RotatedMNIST in domain generalization and probing studies

In domain generalization, RotatedMNIST is valuable precisely because the domain variable is explicit and low-dimensional. One study defines transferability through excess-risk stability and uses the fixed-angle domains $\{0^\circ,15^\circ,30^\circ,45^\circ,60^\circ,75^\circ\}$ to quantify whether near-optimal source hypotheses remain near-optimal on target [2106.03632]. Its transfer measures
\[
T_{H_f}(S\|T),\quad T_{H_f}(S,T),\quad T^{\mathrm r}_{H_f}(S,T)
\]
compare excess risks rather than merely marginal discrepancies, and the resulting target bound
\[
\epsilon_T(h)\le \epsilon_S(h)+\epsilon_T^*-\epsilon_S^*+T_{H_f}(S\|T)
\]
is used to motivate a minimax training objective over perturbations of the classifier [2106.03632]. Empirically, many algorithms that perform well on source rotations become brittle under small classifier perturbations, whereas CORAL, Spectral Decoupling, and the proposed Transfer algorithm are more robust [2106.03632].

RotatedMNIST has also been used to interpret out-of-domain generalization internally. OOD-Probe attaches a linear domain classifier to each hidden layer of a four-layer CNN and asks how linearly decodable the rotation domain is from intermediate representations [2208.12352]. The main observation is a decrease-with-depth pattern: lower layers encode rotation most linearly, and probe accuracy drops upward through the network [2208.12352]. Aggregated Pearson correlations between probe accuracy and OOD generalization accuracy are strongest in the lowest layers, specifically 0.7955 at Probe\_0, 0.8982 at Probe\_1, and 0.7451 at Probe\_2, all significant at $p<0.01$, while higher probes are weak or non-significant [2208.12352]. Even when high-level feature maps appear visually abstract, domain prediction can remain strong; Probe 3 in the ERM example reaches .75 accuracy on predicting the rotation domain [2208.12352].

These results complicate a common misconception that good OOD generalization must correspond to purging domain information from the representation. On RotatedMNIST, high-performing models do not eliminate rotation information; rather, early-layer linear decodability of rotation correlates positively with leave-one-domain-out performance [2208.12352]. This suggests that encoding the nuisance factor in a stable, organized way may be more useful than suppressing it entirely.

## 6. RotatedMNIST in continual learning

RotatedMNIST has become a canonical continual-learning benchmark because it defines a sequence of tasks with identical labels but systematically changing input distributions.

A replicated and extended study of Elastic Weight Consolidation uses three-task sequences such as $0^\circ,40^\circ,90^\circ$ and ten-task sequences from $0^\circ$ to $90^\circ$ in steps of $10^\circ$ [2507.10485]. The model is a fully connected network with two hidden layers of 400 ReLU units, optional batch normalization, and a single 10-class output layer, trained with mini-batch SGD with momentum [2507.10485]. EWC consistently reduces forgetting relative to SGD and naïve L2 penalties, especially when angle jumps are large [2507.10485]. Exact tabulated RotatedMNIST accuracies are not provided in that study, but the qualitative conclusion is that EWC preserves early tasks better than SGD or L2, while exhibiting the expected stability–plasticity trade-off [2507.10485].

Supermasks in Superposition defines 36 tasks at $10^\circ$ increments from $0^\circ$ to $350^\circ$, trains a binary mask per task on a fixed random MLP with two hidden layers of size 1024, and infers task identity at test time by minimizing output entropy over a linear superposition of masks [2006.14769]. On this benchmark, SupSup in the harder task-identity-unknown setting achieves 96.13% average accuracy, compared with 95.87% for PSP and 92.40% for BatchEnsemble in the task-identity-given setting; the upper bound of separate models is 98.01%, while a forgetting-prone lower bound yields 48.71% [2006.14769]. Because neighboring rotation tasks are highly similar, the study reports that one-shot task inference is insufficient and that a binary elimination procedure using the full batch is necessary [2006.14769].

A meta-learned transformer optimizer has also been evaluated on a ten-task RotatedMNIST stream with a single convolutional layer and support/query episodic adaptation [2408.03219]. In that setting, the proposed method attains average accuracy 62.09 ± 1.78, backward transfer $-1.70 \pm 1.08$, and forward transfer 66.07 ± 0.61, whereas DER++ reaches higher average accuracy at 73.67 ± 0.75 but much worse backward transfer at $-18.91 \pm 0.70$ [2408.03219]. This result foregrounds a characteristic tension in continual RotatedMNIST benchmarks: strong forward adaptation to new rotations often comes at the cost of retaining earlier ones.

The continual-learning literature therefore uses RotatedMNIST not merely as a toy benchmark but as a controlled test of how well an algorithm can separate reusable digit structure from rotation-specific adaptation.

## 7. Limitations, ambiguities, and recurring misconceptions

Several recurring issues surround the interpretation of RotatedMNIST results.

First, “RotatedMNIST” often refers to different tasks. Some works rotate both train and test images by random continuous angles [1705.08623] [1906.09986]. Others train on unrotated MNIST and test on rotated versions to isolate architectural invariance [2211.11812] [2602.20930]. Domain generalization papers use fixed-angle domains [2106.03632], whereas continual-learning papers define sequential rotation tasks [2507.10485] [2006.14769]. Directly comparing numbers across these protocols can therefore be misleading.

Second, exact invariance is rarely identical to empirical invariance. DREN proves exact equivariance only for rotations by multiples of $90^\circ$ in the group
\[
\mathfrak R=\{R_\theta\mid \theta=k\pi/2,\;k\in\mathbb Z\},
\]
yet still performs strongly on continuously rotated data [1705.08623]. RIC-CNN is invariant to arbitrary rotations around the input center in theory, but pooling and interpolation introduce small periodic errors, producing accuracy dips toward $45^\circ$ and recovery toward $90^\circ$ [2211.11812]. CyCNN likewise relies on centered objects and approximate translation invariance after polar remapping; off-center digits or distortions near the origin can weaken the intended rotation-to-translation equivalence [2007.10588].

Third, canonicalization methods can fail when orientation is ambiguous. GID estimates a global orientation from the intensity-weighted resultant of pixel directions; if the intensity distribution is nearly isotropic, the vector sum can be near zero and the orientation becomes unstable [2602.20930]. ERI-RBM and $\theta$-RBM rely on dominant orientation estimation via gradient histograms or HOG, and coarse orientation binning or noisy assignments can degrade performance [1604.07045] [1606.08805]. Digits such as 0, 6, 8, and 9 also create genuine orientation-class ambiguities, a point explicitly noted in the per-class analysis of RIC-CNN [2211.11812].

Fourth, strong performance on RotatedMNIST does not imply broad rotation robustness in natural images. Several methods exploit assumptions specific to MNIST: centered foreground objects, simple black backgrounds, limited texture, and stable global orientation statistics [2602.20930] [2007.10588]. This suggests that RotatedMNIST is best treated as a controlled geometry benchmark rather than a complete proxy for real-world rotational variability.

Finally, the benchmark can reward different inductive biases depending on protocol. When train and test both contain full random rotations, augmentation-based methods can already approach ceiling performance; for example, conventional CNNs trained with rotation augmentation reach about 99.5% on MNIST-r in one CyCNN study, leaving little room for architectural gains [2007.10588]. By contrast, the unrotated-train, rotated-test setting sharply separates architecture-driven invariance from augmentation effects [2211.11812].

RotatedMNIST remains influential precisely because these distinctions are visible. It exposes whether a method is exploiting data augmentation, hard-coded symmetry, canonicalization, latent-variable optimization, or memory mechanisms, and it does so under a transformation that is mathematically simple but algorithmically revealing.

Source: https://www.emergentmind.com/topics/rotatedmnist