PermutedMNIST: A Continual Learning Benchmark

• PermutedMNIST is a synthetic benchmark that applies fixed random permutations to MNIST images, erasing local spatial structures.
• It evaluates continual learning by creating sequential tasks that measure performance drops and catastrophic forgetting in neural networks.
• Architectural studies show that while CNNs suffer from disrupted locality, fully connected networks maintain robustness, with methods like EWC mitigating forgetting.

PermutedMNIST is a canonical synthetic benchmark in the study of continual, incremental, and robust learning in neural networks. It systematically disrupts the spatial locality of the classic MNIST handwritten digit dataset by applying fixed randomized permutations of the input pixels, creating input distributions in which all local image structure is erased. PermutedMNIST plays a critical role in identifying the fragility of convolutional neural networks (CNNs) to non-naturalistic data and in the empirical assessment of algorithms for alleviating catastrophic forgetting in sequential learning contexts (Ivan, 2019, Loke et al., 14 Jul 2025).

1. Formal Definition and Generation

PermutedMNIST is derived from the standard MNIST dataset, which consists of $28\times28$ gray-scale images, typically represented as $x \in \mathbb{R}^{784}$. To generate a PermutedMNIST task, a fixed random permutation $\pi$ (or, more generally, a permutation matrix $P \in \{0,1\}^{784 \times 784}$) is drawn once and applied to every input vector, maintaining correspondence between training and test sets. For each image $x$, the permuted input $x'$ is given by

$x'_j = x_{\pi^{-1}(j)},\quad j = 1, \ldots, 784,$

ensuring all images in a given task are identically shuffled. For applications in continual or multitask learning, a sequence of tasks is generated by sampling distinct permutations $(\pi_1, \pi_2, \ldots)$, so each task presents a structurally unique version of MNIST while retaining the original label semantics (Loke et al., 14 Jul 2025).

2. Evaluation Contexts and Metrics

PermutedMNIST is predominantly used in two core evaluation settings:

• Robustness of architectures to non-local input structure: Here, the interest is in how inductive biases—especially convolutional locality—affect generalization when neighborhood relations are corrupted (Ivan, 2019).
• Continual learning with non-overlapping input distributions: Each PermutedMNIST task constitutes a distribution shift, stressing the network's ability to learn new mappings without overwriting prior knowledge, a phenomenon quantified as catastrophic forgetting (Loke et al., 14 Jul 2025).

Standard metrics include:

• Per-task accuracy $A_{t \gets k}$: Test accuracy on task $k$ after training up through task $x \in \mathbb{R}^{784}$0.
• Average retained accuracy $x \in \mathbb{R}^{784}$1 after $x \in \mathbb{R}^{784}$2 tasks.
• Forgetting measure $x \in \mathbb{R}^{784}$3, the drop in accuracy from after training on task $x \in \mathbb{R}^{784}$4 to after all subsequent tasks (Loke et al., 14 Jul 2025).

3. Architectural Benchmarks and Comparative Performance

Multiple architectures are benchmarked on PermutedMNIST:

• Convolutional Neural Networks: A VGG-like CNN with $x \in \mathbb{R}^{784}$5 convolutions and increasing channel depth ($x \in \mathbb{R}^{784}$6), feeding into fully-connected layers. On standard MNIST, such architectures achieve $x \in \mathbb{R}^{784}$7 test accuracy, but this drops to $x \in \mathbb{R}^{784}$8 on the permuted version. On CIFAR-10 and Fashion-MNIST permutations, the performance degradation is more marked (Ivan, 2019).
• Fully Connected Networks (MLPs): Architectures using only dense layers (e.g., two hidden layers of $x \in \mathbb{R}^{784}$9 or $\pi$0 units) are not adversely affected by permutation, showing robust generalization—$\pi$1 on vanilla MNIST versus $\pi$2 on PermutedMNIST (Ivan, 2019, Loke et al., 14 Jul 2025).
• Dilated Convolutions: To address the loss of local correlation, two layers of $\pi$3 dilated convolutions (dilation rate $\pi$4) partially restore performance, with accuracy gains (e.g., $\pi$5 on permuted CIFAR-10, up from $\pi$6 with regular CNNs) (Ivan, 2019).

Model	MNIST (Natural)	MNIST (Permuted)
CNN	99.5%	98.2%
MLP	98.7%	98.6%

4. Theoretical Implications and Inductive Biases

PermutedMNIST exposes the precise contribution of architectural priors, particularly the strong locality and translation invariance bias of CNNs. On natural images, convolutional kernels exploit statistical regularities of local pixel neighborhoods to construct hierarchical features. A global pixel permutation destroys all such local motifs; as a result, the receptive fields of small convolutional kernels sample essentially noise, with no meaningful structure to aggregate. The two components of performance in such architectures—optimization-based discrimination (shared with MLPs) and exploitation of local self-similarity (unique to convolutions)—reduce to the former only, where the convolutional prior becomes a liability (Ivan, 2019).

Dilated convolutions, by expanding the effective receptive field, can capture longer-range interactions introduced by the permutation, but this comes with potential trade-offs in parameter count and depth requirements. The inability of standard convolutions to adapt to such regimes highlights the inadequacy of architectures when their implicit priors are violated.

5. PermutedMNIST in Catastrophic Forgetting Studies

PermutedMNIST is a benchmark of choice for quantifying catastrophic forgetting in sequential training. The absence of shared local structure between tasks (distinct permutations per task) ensures that no convolutional or spatial continuity can be exploited to facilitate transfer, modeling true task interference. Experimental protocols commonly include 3-task and 10-task learning regimes. Key findings include:

• Standard SGD yields severe forgetting after multiple tasks—a drop from initial $\pi$7 accuracy to as low as $\pi$8 on the first task after learning ten permutations.
• Weight regularization methods like L2 reduce, but do not eliminate, forgetting (e.g., retaining $\pi$9 on Task 1 after 3 tasks).
• Elastic Weight Consolidation (EWC), which penalizes changes to parameters important for previous tasks using a Fisher information-based penalty, retains substantially more—$P \in \{0,1\}^{784 \times 784}$0 after 3 tasks and $P \in \{0,1\}^{784 \times 784}$1 after 10—at the cost of slightly reduced plasticity and slower adaptation to new tasks (Loke et al., 14 Jul 2025).

Method	Task-1 Accuracy After 3 Tasks	Accuracy After 10 Tasks	Forgetting (Task 1, 10 Tasks)
SGD	62%	46%	84%
L2	75%	~	~
EWC	90%	74%	34%

6. Insights from Dropout, Regularization, and Hyperparameters

Dropout, when added to EWC-regularized training (input $P \in \{0,1\}^{784 \times 784}$2, hidden $P \in \{0,1\}^{784 \times 784}$3), slightly reduces maximum per-task performance (by $P \in \{0,1\}^{784 \times 784}$4), but increases parameter flexibility and guards against overfitting to any single permutation. This leads to more robust generalization in sequential settings. The strength of the EWC penalty $P \in \{0,1\}^{784 \times 784}$5 directly trades off forgetting against learnability: larger values ($P \in \{0,1\}^{784 \times 784}$6) strongly protect previous knowledge, while smaller values ($P \in \{0,1\}^{784 \times 784}$7–$P \in \{0,1\}^{784 \times 784}$8) allow faster acquisition of new tasks but increase forgetting by $P \in \{0,1\}^{784 \times 784}$9 (Loke et al., 14 Jul 2025).

7. Broader Impact and Applications

PermutedMNIST has broader implications for architecture design in domains where locality is absent or intentionally obfuscated, such as encrypted or privacy-sensitive imaging, or synthetic data in which spatial adjacency carries no semantic weight. Its role as a continued benchmark persists in studies of continual, transfer, and meta-learning, due to its tunable difficulty and ability to isolate specific algorithmic weaknesses. Dilated or non-convolutional architectures may be preferable for settings resembling PermutedMNIST, where long-range pixel–pixel dependencies, rather than local patterns, predominate (Ivan, 2019).