Permuted MNIST: Invariance and Robustness

• Permuted MNIST is a benchmark that uses a fixed random permutation of MNIST pixels to eliminate natural spatial correlations and test model robustness.
• Studies show that while multilayer perceptrons maintain similar performance on permuted images, convolutional networks suffer accuracy drops due to disrupted local features.
• Advanced methods like permutation-equivariant layers and differentiable permutation learning enhance performance by recovering latent structures in highly unstructured data.

Permuted MNIST refers to a benchmark in which the pixels of each image in the MNIST dataset are rearranged according to a fixed random permutation. This transformation destroys the original spatial correlations intrinsic to the natural MNIST images, creating a task that critically tests the permutation invariance and robustness of machine learning models. Permuted MNIST is frequently used to evaluate the ability of neural architectures—particularly multilayer perceptrons (MLPs), convolutional neural networks (CNNs), and permutation-equivariant models—to extract meaningful representations from highly unstructured or non-local data.

1. Task Definition and Mathematical Formulation

Permuted MNIST modifies the standard MNIST dataset, whose images $x \in \mathbb{R}^{28 \times 28}$, by applying a fixed random permutation $\pi$ to the pixel indices. The permutation is typically represented in Cauchy's two-line notation as

$$\sigma = \begin{pmatrix} 1\quad 2\quad 3\quad \dots\quad n^2\ \pi(1)\quad \pi(2)\quad \pi(3)\quad \dots\quad \pi(n^2) \end{pmatrix}$$

where $n=28$ for MNIST, and $\pi$ is sampled uniformly from the symmetric group $S_{n^2}$ and held fixed for all images in the dataset. The permuted image $x'$ is then such that $x'_{\pi(i)} = x_{i}$ for every pixel index $i$. The classification labels remain unchanged.

The task thus becomes: given $x'$ (with lost spatial locality), predict the correct digit label $y \in \{0,\dots,9\}$.

2. Architectural Implications: MLPs vs CNNs

The destruction of locality in Permuted MNIST has a pronounced impact on architectural choices:

• MLPs: Because fully connected layers treat each input feature independently and do not assume any spatial structure, their classification accuracy remains nearly unaffected after permutation. Experimental results show that MLPs achieve similar performance on both standard and permuted MNIST, indicating their insensitivity to input feature order (Ivan, 2019).
• CNNs: Convolutional networks rely on local receptive fields and hierarchically learn spatial features. When the pixel permutation is applied, local neighborhoods are disrupted, and the overlapping filters in convolutional layers lose their semantic meaning. Test accuracy for CNNs drops starkly on permuted data; for instance, standard CNNs can fall from 99.5% (natural MNIST) to 98.2% (permuted MNIST), and from ~89% to ~57% on CIFAR10 when subjected to full random permutation (Ivan, 2019). Dilated convolutions, which artificially widen receptive fields by spacing kernel elements, partially recover long-range dependencies and improve accuracy on permuted versions; experimental data show an increase from ~57% to ~62% on permuted CIFAR10 with dilated layers.

This highlights the architectural rigidity of CNNs and the structural flexibility of MLPs on unstructured inputs.

3. Permutation-Equivariant and Permutation-Invariant Models

Permutation invariance and equivariance are crucial design principles for models addressing tasks like Permuted MNIST:

• Permutation-Invariant Functions: $f(\pi x) = f(x)$ for any permutation $\pi$.
• Permutation-Equivariant Functions: $g(\pi x) = \pi g(x)$ for any permutation $\pi$.

Permutation-equivariant layers are obtained by parameter-sharing to enforce symmetry. A fully connected equivariant layer requires its weight matrix $\Theta$ to commute with all permutation matrices, which is proven to be achieved only if

$\Theta = \lambda I + \gamma (11^T)$

with $\lambda,\gamma \in \mathbb{R}$, $I$ the identity, and $1$ the all-ones vector (Ravanbakhsh et al., 2016).

Variants such as

$f(x) = \sigma(\lambda x - \gamma (\max_n x_n) 1)$

(max-normalization) often yield superior performance.

Explicit incorporation of these symmetry constraints enables networks to remain robust to input order and enhances generalization when permuted structures form, as in MNIST-digit summation tasks—with permutation-invariant models reaching >80% accuracy for $N=6$ digit sets, while non-invariant architectures falter (Ravanbakhsh et al., 2016).

4. Differentiable Permutation Learning

Permutation-optimization modules have emerged to learn optimal orderings end-to-end. The central idea is to define a pairwise ordering cost $F(x_i,x_j)$, forming an anti-symmetric matrix of preferences, and to optimize a (soft) permutation matrix $P$ (via the Sinkhorn operator) so as to minimize a cost function $c(P)$ that reflects the “goodness” of the permutation for the downstream task (Zhang et al., 2018):

$$P = S(\tilde{P}), \quad \tilde{P}^{(t+1)} = \tilde{P}^{(t)} - \eta \frac{\partial c(P^{(t)})}{\partial P^{(t)}}$$

Such modules have demonstrated state-of-the-art performance on sorting tasks, image mosaic reconstruction, and visual reasoning. Application to permuted MNIST suggests that a model might learn to recover or exploit latent pixel structure by optimally reordering the scrambled inputs, potentially enhancing classification even with severely altered spatial arrangements.

5. Adversarial Robustness: Permutation Phase Defense

Permutation Phase Defense (PPD) leverages random pixel permutation (using a secret permutation seed) combined with Fourier phase representation to thwart adversarial attacks (Jafarnia-Jahromi et al., 2018). The randomized permutation acts analogously to a cryptographic key, making it unfeasible for an adversary to anticipate the spatial transformation applied to the input. Transforming permuted images into their phase domain further obscures structural information.

Empirical results on MNIST indicate that PPD maintains test accuracy above 96% on clean images, and ensembles with varied seeds achieve nearly 98%. Under strong adversarial perturbations (FGSM, BIM, PGD, MIM), PPD’s accuracy degrades gradually, matching the drop seen with random noise, and outperforms defenses such as PGD adversarial training, especially under high $\ell_\infty$ attack budgets.

6. Permutation Invariance in Loss Landscapes

Recent work demonstrates that independently trained fully connected networks (via SGD from different initializations) can be aligned using neuron permutations such that their weight interpolation forms a path in parameter space with no significant loss barrier (Benzing et al., 2022). This leverages the inherent permutation invariance of the architecture and can be extended to ensembles trained on different permutations of the input (e.g., various versions of Permuted MNIST), providing a rationale for averaging aligned weights.

In contrast, convolutional architectures do not exhibit this valley connectivity after permutation alignment, and their solutions remain more diverse, particularly under large learning rates.

7. Re-Synchronization for Integrity and Watermarking

Permuted MNIST models, especially multi-layer perceptrons and other architectures with redundant solutions, are vulnerable to integrity attacks via neuron permutations. A robust method for re-synchronization matches the internal weight vectors using cosine similarity to estimate and invert the permutation matrix, realigning both the permuted layer and subsequent channels (Trias et al., 2023). This approach maintains high re-synchronization success rates, even with fine-tuning, quantization, or pruning, and is critical for extracting embedded watermark signatures in white-box watermarking applications.

Cosine-similarity-based matching is more resilient than parameter norm ranking, particularly under adversarial changes that preserve directional weight information. Combined with $\ell_2$-norm monitoring, this dual approach counters attempts to degrade model integrity via scalar modifications.

8. Permutation Equivariance in Quantum Machine Learning

Permutation-equivariant quantum convolutional neural networks (EQCNNs) encode permutation invariance at the quantum circuit level. By designing architectures that commute with all elements of $S_n$ or relevant subgroups (via shared two-qubit operations and careful pixel-to-qubit embedding), EQCNNs achieve classification accuracy on MNIST and graph states that exceeds non-equivariant quantum models (Das et al., 28 Apr 2024).

Maintaining permutational symmetry in EQCNNs not only facilitates generalization for permuted datasets, but also reduces overfitting and optimizes learning efficiency. The permutation structure, explicitly protected in the circuit design, ensures that label invariance is upheld under arbitrary input rearrangement and supports future extensions to complex permutation symmetries and high-dimensional quantum inputs.

Summary Table: Architectural Performance on Permuted MNIST

Architecture	Sensitivity to Permutation	Typical Test Accuracy (Permuted)
Standard CNN	High	98.2% (MNIST), ~57% (CIFAR10)
Dilated CNN	Moderate	up to ~62% (CIFAR10)
MLP	Low	Nearly unchanged
Permutation-equivariant Layer	Low	>80% (for multi-digit summation)
Permutation-Optimization Module	Low	State-of-the-art (varies)

The Permuted MNIST benchmark elucidates the importance of permutation invariance in model design, highlights limitations of spatially biased architectures, and motivates innovative solutions from parameter sharing and optimized permutations to robust phase-based defenses and quantum symmetry protection. It remains a crucial tool for probing the capacity of modern learning systems to operate on highly non-local, unstructured data.