Permuted-MNIST Sequence Benchmark

• The paper shows the TCN's superior performance with 97.2% accuracy in capturing long-range dependencies, outperforming LSTM and GRU.
• It transforms MNIST images into permuted sequences that remove local spatial correlations, testing memory over 784 sequential timesteps.
• Empirical findings highlight that deep dilated causal convolutions and residual blocks in TCNs enhance gradient stability and accelerate convergence.

The Permuted-MNIST Sequence task is a widely adopted benchmark in sequence modeling that assesses the ability of learning algorithms to capture long-range dependencies in high-dimensional, non-natural input sequences. It is a variant of the canonical MNIST handwritten digit classification problem, employing a permutation of input indices to destroy local spatial coherence and requiring the model to integrate dispersed information over a long temporal horizon.

1. Task Definition and Data Preparation

The Permuted-MNIST Sequence task originates from the standard MNIST dataset, which consists of $28 \times 28$ grayscale images of handwritten digits ($10$ classes). To form sequence inputs, each MNIST image is flattened in raster (row-major) order, yielding a vector $\mathbf{x} = (x_0, x_1, \ldots, x_{783})$ of length $784$.

A single fixed random permutation $\pi$ of the indices $\{0, \ldots, 783\}$ is sampled uniformly at the start of the experiment and applied to every image. The permuted input sequence for each image is constructed as $x'_t = x_{\pi(t)}$ for $t = 0, \ldots, 783$. This permutation is held constant across all examples for a given experiment instance.

During training and evaluation, the model receives the sequence of $784$ scalar pixel values in this permuted order, with one value presented at each time step. After consuming the whole sequence, the model outputs a single 10-way softmax prediction $\hat{y}$. The objective is to minimize the cross-entropy loss between $10$0 and the true label $10$1 (Bai et al., 2018).

2. Architectural Benchmarks: Convolutional vs. Recurrent Approaches

The permuted input sequence ablates the local spatial correlations present in the unpermuted MNIST, transforming the problem into an integration challenge over long sequences. Conventional sequence models are evaluated under this setup. Notably, the Temporal Convolutional Network (TCN) has been systematically compared to canonical recurrent neural networks (RNNs), including Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU) architectures.

The TCN used for Permuted-MNIST is constructed as a deep stack of $10$2 one-dimensional causal, dilated convolutional layers, each with a kernel size $10$3 and $10$4 channels. Each layer employs exponentially increasing dilation factors $10$5 for $10$6. Residual connections are present between blocks, and every convolutional filter is normalized by weight normalization. All nonlinearity is realized by ReLU activation. No dropout or gradient clipping is applied in training. Optimization is performed using Adam with a learning rate of $10$7 and batch size $10$8 (Bai et al., 2018).

3. Technical Details: Dilated Causal Convolutions and Residual Blocks

The central operator in TCNs is the dilated causal convolution. Specifically, at layer $10$9 with dilation $\mathbf{x} = (x_0, x_1, \ldots, x_{783})$0 and filter $\mathbf{x} = (x_0, x_1, \ldots, x_{783})$1 of size $\mathbf{x} = (x_0, x_1, \ldots, x_{783})$2, the output at position $\mathbf{x} = (x_0, x_1, \ldots, x_{783})$3 is computed as:

$\mathbf{x} = (x_0, x_1, \ldots, x_{783})$4

with zero padding applied for $\mathbf{x} = (x_0, x_1, \ldots, x_{783})$5.

Each residual block comprises two successive layers of the sequence: (dilated causal convolution $\mathbf{x} = (x_0, x_1, \ldots, x_{783})$6 weight normalization $\mathbf{x} = (x_0, x_1, \ldots, x_{783})$7 ReLU), optionally succeeded by spatial dropout (the dropout rate is zero in this setup). Letting $\mathbf{x} = (x_0, x_1, \ldots, x_{783})$8 be the input and $\mathbf{x} = (x_0, x_1, \ldots, x_{783})$9 the output after transformation, the block provides:

$784$0

If $784$1 and $784$2 have mismatched channel dimensions, a $784$3 convolution $784$4 is used; otherwise, $784$5 (Bai et al., 2018).

4. Receptive Field Analysis and Memory Capacity

For a TCN with $784$6 layers, kernel size $784$7, and exponentially increasing dilations $784$8, the effective receptive field $784$9—the maximum number of timesteps in the past influencing any output—has the closed form

$\pi$0

For $\pi$1, $\pi$2, $\pi$3. This receptive field exceeds the sequence length ($\pi$4), ensuring that the TCN can access all input pixels when performing classification. A plausible implication is that the TCN's design permits information to propagate over the required number of steps for the task (Bai et al., 2018).

5. Empirical Results and Comparative Evaluations

Empirical evaluation on Permuted-MNIST demonstrates significant differences in performance between architectural paradigms. The following table summarizes the test accuracy and parameter counts for each model class, as reported by (Bai et al., 2018):

Model	#Params	Test Accuracy (%)
Vanilla RNN	≈70 K	25.3
LSTM	≈70 K	85.7
GRU	≈70 K	87.3
TCN	≈70 K	97.2

The TCN surpasses both vanilla and gated RNNs (LSTM, GRU) in final accuracy, despite having a comparable parameter budget and accessing each pixel only once in a fixed permutation. Key hyperparameters contributing to TCN performance include batch size $\pi$5, Adam optimizer with a $\pi$6 learning rate, and the absence of dropout.

6. Interpretations: Memory, Gradient Stability, and Model Selection

Despite RNNs' theoretical "infinite" memory, issues with vanishing or exploding gradients limit effective long-range dependency learning in practice, especially on long input sequences such as the length-784 Permuted-MNIST. LSTM and GRU architectures exhibit difficulty in retaining precise pixel-level information from earlier in the sequence.

The TCN, by virtue of its deep dilated convolutional structure and residual learning, achieves a much larger effective receptive field and more stable gradients over long temporal extents. Empirically, this leads to faster convergence and higher classification accuracy, not only on Permuted-MNIST but also on other long-memory benchmarks and real-world tasks such as the copy memory problem and the LAMBADA language modeling task (Bai et al., 2018).

Convolutional architectures with dilation and residual connections thus represent a robust alternative to canonical recurrent networks for sequence modeling tasks characterized by long-range dependencies. The research concludes that such architectures should be the natural starting point for sequence modeling when long-term memory is critical.