Pretraining Recurrent Networks without Recurrence

Abstract: Training recurrent neural networks (RNNs) requires assigning credit across long sequences of computations. Standard backpropagation through time (BPTT) addresses this problem poorly: it is sequential in time, limiting parallelism, and suffers from vanishing or exploding gradients, making long-range associations difficult to learn. We propose Supervised Memory Training (SMT), a method for training nonlinear RNNs that sidesteps recurrent credit propagation entirely by reducing RNN training to supervised learning on one-step memory transition labels $(m_t, x_{t+1}) \rightarrow m_{t+1}$. SMT acquires these memory labels by training a Transformer-based encoder on a predictive state objective--retaining only information from the past necessary to predict the future. By decoupling what to remember from how to update memory, SMT enables time-parallel RNN training with a stable $O(1)$ length gradient path between any two tokens--without ever unrolling the RNN. We find that SMT outperforms BPTT when pretraining various RNN architectures on tasks like language modeling and pixel sequence modeling. SMT enables nonlinear RNNs to better capture long-range dependencies and train in parallel, potentially unlocking the scaling of models that build temporal abstractions of past experience.

• The paper's main contribution is SMT, which decouples memory representation from dynamics to achieve an O(1) gradient path and stable long-range dependency learning.
• It introduces a dual-phase approach with DMT to mitigate exposure bias, ensuring robust autoregressive performance during inference.
• Empirical results demonstrate SMT’s superior performance on synthetic benchmarks and real-world tasks, with predictable scaling and enhanced memory efficiency compared to BPTT.

Pretraining Recurrent Networks without Recurrence: A Summary

Introduction and Motivation

"Pretraining Recurrent Networks without Recurrence" (2606.06479) introduces Supervised Memory Training (SMT), a non-recurrent pretraining method for nonlinear RNNs. The central premise is the decoupling of memory representation ("what to remember") from memory dynamics ("how to update memory"), thereby sidestepping the limitations of Backpropagation Through Time (BPTT), particularly its long credit assignment path and gradient instability. Instead, SMT leverages predictive state representations derived from a time-parallel Transformer encoder-decoder, using these as direct supervision targets for the RNN’s one-step update.

Figure 1: BPTT vs SMT. SMT provides an $\mathcal{O}(1)$ gradient path between tokens, stabilizing training and facilitating learning of long-range dependencies.

The proposal is motivated by the observation that BPTT is both time-sequential and highly susceptible to vanishing/exploding gradients, which fundamentally restricts effective credit assignment in long-sequence tasks. In contrast, SMT provides a time-parallel training regime for nonlinear RNNs, reducing the longest credit path length from $\mathcal{O}(T)$ to $\mathcal{O}(1)$.

Methodology: SMT and DMT

Supervised Memory Training (SMT)

SMT reframes the RNN pretraining problem into supervised learning on memory transitions using memory states produced by an encoder (typically a Transformer). Formally, it constructs memory targets $m_t = \mathcal{E}_\phi(\mathbf{x}_{\leq t})$, with $\mathcal{E}_\phi$ trained to produce predictive states sufficient for future prediction via a decoder $\mathcal{D}_\psi$. The RNN then learns $f_\theta(m_t, x_{t+1}) \rightarrow m_{t+1}$ via MSE. To maintain discriminability, a uniformity loss is included.

This protocol decouples the memory bottleneck (compressed via the encoder) from temporal dynamics (modeled by the RNN), opening up full parallelism in training and a gradient topology that mitigates vanishing and exploding phenomena.

DAgger Memory Training (DMT)

While SMT reduces one-step error for the RNN, autoregressive unrolling at inference leads to compounding drift due to train-test mismatch ("exposure bias"). A lightweight DMT phase analogous to on-policy imitation (DAgger) is employed post-SMT: here, the RNN is exposed to its own rollout distribution and is trained to align its generated trajectory with the encoder’s reference trajectory, further minimizing accumulated error.

Empirical Evaluation

Synthetic Benchmarks

On a diverse suite of synthetic benchmarks probing gradient stability, memory capacity, state tracking, associative recall, and in-context learning, SMT→DMT outperforms BPTT across all tested dimensions and task settings. BPTT struggles with long sequences, memory utilization, and credit assignment, whereas SMT remains robust to sequence length, with performance largely agnostic to horizon in all but associative recall.

Figure 2: SMT→DMT dominates BPTT across synthetic measures relevant to both gradient flow and memory utilization.

Realistic Domains: Language and Pixel Sequence Modeling

For character-level next-token prediction (TinyStories) and sparse pixel sequence modeling (MNIST, Sketchy), SMT-pretrained RNNs surpass BPTT baselines given identical compute and data constraints. Notably, in challenging rasterized image modeling, BPTT RNNs with both vanilla and gated (GRU) architectures fail to model global structure, whereas SMT-pretrained RNNs generate globally coherent samples and effectively propagate long-range dependencies, as visualized in sequence generations.

Figure 3: SMT and SMT→DMT show superior sequential compute and, on MNIST, superior data efficiency compared to BPTT, especially as depth increases.

Scaling Laws and Compression

The SMT approach exhibits classic scaling behaviors: increasing context length and memory dimensionality yield monotonic performance gains; growing model capacity allows the RNN to better match the encoder teacher, with the performance gap narrowing with scale. Crucially, under fixed loss, additional compute resources allow further compression of the memory state, linking compute and information efficiency—relevant for practical instantiations where fixed-memory (bounded hardware) inference is a requirement.

Figure 4: Scaling context and memory size under SMT→DMT leads to predictable improvements in TinyStories perplexity.

Ablations and Analysis

Credit Assignment and Gradient Flow

Empirical gradient analysis on synthetic tasks (e.g., "needle retrieval") confirms that BPTT gradients vanish rapidly over time, especially with increasing sequence length or unfavorable initialization, while SMT gradients remain initialization-agnostic and uniformly distributed across all timesteps. This key property enables SMT to avoid BPTT’s recency bias, supporting effective long-horizon credit assignment.

Markovianity and Memory Geometry

Markovianity of the learned memory states—ideal for RNN operation—is encouraged by including the explicit transition supervision in SMT but is, in principle, already induced by the predictive state objective. Empirical visualizations show task-dependent geometry in the latent memory manifold: effective state minimization in retrieval and FSM induction, tree-like embeddings for string copy, and more complex behavior for higher-level tasks.

Figure 5: Memory space geometry learned by SMT varies with task: FSM-like collapse for retrieval, tree-like expansion for string copy.

Implications and Future Directions

SMT enables fixed-memory, nonlinear RNNs to be trained using time-parallel architectures, removing limitations associated with gradient flow in BPTT. This not only improves computational efficiency and scalability for longer sequences but also aligns deep sequence models with the compressed memory paradigm found in biological systems, in contrast to unbounded-history attention mechanisms of standard Transformers. SMT is particularly well suited for pretraining, after which further adaptation (post-training or even BPTT-based fine-tuning) can augment expressivity and task-specific performance.

A significant theoretical implication is that the expressivity of an SMT-pretrained RNN is upper bounded by the capacity of its encoder-decoder teacher; downstream adaptation is necessary for tasks requiring computation depth beyond the teacher’s training context or representational class. Another important result is the robust length generalization of SMT-pretrained RNNs: in synthetic state tracking, they generalize more strongly to sequence lengths beyond training than their Transformer teachers—a critical property for lifelong or open-horizon RL domains.

This study raises questions regarding optimal encoder design for specific downstream RNN architectures, curriculum learning for memory state induction, and interaction with newer recurrence-based architectures (e.g., Mamba, RNN hybrids). Moreover, it suggests a revision of current assumptions about the necessity of BPTT in high-capacity, resource-constrained sequence modeling.

Conclusion

This work establishes SMT as an effective, parallelizable paradigm for pretraining nonlinear RNNs, outperforming BPTT in both synthetic and real-world tasks, and enabling fixed-memory architectures to learn and utilize long-range dependencies. The combination of SMT and DMT provides a compelling solution to the train-test mismatch and gradient instability issues plaguing traditional RNN training pipelines. Future efforts should investigate scalable post-training, extension to additional architectures, and principled integration with current state-of-the-art sequence models.