Weight-Space Adaptive Recurrent Prediction

• Weight-Space Adaptive Recurrent Prediction is a modeling paradigm that represents the recurrent state as evolving parameters of a root network, enabling high-fidelity memory and meta-adaptation.
• Its linear recurrent update unrolls as a convolution, decoupling nonlinearity to the decoding stage and facilitating efficient parallel analysis.
• Empirical evaluations across tasks like image completion, forecasting, and dynamical systems demonstrate its superior performance and gradient-free test-time adaptation.

Weight-Space Adaptive Recurrent Prediction (WARP) is a modeling paradigm in which the “hidden state” of a sequence model is explicitly represented as the evolving weights of a subordinate (“root”) neural network rather than a fixed-dimensional latent vector. This approach eliminates the conventional dichotomy between fixed neural parameters and learned temporal dynamics by recasting the recurrent “state” in terms of weight-space evolution, enabling high-fidelity memory, gradient-free test-time adaptation, and seamless incorporation of domain-specific priors. WARP’s generic linear recurrence in weight-space unifies concepts from recurrence, state-space modeling, meta-learning, and continual learning, and is empirically validated across tasks in classification, forecasting, sequence completion, and dynamical system reconstruction (Nzoyem et al., 1 Jun 2025).

1. Weight-Space State Representation and Recurrence

Traditional recurrent neural networks (RNNs) maintain a hidden state $h_t$ with updates of the form $h_t = f(h_{t-1}, x_t)$, blending input and memory via nonlinear transformation. In WARP, the hidden state is the parameter vector $\theta_t$ of a root network, which evolves according to a linear relation:

$$\theta_t = A \cdot \theta_{t-1} + B \cdot \Delta x_t, \qquad \Delta x_t = x_t - x_{t-1}$$

where $A$ and $B$ are learnable transition matrices and the initial state is provided as

$\theta_0 = \varphi(x_0)$

via an initial encoding network. Output at each time step is produced by evaluating the root network parameterized by $\theta_t$ at a normalized time variable $\tau$,

$y_t = \text{RootNet}_{\theta_t}(\tau)$

with $\tau = t/(T-1)$ for a sequence of length $T$. This weight-space “state” enables recurrent tracking of arbitrarily rich temporal dependencies.

2. Linear Evolution and Decoupling of Nonlinearity

The WARP update is fundamentally linear in the weight-space variables, in contrast to classical RNN updates where recurrences are rendered nonlinear by gates and activation functions. The strictly linear transition admits unrolling:

$$\theta_t = A^t \theta_0 + \sum_{\ell=0}^{t-1} A^\ell B \Delta x_{t-\ell}$$

which can be interpreted as a convolution:

$\theta_{0:T} = K * \Delta x_{0:T}$

with convolutional kernel $K = [B, AB, A^2B, \dots, A^{T-1}B]$. Nonlinearity enters exclusively at the “decoding” stage—mapping weights back into output space—either through generic root neural network models or through specifically parameterized, domain-informed decoders. This division preserves expressive representation while facilitating efficient parallel decoding and analysis.

3. Adaptive Memory and Test-Time Dynamics

Representing recurrent state as network weights confers extraordinarily high memory resolution, as the evolving $\theta_t$ spans a space of much higher dimensionality than standard latent states. Empirical evidence indicates that WARP’s weight trajectory, when visualized via principal component analysis (PCA), closely mimics the iterative path of gradient descent—implying the recurrence embodies a form of dynamic, meta-learned adaptation in parameter space. The model exhibits gradient-free adaptation at test time; it can update its internal representation in response to incoming data without runtime backpropagation, adjusting to regime shifts or out-of-distribution conditions via deterministic weight evolution.

4. Integration of Domain-Specific Physical Priors

WARP’s architecture facilitates integration of explicit inductive biases by parameterizing the output of the root network to encode scientific or domain knowledge. For example:

• For oscillatory dynamics (e.g., sine wave tracking), the root network generates a phase parameter $\phi$ so the model output becomes $\sin(2\pi \tau + \phi)$.
• For mass-spring-damper (MSD) systems, outputs are formed via $E(\tau)x_0$ where $E(\tau) = \exp(\tau A)$ is the matrix exponential representing physical propagation. Hard-coding such physically informed forms into the root decoder results in improved generalization, lower sample complexity, and interpretable model behavior, especially in scientific machine learning tasks.

5. Empirical Performance across Sequential Tasks

Comprehensive experiments validate WARP’s capability:

• Image sequence completion (MNIST, Fashion MNIST, CelebA): WARP matches or surpasses GRU, LSTM, S4, and ConvCNP in mean squared error (MSE) and bits-per-dimension (BPD).
• Energy forecasting (ETT benchmark): WARP demonstrates comparable or superior accuracy to time series transformers.
• Dynamical systems reconstruction (MSD, MSD-Zero, Lotka–Volterra, SINE): In both vanilla and physically informed modes, WARP significantly outperforms classical RNNs in MSE and mean absolute error (MAE).
• Time series classification (UEA archive, Spirals): WARP maintains robust performance rivaling neural controlled differential equations (Neural CDEs) and domain-specialized models. This breadth of validation underlines WARP’s generality for both canonical and domain-specific sequential prediction.

6. Analysis of Weight Trajectories

The explicit modeling of hidden state as weight trajectories allows unprecedented insight into model memory. PCA visualizations show that weight evolution in WARP traces a path analogous to steps in gradient descent, suggesting that the recurrence is “meta-learning” adaptation dynamics. Correlation structures between $\theta_t$ and normalized time $\tau$ reveal strong linear relationships, facilitated by the diagonal readout $y_t = \theta_t(\tau)$, enabling temporal interpretability. This explicitness is not attainable in classical latent-state RNNs, providing new avenues for analysis and scientific interpretability.

7. Ablation and Architectural Insights

Critical components of WARP are substantiated via extensive ablations:

• Removal of root network decoding results in catastrophic failure (SINE reconstruction), demonstrating the necessity of explicit weight-space representation.
• Fixed vs. variable evaluation points in the decoder: Disallowing $\tau$ variability mildly degrades results, confirming the importance of diagonal time-alignment in decoding.
• Use of “initial network” encoding: Directly optimizing $\theta_0$ (bypassing the initial network $\varphi$) significantly impairs performance on complex datasets, evidencing the value of input-to-weight-space mapping.
• Reduction of transition matrices $A$ to simplified (diagonal/low-rank) forms reduces expressivity and accuracy; dense, full-rank $A$ is essential for complex temporal structures.
• Removing stochastic reparameterization during training impairs high-frequency signal reconstruction, highlighting its importance for capturing fine patterns. These analyses identify weight-space state, initial encoding, flexible evaluation, and full matrix transitions as necessary for WARP’s performance.

Conclusion

WARP—Weight-Space Adaptive Recurrent Prediction—unifies sequence modeling, meta-learning, and scientific machine learning by recasting the recurrent state as an explicitly evolving parameter vector of a functional “root” network. Its architecture leverages linear weight recurrences, gradient-free test-time adaptation, inductive prior integration, and weight-space interpretability. Empirical and ablation results validate its competitive or superior performance across a spectrum of machine learning tasks, signifying a substantive shift in sequential modeling frameworks (Nzoyem et al., 1 Jun 2025).