Create and explore imaginative 3D worlds with our new product Dreamfield (opens in a new tab)
Papers
Topics
Authors
Recent
Search
2000 character limit reached

A unified theory of feature learning in RNNs and DNNs

Published 17 Feb 2026 in cs.LG and cond-mat.dis-nn | (2602.15593v1)

Abstract: Recurrent and deep neural networks (RNNs/DNNs) are cornerstone architectures in machine learning. Remarkably, RNNs differ from DNNs only by weight sharing, as can be shown through unrolling in time. How does this structural similarity fit with the distinct functional properties these networks exhibit? To address this question, we here develop a unified mean-field theory for RNNs and DNNs in terms of representational kernels, describing fully trained networks in the feature learning (μμP) regime. This theory casts training as Bayesian inference over sequences and patterns, directly revealing the functional implications induced by the RNNs' weight sharing. In DNN-typical tasks, we identify a phase transition when the learning signal overcomes the noise due to randomness in the weights: below this threshold, RNNs and DNNs behave identically; above it, only RNNs develop correlated representations across timesteps. For sequential tasks, the RNNs' weight sharing furthermore induces an inductive bias that aids generalization by interpolating unsupervised time steps. Overall, our theory offers a way to connect architectural structure to functional biases.

Summary

  • The paper develops a Bayesian mean-field kernel theory showing that RNNs and DNNs differ through a masking operation on temporal correlations, with predictions validated against finite-width SGLD simulations.
  • The theory identifies a second-order transition exclusive to RNNs: for a four-step linear model, temporal coherence emerges at critical signal strength λ*=8 with order-parameter exponent β=1/2.
  • The paper shows that recurrent weight sharing propagates label information across unsupervised timesteps, enabling RNNs to generalize from fewer supervised sequence steps despite having a smaller hypothesis class than DNNs.

The paper develops a mean-field theory that treats recurrent neural networks (RNNs) and deep feedforward networks (DNNs) within a single framework for feature learning, describing fully trained networks in the maximal-update-parametrization (μ\muP) regime in terms of representational kernels (2602.15593). The central observation is architectural: unrolling an RNN in time yields a DNN whose layers correspond to timesteps, with the sole structural difference being weight sharing across layers. The theory makes this difference explicit as a masking operator on temporal correlations and derives its functional consequences: a signal-strength-controlled phase transition toward temporally coherent representations unique to RNNs, and an interpolation-based inductive bias that gives RNNs a generalization advantage on sequential tasks.

Unified kernel theory of trained RNNs and DNNs

The architecture under consideration is ht=W(t−1)ϕ(ht−1)+Uxt−1h^t = W^{(t-1)}\phi(h^{t-1}) + Ux^{t-1}, ft+1=Vϕ(ht)f^{t+1} = V\phi(h^t), where W(t)W^{(t)} is shared over tt for an RNN and layer-dependent for a DNN. Training is modeled via stochastic gradient Langevin dynamics (SGLD) — gradient descent on a mean-squared loss plus i.i.d. Gaussian noise of strength 2K\sqrt{2K} and weight decay — whose stationary distribution coincides with the Bayesian posterior combining a Gaussian weight prior (with variances GU∝u/DG_U \propto u/D, GW∝w/NG_W \propto w/N, GV∝v/N2G_V \propto v/N^2) and a likelihood equivalent to noisy labels. This recovers the μ\muP scaling from two requirements: bounded pre-activations and ht=W(t−1)ϕ(ht−1)+Uxt−1h^t = W^{(t-1)}\phi(h^{t-1}) + Ux^{t-1}0 changes to the representation kernel.

Marginalizing out the weights yields a posterior over hidden activities,

ht=W(t−1)ϕ(ht−1)+Uxt−1h^t = W^{(t-1)}\phi(h^{t-1}) + Ux^{t-1}1

where the representational kernel ht=W(t−1)ϕ(ht−1)+Uxt−1h^t = W^{(t-1)}\phi(h^{t-1}) + Ux^{t-1}2 self-averages at large width ht=W(t−1)ϕ(ht−1)+Uxt−1h^t = W^{(t-1)}\phi(h^{t-1}) + Ux^{t-1}3 and concentrates on its posterior expectation. Two conjugate "tilts" structure this distribution: ht=W(t−1)ϕ(ht−1)+Uxt−1h^t = W^{(t-1)}\phi(h^{t-1}) + Ux^{t-1}4 pulls representations toward labels, while ht=W(t−1)ϕ(ht−1)+Uxt−1h^t = W^{(t-1)}\phi(h^{t-1}) + Ux^{t-1}5 enforces consistency with the forward pass under random weights. The only architectural distinction between RNN and DNN enters through the masking operator ht=W(t−1)ϕ(ht−1)+Uxt−1h^t = W^{(t-1)}\phi(h^{t-1}) + Ux^{t-1}6, which preserves all temporal correlations for the RNN but projects onto the diagonal for the DNN.

Empirical validation uses SGLD training in weight space compared against the theoretical kernels via centered kernel alignment (CKA), showing convergence already at large finite widths for a sinusoidal regression task with erf activation. The trained kernels exhibit non-trivial off-diagonal temporal structure that the theory reproduces accurately, including along the anti-diagonal autocorrelation.

Phase transition to temporal coherence

For endpoint-supervised tasks — input at the first layer, supervision at the last — typical of DNNs, the theory reveals two regimes separated by a critical learning-signal strength ht=W(t−1)ϕ(ht−1)+Uxt−1h^t = W^{(t-1)}\phi(h^{t-1}) + Ux^{t-1}7 relative to the prior scales ht=W(t−1)ϕ(ht−1)+Uxt−1h^t = W^{(t-1)}\phi(h^{t-1}) + Ux^{t-1}8. In the weak-signal regime, the diagonal (temporally incoherent) solution is the unique stationary point, and RNN and DNN kernels coincide — although both differ from the lazy NNGP kernels at initialization, so feature learning occurs even below threshold. Above the critical ht=W(t−1)ϕ(ht−1)+Uxt−1h^t = W^{(t-1)}\phi(h^{t-1}) + Ux^{t-1}9, only the RNN admits additional stationary points with non-vanishing off-diagonal kernel entries: a temporally coherent phase in which inner products between hidden states at distinct timesteps align, accompanied by an outlier eigenvalue in the recurrent weight spectrum. The pattern-by-pattern kernel at the last timestep, which determines predictor statistics via ft+1=Vϕ(ht)f^{t+1} = V\phi(h^t)0, retains qualitatively similar class-block structure in both architectures; hence predictors differ mainly in scale, not form.

Analytical treatment of the minimal case ft+1=Vϕ(ht)f^{t+1} = V\phi(h^t)1 with linear activation establishes this as a second-order phase transition with mean-field critical exponent ft+1=Vϕ(ht)f^{t+1} = V\phi(h^t)2: the off-diagonal order parameter satisfies ft+1=Vϕ(ht)f^{t+1} = V\phi(h^t)3 near the exactly computed threshold ft+1=Vϕ(ht)f^{t+1} = V\phi(h^t)4. A Landau expansion of the effective action ft+1=Vϕ(ht)f^{t+1} = V\phi(h^t)5 exposes an energy–entropy mechanism. In both architectures, label fitting raises the final-timestep variance ("energy"), penalizing the KL term against the NNGP forward-pass covariance. In the DNN, the masking prevents temporal correlations from compensating this penalty, so the entropic ft+1=Vϕ(ht)f^{t+1} = V\phi(h^t)6 term keeps them suppressed at any ft+1=Vϕ(ht)f^{t+1} = V\phi(h^t)7. In the RNN, correlations can offset the variance increase, and once ft+1=Vϕ(ht)f^{t+1} = V\phi(h^t)8 is large enough their marginal entropic cost falls below the KL cost, destabilizing the diagonal solution. Direct SGLD simulations track the theoretical order-parameter curve closely.

Inductive bias for sequential tasks

On sequential tasks ft+1=Vϕ(ht)f^{t+1} = V\phi(h^t)9 — instantiated minimally as a teacher RNN rotating its input by a fixed angle per timestep, learned from a single pattern with varying numbers of supervised timesteps — the functional consequences become sharper. Because the number of patterns is one, supervised timesteps play the role of sample count, and RNNs reach low generalization error from markedly fewer supervised steps than DNNs. Mechanistically, the DNN's posterior covariance remains diagonal outside supervised timesteps, so prediction there reverts to the prior mean (W(t)W^{(t)}0); the RNN instead develops correlations in W(t)W^{(t)}1 matching the target kernel after few samples, accompanied by recovery of the teacher's conjugate eigenvalue pair in W(t)W^{(t)}2.

A perturbative analysis in label strength around the diagonal NNGP kernel identifies the underlying mechanism. At second order in W(t)W^{(t)}3, the RNN correction takes the form

W(t)W^{(t)}4

which propagates label information across unsupervised timesteps through resolvent propagators — effectively interpolating between observed points. The corresponding DNN expression contains only entry-wise products and vanishes for pairs involving unsupervised timesteps. Equivalently, the first-order response inherits eigenvector structure from W(t)W^{(t)}5 that the DNN's second-order masking destroys. Notably, this advantage arises despite the RNN's hypothesis class being a strict subset of the DNN's, illustrating that task-model alignment depends on adaptivity to task structure, not expressivity alone — consistent with no-free-lunch reasoning.

Limitations and open questions

The authors state several concessions directly. The theory marginalizes over weights and therefore abstracts away learning dynamics and weight-space structure, describing only converged networks. SGLD replaces structured mini-batch noise in SGD with i.i.d. noise, an approximation whose fidelity in practice remains untested here. The analytical results rely on linear activations and orthogonal inputs in deliberately minimal settings; the phase transition persists for nonlinear activations, but proportional limits such as W(t)W^{(t)}6, pattern-structure effects, and cumulative nonlinearity across depth are not analyzed even though the framework could in principle accommodate them. Open questions include whether the exact critical W(t)W^{(t)}7 and exponent generalize beyond W(t)W^{(t)}8 — no closed-form solution exists for larger depths — and how the identified mechanisms interact with correlated structure over patterns.

Conclusion

This work provides a unified, Bayesian-inference-based kernel theory for trained RNNs and DNNs in which weight sharing reduces to a masking operation on temporal correlations. It yields three concrete results: equivalence of the two architectures' representations below a critical learning-signal strength, an analytically characterized second-order transition to temporal coherence exclusive to RNNs with critical point W(t)W^{(t)}9 at depth four, and a message-propagation mechanism by which weight sharing interpolates labels across unsupervised timesteps to improve sample efficiency on sequential tasks. The framework offers a principled route from architectural constraints to functional inductive biases.

Paper to Video (Beta)

No one has generated a video about this paper yet.

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Tweets

Sign up for free to view the 2 tweets with 5 likes about this paper.