Temporal Credit Is Free

Abstract: Recurrent networks do not need Jacobian propagation to adapt online. The hidden state already carries temporal credit through the forward pass; immediate derivatives suffice if you stop corrupting them with stale trace memory and normalize gradient scales across parameter groups. An architectural rule predicts when normalization is needed: \b{eta}2 is required when gradients must pass through a nonlinear state update with no output bypass, and unnecessary otherwise. Across ten architectures, real primate neural data, and streaming ML benchmarks, immediate derivatives with RMSprop match or exceed full RTRL, scaling to n = 1024 at 1000x less memory.

• The paper validates immediate gradient computations for online learning in recurrent networks, substituting full-jacobian propagation
• Increased the significant reduction of memory from O(n^3) to O (n^2), demonstrated key optimizations such as 1000x faster training with immediate gradients
• Key to this is properly calibrating optimizer and trace decay parameters to eliminate the warning signal of the gradient and thereby careful consideration of the memory optimal training

Overview

This paper argues that Jacobian propagation—the defining cost of real-time recurrent learning (RTRL)—is unnecessary for online adaptation in recurrent networks. The author's central claim is that the hidden state already carries temporal information through the forward pass, so the immediate derivative, computed through that state, suffices for gradient-based online learning. What has historically been framed as a temporal credit assignment problem is reframed as a gradient scaling problem: eligibility traces fail not because they lack the Jacobian term $J_t P_{t-1}$, but because of two miscalibrations—trace decay that overweights stale gradients by 85× relative to the measured per-neuron self-propagation factor, and the absence of per-parameter normalization in the face of a 100× gradient scale gap between recurrent and output weights (2603.28750).

The practical consequence is stark: immediate derivatives with RMSprop or Adam match or exceed full RTRL across ten architectures, primate BCI data, and streaming benchmarks, while reducing memory from $O(n^3)$ to $O(n^2)$—at $n=1024$, 12.6 MB versus 12.9 GB for full RTRL, a 1000× reduction.

The trace decay mismatch

The investigation began with an accidental observation: propagating a single random Jacobian column ($k=1$) recovered 86% of full RTRL performance at $n=64$. Due to a ring-mask implementation, this configuration computes only the immediate derivative with zero temporal propagation. If zero propagation recovers most of the signal, standard eligibility traces—which also discard the Jacobian—should work; their failure demands another explanation.

The explanation is quantitative. In a trained vanilla RNN ($n=64$), the measured per-neuron self-propagation factor $\text{diag}(1-h_t^2)\cdot\text{diag}(W_{hh})$ is approximately 0.01 per step. The paper is careful here: although the spectral radius of the full Jacobian is much higher (0.37–0.93), scalar trace decay is a per-neuron approximation and should be compared against the diagonal, not the spectrum. With $\lambda = 0.95$, information from ten steps ago is weighted at $\approx 0.60$ by the trace while its true influence is $O(n^3)$0. At timestep 500, the accumulated trace is 6.7× larger than the immediate derivative and only weakly aligned with it (cosine similarity 0.577). The trace supplies stale gradients at incorrect scale rather than useful temporal credit.

A decay sweep on a sine frequency-shift task confirms this empirically. Decay values from 0.001 to 0.5 yield successful adaptation (102–130% recovery at $O(n^3)$1; 163–200% at $O(n^3)$2), where recovery above 100% indicates outperforming the full RTRL reference. Decay 0.95 produces exactly 0% recovery on every seed at every scale, with a sharp transition near 0.7 (77 ± 56%, indicating threshold behavior). The pattern generalizes: on cross-session BCI data, decay 0.0 achieves 74% recovery versus 46% for 0.95; on Lorenz chaos, 113% versus 0%; on a character-level language domain shift, decay 0.0 matches BPTT to three decimal places (cross-entropy 2.716 vs 2.718). One nuance deserves note: at $O(n^3)$3, decay 0.5 outperforms decay 0.001 by 28 percentage points, suggesting mild regularization from residual trace memory at small scales—but this advantage vanishes by $O(n^3)$4, motivating zero as the recommended default.

The role of second-moment normalization

Correcting the decay alone is insufficient: with $O(n^3)$5, SGD recovers −1%, no better than a frozen model. The immediate derivative points in a useful direction, but SGD cannot exploit it because recurrent weight gradients are roughly 100× smaller than output weight gradients—a direct consequence of tanh compression through the recurrence versus an uncompressed linear output path.

Optimizer isolation experiments make the mechanism unambiguous. Every configuration containing second-moment normalization adapts; every configuration without it fails. Momentum adds nothing: $O(n^3)$6-only recovery (−34%) is identical to raw SGD. RMSprop alone reaches 179% on a delayed task, exceeding full Adam (147%), confirming that $O(n^3)$7 without momentum or bias correction is the active component. A critical control rules out the objection that $O(n^3)$8's running second moment constitutes temporal credit assignment through another channel: training a model to convergence with Adam, then switching to SGD at the distribution shift, yields only 6% recovery, whereas switching to Adam or $O(n^3)$9-only yields 100% and 93%. The effect is about what the optimizer does at each adaptation step—scale calibration via $O(n^2)$0—not about what the model learned during training.

An architectural rule for when $O(n^2)$1 is required

The paper proposes a predictive rule: per-parameter normalization is necessary when two conditions hold simultaneously—(i) the recurrent state update involves nonlinear or multiplicative transformations that compress gradients, and (ii) no parallel output path exists with sufficient capacity to adapt independently. Nine architectures tested on stable tasks confirm the rule:

Architecture	Adam/$O(n^2)$2	SGD	Gradient ratio	Bypass
Vanilla RNN	102%	−34%	100×	None
LSTM	99.7%	59.4%	~31×	None
CTRNN	89.8%	−0.1%	278×	None
xLSTM (sLSTM)	91.5%	60.0%	~24×	None
RWKV	adapts	0%	∞	None
RetNet	93%	−17%	~64×	None
Vanilla GRU	adapts	adapts	5–19×	Update gate
SSM (S4)	adapts	adapts	<6×	$O(n^2)$3/$O(n^2)$4 output
Block-GRU+MLP	adapts	50× better	50000×	MLP output

Two findings within this table merit emphasis. First, LSTM's cell-state highway—the classic remedy for vanishing gradients in BPTT—does not bypass the scale mismatch: sigmoid gating still compresses recurrent gradients 31×, and SGD achieves only 8% recovery on Lorenz. Second, three architectures break the pattern exactly as the rule predicts: GRUs adapt under all optimizers because the update gate creates a linear interpolation path around the nonlinear candidate; S4 adapts through its direct $O(n^2)$5/$O(n^2)$6 output matrices; and DreamerV3's block-GRU+MLP adapts through its MLP head, on which SGD actually outperforms Adam by 50× on Lorenz—normalization actively degrades performance when gradients are already well-scaled.

On chaotic dynamics the rule admits exceptions. RWKV shows inverted behavior: all $O(n^2)$7-containing optimizers diverge on every Lorenz seed, while SGD survives (293% recovery) because its recurrent gradients are literally zero under SGD, effectively freezing the recurrence and adapting only through the output predictor. Normalizing near-zero recurrent gradients amplifies them through the multiplicative key-value interaction, destabilizing updates. The rule therefore holds reliably on stable tasks but can fail predictably on chaotic tasks with multiplicative state updates. A pretrained transformer world model (LeWM) provides further confirmation from the other direction: residual connections keep layer-wise gradient norms within 6× even across 20 rollout steps, and no $O(n^2)$8 requirement appears.

Scaling and validation

The method scales cleanly. At $O(n^2)$9, immediate derivatives require 12.6 MB against 12.9 GB for full RTRL, which cannot run on a single GPU; recovery is 378 ± 76%, though the paper candidly notes this figure reflects instability in the RTRL reference itself, and absolute post-shift MSE is the more informative metric at this scale. On cross-session BCI decoding with seven months of electrode drift, RMSprop ($n=1024$0) achieves 106 ± 18% recovery—all five seeds exceed 84%—versus 85% for sparse RTRL ($n=1024$1) and 50% for full RTRL, two of whose seeds partially diverge. The shorter effective window of RMSprop (~100 steps) suits the timescale of neural drift better than Adam's (~1000 steps). On streaming classification, the RNN beats Adaptive Random Forest by 8.3 pp on the Hyperplane generator (0.925 vs 0.842), though trees retain their advantage on spatially structured tasks (RandomRBF, Electricity).

For pretrained models, the same requirement emerges through LoRA adapters, driven by depth-induced gradient attenuation rather than activation compression: measured LoRA gradient norms on Mamba span 51.8× between layers, with last-layer gradients 12× larger than mid-network ones. Across five LMs from 124M to 7B parameters, Adam $n=1024$2 achieves 21–134% recovery on single-pass code-to-Wikipedia shifts while SGD achieves zero everywhere; rank-16 adapters push GPT-2 to 136.8% recovery. Sequential Code→Wiki→Code→Wiki streaming shows no forgetting—returning to Code yields lower cross-entropy than the first exposure—and faster readaptation on second Wiki exposure. Test-time adaptation on CIFAR-10-C sharpens the rule's scope: adapting homogeneous BN parameters shows only a 2.6 pp Adam–SGD gap, while depth-heterogeneous LoRA adapters widen it to 7.6 pp, confirming that parameter heterogeneity across depths, not adaptation per se, drives the requirement.

Limitations and open questions

The paper states its limits explicitly. All from-scratch experiments use single-layer networks; in two-layer RNNs, per-layer Jacobians remain isotropic (condition numbers 1.7 and 2.5) but the cross-layer Jacobian is severely anisotropic (condition number 3773), leaving credit assignment from output back through input-layer recurrence as an unresolved bottleneck. The claim concerns gradient computation, not long-range credit: tasks span dependencies up to 50 steps, and if the hidden state loses information from step $n=1024$3, no gradient method recovers it. Direct online updates to pretrained foundation models produce near-zero adaptation regardless of optimizer; the finding applies only through adapters. Finally, whether immediate derivatives suffice under non-gradient learning rules such as predictive coding remains untested, and the engineering of foundation-model online adaptation (update frequency, adapter rank, long-stream stability) is unexplored.

Conclusion

This paper demonstrates that the perceived necessity of Jacobian propagation in online recurrent learning rests on two correctable miscalibrations: trace decay set far above the true self-propagation factor, and optimizers lacking per-parameter scale normalization. With zero decay and $n=1024$4-containing optimization, immediate derivatives match or exceed full RTRL across synthetic, neural, streaming, and pretrained-model settings at up to 1000× less memory. The accompanying architectural rule—$n=1024$5 is required when gradients must traverse a nonlinear state update without an output bypass—predicts optimizer requirements across nine architectures and extends to depth-induced heterogeneity in pretrained models. The principal open problem is deep recurrence, where cross-layer anisotropy breaks the isotropy argument underlying the method's success in single-layer networks.