NatSR: Natural Score-driven Replay

• The paper introduces robust methodologies combining Student’s t likelihood and score-based filtering to achieve efficient continual adaptation with reduced KL divergence.
• NatSR is a framework that unifies second-order natural gradient methods with replay buffers and dynamic scaling to handle outliers and regime shifts in online learning.
• By integrating a fixed-size replay buffer with a dynamic scale heuristic, NatSR demonstrates significant performance gains in both time-series forecasting and RL post-training.

Natural Score-driven Replay (NatSR) refers to two distinct but conceptually related algorithmic frameworks introduced for online continual learning (OCL) in time-series forecasting and for prioritized replay in reinforcement learning (RL) post-training of LLMs. Both variants leverage score-based measures—whether in the context of statistical filtering or task-level advantage variance—to guide robust adaptation and efficient replay, yielding improved empirical performance with lightweight, theoretically principled mechanisms.

1. Foundations: Score-driven Methods and Natural Gradients

The core of NatSR in time series OCL (Urettini et al., 19 Jan 2026) is the interpretation of neural network optimization as a parameter filtering problem within a state-space or Generalized Autoregressive Score (GAS) model. Standard stochastic gradient descent is reframed as a filtering update for the time-varying parameter vector $w_t$:

$$y_t \sim p(y_t\,|\,\theta_t),\qquad \theta_{t+1} = \omega + A s_t + B \theta_t,$$

where the scaled score $s_t = S_t \nabla_t$ incorporates the likelihood score $$\nabla_t = \partial \log p(y_t\,|\,\theta_t)/\partial \theta_t$$ and a scaling matrix $S_t$ (typically the inverse Fisher information). Identifying $\theta_t$ with network weights and choosing $A = \eta$, $B = 1$, $S_t = I_t^{-1}$, and $\omega = 0$, the update reduces to natural gradient ascent:

$$w_{t+1} = w_t + \eta\, I_t^{-1}(w_t)\, \nabla_w \ell(y_t\,|\,x_t, w_t),$$

where $\ell(\cdot)$ denotes the sample log-likelihood, and $I_t$ is the Fisher information matrix:

$$I_t(w) = \mathbb{E}_{y_t|x_t}\left[ \nabla_w \log p(y_t|w) \nabla_w \log p(y_t|w)^\top \right].$$

Proposition 1 establishes that, under regularity and Lipschitz smoothness of the expected log-likelihood, the natural gradient update contracts (in expectation) toward the information-theoretic pseudo-true parameter $w_t^*$ minimizing $KL[q_t\,||\,p(\cdot\,|\,\theta)]$, and strictly reduces the Kullback--Leibler divergence to the true conditional law. This ensures theoretical optimality among first-order updates for continual adaptation.

2. Robustness through Student's t Likelihood

Standard Gaussian likelihoods in natural gradient methods can lead to unbounded updates in the presence of outliers. NatSR addresses this via a Student’s t likelihood with $\nu$ degrees of freedom and scale $s$, modeling the residuals as

$$p(y|x,\theta,s,\nu) = \frac{\Gamma((\nu+1)/2)}{\Gamma(\nu/2)\sqrt{\pi\nu}s}\left[1+\frac{(y-f_w(x))^2}{\nu s^2}\right]^{-(\nu+1)/2}.$$

The heavy tails result in the per-sample negative log-likelihood

$$\ell_t(y, f) = \text{const} + \frac{\nu+1}{2}\log\left[1 + \frac{(y-f)^2}{\nu s^2}\right].$$

As the error increases, the Student’s t score decays, conferring robustness. Theorem 1 gives a uniform $\ell_2$ bound on the natural gradient when including Tikhonov regularization $\tau I$:

$$\left\| I_t(w)^{-1} \nabla_w \ell_t \right\|_2 \leq \frac{1}{4}\sqrt{\frac{(\nu+1)(\nu+3)m}{\tau \nu}},$$

where $m$ is the output dimension. This boundedness is not generally achieved with a Gaussian loss.

3. NatSR Algorithmic Architecture

NatSR in time-series OCL (Urettini et al., 19 Jan 2026) combines three core ingredients: the second-order robust Student-t natural gradient, a fixed-size replay buffer, and a dynamic scale heuristic to control adaptation rates during regime shifts.

Replay Buffer

At each $t$, a new sample $N_t = (x_t, y_t)$ arrives, and a batch $B_t$ of prior samples is drawn from the buffer $\mathcal{B}$ (maintained via reservoir sampling). The loss is

$$L_t(w, s) = \ell_t(N_t; w, s) + \lambda \, \ell_t(B_t; w, s),$$

with buffer weight $\lambda > 0$. The corresponding natural gradient update is computed using Fisher–information approximations for both the current and replay batches, along with a Tikhonov regularizer.

Dynamic Scale Heuristic

The scale parameter $s^2$ is updated using a score-driven rule:

$$s^2_{t+1} = s^2_t + \alpha_s \cdot s^2_t\,\nu(e_t^2 - s^2_t)/(\nu s^2_t + e_t^2),$$

where $e_t = y_t - f_w(x_t)$. Persistently large errors increase $s^2$, relaxing the regularization and accelerating adaptation to new regimes; small errors contract $s^2$, tightening update bounds and recovering stability.

Computational Considerations

The algorithm leverages K-FAC approximations of the Fisher matrix (holding two Kronecker factors per layer) and periodic updates (e.g., on large errors only). The replay buffer is small ($M\ll t$), so memory and computation scale modestly beyond standard gradient methods.

4. NatSR for Prioritized Replay in RL Post-Training

A distinct but related application of NatSR appears in RL fine-tuning for LLMs (Fatemi, 6 Jan 2026). Here, NatSR denotes a problem-level prioritized replay scheme for group-regularized policy optimization (GRPO) based RL post-training.

Priority Score Definition

For each problem $i$, an empirical success rate $p_i = \frac{1}{N}\sum_{k=1}^N r_{i,k}$ is maintained ($r_{i,k} \in \{0,1\}$ binary reward). The per-response advantage is $A_{i,k} = r_{i,k} - p_i$. Informativeness is quantified by mean squared advantage, leading to the priority score:

$$\omega_i = p_i (1 - p_i),\qquad \text{maximized at } p_i = 1/2.$$

Only problems with intermediate $p_i$'s contribute non-trivial gradients in GRPO, so prioritizing by $\omega_i$ targets maximal learning signal.

Scheduling and Buffer Management

NatSR maintains a max-heap $H$ of "learnable" problems keyed by $\omega_i$, and solved/unsolved pools $S$ and $U$. Each step, the $C$ highest-priority problems are selected, updated with fresh rollouts and gradients, then moved between $H$, $S$, and $U$ by empirical criteria. Periodic retesting mitigates forgetting and starvation. Priority statistics are smoothed by an EMA to suppress sampling noise.

Integration and Hyperparameters

NatSR is readily composable with the GRPO loop, requiring only upstream modification to task sampling. Because the focus on high-$\omega_i$ problems increases expected gradient magnitude, a reduced learning rate is usually required. The framework requires negligible additional memory per problem and heap operations are $O(\log M)$.

5. Empirical Evaluation

Time Series Forecasting

NatSR (Urettini et al., 19 Jan 2026) was evaluated on seven multivariate, non-stationary datasets (ETTm$_1$, ETTm$_2$, ETTh$_1$, ETTh$_2$, ECL, Traffic, WTH). Using Mean Absolute Scaled Error (MASE) as the metric, NatSR (with $\nu=50$, SGD) attained the best MASE on five datasets and the second-best on two, outperforming Online Gradient Descent (OGD), Experience Replay (ER), DER++, FSNET, and OneNet. Ablation showed that omitting replay or the dynamic scale heuristic harms accuracy by up to 13% and 6%, respectively; omitting both degrades performance by up to 19%. Training time remains only a few minutes on a single GPU.

RL Post-Training

For RL post-training (Fatemi, 6 Jan 2026), NatSR was applied to GRPO fine-tuning of Qwen2.5-7B on the GURU math dataset. Compared to uniform sampling, prioritized replay via NatSR consistently yielded higher pass@1 and pass@4 metrics (early-stage gains of +3–5% absolute pass@1), with the heap focusing on problems providing the strongest learning signal. Empirical results demonstrate rapid convergence and robust task coverage. The heap and solved/unsolved pools remain lightweight, and the prioritization is adaptive and non-parametric.

6. Connections and Implications

NatSR unifies econometric score-driven filtering and second-order natural gradient methods in a single framework for online adaptation. The use of Student's t likelihood aligns robust optimization and heavy-tailed error modeling. The replay buffer and dynamic volatility scale recover long-term memory and rapid regime adaptation without task labels or explicit regime-change detection. In RL, NatSR exploits advantage variance to maximize expected learning signal rather than following pre-determined curricula.

A plausible implication is that this general score-driven replay principle may be extensible to other data modalities and continual learning settings requiring both robustness and sample efficiency. The modularity of the prioritized replay mechanisms (e.g., heap-based scheduling, task pools, adaptive smoothing) admits straightforward adaptation to large-scale or high-throughput environments.

7. Summary Table of NatSR Instantiations

Context	Key Mechanism	Reference
Time Series OCL	Student-t natural gradient + buffer + dynamic scale	(Urettini et al., 19 Jan 2026)
RL Post-training (GRPO)	Advantage-variance prioritized replay, heap/pools, EMA smoothing	(Fatemi, 6 Jan 2026)

NatSR thus denotes both a robust, theoretically justified online learning algorithm for forecasting and a simple, non-parametric prioritized replay schedule for RL fine-tuning, each leveraging the central principle of score-driven selection or update.