LMNT: Deterministic Noise Training

• LMNT is a deterministic, noise-inspired regularization framework that stabilizes time series forecasting of chaotic systems by approximating the effects of multi-noise training.
• It leverages an analytical linearization of the model response to input perturbations, dramatically reducing computational costs compared to stochastic noise injection.
• Empirical validation using reservoir computing on the Kuramoto–Sivashinsky system demonstrated extended prediction valid times and high climate fidelity.

Linearized Multi-Noise Training (LMNT) is a noise-inspired, deterministic regularization framework for stabilizing and improving the predictive skill of machine learning models deployed for time series forecasting of chaotic dynamical systems. LMNT was introduced to address the limitations of stochastic noise injection approaches, most notably by providing a mathematically principled procedure that closely approximates the stabilizing effects of input noise, but with orders-of-magnitude greater computational efficiency and reproducibility. Its development and validation are detailed in the context of reservoir computing applied to the Kuramoto–Sivashinsky equation, with demonstrable advantages in both short-term forecasting and long-term climate fidelity (Wikner et al., 2022).

1. Motivation and Theoretical Basis

In closed-loop forecasting of chaotic systems, models are trained to forecast one time step ahead (“open-loop”) and then iteratively used to predict future states (outputs become subsequent inputs). Standard training is susceptible to error amplification transverse to the data manifold—a phenomenon known as “climate instability”—where trajectory error accumulates rapidly and predictions diverge from the true attractor.

A central insight is that injecting random input noise during training encourages the model to contract perturbations transverse to the attractor, promoting stability. This effect is especially significant in recurrent architectures, such as reservoir computers, where the feedback coupling decouples the learned attractor from the true system’s natural invariances. However, naive stochastic noise injection is computationally demanding—requiring many perturbed forward passes for each time step—and introduces randomness into loss landscapes, complicating hyperparameter selection.

LMNT deterministically approximates the effect of training with small, independent noise perturbations over the memory horizon of the reservoir or RNN. It replaces the need for Monte Carlo sampling by analytically linearizing the model response to input noise, thus enabling single-pass, reproducible training with equivalent regularizing properties.

2. Mathematical Formulation

LMNT builds on the regularized least-squares loss for one-step-ahead forecasting: $$L_0(W) = \frac{1}{T_\text{train}} \sum_{j=0}^{T_\text{train}-1} \| W s_j - v_j \|_2^2 + \beta_T \|W\|_F^2,$$ where $s_j$ is the feature (e.g., reservoir) state at time $j$, $v_j$ the target, and $\beta_T$ the Tikhonov (ridge) weight.

When input noise $\sqrt{\beta_N} \gamma_j$ is injected, the expected loss decomposes into a bias term (mean feature) and a variance term, which, in the small-noise and large-sample ($P \to \infty$) regime, admits a tractable deterministic approximation: $$L_\text{LMNT}(W) = \frac{1}{T_\text{train}}\sum_{j=0}^{T_\text{train}-1} \|W s_j - v_j\|_2^2 + \beta_T \|W\|_F^2 + \frac{\beta_L}{T_\text{train}-K} \sum_{j=K}^{T_\text{train}-1} \sum_{k=j-K+1}^j \|W \nabla_{u_k}s_j\|_F^2,$$ with $\beta_L = \beta_N$ and $K$ the memory window. In matrix notation: $$L_\text{LMNT}(W) = \frac{1}{T_\text{train}} \| W S - V \|_F^2 + \beta_T\,\mathrm{Tr}(W W^\top) + \beta_L\,\mathrm{Tr}(W R_L W^\top),$$ where $R_L$ is the accumulated input-feature Jacobian covariance: $$R_L = \frac{1}{T_\text{train} - K} \sum_{j=K}^{T_\text{train}-1} \sum_{k = j-K+1}^j (\nabla_{u_k} s_j) (\nabla_{u_k} s_j)^\top.$$

LMNT thus regularizes not just the weights, but also the sensitivity of the model to small input perturbations, extended over a finite memory horizon.

3. Implementation in Reservoir Computing

LMNT was instantiated in the context of reservoir computing, where the open-loop reservoir state update is

$$r(t) = (1-\alpha) r(t-\Delta t) + \alpha\, \tanh(A r(t-\Delta t) + B u_\text{in}(t) + C)$$

and the feature vector is $s(t) = [1; u_\text{in}(t); r(t); r(t)^2 ]$.

The key computational step is evaluating $\nabla_{u_k}s_j$ for $k = j-K+1, \ldots, j$, using analytical Jacobians for the reservoir map. The sparsity of the network matrix $A$ ensures computational efficiency of this step. With $S$ and $V$ assembled from noiseless trajectories, the optimal readout $W$ is given by a single linear solve: $$W \left( \frac{1}{T_\text{train}} SS^\top + \beta_T I + \beta_L R_L \right) = \frac{1}{T_\text{train}} V S^\top.$$ This procedure enables very fast evaluation of candidate regularization hyperparameters without repeated forward passes through the reservoir.

4. Hyperparameter Selection and Computational Considerations

Reservoir computing with LMNT requires setting structural (e.g., node count $N$, spectral radius $\rho$, input scaling $\sigma$, bias $\theta$, leaking rate $\alpha$, in-degree $\langle d \rangle$) and regularization ($\beta_T$, $\beta_L$, memory $K$) hyperparameters. The crucial advantage of LMNT is that, once $R_L$ is computed from the noiseless training trajectory, hyperparameter sweeps over $\beta_L$ (and $\beta_T$) require only rescaling in the matrix solve. There is no need to re-run the reservoir or generate new noise samples per candidate setting.

A typical tuning protocol is a coarse logarithmic grid search over $\beta_T$ and $\beta_L$, evaluating model stability (defined as sustained climate prediction) and median prediction valid time; the pair on the stability boundary with maximal skill is selected.

5. Empirical Validation: Kuramoto–Sivashinsky System

LMNT was validated on the Kuramoto–Sivashinsky (KS) equation, a canonical spatiotemporal chaotic PDE. Using a reservoir with $N=500$, $\rho=0.6$, $\sigma=\theta=0.1$, $\alpha=1.0$, and training on $2 \times 10^4$ steps (ca. 240 Lyapunov times), several regularization strategies were benchmarked.

Key results are summarized as follows:

Regularization	Fraction Stable	Median VT (Lyap)	Median $\bar\epsilon_\text{map}$
None	0/1000	0.05 ± 0.01	∞
Jacobian only	0/1000	0.25 ± 0.01	∞
Tikhonov only ($\beta_T=10^{-6}$)	565/1000	0.71 ± 0.02	0.646 ± 0.022
Jacobian+Tikhonov	1000/1000	2.88 ± 0.02	$9.16 \times 10^{-3}$
Noise+Tikhonov	1000/1000	4.24 ± 0.04	$2.77 \times 10^{-3}$
LMNT+Tikhonov ($\beta_L=10^{-7.4}$, $\beta_T=10^{-16.5}$)	1000/1000	4.27 ± 0.04	$2.75 \times 10^{-3}$

Both noise training and LMNT achieved high fractions of stable predictions, long valid times ($\sim4.3$ Lyapunov times), and climate errors well below threshold. Power spectral density (PSD) analyses showed that LMNT and noise-regularized reservoirs reproduced the true KS spectrum with near-perfect fidelity.

6. Generalization and Practical Guidelines

LMNT applies directly to any RNN or feedforward model with memory (e.g., LSTM, GRU, delay-coordinate networks). The requirements are: derivation of relevant input-to-feature Jacobians $\nabla_{u(t-k)} s(t)$ (over a memory window $K$), accumulation of $R_L$ as above, and addition of $\beta_L \mathrm{Tr}[W R_L W^\top]$ to the regularized loss.

Key practical considerations are:

• The memory window $K$ should span the model’s effective fading memory, typically $K \in [4,10]$.
• LMNT may be combined with Tikhonov regularization as needed.
• For large $T_\text{train}$ or $K$, one may subsample time indices or use mean state approximations for $R_L$ to reduce computation.
• Hyperparameter sweeps are vastly accelerated, allowing routine optimization of regularization parameters.

A plausible implication is that LMNT enables scaling noise-inspired regularization to large, modern reservoirs or recurrent models where stochastic noise injection would be prohibitively expensive.

7. Strengths, Limitations, and Significance

LMNT exhibits several strengths: it precisely replicates the stabilizing effect of multi-noise training in a deterministic, single-pass procedure; it supports efficient and reproducible hyperparameter selection; and it empirically delivers both increased prediction valid times and climate fidelity in dynamical forecasting tasks.

Potential limitations include the computational overhead associated with accumulating Jacobian covariance matrices for all time steps and memory window entries, particularly in very large-scale or long-horizon contexts. Subsampling and mean-trajectory approximations mitigate these costs without significant degradation of regularization effect.

LMNT constitutes a principled, model-agnostic regularization method for stabilizing machine learning-based forecasting of chaotic systems, matching or exceeding the performance of classical noise injection while offering tractability required for large-scale applications (Wikner et al., 2022).