Backpropagation Algorithms and Reservoir Computing in Recurrent Neural Networks for the Forecasting of Complex Spatiotemporal Dynamics (1910.05266v2)

Abstract: We examine the efficiency of Recurrent Neural Networks in forecasting the spatiotemporal dynamics of high dimensional and reduced order complex systems using Reservoir Computing (RC) and Backpropagation through time (BPTT) for gated network architectures. We highlight advantages and limitations of each method and discuss their implementation for parallel computing architectures. We quantify the relative prediction accuracy of these algorithms for the longterm forecasting of chaotic systems using as benchmarks the Lorenz-96 and the Kuramoto-Sivashinsky (KS) equations. We find that, when the full state dynamics are available for training, RC outperforms BPTT approaches in terms of predictive performance and in capturing of the long-term statistics, while at the same time requiring much less training time. However, in the case of reduced order data, large scale RC models can be unstable and more likely than the BPTT algorithms to diverge. In contrast, RNNs trained via BPTT show superior forecasting abilities and capture well the dynamics of reduced order systems. Furthermore, the present study quantifies for the first time the Lyapunov Spectrum of the KS equation with BPTT, achieving similar accuracy as RC. This study establishes that RNNs are a potent computational framework for the learning and forecasting of complex spatiotemporal systems.

• The paper demonstrates that reservoir computing outperforms BPTT-trained RNNs for full-state forecasting while GRU and LSTM excel in reduced-order scenarios.
• It introduces a parallel forecasting architecture that scales predictions for high-dimensional systems with local interactions, enhancing both performance and efficiency.
• The study reveals that both RC and GRU models can effectively estimate key chaotic properties like positive Lyapunov exponents despite challenges in capturing zero exponents.

This paper, "Backpropagation Algorithms and Reservoir Computing in Recurrent Neural Networks for the Forecasting of Complex Spatiotemporal Dynamics" (Backpropagation Algorithms and Reservoir Computing in Recurrent Neural Networks for the Forecasting of Complex Spatiotemporal Dynamics, 2019), investigates and compares the effectiveness of Recurrent Neural Networks (RNNs) trained with Backpropagation Through Time (BPTT) and Reservoir Computing (RC) for forecasting complex spatiotemporal dynamics. The paper uses benchmark systems like the Lorenz-96 model and the Kuramoto-Sivashinsky (KS) equation to evaluate predictive performance, ability to capture long-term statistics, scalability, and computational efficiency.

The paper focuses on recurrent neural network architectures designed for time-series forecasting. The core idea is to use a hidden state to encode historical information. The architectures compared are Long Short-Term Memory (LSTM), Gated Recurrent Units (GRU), Unitary RNNs, and a standard RNN used within the Reservoir Computing framework.

The training methods differ significantly:

• BPTT: Applied to LSTM, GRU, and Unitary networks. This involves optimizing all network weights (input, recurrent, output) using gradient descent. Key implementation details include truncated BPTT (sequence length $\kappa_1$), error accumulation ($\kappa_2$), stateful training (propagating hidden state between batches), regularization (Zoneout for recurrent weights, variational dropout for output weights), learning rate scheduling, and validation-based early stopping. Adding Gaussian noise to training data is also explored as a regularization technique.
• RC: Uses a simple RNN cell where the recurrent weights are randomly initialized and fixed according to specific properties (echo state property, controlled spectral radius $\rho$, sparse connections with degree $d$). Only the output weights are trained, typically using linear regression (specifically, regularized least-squares regression with Tikhonov regularization $\eta$). Training is computationally much faster than BPTT as it avoids backpropagation through the recurrent connections. The paper utilizes a memory-efficient batched approach for RC training to handle large datasets. Augmenting the hidden state with quadratic terms is also used based on prior work to potentially enrich the dynamics.

The performance comparison relies on several metrics:

• Normalized Root Mean Square Error (NRMSE): Measures the prediction error normalized by the standard deviation of the data.
• Valid Prediction Time (VPT): Defined as the longest time horizon for which the NRMSE remains below a certain threshold (set to 0.5 in the paper), normalized by the system's Maximal Lyapunov Exponent (MLE). This quantifies the short-term forecasting skill relative to the system's chaotic timescale.
• Power Spectral Density (PSD): Used to assess the ability of the models to reproduce the long-term statistical properties (climate) of the dynamical system.
• Lyapunov Spectrum (LS): Calculated using the trained models as surrogates to verify if they capture the chaotic nature of the system, particularly the rate at which trajectories diverge.

Key Findings:

1. Forecasting Reduced Order vs. Full State Dynamics (Lorenz-96):
   • When the full state of the system is available for training ($d_o=40$), RC networks exhibit superior predictive performance (higher VPT) and accurately capture the long-term statistics (PSD) compared to BPTT-trained RNNs. RC training is also significantly faster.
   • However, forecasting a reduced-order observable (obtained via SVD, $d_o=35$), which requires the network to implicitly reconstruct the full dynamics and capture temporal dependencies, is more challenging. In this scenario, GRU and LSTM networks show superior forecasting abilities and stability. They achieve higher VPT and are more likely to produce stable iterative forecasts that do not diverge from the attractor, unlike many RC and Unitary models. GRU/LSTM also exhibit better generalization (lower overfitting) in the reduced-order case due to effective regularization techniques. Unitary RNNs generally perform poorly and tend to diverge in most tested scenarios.
2. Scalability via Parallel Forecasting (Lorenz-96, Kuramoto-Sivashinsky):
   • For high-dimensional systems with local interactions, a parallel architecture is proposed where multiple independent RNNs/RCs forecast local regions of the state. This significantly improves scalability and prediction horizon compared to a single large model trying to predict the entire high-dimensional state simultaneously.
   • In this parallel setting (tested on full state Lorenz-96, $d_o=40$, and KS, $d_o=512$), both parallel BPTT networks (GRU/LSTM) and RC networks demonstrate improved forecasting performance and reproduce the correct power spectra.
   • RC training remains faster than BPTT training for parallel models. However, RC often requires a larger number of hidden units per network member to achieve similar VPT as GRU/LSTM. Memory requirements per processing unit can be high for large RC reservoirs.
3. Lyapunov Exponent Calculation (Kuramoto-Sivashinsky):
   • Both trained RC and GRU models can serve as effective surrogates for computing the Lyapunov spectrum from time-series data.
   • They accurately capture the positive Lyapunov exponents, which are crucial for characterizing chaos. The estimated Kaplan-Yorke dimension also matches the true value.
   • However, both RC and GRU surrogates struggle to accurately reproduce the zero Lyapunov exponents of the system.
   • GRU shows slightly better agreement with the true spectrum for positive LEs compared to RC, but at the cost of significantly longer training times.

Implementation Considerations:

• Hyperparameter Tuning: Performance is highly sensitive to hyperparameters. Extensive grid search was performed.
• BPTT Specifics: Truncated BPTT length and error accumulation steps ($\kappa_1, \kappa_2$) are important for capturing temporal dependencies, particularly in reduced-order scenarios. Stateful training is crucial for capturing dependencies longer than the batch size.
• RC Specifics: Spectral radius ($\rho$), input scaling ($\omega$), and Tikhonov regularization ($\eta$) are critical hyperparameters. Memory-efficient training methods (batched approach for correlation matrices) are necessary for large reservoirs. Noise regularization is found to be beneficial for RC's generalization in the full-state case.
• Parallelization: Assumes local interactions. Requires communication between network members during iterative forecasting (e.g., using MPI). Can exploit translational invariance in homogeneous systems for efficiency (train only one model).
• Computational Resources: BPTT training is generally more computationally expensive and time-consuming but may require fewer hidden units for reduced-order tasks. RC training is faster but can be memory-intensive for large reservoirs, especially the matrix inversion step in linear regression.

Limitations and Future Work:

• Inability of models to capture zero Lyapunov exponents.
• Potential for spurious dynamics in non-invertible models.
• Difficulty in hyperparameter tuning for Unitary RNNs leading to divergence.
• Exploring hybrid models combining RC and BPTT approaches.
• Investigating data efficiency requirements for different architectures.
• Studying memory capacity of the networks.
• Developing more sophisticated initialization schemes for BPTT networks based on RC principles.
• Addressing memory and numerical challenges in large-scale RC training.
• Evaluating invertible RNNs.

In summary, the paper demonstrates that RNNs (especially GRU and LSTM) and RC are powerful data-driven tools for forecasting chaotic systems. RC excels when the full state is observed, offering fast training and high accuracy. GRU and LSTM are more robust and perform better when only a reduced-order observable is available. Parallel architectures effectively scale both methods to high-dimensional systems with local interactions. Both approaches can also be used to estimate the chaotic properties of the system, such as the positive Lyapunov exponents.