Residual Reservoir Memory Networks (ResRMNs)

• ResRMNs are dual-reservoir models that integrate a linear memory module with a non-linear residual branch using untrained recurrent dynamics.
• They employ temporal residual orthogonal connections and train only the readout via ridge regression, maintaining stable long-term propagation.
• Empirical results on time-series tasks show ResRMNs can improve accuracy by up to 20.7% compared to traditional ESNs.

Searching arXiv for the primary and closely related papers to ground the article in the latest literature. First, I’ll verify the primary paper on ResRMNs and then gather adjacent work on DeepResESNs, RMNs, and reservoir-memory systems. Residual Reservoir Memory Networks (ResRMNs) are a class of untrained recurrent neural networks in the Reservoir Computing (RC) paradigm that combine a linear memory reservoir with a non-linear residual reservoir. The non-linear component is implemented as a Residual Echo State Network (ResESN) with temporal residual orthogonal connections, while only the readout is trained, typically by ridge regression. The architecture was introduced to improve long-term memory propagation without abandoning the training simplicity of RC, and was evaluated on time-series classification and pixel-level $1$-D classification tasks (Pinna et al., 13 Aug 2025).

1. Conceptual origin and model class

ResRMNs arise at the intersection of several strands of work. Standard Echo State Networks (ESNs) and related RC models keep recurrent dynamics fixed after initialization and train only the output map. In that setting, the reservoir supplies a high-dimensional temporal representation, but conventional leaky ESNs use a single reservoir and can be limited in long-range temporal propagation. ResRMNs depart from that baseline by using two reservoirs instead of one, by introducing temporal residual connections in the non-linear reservoir, and by imposing an orthogonal structure on the residual branch (Pinna et al., 13 Aug 2025).

The model is also closely related to two earlier lines of research. First, Residual Memory Networks (RMNs) were proposed as a feed-forward architecture with residual and time delayed connections, where the number of layers signifies both hierarchical processing depth and temporal depth. In RMN, a layer-specific transform $W_l$ processes the current time step, while a shared transform $W_s$ processes delayed hidden representations; the authors explicitly characterize RMN as “a feed-forward approach to learning long temporal dependencies” and report “significantly less” computational complexity than deep recurrent networks (Baskar et al., 2018). Second, Deep Residual Echo State Networks (DeepResESNs) introduced stacked untrained residual recurrent layers with orthogonal temporal skip connections, providing a direct reservoir-computing precursor to ResRMN-style design (Pinna et al., 28 Aug 2025).

This positioning distinguishes ResRMNs from explicit-memory reservoir systems such as Reservoir Memory Machines, which augment an ESN with an external memory and discrete read/write control, and from fully trained residual RNNs, which learn recurrent dynamics by backpropagation through time. ResRMNs remain entirely within RC: the recurrent part is frozen, the memory mechanism is internal to the reservoir architecture, and the trained component is the readout (Paaßen et al., 2020).

2. Architectural formulation

The ResRMN architecture consists of two untrained recurrent components.

The first is a linear memory reservoir:

$$\mathbf{m}(t) = \mathbf{V_m}\mathbf{m}(t - 1) + \mathbf{V_x}\mathbf{x}(t).$$

Here $\mathbf{m}(t) \in \mathbb{R}^{N_m}$ is the memory state, $\mathbf{V_m} \in \mathbb{R}^{N_m \times N_m}$ is the recurrent matrix, and $\mathbf{V_x} \in \mathbb{R}^{N_m \times N_x}$ is the input matrix. In the implementation described for ResRMN, $\mathbf{V_m}$ is initialized with a cyclic orthogonal structure, described as a permutation-like ring shift matrix with spectral radius $1$, intended to support long memory propagation (Pinna et al., 13 Aug 2025).

The second component is a non-linear residual reservoir:

$$\mathbf{h}(t) = \alpha\,\mathbf{O}\,\mathbf{h}(t - 1) + \beta\,\tanh \big( \mathbf{W_h}\,\mathbf{h}(t - 1) + \mathbf{W_m}\,\mathbf{m}(t) + \mathbf{W_x}\,\mathbf{x}(t) + \mathbf{b_h} \big).$$

In this equation, $W_l$0 is the hidden state, $W_l$1 is an orthogonal matrix, $W_l$2 is the recurrent weight matrix, $W_l$3 connects the memory reservoir to the non-linear reservoir, $W_l$4 is the input matrix, $W_l$5 is the bias, and $W_l$6, $W_l$7 are independent scaling coefficients (Pinna et al., 13 Aug 2025).

The readout is linear:

$W_l$8

with $W_l$9 trained by ridge regression. This separation between fixed recurrent dynamics and trained output map is the defining RC training protocol and is retained unchanged in ResRMN (Pinna et al., 13 Aug 2025).

Functionally, the architecture combines a module intended to preserve information over time with a module intended to generate richer non-linear temporal features. The linear part keeps information alive over long horizons, while the residual non-linear part is designed to avoid over-damping state evolution. The independent coefficients $W_s$0 and $W_s$1 allow interpolation between mostly linear propagation, mostly non-linear transformation, and a balanced hybrid (Pinna et al., 13 Aug 2025).

3. Orthogonal residual connections and stability analysis

A central design variable in ResRMN is the orthogonal matrix $W_s$2 used in the residual temporal branch. The paper studies three configurations:

Variant	$W_s$3	Note
ResRMN$W_s$4	random orthogonal matrix	obtained by QR decomposition of a random matrix
ResRMN$W_s$5	cyclic orthogonal matrix $W_s$6	ring-shift permutation
ResRMN$W_s$7	identity matrix	equivalent to RMN when $W_s$8

The cyclic orthogonal matrix is written as

$W_s$9

The authors also note two reductions of the model class: ResRMN$$\mathbf{m}(t) = \mathbf{V_m}\mathbf{m}(t - 1) + \mathbf{V_x}\mathbf{x}(t).$$0 becomes equivalent to an RMN when $$\mathbf{m}(t) = \mathbf{V_m}\mathbf{m}(t - 1) + \mathbf{V_x}\mathbf{x}(t).$$1, and if $$\mathbf{m}(t) = \mathbf{V_m}\mathbf{m}(t - 1) + \mathbf{V_x}\mathbf{x}(t).$$2 or $$\mathbf{m}(t) = \mathbf{V_m}\mathbf{m}(t - 1) + \mathbf{V_x}\mathbf{x}(t).$$3, ResRMN reduces to a ResESN (Pinna et al., 13 Aug 2025).

The paper studies the dynamics through Jacobian-based linear stability analysis. Let

$$\mathbf{m}(t) = \mathbf{V_m}\mathbf{m}(t - 1) + \mathbf{V_x}\mathbf{x}(t).$$4

The Jacobian of the combined system is

$$\mathbf{m}(t) = \mathbf{V_m}\mathbf{m}(t - 1) + \mathbf{V_x}\mathbf{x}(t).$$5

with

$$\mathbf{m}(t) = \mathbf{V_m}\mathbf{m}(t - 1) + \mathbf{V_x}\mathbf{x}(t).$$6

Because this Jacobian has lower triangular block form, its eigenvalues are the union of the eigenvalues of the diagonal blocks, and the spectral radius becomes

$$\mathbf{m}(t) = \mathbf{V_m}\mathbf{m}(t - 1) + \mathbf{V_x}\mathbf{x}(t).$$7

For the zero-input, zero-bias case, the necessary stability conditions given are

$$\mathbf{m}(t) = \mathbf{V_m}\mathbf{m}(t - 1) + \mathbf{V_x}\mathbf{x}(t).$$8

This decomposition is significant because it implies that the linear memory reservoir and the residual non-linear reservoir can be tuned independently while the overall spectrum remains controlled (Pinna et al., 13 Aug 2025).

The spectral interpretation depends on the choice of $$\mathbf{m}(t) = \mathbf{V_m}\mathbf{m}(t - 1) + \mathbf{V_x}\mathbf{x}(t).$$9. The paper states that random orthogonal and cyclic orthogonal configurations have similar eigenvalue spread around the unit circle, whereas the identity leads to a more skewed eigenvalue distribution concentrated near the positive real axis. This is presented as a source of different dynamical biases and representation diversity. The related DeepResESN work likewise emphasized that orthogonal configuration materially affects dynamics, and derived necessary and sufficient conditions for stable dynamics in deep residual reservoirs (Pinna et al., 28 Aug 2025).

4. Memory propagation and dynamical interpretation

The mechanism claimed for ResRMN combines two distinct forms of temporal processing. The linear memory reservoir is “naturally good at retaining and propagating information,” while the orthogonal residual branch in the non-linear reservoir helps preserve norms and therefore maintain stable propagation over many time steps. The model is explicitly designed to work near the edge of stability, where the system is expressive enough to represent long dependencies without becoming unstable (Pinna et al., 13 Aug 2025).

This memory interpretation is consistent with the broader RC literature. Reservoir computing has been described as a setting in which the reservoir state contains “a high-dimensional, nonlinear impulse response function of the inputs,” and as a mechanism that presents “all recent history simultaneously” to downstream modules while avoiding backpropagation through time for recurrent weights (McKee, 2024). In that sense, ResRMN can be understood as separating memory retention and non-linear computation more explicitly than a single-reservoir ESN.

The residual component also admits a dynamical-systems reading. Work on residual networks with skip connections and shared weights showed that residual architectures can be analyzed as systems whose outputs depend on “the integration of the transient dynamics of the residuals,” and explicitly noted a similarity to reservoir networks when classification relies on unstable internal trajectories (Lagzi, 2021). This does not make ResRMN identical to a feed-forward ResNet, but it supports interpreting the residual reservoir as a memory-bearing dynamical substrate whose transient evolution is computationally relevant.

DeepResESN further sharpens this perspective. In that model, the residual branch creates a direct route for state information to persist over time, and deeper residual layers were shown to transform frequency content differently depending on $\mathbf{m}(t) \in \mathbb{R}^{N_m}$0: identity increasingly filters out high frequencies as depth increases, random orthogonal tends to filter lower frequencies, and cyclic orthogonal preserves frequencies relatively uniformly across depth (Pinna et al., 28 Aug 2025). A plausible implication is that the orthogonal choices in ResRMN should likewise induce distinct temporal filtering biases, even though the ResRMN paper studies a dual-reservoir rather than deep-stacked architecture.

5. Empirical evaluation

The primary empirical assessment of ResRMN was conducted on time-series classification from the UCR/UEA archive and on permuted sequential MNIST (psMNIST). The UCR/UEA datasets listed are Adiac, Beef, Blink, Car, DuckDuckGeese (DDG), FordA, FordB, HandMovementDirection (HMD), Libras, Mallat, OSULeaf, and Wine. The compared baselines are LeakyESN, ResESN$\mathbf{m}(t) \in \mathbb{R}^{N_m}$1, ResESN$\mathbf{m}(t) \in \mathbb{R}^{N_m}$2, ResESN$\mathbf{m}(t) \in \mathbb{R}^{N_m}$3, RMN, and the three ResRMN variants (Pinna et al., 13 Aug 2025).

Model selection used random search over up to $\mathbf{m}(t) \in \mathbb{R}^{N_m}$4 configurations with a $\mathbf{m}(t) \in \mathbb{R}^{N_m}$5h runtime limit. Results were averaged over $\mathbf{m}(t) \in \mathbb{R}^{N_m}$6 random initializations for time-series tasks and $\mathbf{m}(t) \in \mathbb{R}^{N_m}$7 random initializations for psMNIST. During selection, the hidden sizes were $\mathbf{m}(t) \in \mathbb{R}^{N_m}$8 and $\mathbf{m}(t) \in \mathbb{R}^{N_m}$9, where $\mathbf{V_m} \in \mathbb{R}^{N_m \times N_m}$0 is sequence length. The searched hyperparameters were input scalings $\mathbf{V_m} \in \mathbb{R}^{N_m \times N_m}$1, memory coupling $\mathbf{V_m} \in \mathbb{R}^{N_m \times N_m}$2, bias scaling $\mathbf{V_m} \in \mathbb{R}^{N_m \times N_m}$3, spectral radius $\mathbf{V_m} \in \mathbb{R}^{N_m \times N_m}$4, leaky rate $\mathbf{V_m} \in \mathbb{R}^{N_m \times N_m}$5 for leaky models, and residual scalings $\mathbf{V_m} \in \mathbb{R}^{N_m \times N_m}$6 for residual models. The readout used ridge regression with SVD solver, the state at the last time step, standard scaling before readout, and regularization $\mathbf{V_m} \in \mathbb{R}^{N_m \times N_m}$7 (Pinna et al., 13 Aug 2025).

The reported findings are specific. On time-series classification, ResRMN is “usually best or among the best,” and ResRMN$\mathbf{V_m} \in \mathbb{R}^{N_m \times N_m}$8 is especially prominent. On average, ResRMN improves over leakyESN by $\mathbf{V_m} \in \mathbb{R}^{N_m \times N_m}$9 test accuracy. The model also generally outperforms both ResESN and RMN. At the dataset level, the paper reports FordA at $\mathbf{V_x} \in \mathbb{R}^{N_m \times N_x}$0 for ResRMN$\mathbf{V_x} \in \mathbb{R}^{N_m \times N_x}$1, compared with $\mathbf{V_x} \in \mathbb{R}^{N_m \times N_x}$2 for leakyESN and $\mathbf{V_x} \in \mathbb{R}^{N_m \times N_x}$3 for RMN; FordB at $\mathbf{V_x} \in \mathbb{R}^{N_m \times N_x}$4 versus $\mathbf{V_x} \in \mathbb{R}^{N_m \times N_x}$5 for leakyESN; Blink at $\mathbf{V_x} \in \mathbb{R}^{N_m \times N_x}$6 for ResRMN$\mathbf{V_x} \in \mathbb{R}^{N_m \times N_x}$7; and Adiac at $\mathbf{V_x} \in \mathbb{R}^{N_m \times N_x}$8 for ResRMN$\mathbf{V_x} \in \mathbb{R}^{N_m \times N_x}$9 (Pinna et al., 13 Aug 2025).

An ablation-style study varied the memory-reservoir size across

$\mathbf{V_m}$0

The reported outcome is that performance drops significantly when $\mathbf{V_m}$1 is too small. With $\mathbf{V_m}$2, all models lose at least $\mathbf{V_m}$3 average performance. This is direct evidence that the explicit memory reservoir is not incidental to the architecture’s behavior on long sequences (Pinna et al., 13 Aug 2025).

For psMNIST, the number of trainable parameters was increased from about $\mathbf{V_m}$4k to $\mathbf{V_m}$5k while keeping $\mathbf{V_m}$6. The paper reports that performance is relatively stable across that parameter range for RMN and ResRMNs, that dual-reservoir architectures generally improve over their single-reservoir counterparts, that identity-based models benefit more from the memory reservoir than orthogonal-based models, and that orthogonal configurations are already strong even without a memory module, while identity configurations lag unless the linear memory reservoir is present (Pinna et al., 13 Aug 2025).

6. Interpretation, misconceptions, and open directions

A common misconception is to treat ResRMNs as ordinary trained residual RNNs. The defining property of the model is the opposite: recurrent dynamics are untrained, no backpropagation through time is used for recurrent weights, and only the output readout is fit, typically by ridge regression. In this respect, ResRMN remains an RC model rather than a conventional deep recurrent architecture (Pinna et al., 13 Aug 2025).

A second misconception is that the residual non-linear reservoir alone defines the model. The empirical ablation on $\mathbf{V_m}$7 argues against that reading: when the memory reservoir is too small, average performance falls by at least $\mathbf{V_m}$8. The architecture is therefore not merely a ResESN with auxiliary structure; it is a dual-reservoir system in which the linear memory module is a principal component of long-range retention (Pinna et al., 13 Aug 2025).

A third misconception is that orthogonality has a uniform effect across tasks. The reported results are more differentiated. On the UCR/UEA classification tasks, ResRMN$\mathbf{V_m}$9 is often strongest. The cyclic-orthogonal configuration in both modules is often worse than the random orthogonal version. On psMNIST, identity-based models benefit more from the memory reservoir than orthogonal-based models, while orthogonal configurations are already strong even without a memory module. This suggests that orthogonal choice is not merely a stability device; it also changes the representational bias of the reservoir (Pinna et al., 13 Aug 2025).

Within the larger literature, ResRMN occupies a specific position. RMN uses residual and delayed shared-weight feed-forward memory layers to make depth signify both hierarchy and temporal context (Baskar et al., 2018). Reservoir Memory Machines combine a reservoir with explicit non-interfering external memory and discrete addressing (Paaßen et al., 2020). Reservoir-based reinforcement-learning memory modules emphasize that a fixed reservoir can simplify optimization by eliminating backpropagation through time and presenting recent history simultaneously to downstream networks (McKee, 2024). ResRMN differs from each of these by coupling an internal linear memory reservoir to an internal residual orthogonal non-linear reservoir, both untrained, inside a single RC formulation.

The open directions identified for ResRMN are concrete. The memory reservoir is currently initialized in a simple cyclic orthogonal form, and other initialization strategies may improve performance. More sophisticated linear memory modules could be explored. The paper also suggests future polar-form eigenspectrum analysis to better understand rotational dynamics. Finally, because performance depends significantly on the size of the memory reservoir, hyperparameter tuning remains important (Pinna et al., 13 Aug 2025).