Deep Residual Echo State Networks

• DeepResESNs are deep reservoir computing models that integrate untrained recurrent layers with temporal residual connections for improved long-term dynamics.
• They use orthogonal residual pathways to preserve hidden-state norms, ensuring stability and enhanced memory capacity.
• Empirical evaluations show significant performance gains in memory, forecasting, and classification tasks compared to traditional ESNs.

Searching arXiv for the specified DeepResESN paper and closely related deep reservoir computing work. Deep Residual Echo State Networks (DeepResESNs) are a class of deep untrained recurrent neural networks within the reservoir computing framework that combine hierarchical reservoir depth with temporal residual recurrent connections. They extend standard Echo State Networks (ESNs), in which the recurrent reservoir is fixed after random initialization and only a linear readout is trained, and they also extend deep reservoir architectures by inserting an orthogonal residual pathway inside each recurrent layer. The stated purpose of this design is to improve stability, memory capacity, and long-term temporal modeling while preserving the fast and efficient learning characteristic of reservoir computing (Pinna et al., 28 Aug 2025).

1. Position within reservoir computing

Echo State Networks are a canonical reservoir-computing model composed of an untrained recurrent reservoir and a trained linear readout. In a LeakyESN, the reservoir update is

$$\mathbf{h}(t) = (1 - \tau)\mathbf{h}(t-1) + \tau \,\phi\Big(\mathbf{W}_h\mathbf{h}(t-1)+\mathbf{W}_x\mathbf{x}(t)+\mathbf{b}\Big).$$

The reservoir weights remain fixed after initialization, and only the readout is learned. This yields cheap and stable training, but shallow ESNs often struggle with long-term dependencies and complex temporal processing (Pinna et al., 28 Aug 2025).

Deep reservoir computing addresses this limitation by stacking multiple untrained reservoirs to construct a hierarchy of temporal representations. A related framework, Deep-ESN, alternates reservoir projections with encoding layers so that each level processes a compressed representation from the previous one; the model is intended to capture multiscale temporal dynamics while retaining the fixed-reservoir training paradigm (Ma et al., 2017). However, the DeepResESN formulation identifies a distinct limitation in plain deep reservoir stacks: forward-propagation degradation, in which signals may attenuate or amplify as they move through depth, thereby harming stability and expressiveness (Pinna et al., 28 Aug 2025).

Residual Echo State Networks (ResESNs) had previously introduced a temporal residual shortcut within a single recurrent reservoir. DeepResESNs generalize that idea to a deep setting: they combine the hierarchical temporal abstraction associated with deep reservoir stacks and the improved information flow associated with residual temporal pathways. In that sense, DeepResESNs are presented as a unified extension of both shallow ResESNs and DeepESNs (Pinna et al., 28 Aug 2025).

2. Layer dynamics and architectural formulation

For layer $l$, DeepResESN uses the update

$$\mathbf{h}^{(l)}(t) = \alpha^{(l)} \mathbf{O}\mathbf{h}^{(l)}(t - 1) + \beta^{(l)} \phi \Big( \mathbf{W}_h^{(l)}\mathbf{h}^{(l)}(t - 1) + \mathbf{W}_x^{(l)}\mathbf{x}^{(l)}(t) + \mathbf{b}^{(l)} \Big),$$

with

$$\mathbf{x}^{(1)}(t)=\mathbf{x}(t), \qquad \mathbf{x}^{(l)}(t)=\mathbf{h}^{(l-1)}(t)\ \text{for}\ l>1.$$

Each recurrent layer therefore receives two contributions from its own past: a residual linear recurrence $\alpha^{(l)}\mathbf{O}\mathbf{h}^{(l)}(t-1)$ and a nonlinear reservoir transformation scaled by $\beta^{(l)}$ (Pinna et al., 28 Aug 2025).

This differs structurally from the standard leaky-integrator ESN update. In place of a leaky convex combination, DeepResESN introduces a temporal skip path mediated by an orthogonal map $\mathbf{O}$. The paper identifies this residual pathway as the main architectural novelty. The deep architecture is consequently not merely a stack of ordinary reservoirs; it is a stack of residual recurrent reservoirs, each of which contains an explicit linear memory-preserving branch (Pinna et al., 28 Aug 2025).

A useful contrast arises with Deep-ESN. Deep-ESN is also hierarchical and untrained, but its depth is organized through alternating reservoirs and encoding layers, together with direct feature links from intermediate encoders to the readout: $$\mathbf{M}(t+1)=[\mathbf{x}_{res}^{(K)}(t+1)^T;\mathbf{u}(t+1)^T;\{\mathbf{x}_{enc}^{(1,\dots,K-1)}(t+1)^T\}].$$ Those connections serve a feature-fusion role, but they are not residual state recursions in the additive hidden-state sense. DeepResESNs instead place residuality inside the temporal recurrence of each layer (Ma et al., 2017).

3. Orthogonal residual configurations

The residual branch in DeepResESN uses an orthogonal matrix $\mathbf{O}$, and the paper studies three specific configurations (Pinna et al., 28 Aug 2025):

Configuration	Symbol	Definition
Random orthogonal	$\mathbf{R}$	Generated by QR decomposition of a random matrix with i.i.d. entries in $l$0
Cyclic orthogonal	$l$1	A fixed-structure permutation-like matrix with ones on the subdiagonal and top-right corner
Identity	$l$2	Identity matrix

Orthogonality matters because orthogonal maps preserve Euclidean norm: $l$3 for any vector $l$4. The immediate consequence is that the residual pathway does not arbitrarily amplify or shrink hidden-state norms. In untrained recurrent networks, this provides a direct mechanism for controlling stability and preserving signal flow through time (Pinna et al., 28 Aug 2025).

The comparison among $l$5, $l$6, and $l$7 is not merely implementation detail; it is used to probe distinct dynamical regimes. The identity matrix yields the simplest residual connection and the strongest “identity memory” effect. The cyclic orthogonal matrix is a deterministic orthogonal map and is reported as often maximizing short-term memory. The random orthogonal matrix represents stochastic orthogonal dynamics. The paper reports that these alternatives can produce markedly different frequency filtering and task behavior (Pinna et al., 28 Aug 2025).

This suggests that the residual branch in DeepResESN is not a generic shortcut whose exact form is unimportant. Rather, the orthogonal structure itself functions as a design variable that shapes temporal encoding behavior.

4. Stability theory and the Echo State Property

A central theoretical concern for reservoir computing is the Echo State Property (ESP), which requires asymptotic independence from initial conditions. In the DeepResESN setting, the ESP is extended as

$l$8

for any input sequence $l$9 and any initial states $$\mathbf{h}^{(l)}(t) = \alpha^{(l)} \mathbf{O}\mathbf{h}^{(l)}(t - 1) + \beta^{(l)} \phi \Big( \mathbf{W}_h^{(l)}\mathbf{h}^{(l)}(t - 1) + \mathbf{W}_x^{(l)}\mathbf{x}^{(l)}(t) + \mathbf{b}^{(l)} \Big),$$0. The meaning is that the effect of initial conditions must vanish asymptotically (Pinna et al., 28 Aug 2025).

The network’s global state is written

$$\mathbf{h}^{(l)}(t) = \alpha^{(l)} \mathbf{O}\mathbf{h}^{(l)}(t - 1) + \beta^{(l)} \phi \Big( \mathbf{W}_h^{(l)}\mathbf{h}^{(l)}(t - 1) + \mathbf{W}_x^{(l)}\mathbf{x}^{(l)}(t) + \mathbf{b}^{(l)} \Big),$$1

Because deeper layers depend only on earlier layers and not vice versa, the Jacobian of the full deep system is block lower-triangular. This yields the spectral characterization

$$\mathbf{h}^{(l)}(t) = \alpha^{(l)} \mathbf{O}\mathbf{h}^{(l)}(t - 1) + \beta^{(l)} \phi \Big( \mathbf{W}_h^{(l)}\mathbf{h}^{(l)}(t - 1) + \mathbf{W}_x^{(l)}\mathbf{x}^{(l)}(t) + \mathbf{b}^{(l)} \Big),$$2

The paper states that

$$\mathbf{h}^{(l)}(t) = \alpha^{(l)} \mathbf{O}\mathbf{h}^{(l)}(t - 1) + \beta^{(l)} \phi \Big( \mathbf{W}_h^{(l)}\mathbf{h}^{(l)}(t - 1) + \mathbf{W}_x^{(l)}\mathbf{x}^{(l)}(t) + \mathbf{b}^{(l)} \Big),$$3

is a necessary condition for the ESP of a DeepResESN. In operational terms, the spectral radius of the global linearization at zero input and zero state must be strictly less than one; otherwise the zero state is not stable and the ESP can fail (Pinna et al., 28 Aug 2025).

A sufficient condition is derived through layer-wise contractivity. For layer $$\mathbf{h}^{(l)}(t) = \alpha^{(l)} \mathbf{O}\mathbf{h}^{(l)}(t - 1) + \beta^{(l)} \phi \Big( \mathbf{W}_h^{(l)}\mathbf{h}^{(l)}(t - 1) + \mathbf{W}_x^{(l)}\mathbf{x}^{(l)}(t) + \mathbf{b}^{(l)} \Big),$$4,

$$\mathbf{h}^{(l)}(t) = \alpha^{(l)} \mathbf{O}\mathbf{h}^{(l)}(t - 1) + \beta^{(l)} \phi \Big( \mathbf{W}_h^{(l)}\mathbf{h}^{(l)}(t - 1) + \mathbf{W}_x^{(l)}\mathbf{x}^{(l)}(t) + \mathbf{b}^{(l)} \Big),$$5

assuming $$\mathbf{h}^{(l)}(t) = \alpha^{(l)} \mathbf{O}\mathbf{h}^{(l)}(t - 1) + \beta^{(l)} \phi \Big( \mathbf{W}_h^{(l)}\mathbf{h}^{(l)}(t - 1) + \mathbf{W}_x^{(l)}\mathbf{x}^{(l)}(t) + \mathbf{b}^{(l)} \Big),$$6 is contractive with coefficient $$\mathbf{h}^{(l)}(t) = \alpha^{(l)} \mathbf{O}\mathbf{h}^{(l)}(t - 1) + \beta^{(l)} \phi \Big( \mathbf{W}_h^{(l)}\mathbf{h}^{(l)}(t - 1) + \mathbf{W}_x^{(l)}\mathbf{x}^{(l)}(t) + \mathbf{b}^{(l)} \Big),$$7. For the first layer,

$$\mathbf{h}^{(l)}(t) = \alpha^{(l)} \mathbf{O}\mathbf{h}^{(l)}(t - 1) + \beta^{(l)} \phi \Big( \mathbf{W}_h^{(l)}\mathbf{h}^{(l)}(t - 1) + \mathbf{W}_x^{(l)}\mathbf{x}^{(l)}(t) + \mathbf{b}^{(l)} \Big),$$8

If every layer is contractive and

$$\mathbf{h}^{(l)}(t) = \alpha^{(l)} \mathbf{O}\mathbf{h}^{(l)}(t - 1) + \beta^{(l)} \phi \Big( \mathbf{W}_h^{(l)}\mathbf{h}^{(l)}(t - 1) + \mathbf{W}_x^{(l)}\mathbf{x}^{(l)}(t) + \mathbf{b}^{(l)} \Big),$$9

then, assuming bounded global state space, the DeepResESN satisfies the ESP for all inputs (Pinna et al., 28 Aug 2025).

These conditions make explicit how the residual and nonlinear branches interact in stability control. The coefficients $$\mathbf{x}^{(1)}(t)=\mathbf{x}(t), \qquad \mathbf{x}^{(l)}(t)=\mathbf{h}^{(l-1)}(t)\ \text{for}\ l>1.$$0 and $$\mathbf{x}^{(1)}(t)=\mathbf{x}(t), \qquad \mathbf{x}^{(l)}(t)=\mathbf{h}^{(l-1)}(t)\ \text{for}\ l>1.$$1 determine the balance between norm-preserving residual recurrence and nonlinear transformation, while $$\mathbf{x}^{(1)}(t)=\mathbf{x}(t), \qquad \mathbf{x}^{(l)}(t)=\mathbf{h}^{(l-1)}(t)\ \text{for}\ l>1.$$2 and $$\mathbf{x}^{(1)}(t)=\mathbf{x}(t), \qquad \mathbf{x}^{(l)}(t)=\mathbf{h}^{(l-1)}(t)\ \text{for}\ l>1.$$3 determine amplification. Orthogonality of $$\mathbf{x}^{(1)}(t)=\mathbf{x}(t), \qquad \mathbf{x}^{(l)}(t)=\mathbf{h}^{(l-1)}(t)\ \text{for}\ l>1.$$4 supports the analysis by ensuring that the residual component is norm-preserving (Pinna et al., 28 Aug 2025).

For context, Deep-ESN also has an ESP guarantee, but it is formulated differently. Under bounded input and encoder matrices, linear encoders for the proof, and $$\mathbf{x}^{(1)}(t)=\mathbf{x}(t), \qquad \mathbf{x}^{(l)}(t)=\mathbf{h}^{(l-1)}(t)\ \text{for}\ l>1.$$5 reservoir units, Deep-ESN inherits ESP when each reservoir recurrent matrix satisfies $$\mathbf{x}^{(1)}(t)=\mathbf{x}(t), \qquad \mathbf{x}^{(l)}(t)=\mathbf{h}^{(l-1)}(t)\ \text{for}\ l>1.$$6. This is a layered extension of classical ESN contraction conditions rather than a residual-specific theorem (Ma et al., 2017). The DeepResESN result is therefore more directly tied to residual recurrence.

5. Memory capacity and frequency-domain behavior

The principal functional claim for DeepResESNs is that they improve memory and long-term temporal modeling by combining temporal residual paths with hierarchical depth. The residual path preserves and propagates state information across time, while the stacked architecture creates multiple temporal abstraction levels (Pinna et al., 28 Aug 2025).

To study this behavior, the paper performs spectral analysis on a multi-sine input

$$\mathbf{x}^{(1)}(t)=\mathbf{x}(t), \qquad \mathbf{x}^{(l)}(t)=\mathbf{h}^{(l-1)}(t)\ \text{for}\ l>1.$$7

with

$$\mathbf{x}^{(1)}(t)=\mathbf{x}(t), \qquad \mathbf{x}^{(l)}(t)=\mathbf{h}^{(l-1)}(t)\ \text{for}\ l>1.$$8

Using FFT on layer states, it finds a configuration-dependent pattern: the identity residual progressively filters out higher frequencies with depth, the random orthogonal residual tends to filter out lower frequencies, and the cyclic orthogonal residual approximately preserves frequencies across depth (Pinna et al., 28 Aug 2025).

These observations show that the residual structure changes how temporal information is encoded layer by layer. The paper further states that $$\mathbf{x}^{(1)}(t)=\mathbf{x}(t), \qquad \mathbf{x}^{(l)}(t)=\mathbf{h}^{(l-1)}(t)\ \text{for}\ l>1.$$9 and $\alpha^{(l)}\mathbf{O}\mathbf{h}^{(l)}(t-1)$0 often give stronger short-term memory than $\alpha^{(l)}\mathbf{O}\mathbf{h}^{(l)}(t-1)$1, whereas $\alpha^{(l)}\mathbf{O}\mathbf{h}^{(l)}(t-1)$2 can be advantageous when preserving exact information is beneficial (Pinna et al., 28 Aug 2025). A plausible implication is that the orthogonal residual matrix acts as a task-dependent inductive bias over the temporal spectrum of the hidden dynamics.

The broader reservoir-computing literature provides a useful comparison. Deep-ESN emphasizes multiscale temporal dynamics emerging from alternating projection and encoding layers; shallow layers tend to have shorter memory and deeper layers longer memory, especially with PCA encoders (Ma et al., 2017). DeepResESN reaches related goals through a different mechanism: rather than interleaving dimension reduction, it retains full recurrent layers while altering their temporal transport properties via residual orthogonal recurrence.

6. Experimental protocol and empirical results

The reported experimental setup uses hidden size $\alpha^{(l)}\mathbf{O}\mathbf{h}^{(l)}(t-1)$3 neurons during model selection, random search over up to 1000 configurations with a maximum of 24 hours, and averages results over 10 random initializations. The readout is trained by ridge regression with an SVD solver. For deep models, the number of layers is chosen from $\alpha^{(l)}\mathbf{O}\mathbf{h}^{(l)}(t-1)$4, and optional concatenation of layer states is considered through $\alpha^{(l)}\mathbf{O}\mathbf{h}^{(l)}(t-1)$5. All reservoirs are fully connected. Hyperparameters explored include $\alpha^{(l)}\mathbf{O}\mathbf{h}^{(l)}(t-1)$6, $\alpha^{(l)}\mathbf{O}\mathbf{h}^{(l)}(t-1)$7, $\alpha^{(l)}\mathbf{O}\mathbf{h}^{(l)}(t-1)$8, $\alpha^{(l)}\mathbf{O}\mathbf{h}^{(l)}(t-1)$9 for leaky models, and $\beta^{(l)}$0 together with inter-layer versions for residual deep models (Pinna et al., 28 Aug 2025).

The benchmark suite covers memory-based tasks, forecasting tasks, and classification tasks. Memory tasks include ctXOR5, ctXOR10, SinMem10, and SinMem20. Forecasting tasks include Lorenz96 with prediction 25 and 50 steps ahead, Mackey-Glass with 1-step and 84-step prediction, and NARMA30 and NARMA60. Classification tasks include UEA/UCR datasets Adiac, Blink, FordA, FordB, Kepler, Libras, Mallat, together with sMNIST and psMNIST treated as time series (Pinna et al., 28 Aug 2025).

The paper reports that DeepResESN is the top-ranked model overall and significantly better than baselines in the critical-difference analysis. Relative to LeakyESN, the reported average performance gains are about $\beta^{(l)}$1 on memory tasks, $\beta^{(l)}$2 on forecasting tasks, and $\beta^{(l)}$3 on classification tasks (Pinna et al., 28 Aug 2025).

On memory tasks, DeepResESNs usually outperform shallow counterparts, with gains becoming larger at longer delays. Random orthogonal and cyclic orthogonal residuals often outperform the identity configuration by a large margin, especially on SinMem10 and SinMem20. The paper states that $\beta^{(l)}$4 and $\beta^{(l)}$5 can reduce error by about one order of magnitude relative to $\beta^{(l)}$6 in some memory tasks (Pinna et al., 28 Aug 2025).

On forecasting tasks, deep models are reported to help more on harder long-horizon settings such as Lz50, MG84, and N60, whereas benefits are smaller or negligible on shorter-horizon tasks such as Lz25, MG, and N30. Configuration differences are weaker in this category, except for MG84, where the identity residual configuration performs best (Pinna et al., 28 Aug 2025).

On classification tasks, deep models again generally outperform shallower ones. The identity configuration often performs best, particularly on Adiac, Blink, FordA, FordB, Kepler, and in some settings sMNIST and psMNIST. The paper hypothesizes that $\beta^{(l)}$7 preserves exact information across layers, which helps classification (Pinna et al., 28 Aug 2025).

These outcomes indicate that the most effective orthogonal residual structure depends on task type. Memory-intensive synthetic tasks favor $\beta^{(l)}$8 or $\beta^{(l)}$9 more often, while several classification tasks favor $\mathbf{O}$0. This task dependence is one of the paper’s central empirical conclusions.

7. Relation to prior deep ESN architectures and interpretation

DeepResESN is positioned as both a generalization and a mechanistic departure. It generalizes DeepESN-style hierarchical reservoir computation by replacing plain stacked recurrent layers with residual recurrent layers, and it generalizes shallow residual ESNs by extending residual temporal recurrence through depth (Pinna et al., 28 Aug 2025).

The relation to Deep-ESN is especially instructive. Deep-ESN demonstrates that deep reservoir stacks can capture multiscale dynamics when they alternate between nonlinear reservoir projection and low-dimensional encoding, while feature links from intermediate encoders to the readout preserve information that would otherwise be lost (Ma et al., 2017). Its theoretical analysis shows that stability is inherited when each reservoir satisfies the usual singular-value contraction condition, and its training complexity remains $\mathbf{O}$1, matching a conventional ESN under the paper’s assumptions (Ma et al., 2017).

DeepResESN addresses a different problem formulation. Instead of introducing encoders or skip-feature fusion into the readout, it modifies the recurrent state dynamics directly through an additive orthogonal residual term. The resulting model is still an untrained RNN within reservoir computing, with only the readout learned, but the internal temporal transport is explicitly residual (Pinna et al., 28 Aug 2025).

A common misconception would be to treat all deep reservoir architectures with intermediate shortcuts as equivalent. The available evidence does not support that equivalence. Deep-ESN’s “feature links” are concatenative shortcuts into the output layer, whereas DeepResESN’s residuality is inside the recurrent update itself (Ma et al., 2017, Pinna et al., 28 Aug 2025). Another misconception would be to assume that identity residuals are always optimal because they preserve information most directly. The experiments instead show a task-dependent tradeoff: identity often helps classification, while random or cyclic orthogonal residuals can dominate on delayed memory tasks (Pinna et al., 28 Aug 2025).

The main contributions attributed to DeepResESN are therefore fourfold: the introduction of a deep untrained recurrent architecture with temporal residual connections, the unification and generalization of DeepESN and ResESN, the systematic study of random orthogonal, cyclic orthogonal, and identity residual structures, and the development of necessary and sufficient ESP-related stability conditions alongside empirical evidence on memory, forecasting, and classification benchmarks (Pinna et al., 28 Aug 2025). Taken together, these results place DeepResESNs within the broader evolution of reservoir computing toward deeper and more dynamically structured untrained recurrent systems.