---
title: Deep Echo State Network (DESN)
url: https://www.emergentmind.com/topics/deep-echo-state-network-desn
type: topic
---

# Deep Echo State Network (DESN)

A Deep Echo State Network (DESN) is a recurrent neural architecture within the reservoir computing paradigm, constructed by stacking multiple echo state network (ESN) reservoirs in a strictly feed-forward manner. Each layer in a DESN is an untrained, fixed, randomly initialized recurrent system—typically with leaky-integrator units—whose nonlinear dynamics transform sequential data. Only the final read-out mapping from the concatenated deep reservoir states to the output is learned (usually via linear or logistic regression). By employing hierarchical stacking and careful spectral scaling to satisfy the Echo State Property (ESP), DESNs enable the extraction of multiscale temporal features, providing a computationally efficient means of modeling complex, structured time-series across classification and regression tasks [1802.06708][2004.08170][1712.04323].

## 1. Mathematical Architecture and Dynamics

A DESN comprises $N_L$ layers, each with $N_R$ leaky-integrator recurrent units. At time $t$, the state of layer $\ell$ is denoted $x^{(\ell)}(t)\in\mathbb{R}^{N_R}$. The recurrent update equations, omitting additive biases, are given by:

- Layer 1:
  $$
  x^{(1)}(t) = (1 - a^{(1)})\,x^{(1)}(t-1) + a^{(1)}\,\tanh\left(W_{\text{in}}\,u(t) + \widehat W^{(1)}\,x^{(1)}(t-1)\right)
  $$
- Layer $\ell>1$:
  $$
  x^{(\ell)}(t) = (1 - a^{(\ell)})\,x^{(\ell)}(t-1) + a^{(\ell)}\,\tanh\left(W^{(\ell)}\,x^{(\ell-1)}(t) + \widehat W^{(\ell)}\,x^{(\ell)}(t-1)\right)
  $$

Where $u(t)\in\mathbb{R}^{N_U}$ is the external input, $W_{\text{in}}$ and $W^{(\ell)}$ are input-to-reservoir and inter-layer weights, and $\widehat W^{(\ell)}$ is the intra-layer recurrent weight matrix. The leaking rate $a^{(\ell)}\in(0,1]$ governs the timescale integration at each layer.

The full network state is the concatenation $x(t) = [x^{(1)}(t); x^{(2)}(t); \dots; x^{(N_L)}(t)]\in\mathbb{R}^{N_L N_R}$.

To ensure the ESP (i.e., input-driven contractivity and asymptotic washout of initial conditions), all intra-layer matrices are rescaled so that
$$
\max_{\ell=1,\dots,N_L} \rho\big((1 - a^{(\ell)}) I + a^{(\ell)}\widehat W^{(\ell)}\big) < 1
$$
where $\rho(\cdot)$ denotes the spectral radius [1802.06708][1712.04323][1909.11022].

## 2. Training Paradigm and Read-Out Learning

DESNs adhere strictly to the reservoir computing principle: all internal weights are fixed post-initialization; only the readout weights are optimized. For time series classification or regression tasks, the DESN output is formed by linearly mapping the concatenated reservoir states. Given an input sequence $s = [u(1), \ldots, u(n)]$, the sequence-level mean state is computed:
$$
\chi(s) = \frac{1}{n} \sum_{t=1}^n x(t)
$$
The network's output is $y(s) = W_{\text{out}}\cdot\chi(s)$, with $W_{\text{out}}\in\mathbb{R}^{N_Y\times(N_L N_R)}$.

Readout weights are obtained via ridge regression:
$$
W = \arg \min_W \|Y_{\text{train}} - W\cdot X_{\text{train}}\|_2^2 + \lambda \|W\|_2^2
$$
where $\lambda$ is a Tikhonov regularization parameter, and $X_{\text{train}}$ collects mean states over the training sequences [1802.06708][2004.08170].

## 3. Hyperparameters, Initialization, and Practical Guidelines

Design of a DESN involves several critical hyperparameters:

| Hyperparameter         | Range / Recommendation                                                         | Effect                                      |
|-----------------------|--------------------------------------------------------------------------------|---------------------------------------------|
| Number of layers ($N_L$)      | 2–10 (10 for challenging multi-scale PD data)                     | Deeper networks capture longer timescales   |
| Reservoir size ($N_R$)        | 10–500 (practical: 10–50/layer for moderate data, up to 2000)     | Larger $N_R$ = more memory/capacity         |
| Leaking rate ($a^{(\ell)}$)   | $0.1$ (slow), layer-specific adjustment for time-scale diversity   | Slower $a$: longer memory at deeper layers  |
| Spectral radius ($\rho$)      | $0.7$–$1.2$ (typical: near $1$)                                    | Near unit radius = near-critical dynamics   |
| Input/inter-layer scaling      | $\sigma,\,\hat{s}$ in $[0.1, 2]$; input scaling small for linearity | Control nonlinearity and memory depth       |
| Readout regularization ($\lambda$) | $[10^{-10}, 1]$ via cross-validation                        | Controls overfitting at output layer        |

Matrix elements are drawn i.i.d. uniformly and rescaled according to these hyperparameter settings. Layer-wise or global cross-validation determines optimal configurations [1802.06708][1712.04323][2004.08170].

## 4. Dynamical Properties and Theoretical Insights

Stacking recurrent reservoirs biases the network’s temporal processing in two key ways:

1. **Multi-timescale representation**: Lower layers respond rapidly to input, encoding high-frequency dynamics; higher layers integrate more slowly, modeling low-frequency, long-term structure. Frequency analysis of linearized stacks shows progressive low-pass filtering in deeper layers—each layer's cutoff shifts toward lower frequencies, formalized by cutoffs $\omega_c^{(i)} \propto (a\,g)^{-i}$ [1712.04323].

2. **Enhanced memory and criticality**: For a fixed total neuron count, deeper arrangements empirically increase short-term memory capacity (MC) and allocate memory across layers. DeepESNs tend toward the “edge of chaos” ($\max_i \lambda_{\max}^{(i)}\approx 0$) in the Lyapunov spectrum, a regime of maximal information-processing richness. Theoretical conditions in terms of local Lipschitz constants or spectral radii guarantee collective contractivity and the ESP [1712.04323][1903.05174].

Layer coupling strength and heterogeneity (through inter-layer scaling) strongly influence the richness of higher-layer representations, as shown by entropy and uncoupled dynamics analyses [1903.05174].

## 5. Network Variants and Topological Extensions

Several architectural enhancements build upon the base DESN design:

- **Wide/parallel DESN**: Multiple independent reservoirs ingest the same external input; their outputs are averaged or concatenated. Memory capacity remains that of a single reservoir, but prediction error is reduced by variance averaging [1908.07063][1908.08380].
- **Series/cascaded DESN**: Reservoirs are cascaded, with each layer processing the prior’s output. This improves feature extraction but typically reduces classical memory capacity.
- **Permutation/structured reservoirs**: Using orthogonal or permutation-based recurrent matrices maximizes short-term memory and improves the conditioning of layer dynamics. The synergy of depth and structure yields best-in-class results for delay-and-chaos prediction tasks [1909.11022].
- **Modular and projection-encoding variants**: Alternation of reservoir and encoding/projection layers (e.g., PCA, autoencoders) combats collinearity and expands the temporal kernel, enabling multiscale encoding with minimal runtime penalty [1711.05255][1808.00523].

A summary table compares these options:

| Variant         | Memory Capacity        | Rich Feature Extraction | Computational Overhead      |
|-----------------|-----------------------|-------------------------|----------------------------|
| Parallel DESN   | Same as shallow       | Redundant/robust        | Low (easy readout)         |
| Series DESN     | Lower than shallow    | Hierarchical            | Low (no BPTT)              |
| Structured Top. | Comparable/higher     | Improved memory, mixing | Slight complexity increase  |

## 6. Benchmark Applications and Empirical Performance

DESNs have been validated across synthetic and real-world domains. For instance:

- **Time-series classification (Parkinson’s spiral drawing)**: Achieved test accuracy of 89.3% (vs. 84.1% shallow ESN) with sensitivity 90.0%, specificity 80.0%, and significant improvement (McNemar $p=0.0032$) [1802.06708].
- **Short-term traffic forecasting (Madrid ATRs)**: Attained $R^2=0.87\,\pm\,0.11$ for 15-min ahead prediction, outperforming shallow ESNs, LSTMs, and classical regression baselines [2004.08170].
- **Multiple superimposed oscillator and chaotic series**: Normalized RMSE reduced by orders of magnitude compared to shallow ESNs; for example, Mackey-Glass 84-step prediction: NRMSE $0.201$ (ESN) vs $0.00517$ (DeepESN) [1908.08380][1712.04323].
- **Pattern recognition and medical diagnosis**: Layering increases performance by up to 7%–10% over single-reservoir baselines for tasks with strong multiscale or nonstationary temporal structure [1712.04323][2012.02974].

## 7. Limitations, Design Challenges, and Future Directions

While DESNs preserve the rapid training and low computational cost of reservoir computing, several limitations are noted:

- **Reservoir weights are unoptimized**: Representational power is constrained by the random nature of reservoirs; performance may lag fully-trained RNNs on massive datasets [2004.08170][1712.04323].
- **Hyperparameter tuning**: Selection of depth, leaking rates, and spectral radii is critical and task-dependent. Systematic model selection methods remain an active research area [2012.02974].
- **Memory capacity and information propagation**: Series-stacked reservoirs lower classical linear memory, while parallel/structured arrangements maintain or increase it [1908.07063][1909.11022].
- **Interpretability and theoretical foundation**: Understanding the mapping from random deep dynamics to task-relevant features is an open problem, especially regarding layerwise information representation, multi-timescale processing, and universal approximation [1712.04323][2012.02974].
- **Scalability and hybridization**: Future extensions include integrating trainable encoders/decoders, leveraging unsupervised representation learning, and parallelized/GPU implementations for very large reservoirs [2012.02974].

DESNs constitute a robust, theoretically principled, and computationally tractable approach for multiscale temporal modeling. Their current and future development is tightly coupled to advances in reservoir theory, hierarchical RNNs, and scalable learning paradigms.

Source: https://www.emergentmind.com/topics/deep-echo-state-network-desn