Parallel Echo State Network (ParalESN)

• Parallel Echo State Network (ParalESN) is a reservoir computing paradigm that uses parallelism to either average predictions across multiple reservoirs or implement structured, diagonal recurrences for temporal data processing.
• The ensemble-style approach reduces prediction error by averaging outputs from independent reservoirs without increasing short-term memory capacity, ensuring a robust variance-reduction strategy.
• The structured operator formulation uses diagonal complex linear recurrences combined with nonlinear mixing to enable efficient parallel temporal processing while maintaining the echo state property and universality.

Searching arXiv for recent and foundational papers on ParalESN and closely related parallel reservoir architectures. Parallel Echo State Network, or ParalESN, denotes a family of reservoir-computing architectures in which parallelism is introduced into the echo state network paradigm. In the literature, the term has referred to two distinct constructions. An earlier formulation studies a parallel deep ESN in which multiple independently randomized reservoirs process the same input sequence and their predictions are averaged, with the stated aim of reducing prediction error rather than increasing short-term memory capacity (Liu et al., 2019). A later formulation introduces ParalESN as a structured-operator reservoir model based on diagonal linear recurrence in the complex space, designed to enable parallel temporal processing through associative scan while preserving the Echo State Property and the universality guarantees associated with traditional ESNs (Pinna et al., 29 Jan 2026).

1. Terminological scope and conceptual orientation

Within reservoir computing, the defining ESN principle is that recurrent dynamics remain fixed after random initialization and only the readout is learned. What distinguishes parallel ESN constructions is the way this fixed recurrent substrate is reorganized. In the 2019 formulation, parallelism is architectural replication: several reservoirs receive the same input, produce separate predictions, and the network outputs the average of those predictions. In the 2026 formulation, parallelism is computational: a structured linear recurrence is chosen so that sequence processing need not remain strictly serial across time. These are related by their adherence to the reservoir-computing philosophy, but they address different bottlenecks and employ different mathematical devices (Liu et al., 2019).

A concise comparison is useful because the shared label can obscure substantive differences.

Formulation	Core mechanism	Stated objective
Parallel deep ESN	Multiple reservoirs in parallel; output averaging	Decrease prediction error while preserving shallow-ESN memory capacity
ParalESN	Diagonal complex linear recurrence; nonlinear mixing; associative scan	Enable parallel processing of temporal data and reduce memory footprint

This distinction also clarifies a common misconception. A parallel ESN is not necessarily a deeper memory machine. In one strand of the literature it is explicitly an ensemble-like averaging scheme; in the other it is a structured linear reservoir designed for parallel temporal computation.

2. Parallel deep ESN as ensemble-style reservoir averaging

The parallel deep ESN proposed in 2019 extends the standard shallow ESN by instantiating $L$ reservoirs that all receive the same input sequence. For a standard shallow ESN, the state and output updates are

$$\boldsymbol x(t+1)=f\!\left(\boldsymbol V\boldsymbol s(t+1)+\boldsymbol W\boldsymbol x(t)\right),$$

$$\boldsymbol y(t+1)=\boldsymbol U\boldsymbol x(t+1).$$

The parallel construction replaces the single reservoir with $L$ separate reservoirs: $$\boldsymbol x^{(l)}(t+1)=f\!\left(\boldsymbol V^{(l)}\boldsymbol s(t+1)+\boldsymbol W^{(l)}\boldsymbol x^{(l)}(t)\right),$$

$$\boldsymbol y^{(l)}(t+1)=\boldsymbol U^{(l)}\boldsymbol x^{(l)}(t+1),$$

for $l\in\{1,2,\ldots,L\}$, and defines the final output by

$$\boldsymbol y(t)=\frac{1}{L}\sum_{l=1}^{L}\boldsymbol y^{(l)}(t).$$

The averaging step is the architectural core. Each branch is an ESN with independently randomized fixed internal weights, and the final prediction is obtained by averaging branch outputs rather than selecting a single reservoir estimate (Liu et al., 2019).

Training follows the usual ESN regime. The internal matrices $\boldsymbol V^{(l)}$ and $\boldsymbol W^{(l)}$ are fixed after random initialization, and only the output matrices $$\boldsymbol x(t+1)=f\!\left(\boldsymbol V\boldsymbol s(t+1)+\boldsymbol W\boldsymbol x(t)\right),$$0 are learned, typically by ridge regression. The authors emphasize that each reservoir has only an $$\boldsymbol x(t+1)=f\!\left(\boldsymbol V\boldsymbol s(t+1)+\boldsymbol W\boldsymbol x(t)\right),$$1 output matrix rather than one large $$\boldsymbol x(t+1)=f\!\left(\boldsymbol V\boldsymbol s(t+1)+\boldsymbol W\boldsymbol x(t)\right),$$2 matrix, so the training cost remains modest relative to a single giant shallow ESN with $$\boldsymbol x(t+1)=f\!\left(\boldsymbol V\boldsymbol s(t+1)+\boldsymbol W\boldsymbol x(t)\right),$$3 neurons. In this sense, the model is best understood as an ensemble of ESNs rather than a single recurrent chain with increased representational depth.

The motivation is explicit: the architecture is intended to stabilize and improve prediction accuracy by reducing the effect of randomness in any single reservoir. This suggests an ensemble variance-reduction interpretation. The design does not attempt to expand the amount of past input that can be stored; it attempts to reduce prediction noise by averaging multiple reservoir-based predictors.

3. Memory capacity analysis and the limits of reservoir parallelization

The 2019 analysis evaluates short-term memory capacity through the standard delayed-reconstruction formalism. Memory at delay $$\boldsymbol x(t+1)=f\!\left(\boldsymbol V\boldsymbol s(t+1)+\boldsymbol W\boldsymbol x(t)\right),$$4 is defined as

$$\boldsymbol x(t+1)=f\!\left(\boldsymbol V\boldsymbol s(t+1)+\boldsymbol W\boldsymbol x(t)\right),$$5

and the total short-term memory capacity is

$$\boldsymbol x(t+1)=f\!\left(\boldsymbol V\boldsymbol s(t+1)+\boldsymbol W\boldsymbol x(t)\right),$$6

For tractability, the derivation assumes a cyclic reservoir weight matrix

$$\boldsymbol x(t+1)=f\!\left(\boldsymbol V\boldsymbol s(t+1)+\boldsymbol W\boldsymbol x(t)\right),$$7

Under this setting, the central theorem states that for a parallel ESN consisting of $$\boldsymbol x(t+1)=f\!\left(\boldsymbol V\boldsymbol s(t+1)+\boldsymbol W\boldsymbol x(t)\right),$$8 single ESNs connected in parallel, the memory capacity is

$$\boldsymbol x(t+1)=f\!\left(\boldsymbol V\boldsymbol s(t+1)+\boldsymbol W\boldsymbol x(t)\right),$$9

The derivation also yields

$$\boldsymbol y(t+1)=\boldsymbol U\boldsymbol x(t+1).$$0

with auxiliary constructions such as $$\boldsymbol y(t+1)=\boldsymbol U\boldsymbol x(t+1).$$1, $$\boldsymbol y(t+1)=\boldsymbol U\boldsymbol x(t+1).$$2, and $$\boldsymbol y(t+1)=\boldsymbol U\boldsymbol x(t+1).$$3, together with

$$\boldsymbol y(t+1)=\boldsymbol U\boldsymbol x(t+1).$$4

Summing over delays recovers the same total capacity expression as for the corresponding shallow ESN (Liu et al., 2019).

The theoretical implication is unambiguous: parallelization does not increase memory capacity in this construction. The internal memory mechanism of each reservoir is unchanged; the architecture duplicates reservoirs and averages outputs. The paper further states that the theoretical memory capacity of an ESN cannot exceed the reservoir size $$\boldsymbol y(t+1)=\boldsymbol U\boldsymbol x(t+1).$$5, interpreted as the maximum storage available for the input stream. Consequently, the benefit of parallelization in this setting is not greater short-term memory but improved predictive stability.

The empirical section reinforces that distinction. On the NARMA benchmark, with reservoir size $$\boldsymbol y(t+1)=\boldsymbol U\boldsymbol x(t+1).$$6, the reported normalized root mean square error values are $$\boldsymbol y(t+1)=\boldsymbol U\boldsymbol x(t+1).$$7 for the traditional shallow ESN, $$\boldsymbol y(t+1)=\boldsymbol U\boldsymbol x(t+1).$$8 for the Parallel ESN, and $$\boldsymbol y(t+1)=\boldsymbol U\boldsymbol x(t+1).$$9 for the Series ESN. The Parallel ESN therefore achieves a 38.5% reduction in NRMSE relative to the traditional shallow ESN, while the series architecture achieves 16.8% reduction. The paper also notes that NRMSE generally decreases as reservoir size increases, that Parallel ESN has the lowest NRMSE among the tested architectures, and that the gain is obtained without increasing memory capacity.

4. ParalESN as structured linear recurrence with nonlinear mixing

The 2026 architecture revisits the meaning of parallelism in reservoir computing. Rather than placing multiple conventional reservoirs side by side, it replaces the usual dense nonlinear reservoir with a structured linear recurrence whose transition operator is diagonal in the complex domain. The stated motivation is to address two bottlenecks of standard ESNs: the necessity of processing temporal data sequentially and the quadratic memory footprint of high-dimensional dense recurrent matrices. A standard ESN is written as

$L$0

with prediction

$L$1

ParalESN instead uses, for layer $L$2,

$L$3

where $L$4 is diagonal and complex-valued, and the leak can be absorbed into

$L$5

Because $L$6 is diagonal, each hidden unit evolves independently in the linear reservoir stage (Pinna et al., 29 Jan 2026).

Independence in the linear stage is not the whole model. ParalESN introduces a nonlinear mixing layer

$L$7

where $L$8 denotes 1-D convolution. This component restores interaction among coordinates and introduces nonlinearity after the parallelizable linear recurrence. The architecture is therefore well characterized as linear parallel reservoir + nonlinear mixing + trained readout.

The input structure is also specialized. The first layer uses a dense complex input matrix $L$9, whereas higher layers use a ring-topology input matrix $$\boldsymbol x^{(l)}(t+1)=f\!\left(\boldsymbol V^{(l)}\boldsymbol s(t+1)+\boldsymbol W^{(l)}\boldsymbol x^{(l)}(t)\right),$$0 given as a shifted sparse matrix. The paper states that this drastically reduces memory cost. Outputs are produced from the mixed states either at every time step or from the final time step: $$\boldsymbol x^{(l)}(t+1)=f\!\left(\boldsymbol V^{(l)}\boldsymbol s(t+1)+\boldsymbol W^{(l)}\boldsymbol x^{(l)}(t)\right),$$1

A central theoretical claim is representational equivalence at the linear-reservoir level. The paper states that any ESN with a linear recurrent matrix $$\boldsymbol x^{(l)}(t+1)=f\!\left(\boldsymbol V^{(l)}\boldsymbol s(t+1)+\boldsymbol W^{(l)}\boldsymbol x^{(l)}(t)\right),$$2 can be transformed into an equivalent complex diagonal ESN by diagonalizing $$\boldsymbol x^{(l)}(t+1)=f\!\left(\boldsymbol V^{(l)}\boldsymbol s(t+1)+\boldsymbol W^{(l)}\boldsymbol x^{(l)}(t)\right),$$3 over $$\boldsymbol x^{(l)}(t+1)=f\!\left(\boldsymbol V^{(l)}\boldsymbol s(t+1)+\boldsymbol W^{(l)}\boldsymbol x^{(l)}(t)\right),$$4, so that if $$\boldsymbol x^{(l)}(t+1)=f\!\left(\boldsymbol V^{(l)}\boldsymbol s(t+1)+\boldsymbol W^{(l)}\boldsymbol x^{(l)}(t)\right),$$5, the recurrence becomes diagonal in transformed coordinates. This underpins the argument that diagonal complex form is not merely a simplification but a structured representation of arbitrary linear reservoirs.

5. Parallel temporal processing, theoretical guarantees, and reported efficiency

The computational novelty of the 2026 ParalESN lies in the observation that the reservoir evolution is linear and can therefore be evaluated using associative scan rather than strictly serial recurrence. The stated complexity analysis gives $$\boldsymbol x^{(l)}(t+1)=f\!\left(\boldsymbol V^{(l)}\boldsymbol s(t+1)+\boldsymbol W^{(l)}\boldsymbol x^{(l)}(t)\right),$$6 for one sequential step, $$\boldsymbol x^{(l)}(t+1)=f\!\left(\boldsymbol V^{(l)}\boldsymbol s(t+1)+\boldsymbol W^{(l)}\boldsymbol x^{(l)}(t)\right),$$7 for a length-$$\boldsymbol x^{(l)}(t+1)=f\!\left(\boldsymbol V^{(l)}\boldsymbol s(t+1)+\boldsymbol W^{(l)}\boldsymbol x^{(l)}(t)\right),$$8 sequence with parallel scan, and parameter storage $$\boldsymbol x^{(l)}(t+1)=f\!\left(\boldsymbol V^{(l)}\boldsymbol s(t+1)+\boldsymbol W^{(l)}\boldsymbol x^{(l)}(t)\right),$$9 rather than $$\boldsymbol y^{(l)}(t+1)=\boldsymbol U^{(l)}\boldsymbol x^{(l)}(t+1),$$0. By contrast, a standard ESN costs $$\boldsymbol y^{(l)}(t+1)=\boldsymbol U^{(l)}\boldsymbol x^{(l)}(t+1),$$1 over a length-$$\boldsymbol y^{(l)}(t+1)=\boldsymbol U^{(l)}\boldsymbol x^{(l)}(t+1),$$2 sequence and stores a dense $$\boldsymbol y^{(l)}(t+1)=\boldsymbol U^{(l)}\boldsymbol x^{(l)}(t+1),$$3 recurrent matrix (Pinna et al., 29 Jan 2026).

The theoretical guarantees are formulated in terms of the Echo State Property and universality. For a one-layer ParalESN with diagonal transition $$\boldsymbol y^{(l)}(t+1)=\boldsymbol U^{(l)}\boldsymbol x^{(l)}(t+1),$$4, the paper proves that

$$\boldsymbol y^{(l)}(t+1)=\boldsymbol U^{(l)}\boldsymbol x^{(l)}(t+1),$$5

The paper then relies on the known result that ESNs with ESP are universal approximators of fading memory filters and states that ParalESN inherits this universality guarantee. A corollary is given that ParalESN with ESP and MLP readout is universal for fading memory filters.

Empirically, the paper reports gains in both memory-oriented tasks and efficiency. On MemCap, ParalESN reaches about 114.5, and ParalESN (deep) about 125.0, compared with 50.6 for ESN and 56.8 for ESN (deep). On sMNIST, the reported accuracies are 96.2% for ParalESN and 97.2% for ParalESN (deep), versus 82.5% for ESN and 91.4% for ESN (deep). On psMNIST, ParalESN gets 96.9%, ParalESN (deep) 95.2%, versus 78.2% for ESN and 82.1% for ESN (deep). The paper also reports that ParalESN typically trains about an order of magnitude faster than traditional ESNs.

The efficiency trade-off is illustrated with 1-D pixel-level classification. On sMNIST, the paper reports: LSTM $$\boldsymbol y^{(l)}(t+1)=\boldsymbol U^{(l)}\boldsymbol x^{(l)}(t+1),$$6, $$\boldsymbol y^{(l)}(t+1)=\boldsymbol U^{(l)}\boldsymbol x^{(l)}(t+1),$$7 min, $$\boldsymbol y^{(l)}(t+1)=\boldsymbol U^{(l)}\boldsymbol x^{(l)}(t+1),$$8 kg emissions; Transformer $$\boldsymbol y^{(l)}(t+1)=\boldsymbol U^{(l)}\boldsymbol x^{(l)}(t+1),$$9, $l\in\{1,2,\ldots,L\}$0 min, $l\in\{1,2,\ldots,L\}$1 kg; LRU $l\in\{1,2,\ldots,L\}$2, $l\in\{1,2,\ldots,L\}$3 min, $l\in\{1,2,\ldots,L\}$4 kg; Mamba $l\in\{1,2,\ldots,L\}$5, $l\in\{1,2,\ldots,L\}$6 min, $l\in\{1,2,\ldots,L\}$7 kg; ParalESN $l\in\{1,2,\ldots,L\}$8, 2.7 min, 0.01 kg, 0.04 kWh; ParalESN (deep) $l\in\{1,2,\ldots,L\}$9, 3.3 min, 0.02 kg, 0.05 kWh. The paper’s interpretation is that ParalESN is not always the absolute best in accuracy, but offers a strong accuracy-efficiency trade-off.

6. Related architectures, broader context, and recurrent points of confusion

Several adjacent lines of work illuminate the broader meaning of parallelism in reservoir computing. Hybrid Backpropagation Parallel Echo State Network (HBP-ESN) splits a full reservoir into $$\boldsymbol y(t)=\frac{1}{L}\sum_{l=1}^{L}\boldsymbol y^{(l)}(t).$$0 sub-reservoirs, concatenates their states, and replaces the linear ESN readout with a deep feedforward network trained by backpropagation, with an M-Ring variant adding cross-talk through a shared matrix $$\boldsymbol y(t)=\frac{1}{L}\sum_{l=1}^{L}\boldsymbol y^{(l)}(t).$$1. The paper reports that the M-Ring structure can achieve similar performance to one large reservoir while decreasing the memory required by an order of magnitude (Evanusa et al., 2020). This is parallel reservoir decomposition, but not the same object as either the 2019 output-averaging Parallel ESN or the 2026 structured-operator ParalESN.

A second related direction is Mod-DeepESN, which studies parallel reservoir pathways, wide versus layered topologies, and neuron partitioning under a fixed budget $$\boldsymbol y(t)=\frac{1}{L}\sum_{l=1}^{L}\boldsymbol y^{(l)}(t).$$2, with

$$\boldsymbol y(t)=\frac{1}{L}\sum_{l=1}^{L}\boldsymbol y^{(l)}(t).$$3

Its conclusions are explicitly task-specific: on Mackey-Glass, an agglomeration of small reservoirs outperforms a single large reservoir and broader topologies are slightly better than deeper ones, whereas for the Melbourne temperature task deeper structures seem to benefit more from increased memory and hierarchy (Carmichael et al., 2019). This suggests that “parallel reservoir pathways” are not uniformly superior; their value depends on the statistics and temporal structure of the target process.

Hybrid architectures have extended the multi-reservoir idea further. Echo State Transformer introduces a Working Memory composed of “several reservoirs working in parallel,” treated as independent memory units with distinct internal dynamics, trainable spectral radius, and adaptive leak rate, then mixed by attention (Bendi-Ouis et al., 25 Jun 2025). All Optical Echo State Network Reservoir Computing is conceptually adjacent because it uses optical routing, beam combiners and splitters, and can increase the number of nodes by increasing the number of parallel fibers, but it is not presented as a dedicated ParalESN architecture (Kaushik et al., 11 Apr 2025).

A persistent theoretical caution is that parallelization should not be conflated with guaranteed stability or guaranteed memory gain. The consistency analysis of ESNs shows that the echo-state property corresponds to complete consistency in the ideal case, yet readouts can remain highly useful even when the reservoir is not fully consistent at the node level (Lymburn et al., 2019). A plausible implication is that future analyses of parallel ESN systems may benefit from measuring not only average prediction error or storage cost, but also the distribution of consistent directions across parallel branches.

Taken together, the literature supports a precise but non-unitary understanding of Parallel Echo State Network. In one formulation it denotes an ensemble-like averaging architecture that improves predictive accuracy without changing the fundamental short-term memory bound. In another it denotes a diagonal-complex, scan-compatible reservoir model that reformulates ESN computation to enable parallel temporal processing while preserving ESN-style theory. Related work on modular reservoirs, hybrid backpropagated readouts, and attention-coupled reservoir banks shows that parallelism in reservoir computing is now a broad design axis rather than a single architecture.