Echo state networks are universal (1806.00797v2)

Abstract: This paper shows that echo state networks are universal uniform approximants in the context of discrete-time fading memory filters with uniformly bounded inputs defined on negative infinite times. This result guarantees that any fading memory input/output system in discrete time can be realized as a simple finite-dimensional neural network-type state-space model with a static linear readout map. This approximation is valid for infinite time intervals. The proof of this statement is based on fundamental results, also presented in this work, about the topological nature of the fading memory property and about reservoir computing systems generated by continuous reservoir maps.

• The paper proves that echo state networks can universally approximate any discrete-time fading memory filter under appropriate conditions.
• It employs a two-step methodology separating internal and external approximations, leveraging the Stone-Weierstrass and Schauder's Fixed Point Theorems.
• The findings expand reservoir computing applications in time-series forecasting, signal processing, and nonlinear dynamical systems.

Universality of Echo State Networks in Approximation of Fading Memory Filters

The paper presented by Grigoryeva and Ortega addresses the topic of Echo State Networks (ESNs) and their role as universal approximants in the domain of discrete-time fading memory filters with uniformly bounded inputs. The primary assertion is that ESNs provide a finite-dimensional, neural network-type state-space model through which any fading memory input/output system in discrete time can be effectively realized over infinite time intervals. This offers a significant contribution to the understanding and application of reservoir computing systems in complex dynamical problems.

Key Contributions and Methodology

The paper demonstrates the universality of ESNs by leveraging their ability to uniformly approximate systems with the fading memory property (FMP). This claim hinges on presenting both theoretical and practical insights into the structure and functionality of ESNs and similar reservoir computing paradigms under the right conditions.

1. Topological Nature of FMP: The authors define the Fading Memory Property as a topological condition rather than a metric-based one. This redefinition implies that any uniform topology can be used for sequences with fading memory, highlighting that the property is independent of any specified rate of memory decay.
2. Reservoir Computing Systems: The paper explores reservoir systems characterized by state-space transformations with linear readouts. A central argument presented is the reliance on neural network structures to achieve universality through continuous reservoir maps that ensure the existence of solutions, and the Echo State Property (ESP), which guarantees uniqueness and stability.
3. Internal and External Approximation: The methodology separates the problem into internal and external approximations. It employs a two-step approach to show universality: first constructing a reservoir filter approximating a non-reservoir system (external), and subsequently showing that any reservoir filter can themselves be approximated by an ESN (internal).
4. Mathematical Rigor: The validity of these approximations is supported by a series of mathematical proofs. These establish conditions like the continuity of reservoir maps, use of the Stone-Weierstrass theorem for neural networks, and Schauder's Fixed Point Theorem that collectively provide a rigorous underpinning to the claims about ESN universality.

Implications and Future Research

The implications of establishing ESN's universality as uniform approximants are substantial. Practically, this means that a wide class of discrete-time systems can be effectively modeled and predicted using ESNs, thereby broadening their applicability in fields such as time-series forecasting, signal processing, and nonlinear dynamic systems.

Theoretically, the characterization of the fading memory property and the resultant topological insights can guide future research in more effectively dealing with memory and forgetting in dynamical systems—key issues in fields reliant on temporal data.

For future research, an intriguing question is to explore further the robustness of ESNs against random variations of their parameters. As noted, ESNs are commonly implemented with random, fixed reservoir weights, focusing training on the linear readout part; understanding why this approach generally works well in practice remains an open area.

Overall, this paper provides a solid theoretical basis for viewing ESNs as a powerful tool in machine learning and dynamical systems theory, ensuring researchers and practitioners alike leverage these networks for their robust approximative capabilities within the context of fading memory filters.