Feature-Based Echo-State Networks: A Step Towards Interpretability and Minimalism in Reservoir Computer (2403.19806v1)
Abstract: This paper proposes a novel and interpretable recurrent neural-network structure using the echo-state network (ESN) paradigm for time-series prediction. While the traditional ESNs perform well for dynamical systems prediction, it needs a large dynamic reservoir with increased computational complexity. It also lacks interpretability to discern contributions from different input combinations to the output. Here, a systematic reservoir architecture is developed using smaller parallel reservoirs driven by different input combinations, known as features, and then they are nonlinearly combined to produce the output. The resultant feature-based ESN (Feat-ESN) outperforms the traditional single-reservoir ESN with less reservoir nodes. The predictive capability of the proposed architecture is demonstrated on three systems: two synthetic datasets from chaotic dynamical systems and a set of real-time traffic data.
- G. Hinton, L. Deng, D. Yu, G. E. Dahl, A. Mohamed, N. Jaitly, A. Senior, V. Vanhoucke, P. Nguyen, T. N. Sainath, and B. Kingsbury, “Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups,” IEEE Signal Processing Magazine, vol. 29, no. 6, pp. 82–97, Nov 2012.
- D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. van den Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot, S. Dieleman, D. Grewe, J. Nham, N. Kalchbrenner, I. Sutskever, T. Lillicrap, M. Leach, K. Kavukcuoglu, T. Graepel, and D. Hassabis, “Mastering the game of Go with deep neural networks and tree search,” Nature, vol. 529, no. 7587, pp. 484–489, 2016.
- A. Fawzi, M. Balog, A. Huang, T. Hubert, B. Romera-Paredes, M. Barekatain, A. Novikov, F. J. R. Ruiz, J. Schrittwieser, G. Swirszcz, D. Silver, D. Hassabis, and P. Kohli, “Discovering faster matrix multiplication algorithms with reinforcement learning,” Nature, vol. 610, no. 7930, pp. 47–53, 2022.
- Z. Lu, J. Pathak, B. Hunt, M. Girvan, R. Brockett, and E. Ott, “Reservoir observers: Model-free inference of unmeasured variables in chaotic systems,” Chaos: An Interdisciplinary Journal of Nonlinear Science, vol. 27, no. 4, pp. 1054–1500, 2017.
- J. Pathak, B. Hunt, M. Girvan, Z. Lu, and E. Ott, “Model-free prediction of large spatiotemporally chaotic systems from data: A reservoir computing approach,” Phys. Rev. Lett., vol. 120, p. 024102, Jan 2018.
- L. Grigoryeva and J.-P. Ortega, “Echo state networks are universal,” Neural Networks, vol. 108, pp. 495 – 508, 2018.
- G. Tanaka, T. Yamane, J. B. Héroux, R. Nakane, N. Kanazawa, S. Takeda, H. Numata, D. Nakano, and A. Hirose, “Recent advances in physical reservoir computing: A review,” Neural Networks, vol. 115, pp. 100–123, 2019.
- K. Fujii and K. Nakajima, “Harnessing disordered-ensemble quantum dynamics for machine learning,” Phys. Rev. Applied, vol. 8, p. 024030, Aug 2017.
- D. Goswami, A. Wolek, and D. A. Paley, “Data-driven estimation using an echo-state neural network equipped with an ensemble kalman filter,” in 2021 American Control Conference (ACC), 2021, pp. 2549–2554.
- D. Goswami, A. Riggins, and D. A. Paley, “Data-driven prediction of urban micromobility: A study of dockless electric scooters [applications of control],” IEEE Control Systems Magazine, vol. 42, no. 5, pp. 18–31, 2022.
- D. Goswami, “Sequential learning from noisy data: Data-assimilation meets the echo-state network,” in 2023 62nd IEEE Conference on Decision and Control (CDC), 2023, pp. 7675–7680.
- W. Maass and H. Markram, “On the computational power of circuits of spiking neurons,” Journal of Computer and System Sciences, vol. 69, no. 4, pp. 593 – 616, 2004.
- H. Jaeger and H. Haas, “Harnessing nonlinearity: Predicting chaotic systems and saving energy in wireless communication,” Science, vol. 304, no. 5667, pp. 78–80, 2004.
- D. Goswami, “Delay embedded echo-state network: A predictor for partially observed systems,” IFAC-PapersOnLine, vol. 56, no. 2, pp. 6826–6832, 2023, 22nd IFAC World Congress.
- Numina, “Numina mobility solutions,” https://numina.co, 2019.
- O. Rössler, “An equation for continuous chaos,” Physics Letters A, vol. 57, no. 5, pp. 397–398, 1976.