Papers
Topics
Authors
Recent
2000 character limit reached

Reservoir Computing Benchmarks: a tutorial review and critique

Published 10 May 2024 in cs.ET, cs.LG, and cs.NE | (2405.06561v2)

Abstract: Reservoir Computing is an Unconventional Computation model to perform computation on various different substrates, such as recurrent neural networks or physical materials. The method takes a 'black-box' approach, training only the outputs of the system it is built on. As such, evaluating the computational capacity of these systems can be challenging. We review and critique the evaluation methods used in the field of reservoir computing. We introduce a categorisation of benchmark tasks. We review multiple examples of benchmarks from the literature as applied to reservoir computing, and note their strengths and shortcomings. We suggest ways in which benchmarks and their uses may be improved to the benefit of the reservoir computing community.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (129)
  1. AMS (2012). Persistence forecast. Glossary of Meteorology. https://glossary.ametsoc.org/wiki/Persistence_forecast, accessed 2024-03-20.
  2. Information processing using a single dynamical node as complex system. Nat. Commun., 2:468.
  3. New results on recurrent network training: unifying the algorithms and accelerating convergence. IEEE Trans. Neural Netw., 11(3):697–709.
  4. McMaster IPIX radar. http://soma.mcmaster.ca/ipix/index.html. Accessed: 2022-4-7.
  5. An echo state network architecture based on volterra filtering and PCA with application to the channel equalization problem. In The 2011 International Joint Conference on Neural Networks, pages 580–587.
  6. An extended echo state network using volterra filtering and principal component analysis. Neural Netw., 32:292–302.
  7. Brownlee, J. (2016). 7 time series datasets for machine learning. https://machinelearningmastery.com/time-series-datasets-for-machine-learning/. Accessed: 2022-4-6.
  8. Parallel photonic information processing at gigabyte per second data rates using transient states. Nat. Commun., 4:1364.
  9. Connectivity, dynamics, and memory in reservoir computing with binary and analog neurons. Neural Computation, 22(5):1272–1311.
  10. Extending reservoir computing with random static projections: a hybrid between extreme learning and RC. https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.226.6672&rep=rep1&type=pdf. Accessed: 2021-11-19.
  11. Model-free control of dynamical systems with deep reservoir computing. J. Phys. Complex., 2(3):035025.
  12. Predictive modeling with echo state networks. In Artificial Neural Networks - ICANN 2008, pages 778–787. Springer Berlin Heidelberg.
  13. A NEAT way for evolving echo state networks. In ECAI 2010, pages 909–914. IOS Press.
  14. Recurrent networks and NARMA modeling. In Proceedings of the 4th International Conference on Neural Information Processing Systems, NIPS’91, pages 301–308. Morgan Kaufmann.
  15. Recurrent neural networks and robust time series prediction. IEEE Transactions on Neural Networks, 5(2):240–254.
  16. Hands-on reservoir computing: a tutorial for practical implementation. Neuromorphic Computing and Engineering, 2(3):032002.
  17. Reservoir computing with biocompatible organic electrochemical networks for brain-inspired biosignal classification. Sci Adv, 7(34).
  18. Dale, M. (2018a). Neuroevolution of hierarchical reservoir computers. In Proceedings of the Genetic and Evolutionary Computation Conference, GECCO ’18, pages 410–417. ACM.
  19. Dale, M. (2018b). Reservoir Computing in Materio. PhD thesis, University of York.
  20. Evolving carbon nanotube reservoir computers. In Amos, M. and Condon, A., editors, UCNC 2016, Manchester, UK, July 2016, LNCS, page 13. Springer.
  21. A substrate-independent framework to characterize reservoir computers. Proceedings of the Royal Society A, 475(20180723).
  22. Information processing capacity of dynamical systems. Sci. Rep., 2:514.
  23. All-optical reservoir computer based on saturation of absorbtion. Optical Express, 22(9):10868–10881.
  24. Reservoir computing with a single delay-coupled non-linear mechanical oscillator. J. Appl. Phys., 124(15):152132.
  25. Speech recognition: Turning theory into practice. IEEE Spectrum, 18(9):26–32.
  26. Processing EMG signals using reservoir computing on an event-based neuromorphic system. In BioCAS 2018, pages 1–4. IEEE.
  27. Reservoir computing using dynamic memristors for temporal information processing. Nat. Commun., 8(1):2204.
  28. All-optical reservoir computing. Opt. Express, 20(20):22783–22795.
  29. All-optical reservoir computing. Optics Express, 20(20):22783–22795.
  30. Farmer, J. D. (1982). Chaotic attractors of an infinite-dimensional dynamical system. Physica D. Nonlinear phenomena, 4(3):366–393.
  31. Pattern recognition in a bucket. In Advances in Artificial Life, pages 588–597. Springer Berlin Heidelberg.
  32. Harnessing Disordered-Ensemble quantum dynamics for machine learning. Phys. Rev. Applied, 8(2):024030.
  33. Gan, T. (2023). Breaking Implicit Assumptions of Physical Delay-Feedback Reservoir Computing. PhD thesis, Physics, Engineering and Technology.
  34. Multi-input multi-output (mimo) delay-feedback reservoir computing: Model-free control of van der pol oscillator (submitted).
  35. Signals to spikes for neuromorphic regulated reservoir computing and EMG hand gesture recognition. In ICONS 2021, number Article 29, pages 1–8. ACM.
  36. Datasheets for datasets. arXiv:1803.09010 [cs.DB].
  37. Gilpin, W. (2021). Chaos as an interpretable benchmark for forecasting and data-driven modelling. CoRR, abs/2110.05266.
  38. Mackey-Glass equation. Scholarpedia, 5(3):6908.
  39. Physiobank, physiotoolkit, and physionet. Circulation, 101(23):e215–e220.
  40. Exploring transfer function nonlinearity in echo state networks. In 2015 IEEE Symposium on Computational Intelligence for Security and Defense Applications (CISDA), pages 1–8.
  41. Four-channels reservoir computing based on polarization dynamics in mutually coupled VCSELs system. Opt. Express, 27(16):23293–23306.
  42. What makes a dynamical system computationally powerful? In New Directions in Statistical Signal Processing: From Systems to Brains, pages 127–154. MIT Press.
  43. Echo state networks with filter neurons and a delay&sum readout. Neural Netw., 23(2):244–256.
  44. Lorenz-like chaos in NH3-FIR lasers (data set A). In Weigend and Gershenfeld, (1994), pages 73–104.
  45. Development of the polysomnographic database on CD-ROM. Psychiatry Clin. Neurosci., 53(2):175–177.
  46. Reservoir computing beyond Memory-Nonlinearity trade-off. Sci. Rep., 7(1):10199.
  47. Jaeger, H. (2002). Short Term Memory in Echo State Networks. Technical Report 152, GMD.
  48. Jaeger, H. (2004a). Supporting online material.
  49. Jaeger, H. (2004b). The echo state approach to recurrent neural networks (presentation). accessed 29 November 2023.
  50. Jaeger, H. (2010). The “echo state” approach to analysing and training recurrent neural networks – with an erratum note1. http://www.faculty.jacobs-university.de/hjaeger/pubs/EchoStatesTechRep.pdf. Accessed: 2021-10-5.
  51. Jaeger, H. (2012). Long short-term memory in echo State Networks: Details of a simulation study. Technical Report 27, Jacobs University.
  52. Harnessing nonlinearity: predicting chaotic systems and saving energy in wireless communication. Science, 304(5667):78–80.
  53. Optimization and applications of echo state networks with leaky-integrator neurons. Neural Networks, 20(3):335–352.
  54. Special issue on echo state networks and liquid state machines. Neural Netw., 20(3):287–289.
  55. Supervised and evolutionary learning of echo state networks. In Parallel Problem Solving from Nature – PPSN X, pages 215–224. Springer.
  56. Unsupervised learning of echo state networks: Balancing the double pole. In Genetic and Evolutionary Computation Conference, GECCO 2008, Proceedings, Atlanta, GA, USA, July 12-16, 2008, pages 869–870. unknown.
  57. Movement generation and control with generic neural microcircuits. In Biologically Inspired Approaches to Advanced Information Technology, First International Workshop, BioADIT 2004, Lausanne, Switzerland, January 29-30, 2004. Revised Selected Papers, volume 3141, pages 258–273. Springer Verlag.
  58. Movement generation with circuits of spiking neurons. Neural Comput., 17(8):1715–1738.
  59. Balanced echo state networks. Neural Netw., 36:35–45.
  60. Unifying framework for information processing in stochastically driven dynamical systems. Physical Review Research, 3(4):043135.
  61. Design and analysis of a neuromemristive reservoir computing architecture for biosignal processing. Front. Neurosci., 9:502.
  62. Kuramoto, Y. (1978). Diffusion-Induced chaos in reaction systems. Progress of Theoretical Physics Supplement, 64:346–367.
  63. Simulating self-learning in photorefractive optical reservoir computers. Sci. Rep., 11(1):2701.
  64. Photonic information processing beyond turing: an optoelectronic implementation of reservoir computing. Opt. Express, 20(3):3241–3249.
  65. MNIST handwritten digit database. http://yann.lecun.com/exdb/mnist/. Accessed: 2022-4-5.
  66. Edge of chaos and prediction of computational performance for neural circuit models. Neural Networks, 20(3):323–334.
  67. TI 46-word. https://catalog.ldc.upenn.edu/LDC93S9. Accessed: 2022-4-7.
  68. Lorenz, E. (1995). Predictability: a problem partly solved. In Seminar on Predictability, 4-8 September 1995, volume 1, pages 1–18, Shinfield Park, Reading. ECMWF, ECMWF.
  69. Lorenz, E. N. (1963). Deterministic nonperiodic flow. Journal of Atmospheric Sciences, 20(2):130–141.
  70. Task Agnostic Metrics for Reservoir Computing.
  71. Reservoir observers: Model-free inference of unmeasured variables in chaotic systems. Chaos, 27(4):041102.
  72. Lukoševičius, M. (2012). A Practical Guide to Applying Echo State Networks. In Montavon, G., Orr, G. B., and Müller, K.-R., editors, Neural Networks: Tricks of the Trade, number 7700 in LNCS, chapter 27, pages 659–686. Springer, 2 edition.
  73. Reservoir computing with swarms. Chaos, 31(3):033121.
  74. Lyon, R. (1982). A computational model of filtering, detection, and compression in the cochlea. In ICASSP ’82. IEEE International Conference on Acoustics, Speech, and Signal Processing, volume 7, pages 1282–1285.
  75. Real-time computing without stable states: a new framework for neural computation based on perturbations. Neural Comput., 14(11):2531–2560.
  76. Oscillations and chaos in physiological control systems. Science, 197(4300):287–289.
  77. Exploiting multiple timescales in hierarchical echo state networks. Frontiers in Applied Mathematics and Statistics, 6.
  78. SpaRCe: Improved learning of reservoir computing systems through sparse representations. IEEE Trans Neural Netw Learn Syst, PP.
  79. Adaptive algorithms for bilinear filtering. In SPIE 2296, Advanced Signal Processing: Algorithms, Architectures, and Implementations V, volume 2296.
  80. Temporal data classification and forecasting using a memristor-based reservoir computing system. Nature Electronics, 2(10):480–487.
  81. A soft body as a reservoir: case studies in a dynamic model of octopus-inspired soft robotic arm. Front. Comput. Neurosci., 7:91.
  82. NASA (2016). NASA greenwich sunspot numbers. https://solarscience.msfc.nasa.gov/greenwch/SN_m_tot_V2.0.txt. Accessed: 2022-6-4.
  83. NGDC (2014). NGDC sunspot numbers. https://www.ngdc.noaa.gov/stp/space-weather/solar-data/solar-indices/sunspot-numbers/group/daily-values-and-means/group-sunspot-numbers_monthly-means(monthrg).txt. Accessed: 2022-6-4.
  84. Supervised learning in spiking neural networks with force training. Nature communications, 8(1):2208.
  85. Nonlinear dynamics of delay systems: an overview. Philosophical Transactions A, 377(2153):20180389.
  86. Optoelectronic reservoir computing. Sci. Rep., 2:287.
  87. Model-Free prediction of large spatiotemporally chaotic systems from data: A reservoir computing approach. Phys. Rev. Lett., 120(2):024102.
  88. Using machine learning to replicate chaotic attractors and calculate lyapunov exponents from data. Chaos, 27(12):121102.
  89. Echo state networks for mobile robot modeling and control. In RoboCup 2003: Robot Soccer World Cup VII, pages 157–168. unknown.
  90. Large-Scale optical reservoir computing for spatiotemporal chaotic systems prediction. Physical Review, 10(041037).
  91. Simple deterministically constructed recurrent neural networks. In Intelligent Data Engineering and Automated Learning – IDEAL 2010, pages 267–274. Springer.
  92. Minimum complexity echo state network. IEEE Trans. Neural Netw., 22(1):131–144.
  93. Structure optimization of reservoir networks. Log. J. IGPL, 18(5):635–669.
  94. Structure optimization of reservoir networks. Logic journal of the IGPL, 18(5):635–669.
  95. The effective rank: A measure of effective dimensionality. In 2007 15th European Signal Processing Conference, pages 606–610.
  96. Salles, R. P. (2021). SantaFe.A: Time series a of the santa fe time series competition in TSPred: Functions for benchmarking time series prediction. https://rdrr.io/cran/TSPred/man/SantaFe.A.html. Accessed: 2022-4-7.
  97. Echo state networks-based reservoir computing for MNIST handwritten digits recognition. In 2016 IEEE Intl Conference on Computational Science and Engineering (CSE) and IEEE Intl Conference on Embedded and Ubiquitous Computing (EUC) and 15th Intl Symposium on Distributed Computing and Applications for Business Engineering (DCABES). IEEE.
  98. Training recurrent networks by evolino. Neural Comput., 19(3):757–779.
  99. On computational power and the Order-Chaos phase transition in reservoir computing. In Advances in Neural Information Processing Systems 21, Proc. 22nd Annual Conf. on Neural Information Processing Systems, pages 1425–1432.
  100. BSA, a fast and accurate spike train encoding scheme. In Proceedings of the International Joint Conference on Neural Networks, 2003., volume 4, pages 2825–2830 vol.4.
  101. Improving reservoirs using intrinsic plasticity. Neurocomputing, 71(7-9):1159–1171.
  102. Echo state networks and neural network ensembles to predict sunspots activity. In ESANN 2009, 17th European Symposium on Artificial Neural Networks, Bruges, Belgium, April 22-24, 2009, Proceedings. unknown.
  103. A Hopf physical reservoir computer. Sci. Rep., 11(1):19465.
  104. The van der pol physical reservoir computer. Neuromorphic Computing and Engineering, 3(2):024004.
  105. SILSO (1749–2022). Raw data - Carrington. https://sidc.be/SILSO/carrington. Accessed: 2022-04-06.
  106. Sivashinsky, G. I. (1977). Nonlinear analysis of hydrodynamic instability in laminar flames—I. derivation of basic equations. Acta Astronautica, 4(11):1177–1206.
  107. Sivashinsky, G. I. (1980). On flame propagation under conditions of stoichiometry. SIAM J. Appl. Math., 39(1):67–82.
  108. Applied Nonlinear Control. Prentice-Hall.
  109. Delay-based reservoir computing: noise effects in a combined analog and digital implementation. IEEE TNNLS, 26(2):388–393.
  110. Stepney, S. (2021). Non-instantaneous information transfer in physical reservoir computing. In UCNC 2021 Espoo, Finland, October 2021, volume 12984 of LNCS, pages 164–176. Springer.
  111. Recent advances in physical reservoir computing: A review. Neural Netw., 115:100–123.
  112. Phoneme recognition with large hierarchical reservoirs. Advances in Neural Information Processing Systems, 23:2307–2315.
  113. van der Pol, B. (1922). Lxxxv. on oscillation hysteresis in a triode generator with two degrees of freedom. The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science, 43(256):700–719.
  114. Lxxii. the heartbeat considered as a relaxation oscillation, and an electrical model of the heart. The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science, 6(38):763–775.
  115. Assessment for automatic speech recognition: II. NOISEX-92: A database and an experiment to study the effect of additive noise on speech recognition systems. Speech Commun., 12(3):247–251.
  116. Memory versus non-linearity in reservoirs. In The 2010 International Joint Conference on Neural Networks (IJCNN), pages 1–8.
  117. An experimental unification of reservoir computing methods. Neural Netw., 20(3):391–403.
  118. Reservoir-based techniques for speech recognition. In The 2006 IEEE International Joint Conference on Neural Network Proceedings, pages 1050–1053.
  119. Isolated word recognition with the liquid state machine: a case study. Inf. Process. Lett., 95(6):521–528.
  120. Quantifying the computational capability of a nanomagnetic reservoir computing platform with emergent magnetisation dynamics. Nanotechnology, 33(48).
  121. High-performance photonic reservoir computer based on a coherently driven passive cavity. Optica, OPTICA, 2(5):438–446.
  122. Data-driven forecasting of high-dimensional chaotic systems with long short-term memory networks. Proc. Math. Phys. Eng. Sci., 474(2213):20170844.
  123. Backpropagation algorithms and Reservoir Computing in Recurrent Neural Networks for the forecasting of complex spatiotemporal dynamics. Neural Networks, 126:191–217.
  124. Time Series Prediction: Forecasting The Future And Understanding The Past. Westview Press.
  125. Modeling systems with internal state using evolino. In GECCO 2025, pages 1795–1802. ACM.
  126. Heterogeneous reservoirs for predicting multivariate physiological data.
  127. Decoupled echo state networks with lateral inhibition. Neural Netw., 20(3):365–376.
  128. Yeo, K. (2019). Data-driven reconstruction of nonlinear dynamics from sparse observation. Journal of Computational Physics, 395:671–689.
  129. Yule, G. U. (1927). On a method of investigating periodicities in disturbed series, with special reference to Wolfer’s sunspot numbers. Philosophical Transactions of the Royal Society A, 226(636-646):267–298.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now

Whiteboard

Paper to Video (Beta)

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up

Tweets

Sign up for free to view the 1 tweet with 0 likes about this paper.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up for Free