Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
162 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Model scale versus domain knowledge in statistical forecasting of chaotic systems (2303.08011v3)

Published 13 Mar 2023 in cs.LG and physics.comp-ph

Abstract: Chaos and unpredictability are traditionally synonymous, yet large-scale machine learning methods recently have demonstrated a surprising ability to forecast chaotic systems well beyond typical predictability horizons. However, recent works disagree on whether specialized methods grounded in dynamical systems theory, such as reservoir computers or neural ordinary differential equations, outperform general-purpose large-scale learning methods such as transformers or recurrent neural networks. These prior studies perform comparisons on few individually-chosen chaotic systems, thereby precluding robust quantification of how statistical modeling choices and dynamical invariants of different chaotic systems jointly determine empirical predictability. Here, we perform the largest to-date comparative study of forecasting methods on the classical problem of forecasting chaos: we benchmark 24 state-of-the-art forecasting methods on a crowdsourced database of 135 low-dimensional systems with 17 forecast metrics. We find that large-scale, domain-agnostic forecasting methods consistently produce predictions that remain accurate up to two dozen Lyapunov times, thereby accessing a new long-horizon forecasting regime well beyond classical methods. We find that, in this regime, accuracy decorrelates with classical invariant measures of predictability like the Lyapunov exponent. However, in data-limited settings outside the long-horizon regime, we find that physics-based hybrid methods retain a comparative advantage due to their strong inductive biases.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (100)
  1. Predictability: a way to characterize complexity. Physics reports 356, 367–474 (2002).
  2. Results of the time series prediction competition at the santa fe institute. In IEEE International Conference on Neural Networks, 1786–1793 (IEEE, 1993).
  3. Systems biology informed deep learning for inferring parameters and hidden dynamics. PLoS Computational Biology 16, e1007575 (2020).
  4. Espeholt, L. et al. Deep learning for twelve hour precipitation forecasts. Nature Communications 13, 5145 (2022).
  5. Colen, J. et al. Machine learning active-nematic hydrodynamics. Proceedings of the National Academy of Sciences 118, e2016708118 (2021).
  6. Zheng, W. et al. Hybrid neural network for density limit disruption prediction and avoidance on j-text tokamak. Nuclear Fusion 58, 056016 (2018).
  7. Time-series forecasting with deep learning: a survey. Philosophical Transactions of the Royal Society A 379, 20200209 (2021).
  8. Introduction to focus issue: When machine learning meets complex systems: Networks, chaos, and nonlinear dynamics. Chaos: An Interdisciplinary Journal of Nonlinear Science 30, 063151 (2020).
  9. Monash time series forecasting archive. Advances in Neural Information Processing Systems 1 (2021).
  10. Chaos as an intermittently forced linear system. Nature communications 8, 19 (2017).
  11. Koopman neural forecaster for time series with temporal distribution shifts. In International Conference on Machine Learning (PMLR, 2023).
  12. Koopman operators for estimation and control of dynamical systems. Annual Review of Control, Robotics, and Autonomous Systems 4, 59–87 (2021).
  13. Next generation reservoir computing. Nature communications 12, 5564 (2021).
  14. Accuracy of neural networks for the simulation of chaotic dynamics: Precision of training data vs precision of the algorithm. Chaos: An Interdisciplinary Journal of Nonlinear Science 30, 113118 (2020).
  15. Robust forecasting using predictive generalized synchronization in reservoir computing. Chaos: An Interdisciplinary Journal of Nonlinear Science 31, 123118 (2021).
  16. Model-free prediction of large spatiotemporally chaotic systems from data: A reservoir computing approach. Physical review letters 120, 024102 (2018).
  17. Model-free prediction of spatiotemporal dynamical systems with recurrent neural networks: Role of network spectral radius. Physical Review Research 1, 033056 (2019).
  18. Data-driven super-parameterization using deep learning: Experimentation with multiscale lorenz 96 systems and transfer learning. Journal of Advances in Modeling Earth Systems 12, e2020MS002084 (2020).
  19. Vlachas, P. R. et al. Backpropagation algorithms and reservoir computing in recurrent neural networks for the forecasting of complex spatiotemporal dynamics. Neural Networks 126, 191–217 (2020).
  20. Real-time computing without stable states: A new framework for neural computation based on perturbations. Neural computation 14, 2531–2560 (2002).
  21. Harnessing nonlinearity: Predicting chaotic systems and saving energy in wireless communication. Science 304, 78–80 (2004).
  22. Karniadakis, G. E. et al. Physics-informed machine learning. Nature Reviews Physics 3, 422–440 (2021).
  23. Neural ordinary differential equations. Advances in neural information processing systems 31 (2018).
  24. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational physics 378, 686–707 (2019).
  25. Robustness of lstm neural networks for multi-step forecasting of chaotic time series. Chaos, Solitons & Fractals 139, 110045 (2020).
  26. Recurrent neural networks for partially observed dynamical systems. Physical Review E 105, 044205 (2022).
  27. Learning chaotic dynamics using tensor recurrent neural networks. In Proceedings of the ICML, vol. 17 (2017).
  28. Gilpin, W. Generative learning for nonlinear dynamics. arXiv preprint arXiv:2311.04128 (2023).
  29. Do we really need deep learning models for time series forecasting? arXiv preprint arXiv:2101.02118 (2021).
  30. Zhou, H. et al. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAI conference on artificial intelligence, vol. 35, 11106–11115 (2021).
  31. Challu, C. et al. N-HiTS: Neural hierarchical interpolation for time series forecasting. In Proceedings of the AAAI Conference on Artificial Intelligence (2023).
  32. Forecasting: principles and practice (OTexts, 2018).
  33. A look at the evaluation setup of the m5 forecasting competition. arXiv preprint arXiv:2108.03588 (2021).
  34. The m4 competition: 100,000 time series and 61 forecasting methods. International Journal of Forecasting 36, 54–74 (2020).
  35. M5 accuracy competition: Results, findings, and conclusions. International Journal of Forecasting 38, 1346–1364 (2022).
  36. Adaptive, locally linear models of complex dynamics. Proceedings of the National Academy of Sciences 116, 1501–1510 (2019).
  37. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint arXiv:1803.01271 (2018).
  38. Yik, J. et al. Neurobench: Advancing neuromorphic computing through collaborative, fair and representative benchmarking. arXiv preprint arXiv:2304.04640 (2023).
  39. Gilpin, W. Chaos as an interpretable benchmark for forecasting and data-driven modelling. Advances in Neural Information Processing Systems 1 (2021).
  40. Nonlinear time series analysis, vol. 7 (Cambridge university press, 2004).
  41. Sprott, J. C. Some simple chaotic flows. Physical review E 50, R647 (1994).
  42. Benchmarking sparse system identification with low-dimensional chaos. arXiv preprint arXiv:2302.10787 (2023).
  43. Time series feature extraction on basis of scalable hypothesis tests (tsfresh–a python package). Neurocomputing 307, 72–77 (2018).
  44. Makowski, D. et al. Neurokit2: A python toolbox for neurophysiological signal processing. Behavior research methods 1–8 (2021).
  45. Umap: Uniform manifold approximation and projection. Journal of Open Source Software 3, 861 (2018).
  46. Highly comparative time-series analysis: the empirical structure of time series and their methods. Journal of the Royal Society Interface 10, 20130048 (2013).
  47. Herzen, J. et al. Darts: User-friendly modern machine learning for time series. Journal of Machine Learning Research 23, 1–6 (2022).
  48. Are transformers effective for time series forecasting? arXiv preprint arXiv:2205.13504 (2022).
  49. N-BEATS: Neural basis expansion analysis for interpretable time series forecasting. In International Conference on Learning Representations (PMLR, 2020).
  50. Time series analysis by state space methods, vol. 38 (OUP Oxford, 2012).
  51. Learning from predictions: Fusing training and autoregressive inference for long-term spatiotemporal forecasts. arXiv preprint arXiv:2302.11101 (2023).
  52. Gilpin, W. Recurrences reveal shared causal drivers of complex time series. arXiv preprint arXiv:2301.13516 (2023).
  53. An analysis of deep neural network models for practical applications. arXiv preprint arXiv:1605.07678 (2016).
  54. Temporal convolutional networks for action segmentation and detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 156–165 (2017).
  55. Cvitanović, P. Invariant measurement of strange sets in terms of cycles. Physical Review Letters 61, 2729 (1988).
  56. Prediction of chaotic time series using recurrent neural networks and reservoir computing techniques: A comparative study. Machine learning with applications 8, 100300 (2022).
  57. A review of deep learning models for time series prediction. IEEE Sensors Journal 21, 7833–7848 (2019).
  58. Catch-22s of reservoir computing. Physical Review Research 5, 033213 (2023).
  59. Teaching recurrent neural networks to infer global temporal structure from local examples. Nature Machine Intelligence 3, 316–323 (2021).
  60. Learning continuous chaotic attractors with a reservoir computer. Chaos: An Interdisciplinary Journal of Nonlinear Science 32, 011101 (2022).
  61. Generating coherent patterns of activity from chaotic neural networks. Neuron 63, 544–557 (2009).
  62. Predicting trends in the quality of state-of-the-art neural networks without access to training or testing data. Nature Communications 12, 4122 (2021).
  63. Brown, T. et al. Language models are few-shot learners. Advances in neural information processing systems 33, 1877–1901 (2020).
  64. No free lunch theorems for optimization. IEEE transactions on evolutionary computation 1, 67–82 (1997).
  65. Defining chaos. Chaos: An Interdisciplinary Journal of Nonlinear Science 25 (2015).
  66. Stability analysis of chaotic systems from data. Nonlinear Dynamics 111, 8799–8819 (2023).
  67. Cvitanovic, P. et al. Chaos: classical and quantum. ChaosBook. org (Niels Bohr Institute, Copenhagen 2005) 69, 25 (2005).
  68. Reconstruction, forecasting, and stability of chaotic dynamics from partial data. arXiv preprint arXiv:2305.15111 (2023).
  69. Temporal convolutional networks: A unified approach to action segmentation. In European Conference on Computer Vision, 47–54 (Springer, 2016).
  70. Alexandrov, A. et al. GluonTS: Probabilistic and Neural Time Series Modeling in Python. J. Mach. Learn. Res. 21, 1–6 (2020).
  71. Tanaka, G. et al. Recent advances in physical reservoir computing: A review. Neural Networks 115, 100–123 (2019).
  72. Bollt, E. On explaining the surprising success of reservoir computing forecaster of chaos? the universal machine learning dynamical system with contrast to var and dmd. Chaos: An Interdisciplinary Journal of Nonlinear Science 31, 013108 (2021).
  73. Adjoint dynamics of stable limit cycle neural networks. In 2019 53rd Asilomar Conference on Signals, Systems, and Computers, 884–887 (IEEE, 2019).
  74. Vaswani, A. et al. Attention is all you need. Advances in neural information processing systems 30 (2017).
  75. Long short-term memory. Neural Computation 9, 1735–1780 (1997).
  76. Unsupervised scalable representation learning for multivariate time series. Advances in Neural Information Processing Systems 32 (2019).
  77. Gardner Jr, E. S. Exponential smoothing: The state of the art. Journal of forecasting 4, 1–28 (1985).
  78. The theta model: a decomposition approach to forecasting. International journal of forecasting 16, 521–530 (2000).
  79. The elements of statistical learning: data mining, inference, and prediction, vol. 2 (Springer, 2009).
  80. Friedman, J. H. Greedy function approximation: a gradient boosting machine. Annals of statistics 1189–1232 (2001).
  81. Why do tree-based models still outperform deep learning on tabular data? arXiv preprint arXiv:2207.08815 (2022).
  82. Sheikholeslami, S. et al. Autoablation: Automated parallel ablation studies for deep learning. In Proceedings of the 1st Workshop on Machine Learning and Systems, 55–61 (2021).
  83. Harvey, A. C. Forecasting, structural time series models and the Kalman filter (Cambridge university press, 1990).
  84. Löning, M. et al. sktime: A unified interface for machine learning with time series. arXiv preprint arXiv:1909.07872 (2019).
  85. Luko𝐬𝐬\mathbf{s}bold_sevi𝐜𝐜\mathbf{c}bold_cius, M. A practical guide to applying echo state networks. Neural Networks: Tricks of the Trade: Second Edition 659–686 (2012).
  86. Descending through a crowded valley-benchmarking deep learning optimizers. In International Conference on Machine Learning, 9367–9376 (PMLR, 2021).
  87. Pmlb: a large benchmark suite for machine learning evaluation and comparison. BioData mining 10, 1–13 (2017).
  88. Another look at measures of forecast accuracy. International journal of forecasting 22, 679–688 (2006).
  89. Pérez-Cruz, F. Estimation of information theoretic measures for continuous random variables. Advances in neural information processing systems 21 (2008).
  90. Estimating mutual information. Physical review E 69, 066138 (2004).
  91. Sample estimate of the entropy of a random vector. Problemy Peredachi Informatsii 23, 9–16 (1987).
  92. Evans, D. A computationally efficient estimator for mutual information. Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences 464, 1203–1215 (2008).
  93. Nonparametric k-nearest-neighbor entropy estimator. Physical Review E 93, 013310 (2016).
  94. Characterization of strange attractors. Physical Review Letters 50, 346 (1983).
  95. Liapunov exponents from time series. Physical Review A 34, 4971 (1986).
  96. Lyapunov exponents in chaotic systems: their importance and their evaluation using observed data. International Journal of Modern Physics B 5, 1347–1375 (1991).
  97. Rocket: exceptionally fast and accurate time series classification using random convolutional kernels. Data Mining and Knowledge Discovery 34, 1454–1495 (2020).
  98. Brain entropy, fractal dimensions and predictability: A review of complexity measures for eeg in healthy and neuropsychiatric populations. European Journal of Neuroscience 56, 5047–5069 (2022).
  99. Automatic differentiation in machine learning: a survey. Journal of Marchine Learning Research 18, 1–43 (2018).
  100. Multivariate multiscale entropy: A tool for complexity analysis of multichannel data. Physical Review E 84, 061918 (2011).
Citations (10)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now