Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
125 tokens/sec
GPT-4o
53 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Stable Neural Stochastic Differential Equations in Analyzing Irregular Time Series Data (2402.14989v5)

Published 22 Feb 2024 in cs.LG and cs.AI

Abstract: Irregular sampling intervals and missing values in real-world time series data present challenges for conventional methods that assume consistent intervals and complete data. Neural Ordinary Differential Equations (Neural ODEs) offer an alternative approach, utilizing neural networks combined with ODE solvers to learn continuous latent representations through parameterized vector fields. Neural Stochastic Differential Equations (Neural SDEs) extend Neural ODEs by incorporating a diffusion term, although this addition is not trivial, particularly when addressing irregular intervals and missing values. Consequently, careful design of drift and diffusion functions is crucial for maintaining stability and enhancing performance, while incautious choices can result in adverse properties such as the absence of strong solutions, stochastic destabilization, or unstable Euler discretizations, significantly affecting Neural SDEs' performance. In this study, we propose three stable classes of Neural SDEs: Langevin-type SDE, Linear Noise SDE, and Geometric SDE. Then, we rigorously demonstrate their robustness in maintaining excellent performance under distribution shift, while effectively preventing overfitting. To assess the effectiveness of our approach, we conduct extensive experiments on four benchmark datasets for interpolation, forecasting, and classification tasks, and analyze the robustness of our methods with 30 public datasets under different missing rates. Our results demonstrate the efficacy of the proposed method in handling real-world irregular time series data.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (74)
  1. Neural continuous-discrete state space models for irregularly-sampled time series. arXiv preprint arXiv:2301.11308, 2023.
  2. Stabilization and destabilization of nonlinear differential equations by noise. IEEE Transactions on Automatic Control, 53:683–691, 2008.
  3. The uea multivariate time series classification archive, 2018. arXiv preprint arXiv:1811.00075, 2018.
  4. From detection of individual metastases to classification of lymph node status at the patient level: the camelyon17 challenge. IEEE transactions on medical imaging, 38(2), 2018.
  5. On stochastic gradient Langevin dynamics with dependent data streams: The fully nonconvex case. SIAM Journal on Mathematics of Data Science, 3(3):959–986, 2021.
  6. Recurrent neural networks for multivariate time series with missing values. Scientific Reports, 8(1):6085, 2018.
  7. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018.
  8. Doctor ai: Predicting clinical events via recurrent neural networks. In Machine learning for healthcare conference, pp.  301–318. PMLR, 2016.
  9. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555, 2014.
  10. Gru-ode-bayes: Continuous modeling of sporadically-observed time series. Advances in neural information processing systems, 32, 2019.
  11. Augmented neural odes. Advances in Neural Information Processing Systems, 32, 2019.
  12. Efficient and accurate estimation of lipschitz constants for deep neural networks. Advances in Neural Information Processing Systems, 32, 2019.
  13. Efficient discovery of variable-length time series motifs with large length range in million scale time series. arXiv preprint arXiv:1802.04883, 2018.
  14. Deep learning-based numerical methods for high-dimensional parabolic partial differential equations and backward stochastic differential equations. Communications in mathematics and statistics, 5(4):349–380, 2017.
  15. Long short-term memory. Neural computation, 9(8):1735–1780, 1997.
  16. Strong and weak divergence in finite time of euler’s method for stochastic differential equations with non-globally lipschitz continuous coefficients. Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences, 467:1563–1576, 2011.
  17. Ace-node: Attentive co-evolving neural ordinary differential equations. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pp.  736–745, 2021.
  18. Exit: Extrapolation and interpolation-based neural controlled differential equations for time-series classification and forecasting. In Proceedings of the ACM Web Conference 2022, pp.  3102–3112, 2022.
  19. Learnable path in neural controlled differential equations. arXiv preprint arXiv:2301.04333, 2023a.
  20. Attentive neural controlled differential equations for time-series classification and forecasting. Knowledge and Information Systems, pp.  1–31, 2023b.
  21. Neural jump stochastic differential equations. Advances in Neural Information Processing Systems, 32, 2019.
  22. Eamonn Keogh. Efficiently finding arbitrarily scaled patterns in massive time series databases. In European Conference on Principles of Data Mining and Knowledge Discovery, pp.  253–265. Springer, 2003.
  23. Rafail Khasminskii. Stochastic Stability of Differential Equations. Springer, 2011.
  24. Neural controlled differential equations for irregular time series. Advances in Neural Information Processing Systems, 33:6696–6707, 2020.
  25. Neural sdes as infinite-dimensional gans. In International Conference on Machine Learning, pp.  5453–5463. PMLR, 2021a.
  26. Efficient and accurate gradients for neural sdes. Advances in Neural Information Processing Systems, 34:18747–18761, 2021b.
  27. The pathwise convergence of approximation schemes for stochastic differential equations. LMS journal of Computation and Mathematics, 10:235–253, 2007.
  28. Sde-net: Equipping deep neural networks with uncertainty estimates. International Conference on Machine Learning, 37, 2020.
  29. Lipschitz constant estimation of neural networks via sparse polynomial optimization. arXiv preprint arXiv:2004.08688, 2020.
  30. Mind the gap: Assessing temporal generalization in neural language models. Advances in neural information processing systems, 2021.
  31. Learning long-term dependencies in irregularly-sampled time series. arXiv preprint arXiv:2006.04418, 2020.
  32. Multi-view integrative attention-based deep representation learning for irregular clinical time-series data. IEEE Journal of Biomedical and Health Informatics, 2022.
  33. Tackling the widespread and critical impact of batch effects in high-throughput data. Nature Reviews Genetics, 11(10), 2010.
  34. Detecting and adapting to irregular distribution shifts in bayesian online learning. Advances in neural information processing systems, 2021.
  35. Scalable gradients for stochastic differential equations. In International Conference on Artificial Intelligence and Statistics, pp.  3870–3882. PMLR, 2020.
  36. Tune: A research platform for distributed model selection and training. arXiv preprint arXiv:1807.05118, 2018.
  37. Polygonal unadjusted langevin algorithms: Creating stable and efficient adaptive algorithms for neural networks. arXiv preprint arXiv:2105.13937, 2021.
  38. Non-asymptotic estimates for tusla algorithm for non-convex learning with applications to neural networks with relu activation function. IMA Journal of numerical analysis, 2023a.
  39. Langevin dynamics based algorithm e-thε𝜀\varepsilonitalic_εo poula for stochastic optimization problems with discontinuous stochastic gradient. arXiv preprint arXiv:2210.13193, 2023b.
  40. Neural sde: Stabilizing neural ode networks with stochastic noise. arXiv preprint arXiv:1906.02355, 2019.
  41. sktime: A unified interface for machine learning with time series. arXiv preprint arXiv:1909.07872, 2019.
  42. X. Mao. Stochastic stabilisation and destabilisation. Systems and Control Letters, 23:279–290, 1994.
  43. Xuerong Mao. Stochastic differential equations and applications. Elsevier, 2007.
  44. Ergodicity for sdes and approximations: locally lipschitz vector fields and degenerate noise. Stochastic Processes and Their Applications, 101(2):185–232, 2002.
  45. Recurrent neural networks: design and applications. CRC press, 1999.
  46. The effect of natural distribution shift on question answering models. International Conference on Machine Learning, 2020.
  47. Accuracy on the line: on the strong correlation between out-of-distribution and in-distribution generalization. International Conference on Machine Learning, 2021.
  48. Ray: A distributed framework for emerging {{\{{AI}}\}} applications. In 13th {normal-{\{{USENIX}normal-}\}} Symposium on Operating Systems Design and Implementation ({normal-{\{{OSDI}normal-}\}} 18), pp.  561–577, 2018.
  49. Neural controlled differential equations for online prediction tasks. arXiv preprint arXiv:2106.11028, 2021a.
  50. Neural rough differential equations for long time series. In International Conference on Machine Learning, pp.  7829–7838. PMLR, 2021b.
  51. Discrete event, continuous time rnns. arXiv preprint arXiv:1710.04110, 2017.
  52. Stochastic differential equations. Springer, 2003.
  53. Can you trust your model’s uncertainty? evaluating predictive uncertainty under dataset shift. Advances in neural information processing systems, 33, 2019.
  54. Neural markov controlled sde: Stochastic optimization for continuous-time data. In International Conference on Learning Representations, 2021.
  55. Non-convex learning via stochastic gradient Langevin dynamics: a nonasymptotic analysis. In In Conference on Learning Theory, pp.  1674–1703. PMLR, 2017.
  56. Early prediction of sepsis from clinical data: the physionet/computing in cardiology challenge 2019. In 2019 Computing in Cardiology (CinC), pp.  Page–1. IEEE, 2019.
  57. Exponential convergence of langevin distributions and their discrete approximations. Bernoulli, 2(4):341–363, 1996.
  58. Andreas Rößler. Runge–kutta methods for stratonovich stochastic differential equation systems with commutative noise. Journal of computational and applied mathematics, 164:613–627, 2004.
  59. Latent ordinary differential equations for irregularly-sampled time series. Advances in neural information processing systems, 32, 2019.
  60. Learning representations by back-propagating errors. nature, 323(6088):533–536, 1986.
  61. Multi-time attention networks for irregularly sampled time series. arXiv preprint arXiv:2101.10318, 2021.
  62. Predicting in-hospital mortality of icu patients: The physionet/computing in cardiology challenge 2012. In 2012 Computing in Cardiology, pp.  245–248. IEEE, 2012.
  63. Diffusion for global optimization in rnsuperscript𝑟𝑛r^{n}italic_r start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT. SIAM Journal on Control and Optimization, 25(3):737–753, 1987.
  64. Time series classification for varying length series. arXiv preprint arXiv:1910.04341, 2019.
  65. Deepmind control suite. arXiv preprint arXiv:1801.00690, 2018.
  66. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems, pp.  5026–5033. IEEE, 2012.
  67. Neural stochastic differential equations: Deep latent gaussian models in the diffusion limit. arXiv preprint arXiv:1905.09883, 2019.
  68. Attention is all you need. Advances in neural information processing systems, 30, 2017.
  69. Lipschitz regularity of deep neural networks: analysis and efficient estimation. Advances in Neural Information Processing Systems, 31, 2018.
  70. Pete Warden. Speech commands: A dataset for limited-vocabulary speech recognition. arXiv preprint arXiv:1804.03209, 2018.
  71. Global convergence of Langevin dynamics based algorithms for nonconvex optimization. Advances in Neural Information Processing Systems, 32, 2018.
  72. Detecting time series motifs under uniform scaling. In Proceedings of the 13th ACM SIGKDD international conference on Knowledge discovery and data mining, pp.  844–853, 2007.
  73. A hitting time analysis of stochastic gradient langevin dynamics. In In Conference on Learning Theory, pp.  1980–2022. PMLR, 2017.
  74. Domain adaptation under missingness shift. International Conference on Artificial Intelligence and Statistics, 2023.
Citations (8)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets