Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
153 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

EgPDE-Net: Building Continuous Neural Networks for Time Series Prediction with Exogenous Variables (2208.01913v2)

Published 3 Aug 2022 in cs.LG

Abstract: While exogenous variables have a major impact on performance improvement in time series analysis, inter-series correlation and time dependence among them are rarely considered in the present continuous methods. The dynamical systems of multivariate time series could be modelled with complex unknown partial differential equations (PDEs) which play a prominent role in many disciplines of science and engineering. In this paper, we propose a continuous-time model for arbitrary-step prediction to learn an unknown PDE system in multivariate time series whose governing equations are parameterised by self-attention and gated recurrent neural networks. The proposed model, \underline{E}xogenous-\underline{g}uided \underline{P}artial \underline{D}ifferential \underline{E}quation Network (EgPDE-Net), takes account of the relationships among the exogenous variables and their effects on the target series. Importantly, the model can be reduced into a regularised ordinary differential equation (ODE) problem with special designed regularisation guidance, which makes the PDE problem tractable to obtain numerical solutions and feasible to predict multiple future values of the target series at arbitrary time points. Extensive experiments demonstrate that our proposed model could achieve competitive accuracy over strong baselines: on average, it outperforms the best baseline by reducing $9.85\%$ on RMSE and $13.98\%$ on MAE for arbitrary-step prediction.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (48)
  1. D. Chen, L. Chen, Y. Zhang, B. Wen, and C. Yang, “A multiscale interactive recurrent network for time-series forecasting,” IEEE Transactions on Cybernetics, vol. 52, no. 9, pp. 8793–8803, 2021.
  2. J. Wang, T. Sun, B. Liu, Y. Cao, and H. Zhu, “CLVSA: A convolutional LSTM based variational sequence-to-sequence model with attention for predicting trends of financial markets,” in Proceedings of the 28th International Joint Conference on Artificial Intelligence, 2021, pp. 3705–3711.
  3. X. Xu and M. Yoneda, “Multitask air-quality prediction based on lstm-autoencoder model,” IEEE transactions on cybernetics, vol. 51, no. 5, pp. 2577–2586, 2019.
  4. A. W. Mulyadi, E. Jun, and H.-I. Suk, “Uncertainty-aware variational-recurrent imputation network for clinical time series,” IEEE Transactions on Cybernetics, vol. 52, no. 9, pp. 9684–9694, 2021.
  5. Z. Xu, Y. Kang, Y. Cao, and Z. Li, “Spatiotemporal graph convolution multifusion network for urban vehicle emission prediction,” IEEE Transactions on Neural Networks and Learning Systems, vol. 32, no. 8, pp. 3342–3354, 2021.
  6. R. Yu, S. Zheng, A. Anandkumar, and Y. Yue, “Long-term forecasting using higher order tensor RNNs,” arXiv preprint arXiv:1711.00073, 2017.
  7. I. Fox, L. Ang, M. Jaiswal, R. Pop-Busui, and J. Wiens, “Deep multi-output forecasting: Learning to accurately predict blood glucose trajectories,” in Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD 2018, Y. Guo and F. Farooq, Eds., 2018, pp. 1387–1395.
  8. P. Gao, X. Yang, K. Huang, R. Zhang, and J. Y. Goulermas, “Explainable tensorized neural ordinary differential equations for arbitrary-step time series prediction,” IEEE Transactions on Knowledge and Data Engineering, 2022.
  9. T. Q. Chen, Y. Rubanova, J. Bettencourt, and D. K. Duvenaud, “Neural ordinary differential equations,” in Advances in the 31st Neural Information Processing Systems, vol. 31, 2018, pp. 6572–6583.
  10. Y. Rubanova, R. T. Q. Chen, and D. Duvenaud, “Latent ODEs for irregularly-sampled time series,” CoRR, vol. abs/1907.03907, 2019.
  11. M. Poli, S. Massaroli, J. Park, A. Yamashita, H. Asama, and J. Park, “Graph neural ordinary differential equations,” arXiv preprint arXiv:1911.07532, 2019.
  12. L.-P. Xhonneux, M. Qu, and J. Tang, “Continuous graph neural networks,” in Proceedings of the 37th International Conference on Machine Learning, 2020, pp. 10 432–10 441.
  13. Y. Li, H. Yi, C. M. Bender, S. Shan, and J. B. Oliva, “Exchangeable neural ODE for set modeling,” in Advances in the 33rd Neural Information Processing Systems, vol. 33, 2020, pp. 6936–6946.
  14. W. Grathwohl, R. T. Q. Chen, J. Bettencourt, I. Sutskever, and D. Duvenaud, “FFJORD: free-form continuous dynamics for scalable reversible generative models,” in Proceedings of the 7th International Conference on Learning Representations, 2019.
  15. P. Guo, K. Huang, and Z. Xu, “Partial differential equations is all you need for generating neural architectures–a theory for physical artificial intelligence systems,” arXiv preprint arXiv:2103.08313, 2021.
  16. J. Heo, H. B. Lee, S. Kim, J. Lee, K. J. Kim, E. Yang, and S. J. Hwang, “Uncertainty-aware attention for reliable interpretation and prediction,” in Advances in the 31st Neural Information Processing Systems, vol. 31, 2018, pp. 917–926.
  17. T. Guo, T. Lin, and N. Antulov-Fantulin, “Exploring interpretable LSTM neural networks over multi-variable data,” in Proceedings of the 36th International Conference on Machine Learning, vol. 97, 2019, pp. 2494–2504.
  18. G. Lai, W.-C. Chang, Y. Yang, and H. Liu, “Modeling long-and short-term temporal patterns with deep neural networks,” in Proceedings of the 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, 2018, pp. 95–104.
  19. M. Binkowski, G. Marti, and P. Donnat, “Autoregressive convolutional neural networks for asynchronous time series,” in Proceedings of the 35th International Conference on Machine Learning, vol. 80, 2018, pp. 579–588.
  20. J. Cheng, K. Huang, and Z. Zheng, “Towards better forecasting by fusing near and distant future visions,” in Proceedings of the 34th Association for the Advance of Artificial Intelligence Conference on Artificial Intelligence, vol. 34, no. 04, 2020, pp. 3593–3600.
  21. H. Akaike, “Fitting autoregressive models for prediction,” Annals of the institute of Statistical Mathematics, vol. 21, no. 1, pp. 243–247, 1969.
  22. C. A. Sims, “Macroeconomics and reality,” Econometrica: journal of the Econometric Society, pp. 1–48, 1980.
  23. Z. Zhang, T. Fu, Z. Yan, L. Jin, L. Xiao, Y. Sun, Z. Yu, and Y. Li, “A varying-parameter convergent-differential neural network for solving joint-angular-drift problems of redundant robot manipulators,” IEEE/ASME Transactions on Mechatronics, vol. 23, no. 2, pp. 679–689, 2018.
  24. Z. Zhang, L. Zheng, Z. Chen, L. Kong, and H. R. Karimi, “Mutual-collision-avoidance scheme synthesized by neural networks for dual redundant robot manipulators executing cooperative tasks,” IEEE Transactions on Neural Networks and Learning Systems, vol. 32, no. 3, pp. 1052–1066, 2020.
  25. Z. Zhang, B. Chen, S. Xu, G. Chen, and J. Xie, “A novel voting convergent difference neural network for diagnosing breast cancer,” Neurocomputing, vol. 437, pp. 339–350, 2021.
  26. I. Fox, L. Ang, M. Jaiswal, R. Pop-Busui, and J. Wiens, “Deep multi-output forecasting: Learning to accurately predict blood glucose trajectories,” in Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2018, pp. 1387–1395.
  27. C. Zang, “Deep learning in multiple multistep time series prediction,” CoRR, vol. abs/1710.04373, 2017.
  28. P. Kidger, J. Morrill, J. Foster, and T. J. Lyons, “Neural controlled differential equations for irregular time series,” in Advances in the 33rd Neural Information Processing Systems, vol. 33, 2020, pp. 6696–6707.
  29. S. Y. Jhin, J. Lee, M. Jo, S. Kook, J. Jeon, J. Hyeong, J. Kim, and N. Park, “Exit: Extrapolation and interpolation-based neural controlled differential equations for time-series classification and forecasting,” in Proceedings of the ACM Web Conference 2022, 2022, pp. 3102–3112.
  30. C. Finlay, J. Jacobsen, L. Nurbekyan, and A. M. Oberman, “How to train your neural ODE: the world of jacobian and kinetic regularization,” in Proceedings of the 37th International Conference on Machine Learning, vol. 119, 2020, pp. 3154–3164.
  31. V. Iakovlev, M. Heinonen, and H. Lähdesmäki, “Learning continuous-time PDEs from sparse data with graph neural networks,” in Proceedings of the 9th International Conference on Learning Representations, 2021.
  32. Z. Long, Y. Lu, X. Ma, and B. Dong, “Pde-net: Learning pdes from data,” in Proceedings of the 35th International Conference on Machine Learning, vol. 80, 2018, pp. 3214–3222.
  33. Z. Long, Y. Lu, and B. Dong, “PDE-Net 2.0: Learning PDEs from data with a numeric-symbolic hybrid deep network,” Journal of Computational Physics, vol. 399, p. 108925, 2019.
  34. Y. Luo, C. Xu, Y. Liu, W. Liu, S. Zheng, and J. Bian, “Learning differential operators for interpretable time series modeling,” in Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2022, pp. 1192–1201.
  35. E. Choi, M. T. Bahadori, J. Sun, J. Kulas, A. Schuetz, and W. Stewart, “Retain: An interpretable predictive model for healthcare using reverse time attention mechanism,” in Advances in the 29th Neural Information Processing Systems, vol. 29, 2016, pp. 3504–3512.
  36. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” in Advances in the 30th Neural Information Processing Systems, vol. 30, 2017, pp. 5998–6008.
  37. H. Zhou, S. Zhang, J. Peng, S. Zhang, J. Li, H. Xiong, and W. Zhang, “Informer: Beyond efficient transformer for long sequence time-series forecasting,” arXiv preprint arXiv:2012.07436, 2020.
  38. G. Bao, X. Ye, Y. Zang, and H. Zhou, “Numerical solution of inverse problems by weak adversarial networks,” Inverse Problems, vol. 36, no. 11, p. 115003, 2020.
  39. C. Beck, S. Becker, P. Cheridito, A. Jentzen, and A. Neufeld, “Deep splitting method for parabolic PDEs,” SIAM Journal on Scientific Computing, vol. 43, no. 5, pp. A3135–A3154, 2021.
  40. J. Sirignano and K. Spiliopoulos, “DGM: A deep learning algorithm for solving partial differential equations,” Journal of Computational Physics, vol. 375, pp. 1339–1364, 2018.
  41. F. Chen, J. Huang, C. Wang, and H. Yang, “Friedrichs learning: Weak solutions of partial differential equations via deep learning,” arXiv preprint arXiv:2012.08023, 2020.
  42. J. Han, A. Jentzen et al., “Deep learning-based numerical methods for high-dimensional parabolic partial differential equations and backward stochastic differential equations,” Communications in Mathematics and Statistics, vol. 5, no. 4, pp. 349–380, 2017.
  43. D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in Proceedings of the 3rd International Conference on Learning Representations, 2015.
  44. F. Zamora-Martinez, P. Romeu, P. Botella-Rocamora, and J. Pardo, “On-line learning of indoor temperature forecasting models towards energy efficiency,” Energy and Buildings, vol. 83, pp. 162–172, 2014.
  45. A. Jain, “HomeStead(US) Electricity Consumption,” https://www.kaggle.com/datasets/unajtheb/homesteadus-electricity-consumption.
  46. M. Jin, Y. Zheng, Y.-F. Li, S. Chen, B. Yang, and S. Pan, “Multivariate time series forecasting with dynamic graph neural odes,” IEEE Transactions on Knowledge and Data Engineering, 2022.
  47. J. Choi, H. Choi, J. Hwang, and N. Park, “Graph neural controlled differential equations for traffic forecasting,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 36, no. 6, 2022, pp. 6367–6374.
  48. Z. Fang, Q. Long, G. Song, and K. Xie, “Spatial-temporal graph ODE networks for traffic flow forecasting,” in Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining.   ACM, 2021, pp. 364–373.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now