Papers
Topics
Authors
Recent
Search
2000 character limit reached

Higher-order Cross-structural Embedding Model for Time Series Analysis

Published 30 Oct 2024 in cs.LG and cs.AI | (2410.22984v1)

Abstract: Time series analysis has gained significant attention due to its critical applications in diverse fields such as healthcare, finance, and sensor networks. The complexity and non-stationarity of time series make it challenging to capture the interaction patterns across different timestamps. Current approaches struggle to model higher-order interactions within time series, and focus on learning temporal or spatial dependencies separately, which limits performance in downstream tasks. To address these gaps, we propose Higher-order Cross-structural Embedding Model for Time Series (High-TS), a novel framework that jointly models both temporal and spatial perspectives by combining multiscale Transformer with Topological Deep Learning (TDL). Meanwhile, High-TS utilizes contrastive learning to integrate these two structures for generating robust and discriminative representations. Extensive experiments show that High-TS outperforms state-of-the-art methods in various time series tasks and demonstrate the importance of higher-order cross-structural information in improving model performance.

Authors (3)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (69)
  1. A. Gharehbaghi and M. Lindén, “A deep machine learning method for classifying cyclic time series of biological signals using time-growing neural network,” IEEE transactions on neural networks and learning systems, vol. 29, no. 9, pp. 4102–4115, 2017.
  2. A. Gupta, H. P. Gupta, B. Biswas, and T. Dutta, “Approaches and applications of early classification of time series: A review,” IEEE Transactions on Artificial Intelligence, vol. 1, no. 1, pp. 47–61, 2020.
  3. J. Yang, Y. Xu, H. Cao, H. Zou, and L. Xie, “Deep learning and transfer learning for device-free human activity recognition: A survey,” Journal of Automation and Intelligence, vol. 1, no. 1, p. 100007, 2022.
  4. Z. Liu, W. Pei, D. Lan, and Q. Ma, “Diffusion language-shapelets for semi-supervised time-series classification,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, no. 13, 2024, pp. 14 079–14 087.
  5. Z. Zhao, W. Chen, X. Wu, P. C. Chen, and J. Liu, “LSTM network: a deep learning approach for short-term traffic forecast,” IET intelligent transport systems, vol. 11, no. 2, pp. 68–75, 2017.
  6. S. M. Shahid, S. Ko, and S. Kwon, “Performance comparison of 1d and 2d convolutional neural networks for real-time classification of time series sensor data,” in 2022 International Conference on Information Networking (ICOIN).   IEEE, 2022, pp. 507–511.
  7. Z. Wu, S. Pan, F. Chen, G. Long, C. Zhang, and S. Y. Philip, “A comprehensive survey on graph neural networks,” IEEE transactions on neural networks and learning systems, vol. 32, no. 1, pp. 4–24, 2020.
  8. F. Scarselli, M. Gori, A. C. Tsoi, M. Hagenbuchner, and G. Monfardini, “The graph neural network model,” IEEE transactions on neural networks, vol. 20, no. 1, pp. 61–80, 2008.
  9. L. R. Medsker, L. Jain et al., “Recurrent neural networks,” Design and Applications, vol. 5, no. 64-67, p. 2, 2001.
  10. J. Schmidhuber, S. Hochreiter et al., “Long short-term memory,” Neural Comput, vol. 9, no. 8, pp. 1735–1780, 1997.
  11. K. Cho, B. van Merriënboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio, “Learning phrase representations using RNN encoder–decoder for statistical machine translation,” in Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), A. Moschitti, B. Pang, and W. Daelemans, Eds.   Doha, Qatar: Association for Computational Linguistics, Oct. 2014, pp. 1724–1734. [Online]. Available: https://aclanthology.org/D14-1179
  12. A. Vaswani, “Attention is all you need,” Advances in Neural Information Processing Systems, 2017.
  13. J. Devlin, M. Chang, K. Lee, and K. Toutanova, “BERT: pre-training of deep bidirectional transformers for language understanding,” in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), J. Burstein, C. Doran, and T. Solorio, Eds.   Association for Computational Linguistics, 2019, pp. 4171–4186. [Online]. Available: https://doi.org/10.18653/v1/n19-1423
  14. S. Li, X. Jin, Y. Xuan, X. Zhou, W. Chen, Y.-X. Wang, and X. Yan, “Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting,” Advances in neural information processing systems, vol. 32, 2019.
  15. M. Jin, H. Y. Koh, Q. Wen, D. Zambon, C. Alippi, G. I. Webb, I. King, and S. Pan, “A survey on graph neural networks for time series: Forecasting, classification, imputation, and anomaly detection,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
  16. Y. Wang, Y. Xu, J. Yang, M. Wu, X. Li, L. Xie, and Z. Chen, “Fully-connected spatial-temporal graph for multivariate time-series data,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, no. 14, 2024, pp. 15 715–15 724.
  17. Z. Jia, Y. Lin, J. Wang, R. Zhou, X. Ning, Y. He, and Y. Zhao, “Graphsleepnet: Adaptive spatial-temporal graph convolutional networks for sleep stage classification,” in Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20, C. Bessiere, Ed.   International Joint Conferences on Artificial Intelligence Organization, 7 2020, pp. 1324–1330, main track. [Online]. Available: https://doi.org/10.24963/ijcai.2020/184
  18. Y. Wang, Y. Xu, J. Yang, Z. Chen, M. Wu, X. Li, and L. Xie, “Sensor alignment for multivariate time-series unsupervised domain adaptation,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 37, no. 8, 2023, pp. 10 253–10 261.
  19. Y. Wang, Y. Xu, J. Yang, M. Wu, X. Li, L. Xie, and Z. Chen, “Graph-aware contrasting for multivariate time-series classification,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, no. 14, 2024, pp. 15 725–15 734.
  20. Y. Bengio, P. Simard, and P. Frasconi, “Learning long-term dependencies with gradient descent is difficult,” IEEE transactions on neural networks, vol. 5, no. 2, pp. 157–166, 1994.
  21. S. Siami-Namini, N. Tavakoli, and A. S. Namin, “The performance of lstm and bilstm in forecasting time series,” in 2019 IEEE International conference on big data (Big Data).   IEEE, 2019, pp. 3285–3292.
  22. H. Wang, Y. Zhang, J. Liang, and L. Liu, “Dafa-bilstm: Deep autoregression feature augmented bidirectional lstm network for time series prediction,” Neural Networks, vol. 157, pp. 240–256, 2023.
  23. X. Wang and Y. Zhang, “Multi-step-ahead time series prediction method with stacking lstm neural network,” in 2020 3rd International Conference on Artificial Intelligence and Big Data (ICAIBD).   IEEE, 2020, pp. 51–55.
  24. A. Essien and C. Giannetti, “A deep learning framework for univariate time series prediction using convolutional lstm stacked autoencoders,” in 2019 IEEE International Symposium on INnovations in Intelligent SysTems and Applications (INISTA).   IEEE, 2019, pp. 1–6.
  25. B. Lim, S. Ö. Arık, N. Loeff, and T. Pfister, “Temporal fusion transformers for interpretable multi-horizon time series forecasting,” International Journal of Forecasting, vol. 37, no. 4, pp. 1748–1764, 2021.
  26. D. Li, Y. Tan, Y. Zhang, S. Miao, and S. He, “Probabilistic forecasting method for mid-term hourly load time series based on an improved temporal fusion transformer model,” International Journal of Electrical Power & Energy Systems, vol. 146, p. 108743, 2023.
  27. H. Zhou, S. Zhang, J. Peng, S. Zhang, J. Li, H. Xiong, and W. Zhang, “Informer: Beyond efficient transformer for long sequence time-series forecasting,” in Proceedings of the AAAI conference on artificial intelligence, vol. 35, no. 12, 2021, pp. 11 106–11 115.
  28. S. Bai, J. Z. Kolter, and V. Koltun, “An empirical evaluation of generic convolutional and recurrent networks for sequence modeling,” arXiv preprint arXiv:1803.01271, 2018.
  29. K. Mao, Z. Dou, H. Qian, F. Mo, X. Cheng, and Z. Cao, “Convtrans: Transforming web search sessions for conversational dense retrieval,” in Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, 2022, pp. 2935–2946.
  30. X. Li, Y. Chen, X. Zhang, Y. Peng, D. Zhang, and Y. Chen, “Convtrans-cl: Ocean time series temperature data anomaly detection based context contrast learning,” Applied Ocean Research, vol. 150, p. 104122, 2024.
  31. N. Kitaev, L. Kaiser, and A. Levskaya, “Reformer: The efficient transformer,” in International Conference on Learning Representations, 2020. [Online]. Available: https://openreview.net/forum?id=rkgNKkHtvB
  32. T. Zhou, Z. Ma, Q. Wen, X. Wang, L. Sun, and R. Jin, “Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting,” in International conference on machine learning.   PMLR, 2022, pp. 27 268–27 286.
  33. M. Liu, A. Zeng, M. Chen, Z. Xu, Q. Lai, L. Ma, and Q. Xu, “Scinet: Time series modeling and forecasting with sample convolution and interaction,” Advances in Neural Information Processing Systems, vol. 35, pp. 5816–5828, 2022.
  34. M. Costa, A. L. Goldberger, and C.-K. Peng, “Multiscale entropy analysis of complex physiologic time series,” Physical review letters, vol. 89, no. 6, p. 068102, 2002.
  35. Y. Li, B. Tang, S. Jiao, and Q. Su, “Snake optimization-based variable-step multiscale single threshold slope entropy for complexity analysis of signals,” IEEE Transactions on Instrumentation and Measurement, 2023.
  36. C. Li, K. Zeng, B. Li, G. Li, H. Yang, and S. Li, “Internal short-circuit fault diagnosis for batteries of energy storage stations based on multivariate multiscale sample entropy,” IEEE Transactions on Industrial Electronics, 2024.
  37. Y. Li, R. Yu, C. Shahabi, and Y. Liu, “Diffusion convolutional recurrent neural network: Data-driven traffic forecasting,” in International Conference on Learning Representations, 2018. [Online]. Available: https://openreview.net/forum?id=SJiHXGWAZ
  38. B. Yu, H. Yin, and Z. Zhu, “Spatio-temporal graph convolutional networks: a deep learning framework for traffic forecasting,” in Proceedings of the 27th International Joint Conference on Artificial Intelligence, ser. IJCAI’18.   AAAI Press, 2018, p. 3634–3640.
  39. Z. Wu, S. Pan, G. Long, J. Jiang, and C. Zhang, “Graph wavenet for deep spatial-temporal graph modeling,” in Proceedings of the 28th International Joint Conference on Artificial Intelligence, ser. IJCAI’19.   AAAI Press, 2019, p. 1907–1913.
  40. Z. Wu, S. Pan, G. Long, J. Jiang, X. Chang, and C. Zhang, “Connecting the dots: Multivariate time series forecasting with graph neural networks,” in Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, 2020, pp. 753–763.
  41. L. Bai, L. Yao, C. Li, X. Wang, and C. Wang, “Adaptive graph convolutional recurrent network for traffic forecasting,” Advances in neural information processing systems, vol. 33, pp. 17 804–17 815, 2020.
  42. J. Popović and H. Hoppe, “Progressive simplicial complexes,” in Proceedings of the 24th annual conference on Computer graphics and interactive techniques, 1997, pp. 217–224.
  43. S. Ebli, M. Defferrard, and G. Spreemann, “Simplicial neural networks,” in TDA & Beyond, 2020. [Online]. Available: https://openreview.net/forum?id=nPCt39DVIfk
  44. M. T. Schaub, A. R. Benson, P. Horn, G. Lippner, and A. Jadbabaie, “Random walks on simplicial complexes and the normalized hodge 1-laplacian,” SIAM Review, vol. 62, no. 2, pp. 353–391, 2020.
  45. T. M. Roddenberry, N. Glaze, and S. Segarra, “Principled simplicial neural networks for trajectory prediction,” in International Conference on Machine Learning.   PMLR, 2021, pp. 9020–9029.
  46. M. Gidea and Y. Katz, “Topological data analysis of financial time series: Landscapes of crashes,” Physica A: Statistical mechanics and its applications, vol. 491, pp. 820–834, 2018.
  47. C. W. J. Goh, C. Bodnar, and P. Lio, “Simplicial attention networks,” in ICLR 2022 Workshop on Geometrical and Topological Representation Learning, 2022. [Online]. Available: https://openreview.net/forum?id=ScfRNWkpec
  48. F. Mémoli, Z. Wan, and Y. Wang, “Persistent laplacians: Properties, algorithms and implications,” SIAM Journal on Mathematics of Data Science, vol. 4, no. 2, pp. 858–884, 2022.
  49. M. A. Shabani, A. H. Abdi, L. Meng, and T. Sylvain, “Scaleformer: Iterative multi-scale refining transformers for time series forecasting,” in The Eleventh International Conference on Learning Representations, 2023. [Online]. Available: https://openreview.net/forum?id=sCrnllCtjoE
  50. Y. Zhang and J. Yan, “Crossformer: Transformer utilizing cross-dimension dependency for multivariate time series forecasting,” in The eleventh international conference on learning representations, 2023.
  51. M. Hajij, G. Zamzmi, T. Papamarkou, N. Miolane, A. Guzmán-Sáenz, K. N. Ramamurthy, T. Birdal, T. K. Dey, S. Mukherjee, S. N. Samaga et al., “Topological deep learning: Going beyond graph data,” arXiv preprint arXiv:2206.00606, 2022.
  52. C. Bodnar, “Topological deep learning: Graphs, complexes, sheaves,” Ph.D. dissertation, Apollo - University of Cambridge Repository, 2022. [Online]. Available: https://www.repository.cam.ac.uk/handle/1810/350982
  53. L. Giusti, C. Battiloro, L. Testa, P. Di Lorenzo, S. Sardellitti, and S. Barbarossa, “Cell attention networks,” in 2023 International Joint Conference on Neural Networks (IJCNN).   IEEE, 2023, pp. 1–8.
  54. M. T. Schaub, J.-B. Seby, F. Frantzen, T. M. Roddenberry, Y. Zhu, and S. Segarra, “Signal processing on simplicial complexes,” in Higher-Order Systems.   Springer, 2022, pp. 301–328.
  55. Y. Feng, H. You, Z. Zhang, R. Ji, and Y. Gao, “Hypergraph neural networks,” in Proceedings of the AAAI conference on artificial intelligence, vol. 33, no. 01, 2019, pp. 3558–3565.
  56. Y. You, T. Chen, Y. Sui, T. Chen, Z. Wang, and Y. Shen, “Graph contrastive learning with augmentations,” Advances in neural information processing systems, vol. 33, pp. 5812–5823, 2020.
  57. R. G. Andrzejak, K. Lehnertz, F. Mormann, C. Rieke, P. David, and C. E. Elger, “Indications of nonlinear deterministic and finite-dimensional structures in time series of brain electrical activity: Dependence on recording region and brain state,” Physical Review E, vol. 64, no. 6, p. 061907, 2001.
  58. E. Eldele, M. Ragab, Z. Chen, M. Wu, C. K. Kwoh, X. Li, and C. Guan, “Time-series representation learning via temporal and contextual contrasting,” in Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, 2021, pp. 2352–2359.
  59. R. Dhandapani, I. Mitiche, S. McMeekin, and G. Morison, “A novel bearing faults detection method using generalized gaussian distribution refined composite multiscale dispersion entropy,” IEEE Transactions on Instrumentation and Measurement, vol. 71, pp. 1–12, 2022.
  60. X. Wang, S. Si, and Y. Li, “Hierarchical diversity entropy for the early fault diagnosis of rolling bearing,” Nonlinear Dynamics, vol. 108, no. 2, pp. 1447–1462, 2022.
  61. Y. Chen, B. Hu, E. Keogh, and G. E. Batista, “Dtw-d: time series semi-supervised learning from a single example,” in Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining, 2013, pp. 383–391.
  62. S. Tonekaboni, D. Eytan, and A. Goldenberg, “Unsupervised representation learning for time series with temporal neighborhood coding,” in International Conference on Learning Representations, 2021. [Online]. Available: https://openreview.net/forum?id=8qDwejCuCN
  63. G. Zerveas, S. Jayaraman, D. Patel, A. Bhamidipaty, and C. Eickhoff, “A transformer-based framework for multivariate time series representation learning,” in Proceedings of the 27th ACM SIGKDD conference on knowledge discovery & data mining, 2021, pp. 2114–2124.
  64. J.-Y. Franceschi, A. Dieuleveut, and M. Jaggi, “Unsupervised scalable representation learning for multivariate time series,” Advances in neural information processing systems, vol. 32, 2019.
  65. Z. Yue, Y. Wang, J. Duan, T. Yang, C. Huang, Y. Tong, and B. Xu, “Ts2vec: Towards universal representation of time series,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 36, no. 8, 2022, pp. 8980–8987.
  66. A. Dempster, F. Petitjean, and G. I. Webb, “Rocket: exceptionally fast and accurate time series classification using random convolutional kernels,” Data Mining and Knowledge Discovery, vol. 34, no. 5, pp. 1454–1495, 2020.
  67. E. Qu, Y. Wang, X. Luo, W. He, K. Ren, and D. Li, “Cnn kernels can be the best shapelets,” in The Twelfth International Conference on Learning Representations, 2024.
  68. L. Van der Maaten and G. Hinton, “Visualizing data using t-sne.” Journal of machine learning research, vol. 9, no. 11, 2008.
  69. D. L. Davies and D. W. Bouldin, “A cluster separation measure,” IEEE transactions on pattern analysis and machine intelligence, no. 2, pp. 224–227, 1979.

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up

Tweets

Sign up for free to view the 1 tweet with 0 likes about this paper.

Sign Up for Free