Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
97 tokens/sec
GPT-4o
53 tokens/sec
Gemini 2.5 Pro Pro
44 tokens/sec
o3 Pro
5 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

K-Link: Knowledge-Link Graph from LLMs for Enhanced Representation Learning in Multivariate Time-Series Data (2403.03645v1)

Published 6 Mar 2024 in cs.AI

Abstract: Sourced from various sensors and organized chronologically, Multivariate Time-Series (MTS) data involves crucial spatial-temporal dependencies, e.g., correlations among sensors. To capture these dependencies, Graph Neural Networks (GNNs) have emerged as powerful tools, yet their effectiveness is restricted by the quality of graph construction from MTS data. Typically, existing approaches construct graphs solely from MTS signals, which may introduce bias due to a small training dataset and may not accurately represent underlying dependencies. To address this challenge, we propose a novel framework named K-Link, leveraging LLMs to encode extensive general knowledge and thereby providing effective solutions to reduce the bias. Leveraging the knowledge embedded in LLMs, such as physical principles, we extract a \textit{Knowledge-Link graph}, capturing vast semantic knowledge of sensors and the linkage of the sensor-level knowledge. To harness the potential of the knowledge-link graph in enhancing the graph derived from MTS data, we propose a graph alignment module, facilitating the transfer of semantic knowledge within the knowledge-link graph into the MTS-derived graph. By doing so, we can improve the graph quality, ensuring effective representation learning with GNNs for MTS data. Extensive experiments demonstrate the efficacy of our approach for superior performance across various MTS-related downstream tasks.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (45)
  1. Hybrid deep neural network model for remaining useful life estimation. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 3872–3876.
  2. Human activity recognition on smartphones using a multiclass hardware-friendly support vector machine. In International workshop on ambient assisted living. Springer, 216–223.
  3. Llm4ts: Two-stage fine-tuning for time-series forecasting with pre-trained llms. arXiv preprint arXiv:2308.08469 (2023).
  4. Multi-scale adaptive graph neural network for multivariate time series forecasting. IEEE Transactions on Knowledge and Data Engineering (2023).
  5. Machine remaining useful life prediction via an attention-based deep learning approach. IEEE Transactions on Industrial Electronics 68, 3 (2020), 2521–2531.
  6. Ailin Deng and Bryan Hooi. 2021. Graph neural network-based anomaly detection in multivariate time series. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 35. 4027–4035.
  7. Multivariate time series forecasting via attention-based encoder–decoder framework. Neurocomputing 388 (2020), 269–279. https://doi.org/10.1016/j.neucom.2019.12.118
  8. Time-Series Representation Learning via Temporal and Contextual Contrasting. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21. 2352–2359. https://doi.org/10.24963/ijcai.2021/324
  9. Unsupervised scalable representation learning for multivariate time series. Advances in neural information processing systems 32 (2019).
  10. Approaches and applications of early classification of time series: A review. IEEE Transactions on Artificial Intelligence 1, 1 (2020), 47–61.
  11. GraphSleepNet: Adaptive Spatial-Temporal Graph Convolutional Networks for Sleep Stage Classification.. In IJCAI. 1324–1330.
  12. A Survey on Graph Neural Networks for Time Series: Forecasting, Classification, Imputation, and Anomaly Detection. arXiv preprint arXiv:2307.03759 (2023).
  13. Time-llm: Time series forecasting by reprogramming large language models. arXiv preprint arXiv:2310.01728 (2023).
  14. ISRUC-Sleep: A comprehensive public dataset for sleep researchers. Computer methods and programs in biomedicine 124 (2016), 180–192.
  15. A Directed Acyclic Graph Network Combined With CNN and LSTM for Remaining Useful Life Prediction. IEEE Access 7 (2019), 75464–75475. https://doi.org/10.1109/ACCESS.2019.2919566
  16. Hierarchical attention graph convolutional network to fuse multi-sensor signals for remaining useful life prediction. Reliability Engineering & System Safety 215 (2021), 107878.
  17. Multireceptive Field Graph Convolutional Networks for Machine Fault Diagnosis. IEEE Transactions on Industrial Electronics 68, 12 (2021), 12739–12749. https://doi.org/10.1109/TIE.2020.3040669
  18. Time Series Classification With Multivariate Convolutional Neural Network. IEEE Transactions on Industrial Electronics 66, 6 (2019), 4788–4797. https://doi.org/10.1109/TIE.2018.2864702
  19. Remaining useful life prediction using a novel feature-attention-based end-to-end approach. IEEE Transactions on Industrial Informatics 17, 2 (2020), 1197–1207.
  20. DSTP-RNN: A dual-stage two-phase attention-based recurrent neural network for long-term and multivariate time series prediction. Expert Systems with Applications 143 (2020), 113082. https://doi.org/10.1016/j.eswa.2019.113082
  21. A Deep Adversarial Learning Prognostics Model for Remaining Useful Life Prediction of Rolling Bearing. IEEE Transactions on Artificial Intelligence 2, 4 (2021), 329–340. https://doi.org/10.1109/TAI.2021.3097311
  22. MHCCL: Masked Hierarchical Cluster-Wise Contrastive Learning for Multivariate Time Series. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 37. 9153–9161.
  23. Learning transferable visual models from natural language supervision. In International conference on machine learning. PMLR, 8748–8763.
  24. Language models are unsupervised multitask learners. OpenAI blog 1, 8 (2019), 9.
  25. Damage propagation modeling for aircraft engine run-to-failure simulation. In 2008 International Conference on Prognostics and Health Management. 1–9. https://doi.org/10.1109/PHM.2008.4711414
  26. TEST: Text prototype aligned embedding to activate LLM’s ability for time series. arXiv preprint arXiv:2308.08241 (2023).
  27. Laurens Van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-SNE. Journal of machine learning research 9, 11 (2008).
  28. Attention is all you need. Advances in neural information processing systems 30 (2017).
  29. Multiple convolutional neural networks for multivariate time series prediction. Neurocomputing 360 (2019), 107–119. https://doi.org/10.1016/j.neucom.2019.05.023
  30. Local–Global Correlation Fusion-Based Graph Neural Network for Remaining Useful Life Prediction. IEEE Transactions on Neural Networks and Learning Systems (2023).
  31. Multivariate Time Series Representation Learning via Hierarchical Correlation Pooling Boosted Graph Neural Network. IEEE Transactions on Artificial Intelligence (2023).
  32. SEnsor Alignment for Multivariate Time-Series Unsupervised Domain Adaptation. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 37. 10253–10261.
  33. Fully-Connected Spatial-Temporal Graph for Multivariate Time Series Data. arXiv preprint arXiv:2309.05305 (2023).
  34. Graph-Aware Contrasting for Multivariate Time Series Classification. arXiv preprint arXiv:2309.05202 (2023).
  35. Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. Advances in Neural Information Processing Systems 34 (2021), 22419–22430.
  36. A dual-stage attention-based Conv-LSTM network for spatio-temporal correlation and multivariate time series prediction. International Journal of Intelligent Systems 36, 5 (2021), 2036–2057.
  37. KDnet-RUL: A knowledge distillation framework to compress deep neural networks for machine remaining useful life prediction. IEEE Transactions on Industrial Electronics 69, 2 (2021), 2022–2032.
  38. SGBRT: an edge-intelligence based remaining useful life prediction model for aero-engine monitoring system. IEEE Transactions on Network Science and Engineering 9, 5 (2022), 3112–3122.
  39. Hao Xue and Flora D Salim. 2023. Promptcast: A new prompt-based learning paradigm for time series forecasting. IEEE Transactions on Knowledge and Data Engineering (2023).
  40. Deep learning and transfer learning for device-free human activity recognition: A survey. Journal of Automation and Intelligence 1, 1 (2022), 100007.
  41. Flames2graph: An interpretable federated multivariate time series classification framework. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 3140–3150.
  42. A transformer-based framework for multivariate time series representation learning. In Proceedings of the 27th ACM SIGKDD conference on knowledge discovery & data mining. 2114–2124.
  43. Tapnet: Multivariate time series classification with attentional prototypical network. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 34. 6845–6852.
  44. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 35. 11106–11115.
  45. One Fits All: Power General Time Series Analysis by Pretrained LM. arXiv preprint arXiv:2302.11939 (2023).
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (6)
  1. Yucheng Wang (83 papers)
  2. Ruibing Jin (5 papers)
  3. Min Wu (201 papers)
  4. Xiaoli Li (120 papers)
  5. Lihua Xie (212 papers)
  6. Zhenghua Chen (51 papers)
Citations (4)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now