Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
139 tokens/sec
GPT-4o
47 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

GATGPT: A Pre-trained Large Language Model with Graph Attention Network for Spatiotemporal Imputation (2311.14332v1)

Published 24 Nov 2023 in cs.LG and stat.ML

Abstract: The analysis of spatiotemporal data is increasingly utilized across diverse domains, including transportation, healthcare, and meteorology. In real-world settings, such data often contain missing elements due to issues like sensor malfunctions and data transmission errors. The objective of spatiotemporal imputation is to estimate these missing values by understanding the inherent spatial and temporal relationships in the observed multivariate time series. Traditionally, spatiotemporal imputation has relied on specific, intricate architectures designed for this purpose, which suffer from limited applicability and high computational complexity. In contrast, our approach integrates pre-trained LLMs into spatiotemporal imputation, introducing a groundbreaking framework, GATGPT. This framework merges a graph attention mechanism with LLMs. We maintain most of the LLM parameters unchanged to leverage existing knowledge for learning temporal patterns, while fine-tuning the upper layers tailored to various applications. The graph attention component enhances the LLM's ability to understand spatial relationships. Through tests on three distinct real-world datasets, our innovative approach demonstrates comparable results to established deep learning benchmarks.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (34)
  1. The quiet revolution of numerical weather prediction. Nature, 525(7567):47–55, 2015.
  2. Ai in healthcare: time-series forecasting using statistical, neural, and ensemble architectures. Frontiers in big data, 3:4, 2020.
  3. Stock price prediction based on deep neural networks. Neural Computing and Applications, 32:1609–1628, 2020.
  4. Graph wavenet for deep spatial-temporal graph modeling. arXiv preprint arXiv:1906.00121, 2019.
  5. A review of deep learning models for time series prediction. IEEE Sensors Journal, 21(6):7833–7848, 2019.
  6. Deep learning for time series classification: a review. Data mining and knowledge discovery, 33(4):917–963, 2019.
  7. A review on outlier/anomaly detection in time series data. ACM Computing Surveys (CSUR), 54(3):1–33, 2021.
  8. On the estimation of arima models with missing values. In Time Series Analysis of Irregularly Observed Data: Proceedings of a Symposium held at Texas A & M University, College Station, Texas February 10–13, 1983, pages 9–37. Springer, 1984.
  9. An approach to time series smoothing and forecasting using the em algorithm. Journal of time series analysis, 3(4):253–264, 1982.
  10. knn ensembles with penalized dtw for multivariate time series imputation. In 2016 International Joint Conference on Neural Networks (IJCNN), pages 2774–2781. IEEE, 2016.
  11. The elements of statistical learning: data mining, inference, and prediction, volume 2. Springer, 2009.
  12. Brits: Bidirectional recurrent imputation for time series. Advances in neural information processing systems, 31, 2018.
  13. Recurrent neural networks for multivariate time series with missing values. Scientific reports, 8(1):1–12, 2018.
  14. Generative semi-supervised learning for multivariate time series imputation. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 8983–8991, 2021.
  15. Towards language models that can see: Computer vision through the lens of natural language. arXiv preprint arXiv:2306.16410, 2023.
  16. mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint arXiv:2304.14178, 2023.
  17. Is chatgpt fair for recommendation? evaluating fairness in large language model recommendation. arXiv preprint arXiv:2305.07609, 2023.
  18. Multiple imputation by chained equations: what is it and how does it work? International journal of methods in psychiatric research, 20(1):40–49, 2011.
  19. Paramasivan Arumugam and R Saranya. Outlier detection and missing value in seasonal arima model using rainfall data. Materials Today: Proceedings, 5(1):1791–1799, 2018.
  20. Gain: Missing data imputation using generative adversarial nets. In International conference on machine learning, pages 5689–5698. PMLR, 2018.
  21. Filling the gaps: Multivariate time series imputation by graph neural networks. arXiv preprint arXiv:2108.00298, 2021.
  22. Promptcast: A new prompt-based learning paradigm for time series forecasting. 2022.
  23. Large language models are zero-shot time series forecasters. arXiv preprint arXiv:2310.07820, 2023.
  24. Llm4ts: Two-stage fine-tuning for time-series forecasting with pre-trained llms. arXiv preprint arXiv:2308.08469, 2023.
  25. Test: Text prototype aligned embedding to activate llm’s ability for time series. arXiv preprint arXiv:2308.08241, 2023.
  26. Time-llm: Time series forecasting by reprogramming large language models. arXiv preprint arXiv:2310.01728, 2023.
  27. One fits all: Power general time series analysis by pretrained lm. arXiv preprint arXiv:2302.11939, 2023.
  28. Lag-llama: Towards foundation models for time series forecasting. arXiv preprint arXiv:2310.08278, 2023.
  29. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.
  30. Multiple imputation using chained equations: issues and guidance for practice. Statistics in medicine, 30(4):377–399, 2011.
  31. Temporal regularized matrix factorization for high-dimensional time series prediction. Advances in neural information processing systems, 29, 2016.
  32. Missing traffic data imputation and pattern discovery with a bayesian augmented tensor factorization model. Transportation Research Part C: Emerging Technologies, 104:66–77, 2019.
  33. Gp-vae: Deep probabilistic time series imputation. In International conference on artificial intelligence and statistics, pages 1651–1661. PMLR, 2020.
  34. St-mvl: filling missing values in geo-sensory time series data. In Proceedings of the 25th International Joint Conference on Artificial Intelligence, 2016.
Citations (19)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now