Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
156 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

ComS2T: A complementary spatiotemporal learning system for data-adaptive model evolution (2403.01738v1)

Published 4 Mar 2024 in cs.LG

Abstract: Spatiotemporal (ST) learning has become a crucial technique to enable smart cities and sustainable urban development. Current ST learning models capture the heterogeneity via various spatial convolution and temporal evolution blocks. However, rapid urbanization leads to fluctuating distributions in urban data and city structures over short periods, resulting in existing methods suffering generalization and data adaptation issues. Despite efforts, existing methods fail to deal with newly arrived observations and those methods with generalization capacity are limited in repeated training. Motivated by complementary learning in neuroscience, we introduce a prompt-based complementary spatiotemporal learning termed ComS2T, to empower the evolution of models for data adaptation. ComS2T partitions the neural architecture into a stable neocortex for consolidating historical memory and a dynamic hippocampus for new knowledge update. We first disentangle two disjoint structures into stable and dynamic weights, and then train spatial and temporal prompts by characterizing distribution of main observations to enable prompts adaptive to new data. This data-adaptive prompt mechanism, combined with a two-stage training process, facilitates fine-tuning of the neural architecture conditioned on prompts, thereby enabling efficient adaptation during testing. Extensive experiments validate the efficacy of ComS2T in adapting to various spatiotemporal out-of-distribution scenarios while maintaining efficient inference capabilities.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (71)
  1. Z. Wu, S. Pan, G. Long, J. Jiang, X. Chang, and C. Zhang, “Connecting the dots: Multivariate time series forecasting with graph neural networks,” in KDD, 2020, pp. 753–763.
  2. Q. Huang, L. Shen, R. Zhang, S. Ding, B. Wang, Z. Zhou, and Y. Wang, “Crossgnn: Confronting noisy multivariate time series via cross interaction refinement,” in Thirty-seventh Conference on Neural Information Processing Systems, 2023.
  3. Y. Zheng, L. Zhong, S. Wang, Y. Yang, W. Gu, J. Zhang, and J. Wang, “Diffuflow: Robust fine-grained urban flow inference with denoising diffusion model,” in Proceedings of the 32nd ACM International Conference on Information and Knowledge Management, 2023, pp. 3505–3513.
  4. G. Jin, Y. Liang, Y. Fang, J. Huang, J. Zhang, and Y. Zheng, “Spatio-temporal graph neural networks for predictive learning in urban computing: A survey,” arXiv preprint arXiv:2303.14483, 2023.
  5. S. Wang, J. Cao, and S. Y. Philip, “Deep learning for spatio-temporal data mining: A survey,” IEEE transactions on knowledge and data engineering, vol. 34, no. 8, pp. 3681–3700, 2020.
  6. B. Lu, X. Gan, W. Zhang, H. Yao, L. Fu, and X. Wang, “Spatio-temporal graph few-shot learning with cross-city knowledge transfer,” in Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2022, pp. 1162–1172.
  7. Z. Pan, W. Zhang, Y. Liang, W. Zhang, Y. Yu, J. Zhang, and Y. Zheng, “Spatio-temporal meta learning for urban traffic prediction,” IEEE Transactions on Knowledge and Data Engineering, vol. 34, no. 3, pp. 1462–1476, 2020.
  8. Y. Liu, T. Hu, H. Zhang, H. Wu, S. Wang, L. Ma, and M. Long, “itransformer: Inverted transformers are effective for time series forecasting,” in The Twelfth International Conference on Learning Representations, 2023.
  9. J. Dong, H. Wu, H. Zhang, L. Zhang, J. Wang, and M. Long, “Simmtm: A simple pre-training framework for masked time-series modeling,” Advances in Neural Information Processing Systems, vol. 36, 2024.
  10. H. Wu, T. Hu, Y. Liu, H. Zhou, J. Wang, and M. Long, “Timesnet: Temporal 2d-variation modeling for general time series analysis,” in The Eleventh International Conference on Learning Representations, 2022.
  11. Y. Liu, H. Wu, J. Wang, and M. Long, “Non-stationary transformers: Rethinking the stationarity in time series forecasting,” arXiv preprint arXiv:2205.14415, 2022.
  12. Z. Zhou, Y. Wang, X. Xie, L. Chen, and C. Zhu, “Foresee urban sparse traffic accidents: A spatiotemporal multi-granularity perspective,” IEEE TKDE, 2020.
  13. J. Ji, J. Wang, C. Huang, J. Wu, B. Xu, Z. Wu, J. Zhang, and Y. Zheng, “Spatio-temporal self-supervised learning for traffic flow prediction,” in Proceedings of the AAAI Conference on Artificial Intelligence, 2023.
  14. Y. Li, R. Yu, C. Shahabi, and Y. Liu, “Diffusion convolutional recurrent neural network: Data-driven traffic forecasting,” in International Conference on Learning Representations, 2018.
  15. Z. Zhou, Y. Wang, X. Xie, L. Chen, and H. Liu, “Riskoracle: A minute-level citywide traffic accident forecasting framework,” in Proceedings of the AAAI conference on artificial intelligence, vol. 34, no. 01, 2020, pp. 1258–1265.
  16. H. Wu, H. Zhou, M. Long, and J. Wang, “Interpretable weather forecasting for worldwide stations with a unified deep model,” Nature Machine Intelligence, pp. 1–10, 2023.
  17. R. Castro, Y. M. Souto, E. Ogasawara, F. Porto, and E. Bezerra, “Stconvs2s: Spatiotemporal convolutional sequence to sequence network for weather forecasting,” Neurocomputing, vol. 426, pp. 285–298, 2021.
  18. K. Chen, T. Han, J. Gong, L. Bai, F. Ling, J.-J. Luo, X. Chen, L. Ma, T. Zhang, R. Su et al., “Fengwu: Pushing the skillful global medium-range weather forecast beyond 10 days lead,” arXiv preprint arXiv:2304.02948, 2023.
  19. Y. Zhang, M. Long, K. Chen, L. Xing, R. Jin, M. I. Jordan, and J. Wang, “Skilful nowcasting of extreme precipitation with nowcastnet,” Nature, vol. 619, no. 7970, pp. 526–532, 2023.
  20. Y. Liang, Y. Xia, S. Ke, Y. Wang, Q. Wen, J. Zhang, Y. Zheng, and R. Zimmermann, “Airformer: Predicting nationwide air quality in china with transformers,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 37, no. 12, 2023, pp. 14 329–14 337.
  21. W. Du, L. Chen, H. Wang, Z. Shan, Z. Zhou, W. Li, and Y. Wang, “Deciphering urban traffic impacts on air quality by deep learning and emission inventory,” Journal of Environmental Sciences, vol. 124, pp. 745–757, 2023.
  22. S. Du, T. Li, Y. Yang, and S.-J. Horng, “Deep air quality forecasting using hybrid deep learning framework,” IEEE Transactions on Knowledge and Data Engineering, vol. 33, no. 6, pp. 2412–2424, 2019.
  23. L. Chen, J. Xu, B. Wu, and J. Huang, “Group-aware graph neural network for nationwide city air quality forecasting,” ACM Transactions on Knowledge Discovery from Data, vol. 18, no. 3, pp. 1–20, 2023.
  24. K. Wang, Y. Liang, X. Li, G. Li, B. Ghanem, R. Zimmermann, H. Yi, Y. Zhang, Y. Wang et al., “Brave the wind and the waves: Discovering robust and generalizable graph lottery tickets,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
  25. Y. Wu, X. Wang, A. Zhang, X. He, and T.-S. Chua, “Discovering invariant rationales for graph neural networks,” in International Conference on Learning Representations, 2021.
  26. Q. Wu, H. Zhang, J. Yan, and D. Wipf, “Handling distribution shifts on graphs: An invariance perspective,” in International Conference on Learning Representations, 2022.
  27. S. Li, X. Wang, A. Zhang, Y. Wu, X. He, and T.-S. Chua, “Let invariant rationale discovery inspire graph contrastive learning,” in International Conference on Machine Learning.   PMLR, 2022, pp. 13 052–13 065.
  28. Y. Du, J. Wang, W. Feng, S. Pan, T. Qin, R. Xu, and C. Wang, “Adarnn: Adaptive learning and forecasting of time series,” in Proceedings of the 30th ACM International Conference on Information & Knowledge Management, 2021, pp. 402–411.
  29. Z. Zhou, Q. Huang, K. Yang, K. Wang, X. Wang, Y. Zhang, Y. Liang, and Y. Wang, “Maintaining the status quo: Capturing invariant relations for ood spatiotemporal learning,” in Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2023, p. 3603–3614.
  30. X. Chen, J. Wang, and K. Xie, “Trafficstream: A streaming traffic flow forecasting framework based on graph neural networks and continual learning,” arXiv preprint arXiv:2106.06273, 2021.
  31. B. Wang, Y. Zhang, X. Wang, P. Wang, Z. Zhou, L. Bai, and Y. Wang, “Pattern expansion and consolidation on evolving graphs for continual traffic prediction,” in Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2023, pp. 2223–2232.
  32. Q. Wang, B. Guo, L. Cheng, and Z. Yu, “surban: Stable prediction for unseen urban data from location-based sensors,” Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, vol. 7, no. 3, pp. 1–20, 2023.
  33. Y. Xia, Y. Liang, H. Wen, X. Liu, K. Wang, Z. Zhou, and R. Zimmermann, “Deciphering spatio-temporal graph forecasting: A causal lens and treatment,” in Thirty-seventh Conference on Neural Information Processing Systems, 2023.
  34. H. Yuan, Q. Sun, X. Fu, Z. Zhang, C. Ji, H. Peng, and J. Li, “Environment-aware dynamic graph learning for out-of-distribution generalization,” in Thirty-seventh Conference on Neural Information Processing Systems, 2023.
  35. R. C. O’Reilly, R. Bhattacharyya, M. D. Howard, and N. Ketz, “Complementary learning systems,” Cognitive science, vol. 38, no. 6, pp. 1229–1248, 2014.
  36. D. Kumaran, D. Hassabis, and J. L. McClelland, “What learning systems do intelligent agents need? complementary learning systems theory updated,” Trends in cognitive sciences, vol. 20, no. 7, pp. 512–534, 2016.
  37. J. L. McClelland, B. L. McNaughton, and A. K. Lampinen, “Integration of new information in memory: new insights from a complementary learning systems perspective,” Philosophical Transactions of the Royal Society B, vol. 375, no. 1799, p. 20190637, 2020.
  38. C. S. Lee and A. Y. Lee, “Clinical applications of continual learning machine learning,” The Lancet Digital Health, vol. 2, no. 6, pp. e279–e281, 2020.
  39. E. Arani, F. Sarfraz, and B. Zonooz, “Learning fast, learning slow: A general continual learning method based on complementary learning system,” in International Conference on Learning Representations, 2022.
  40. Z. Zhang, X. Zhao, Q. Liu, C. Zhang, Q. Ma, W. Wang, H. Zhao, Y. Wang, and Z. Liu, “Promptst: Prompt-enhanced spatio-temporal multi-attribute prediction,” in Proceedings of the 32nd ACM International Conference on Information and Knowledge Management, 2023, pp. 3195–3205.
  41. X. Liu, K. Ji, Y. Fu, W. Tam, Z. Du, Z. Yang, and J. Tang, “P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks,” in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), 2022, pp. 61–68.
  42. H. Guo, T. Ruiming, Y. Ye, Z. Li, and X. He, “Deepfm: A factorization-machine based neural network for ctr prediction,” in Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence.   International Joint Conferences on Artificial Intelligence Organization, 2017.
  43. S. Wang, Y. Li, J. Zhang, Q. Meng, L. Meng, and F. Gao, “Pm2. 5-gnn: A domain knowledge enhanced graph neural network for pm2. 5 forecasting,” in Proceedings of the 28th International Conference on Advances in Geographic Information Systems, 2020, pp. 163–166.
  44. Z. Wu, S. Pan, G. Long, J. Jiang, and C. Zhang, “Graph wavenet for deep spatial-temporal graph modeling,” in Proceedings of the 28th International Joint Conference on Artificial Intelligence, 2019, pp. 1907–1913.
  45. S. Guo, Y. Lin, N. Feng, C. Song, and H. Wan, “Attention based spatial-temporal graph convolutional networks for traffic flow forecasting,” in AAAI, vol. 33, no. 01, 2019, pp. 922–929.
  46. S. Wang, H. Miao, H. Chen, and Z. Huang, “Multi-task adversarial spatial-temporal networks for crowd flow prediction,” in Proceedings of the 29th ACM international conference on information & knowledge management, 2020, pp. 1555–1564.
  47. X. Ouyang, Y. Yang, W. Zhou, Y. Zhang, H. Wang, and W. Huang, “Citytrans: Domain-adversarial training with knowledge transfer for spatio-temporal prediction across cities,” IEEE Transactions on Knowledge and Data Engineering, 2023.
  48. Y. Zheng, L. Capra, O. Wolfson, and H. Yang, “Urban computing: concepts, methodologies, and applications,” ACM Transactions on Intelligent Systems and Technology (TIST), vol. 5, no. 3, pp. 1–55, 2014.
  49. Y. Liang, K. Ouyang, Y. Wang, Y. Liu, J. Zhang, Y. Zheng, and D. S. Rosenblum, “Revisiting convolutional neural networks for citywide crowd flow analytics,” in Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2020, Ghent, Belgium, September 14–18, 2020, Proceedings, Part I.   Springer, 2021, pp. 578–594.
  50. F. Amato, F. Guignard, S. Robert, and M. Kanevski, “A novel framework for spatio-temporal prediction of environmental data using deep learning,” Scientific reports, vol. 10, no. 1, p. 22243, 2020.
  51. X. Liu, “Spatial and temporal dependence in house price prediction,” The Journal of Real Estate Finance and Economics, vol. 47, no. 2, pp. 341–369, 2013.
  52. P. Wang, C. Ge, Z. Zhou, X. Wang, Y. Li, and Y. Wang, “Joint gated co-attention based multi-modal networks for subregion house price prediction,” IEEE Transactions on Knowledge and Data Engineering, 2021.
  53. J. Zhang, Y. Zheng, and D. Qi, “Deep spatio-temporal residual networks for citywide crowd flows prediction,” in Thirty-first AAAI conference on artificial intelligence, 2017.
  54. J. Ye, L. Sun, B. Du, Y. Fu, X. Tong, and H. Xiong, “Co-prediction of multiple transportation demands based on deep spatio-temporal neural network,” in Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, 2019, pp. 305–313.
  55. B. Yu, H. Yin, and Z. Zhu, “Spatio-temporal graph convolutional networks: a deep learning framework for traffic forecasting,” in IJCAI, 2018, pp. 3634–3640.
  56. L. Bai, L. Yao, C. Li, X. Wang, and C. Wang, “Adaptive graph convolutional recurrent network for traffic forecasting,” NIPS, vol. 33, 2020.
  57. Y. Wang, H. Wu, J. Zhang, Z. Gao, J. Wang, S. Y. Philip, and M. Long, “Predrnn: A recurrent neural network for spatiotemporal predictive learning,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 2, pp. 2208–2225, 2022.
  58. Z. Yao, Y. Wang, H. Wu, J. Wang, and M. Long, “Modernn: Harnessing spatiotemporal mode collapse in unsupervised predictive learning,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
  59. Y. Wang, Z. Gao, M. Long, J. Wang, and S. Y. Philip, “Predrnn++: Towards a resolution of the deep-in-time dilemma in spatiotemporal predictive learning,” in International Conference on Machine Learning.   PMLR, 2018, pp. 5123–5132.
  60. Y. Yuan, C. Shao, J. Ding, D. Jin, and Y. Li, “A generative pre-training framework for spatio-temporal graph transfer learning,” arXiv preprint arXiv:2402.11922, 2024.
  61. G. I. Parisi, J. Tani, C. Weber, and S. Wermter, “Lifelong learning of spatiotemporal representations with dual-memory recurrent self-organization,” Frontiers in neurorobotics, vol. 12, p. 78, 2018.
  62. G. W. Lindsay, “Attention in psychology, neuroscience, and machine learning,” Frontiers in computational neuroscience, vol. 14, p. 29, 2020.
  63. M. W. Mathis and A. Mathis, “Deep learning tools for the measurement of animal behavior in neuroscience,” Current opinion in neurobiology, vol. 60, pp. 1–11, 2020.
  64. A. Bessadok, M. A. Mahjoub, and I. Rekik, “Graph neural networks in network neuroscience,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 5, pp. 5833–5848, 2022.
  65. N. Gupta et al., “Artificial neural network,” Network and Complex Systems, vol. 3, no. 1, pp. 24–28, 2013.
  66. F. Zenke, B. Poole, and S. Ganguli, “Continual learning through synaptic intelligence,” in International conference on machine learning.   PMLR, 2017, pp. 3987–3995.
  67. J. Zou, Y. Han, and S.-S. So, “Overview of artificial neural networks,” Artificial neural networks: methods and applications, pp. 14–22, 2009.
  68. J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska-Barwinska et al., “Overcoming catastrophic forgetting in neural networks,” Proceedings of the national academy of sciences, vol. 114, no. 13, pp. 3521–3526, 2017.
  69. G. M. Van de Ven, H. T. Siegelmann, and A. S. Tolias, “Brain-inspired replay for continual learning with artificial neural networks,” Nature communications, vol. 11, no. 1, p. 4069, 2020.
  70. L. Wang, X. Zhang, Q. Li, M. Zhang, H. Su, J. Zhu, and Y. Zhong, “Incorporating neuro-inspired adaptability for continual learning in artificial intelligence,” Nature Machine Intelligence, pp. 1–13, 2023.
  71. L. Wang, X. Zhang, H. Su, and J. Zhu, “A comprehensive survey of continual learning: Theory, method and application,” arXiv preprint arXiv:2302.00487, 2023.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now