Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
112 tokens/sec
GPT-4o
8 tokens/sec
Gemini 2.5 Pro Pro
47 tokens/sec
o3 Pro
5 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Caformer: Rethinking Time Series Analysis from Causal Perspective (2403.08572v1)

Published 13 Mar 2024 in cs.LG

Abstract: Time series analysis is a vital task with broad applications in various domains. However, effectively capturing cross-dimension and cross-time dependencies in non-stationary time series poses significant challenges, particularly in the context of environmental factors. The spurious correlation induced by the environment confounds the causal relationships between cross-dimension and cross-time dependencies. In this paper, we introduce a novel framework called Caformer (\underline{\textbf{Ca}}usal Trans\underline{\textbf{former}}) for time series analysis from a causal perspective. Specifically, our framework comprises three components: Dynamic Learner, Environment Learner, and Dependency Learner. The Dynamic Learner unveils dynamic interactions among dimensions, the Environment Learner mitigates spurious correlations caused by environment with a back-door adjustment, and the Dependency Learner aims to infer robust interactions across both time and dimensions. Our Caformer demonstrates consistent state-of-the-art performance across five mainstream time series analysis tasks, including long- and short-term forecasting, imputation, classification, and anomaly detection, with proper interpretability.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (40)
  1. Y. Liu, T. Hu, H. Zhang, H. Wu, S. Wang, L. Ma, and M. Long, “itransformer: Inverted transformers are effective for time series forecasting,” arXiv preprint arXiv:2310.06625, 2023.
  2. N. Passalis, A. Tefas, J. Kanniainen, M. Gabbouj, and A. Iosifidis, “Deep adaptive input normalization for time series forecasting,” IEEE Transactions on Neural Networks and Learning Systems, vol. 31, no. 9, pp. 3760–3765, 2020.
  3. W. Zheng and J. Hu, “Multivariate time series prediction based on temporal change information learning method,” IEEE Transactions on Neural Networks and Learning Systems, vol. 34, no. 10, pp. 7034–7048, 2023.
  4. Z. Wu, D. Zheng, S. Pan, Q. Gan, G. Long, and G. Karypis, “Traversenet: Unifying space and time in message passing for traffic forecasting,” IEEE Transactions on Neural Networks and Learning Systems, vol. 35, no. 2, pp. 2003–2013, 2024.
  5. H. Wu, T. Hu, Y. Liu, H. Zhou, J. Wang, and M. Long, “Timesnet: Temporal 2d-variation modeling for general time series analysis,” in The Eleventh International Conference on Learning Representations, 2023.
  6. J. Xu, H. Wu, J. Wang, and M. Long, “Anomaly transformer: Time series anomaly detection with association discrepancy,” in International Conference on Learning Representations, 2022.
  7. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin, “Attention is all you need,” in Advances in Neural Information Processing Systems, vol. 30.   Curran Associates, Inc., 2017.
  8. Q. Wen, T. Zhou, C. Zhang, W. Chen, Z. Ma, J. Yan, and L. Sun, “Transformers in time series: A survey,” arXiv preprint arXiv:2202.07125, 2022.
  9. Y. Zhang and J. Yan, “Crossformer: Transformer utilizing cross-dimension dependency for multivariate time series forecasting,” in The Eleventh International Conference on Learning Representations, 2023.
  10. Y. Liu, H. Wu, J. Wang, and M. Long, “Non-stationary transformers: Exploring the stationarity in time series forecasting,” in Advances in Neural Information Processing Systems, vol. 35.   Curran Associates, Inc., 2022, pp. 9881–9893.
  11. H. Wu, J. Xu, J. Wang, and M. Long, “Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting,” in Advances in Neural Information Processing Systems, M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, Eds., vol. 34.   Curran Associates, Inc., 2021, pp. 22 419–22 430.
  12. Y. Nie, N. H. Nguyen, P. Sinthong, and J. Kalagnanam, “A time series is worth 64 words: Long-term forecasting with transformers,” in The Eleventh International Conference on Learning Representations, 2023.
  13. F. M. Bianchi, S. Scardapane, S. Løkse, and R. Jenssen, “Reservoir computing approaches for representation and classification of multivariate time series,” IEEE Transactions on Neural Networks and Learning Systems, vol. 32, no. 5, pp. 2169–2179, 2021.
  14. V. Ekambaram, A. Jati, N. Nguyen, P. Sinthong, and J. Kalagnanam, “Tsmixer: Lightweight mlp-mixer model for multivariate time series forecasting,” arXiv preprint arXiv:2306.09364, 2023.
  15. A. Zeng, M. Chen, L. Zhang, and Q. Xu, “Are transformers effective for time series forecasting?” in Proceedings of the AAAI conference on Artificial Intelligence, vol. 37, no. 9, June 2023, pp. 11 121–11 128.
  16. K. Zhang, C. Li, and Q. Yang, “Trid-mae: A generic pre-trained model for multivariate time series with missing values,” in Proceedings of the 32nd ACM International Conference on Information and Knowledge Management.   New York, NY, USA: Association for Computing Machinery, 2023, p. 3164–3173.
  17. T. Zhou, Z. Ma, Q. Wen, X. Wang, L. Sun, and R. Jin, “FEDformer: Frequency enhanced decomposed transformer for long-term series forecasting,” in Proceedings of the 39th International Conference on Machine Learning, K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu, and S. Sabato, Eds., vol. 162.   PMLR, July 2022, pp. 27 268–27 286.
  18. S. Yang, M. Dong, Y. Wang, and C. Xu, “Adversarial recurrent time series imputation,” IEEE Transactions on Neural Networks and Learning Systems, vol. 34, no. 4, pp. 1639–1650, 2023.
  19. M. Liu, S. Ren, S. Ma, J. Jiao, Y. Chen, Z. Wang, and W. Song, “Gated transformer networks for multivariate time series classification,” arXiv preprint arXiv:2103.14438, 2021.
  20. R. Zuo, G. Li, B. Choi, S. S Bhowmick, D. N.-y. Mah, and G. L. Wong, “Svp-t: A shape-level variable-position transformer for multivariate time series classification,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 37, no. 9, June 2023, pp. 11 497–11 505.
  21. Y. Zheng, H. Y. Koh, M. Jin, L. Chi, K. T. Phan, S. Pan, Y.-P. P. Chen, and W. Xiang, “Correlation-aware spatial–temporal graph learning for multivariate time-series anomaly detection,” IEEE Transactions on Neural Networks and Learning Systems, pp. 1–15, 2023.
  22. H. Zhou, S. Zhang, J. Peng, S. Zhang, J. Li, H. Xiong, and W. Zhang, “Informer: Beyond efficient transformer for long sequence time-series forecasting,” vol. 35, no. 12, May 2021, pp. 11 106–11 115.
  23. M. Barahona and C.-S. Poon, “Detection of nonlinear dynamics in short, noisy time series,” Nature, vol. 381, no. 6579, pp. 215–217, 1996.
  24. M. Zečević, D. S. Dhami, P. Veličković, and K. Kersting, “Relating graph neural networks to structural causal models,” arXiv preprint arXiv:2109.04173, 2021.
  25. S. Löwe, D. Madras, R. Zemel, and M. Welling, “Amortized causal discovery: Learning to infer causal graphs from time-series data,” in Proceedings of the First Conference on Causal Learning and Reasoning, ser. Proceedings of Machine Learning Research, vol. 177.   PMLR, April 2022, pp. 509–525.
  26. A. Goyal, A. Lamb, J. Hoffmann, S. Sodhani, S. Levine, Y. Bengio, and B. Schölkopf, “Recurrent independent mechanisms,” in International Conference on Learning Representations, 2021.
  27. J. Runge, A. Gerhardus, G. Varando, V. Eyring, and G. Camps-Valls, “Causal inference for time series,” Nature Reviews Earth & Environment, vol. 4, no. 7, pp. 487–505, July 2023.
  28. D. Zhang, H. Zhang, J. Tang, X.-S. Hua, and Q. Sun, “Causal intervention for weakly-supervised semantic segmentation,” in Advances in Neural Information Processing Systems, vol. 33.   Curran Associates, Inc., 2020, pp. 655–666.
  29. E. D. Abraham, K. D’Oosterlinck, A. Feder, Y. Gat, A. Geiger, C. Potts, R. Reichart, and Z. Wu, “Cebab: Estimating the causal effects of real-world concepts on nlp model behavior,” in Advances in Neural Information Processing Systems, vol. 35.   Curran Associates, Inc., 2022, pp. 17 582–17 596.
  30. L. Castri, S. Mghames, and N. Bellotto, “From continual learning to causal discovery in robotics,” in Proceedings of The First AAAI Bridge Program on Continual Causality, ser. Proceedings of Machine Learning Research, vol. 208.   PMLR, February 2023, pp. 85–91.
  31. B. D. Haig, “What is a spurious correlation?” Understanding Statistics, vol. 2, no. 2, pp. 125–132, 2003.
  32. G. Woo, C. Liu, D. Sahoo, A. Kumar, and S. Hoi, “Etsformer: Exponential smoothing transformers for time-series forecasting,” arXiv preprint arXiv:2202.01381, 2022.
  33. T. Zhang, Y. Zhang, W. Cao, J. Bian, X. Yi, S. Zheng, and J. Li, “Less is more: Fast multivariate time series forecasting with light sampling-oriented mlp structures,” arXiv preprint arXiv:2207.01186, 2022.
  34. S. Makridakis, E. Spiliotis, and V. Assimakopoulos, “The m4 competition: 100,000 time series and 61 forecasting methods,” International Journal of Forecasting, vol. 36, no. 1, pp. 54–74, 2020, m4 Competition.
  35. G. Lai, W.-C. Chang, Y. Yang, and H. Liu, “Modeling long- and short-term temporal patterns with deep neural networks,” in The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval.   New York, NY, USA: Association for Computing Machinery, 2018, p. 95–104.
  36. A. Bagnall, H. A. Dau, J. Lines, M. Flynn, J. Large, A. Bostrom, P. Southam, and E. Keogh, “The uea multivariate time series classification archive, 2018,” arXiv preprint arXiv:1811.00075, 2018.
  37. Y. Su, Y. Zhao, C. Niu, R. Liu, W. Sun, and D. Pei, “Robust anomaly detection for multivariate time series through stochastic recurrent neural network,” in Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining.   New York, NY, USA: Association for Computing Machinery, 2019, p. 2828–2837.
  38. K. Hundman, V. Constantinou, C. Laporte, I. Colwell, and T. Soderstrom, “Detecting spacecraft anomalies using lstms and nonparametric dynamic thresholding,” in Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining.   New York, NY, USA: Association for Computing Machinery, 2018, p. 387–395.
  39. A. P. Mathur and N. O. Tippenhauer, “Swat: A water treatment testbed for research and training on ics security,” in 2016 International Workshop on Cyber-physical Systems for Smart Water Networks (CySWater).   IEEE, 2016, pp. 31–36.
  40. A. Abdulaal, Z. Liu, and T. Lancewicki, “Practical approach to asynchronous multivariate time series anomaly detection and localization,” in Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining.   New York, NY, USA: Association for Computing Machinery, 2021, p. 2485–2494.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets