Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
162 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

MTAD: Tools and Benchmarks for Multivariate Time Series Anomaly Detection (2401.06175v1)

Published 10 Jan 2024 in cs.SE, cs.AI, and cs.LG

Abstract: Key Performance Indicators (KPIs) are essential time-series metrics for ensuring the reliability and stability of many software systems. They faithfully record runtime states to facilitate the understanding of anomalous system behaviors and provide informative clues for engineers to pinpoint the root causes. The unprecedented scale and complexity of modern software systems, however, make the volume of KPIs explode. Consequently, many traditional methods of KPI anomaly detection become impractical, which serves as a catalyst for the fast development of machine learning-based solutions in both academia and industry. However, there is currently a lack of rigorous comparison among these KPI anomaly detection methods, and re-implementation demands a non-trivial effort. Moreover, we observe that different works adopt independent evaluation processes with different metrics. Some of them may not fully reveal the capability of a model and some are creating an illusion of progress. To better understand the characteristics of different KPI anomaly detectors and address the evaluation issue, in this paper, we provide a comprehensive review and evaluation of twelve state-of-the-art methods, and propose a novel metric called salience. Particularly, the selected methods include five traditional machine learning-based methods and seven deep learning-based methods. These methods are evaluated with five multivariate KPI datasets that are publicly available. A unified toolkit with easy-to-use interfaces is also released. We report the benchmark results in terms of accuracy, salience, efficiency, and delay, which are of practical importance for industrial deployment. We believe our work can contribute as a basis for future academic research and industrial application.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (37)
  1. Z. Chen, Y. Kang, L. Li, X. Zhang, H. Zhang, H. Xu, Y. Zhou, L. Yang, J. Sun, Z. Xu, Y. Dang, F. Gao, P. Zhao, B. Qiao, Q. Lin, D. Zhang, and M. R. Lyu, “Towards intelligent incident management: why we need it and how we make it,” in Proceedings of the 28th Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, (ESEC/FSE), 2020, pp. 1487–1497.
  2. X. Zhang, Q. Lin, Y. Xu, S. Qin, H. Zhang, B. Qiao, Y. Dang, X. Yang, Q. Cheng, M. Chintalapati, Y. Wu, K. Hsieh, K. Sui, X. Meng, Y. Xu, W. Zhang, F. Shen, and D. Zhang, “Cross-dataset time series anomaly detection for cloud systems,” in Proceedings of the Annual Technical Conference, (USENIX ATC), 2019, pp. 1063–1076.
  3. Y. Dang, Q. Lin, and P. Huang, “Aiops: real-world challenges and research innovations,” in Proceedings of the 41st International Conference on Software Engineering: Companion Proceedings, (ICSE), 2019, pp. 4–5.
  4. S. He, P. He, Z. Chen, T. Yang, Y. Su, and M. R. Lyu, “A survey on automated log analysis for reliability engineering,” CoRR, vol. abs/2009.07237, 2020.
  5. S. He, Q. Lin, J. Lou, H. Zhang, M. R. Lyu, and D. Zhang, “Identifying impactful service system problems via log analysis,” in Proceedings of the Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, (ESEC/FSE) FSE.   ACM, 2018, pp. 60–70.
  6. W. Gu, J. Liu, Z. Chen, J. Zhang, Y. Su, J. Gu, C. Feng, Z. Yang, and M. Lyu, “Performance issue identification in cloud systems with relational-temporal anomaly detection,” arXiv preprint arXiv:2307.10869, 2023.
  7. K. Hundman, V. Constantinou, C. Laporte, I. Colwell, and T. Söderström, “Detecting spacecraft anomalies using lstms and nonparametric dynamic thresholding,” in Proceedings of the 24th International Conference on Knowledge Discovery & Data Mining, (KDD), 2018, pp. 387–395.
  8. A. Borghesi, A. Bartolini, M. Lombardi, M. Milano, and L. Benini, “Anomaly detection using autoencoders in high performance computing systems,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, no. 01, 2019, pp. 9428–9433.
  9. D. Park, Y. Hoshi, and C. C. Kemp, “A multimodal anomaly detector for robot-assisted feeding using an lstm-based variational autoencoder,” CoRR, vol. abs/1711.00614, 2017.
  10. D. Li, D. Chen, B. Jin, L. Shi, J. Goh, and S.-K. Ng, “Mad-gan: Multivariate anomaly detection for time series data with generative adversarial networks,” in International Conference on Artificial Neural Networks.   Springer, 2019, pp. 703–716.
  11. C. Zhang, D. Song, Y. Chen, X. Feng, C. Lumezanu, W. Cheng, J. Ni, B. Zong, H. Chen, and N. V. Chawla, “A deep neural network for unsupervised anomaly detection and diagnosis in multivariate time series data,” in Proceedings of the 33rd Applications of Artificial Intelligence Conference, (AAAI), 2019, pp. 1409–1416.
  12. A. Siffer, P.-A. Fouque, A. Termier, and C. Largouet, “Anomaly detection in streams with extreme value theory,” in Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2017, pp. 1067–1075.
  13. H. Xu, W. Chen, N. Zhao, Z. Li, J. Bu, Z. Li, Y. Liu, Y. Zhao, D. Pei, Y. Feng, J. Chen, Z. Wang, and H. Qiao, “Unsupervised anomaly detection via variational auto-encoder for seasonal kpis in web applications,” in Proceedings of the 2018 World Wide Web Conference on World Wide Web, (WWW).   ACM, 2018, pp. 187–196.
  14. H. Zhao, Y. Wang, J. Duan, C. Huang, D. Cao, Y. Tong, B. Xu, J. Bai, J. Tong, and Q. Zhang, “Multivariate time-series anomaly detection via graph attention network,” in 20th IEEE International Conference on Data Mining, ICDM 2020, Sorrento, Italy, November 17-20, 2020.   IEEE, 2020, pp. 841–850.
  15. A. Deng and B. Hooi, “Graph neural network-based anomaly detection in multivariate time series,” in Proceedings of the 35th AAAI Conference on Artificial Intelligence, 2021.
  16. M. Braei and S. Wagner, “Anomaly detection in univariate time-series: A survey on the state-of-the-art,” CoRR, vol. abs/2004.00433, 2020.
  17. S. Ramaswamy, R. Rastogi, and K. Shim, “Efficient algorithms for mining outliers from large data sets,” in Proceedings of the 2000 ACM SIGMOD international conference on Management of data, 2000, pp. 427–438.
  18. M. M. Breunig, H.-P. Kriegel, R. T. Ng, and J. Sander, “Lof: identifying density-based local outliers,” in Proceedings of the 2000 ACM SIGMOD international conference on Management of data, 2000, pp. 93–104.
  19. F. T. Liu, K. M. Ting, and Z. Zhou, “Isolation forest,” in Proceedings of the 8th International Conference on Data Mining (ICDM), 2008, pp. 413–422.
  20. T. Pevnỳ, “Loda: Lightweight on-line detector of anomalies,” Machine Learning, vol. 102, no. 2, pp. 275–304, 2016.
  21. Y. Su, Y. Zhao, C. Niu, R. Liu, W. Sun, and D. Pei, “Robust anomaly detection for multivariate time series through stochastic recurrent neural network,” in Proceedings of the 25th International Conference on Knowledge Discovery & Data Mining, (KDD), 2019, pp. 2828–2837.
  22. B. Zong, Q. Song, M. R. Min, W. Cheng, C. Lumezanu, D. Cho, and H. Chen, “Deep autoencoding gaussian mixture model for unsupervised anomaly detection,” in Proceedings of the 6th International Conference on Learning Representations, (ICLR), 2018.
  23. H. Ren, B. Xu, Y. Wang, C. Yi, C. Huang, X. Kou, T. Xing, M. Yang, J. Tong, and Q. Zhang, “Time-series anomaly detection service at microsoft,” in Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2019, pp. 3009–3017.
  24. J. Audibert, P. Michiardi, F. Guyard, S. Marti, and M. A. Zuluaga, “USAD: unsupervised anomaly detection on multivariate time series,” in Proceedings of the 26th SIGKDD Conference on Knowledge Discovery and Data Mining, (KDD).   ACM, 2020, pp. 3395–3404.
  25. H. Ren, B. Xu, Y. Wang, C. Yi, C. Huang, X. Kou, T. Xing, M. Yang, J. Tong, and Q. Zhang, “Time-series anomaly detection service at microsoft,” in Proceedings of the 25th International Conference on Knowledge Discovery & Data Mining, (KDD), 2019, pp. 3009–3017.
  26. S. He, J. Zhu, P. He, and M. R. Lyu, “Experience report: System log analysis for anomaly detection,” in Proceedings of the 27th International Symposium on Software Reliability Engineering, (ISSRE), 2016, pp. 207–218.
  27. M. Du, F. Li, G. Zheng, and V. Srikumar, “Deeplog: Anomaly detection and diagnosis from system logs through deep learning,” in Proceedings of the 2017 Conference on Computer and Communications Security, (CCS).   ACM, 2017, pp. 1285–1298.
  28. A. Siffer, P. Fouque, A. Termier, and C. Largouët, “Anomaly detection in streams with extreme value theory,” in Proceedings of the 23rd SIGKDD International Conference on Knowledge Discovery and Data Mining, (KDD).   ACM, 2017, pp. 1067–1075.
  29. Wikipedia, “Complete-linkage,” http://en.wikipedia.org/wiki/Complete-linkage_clustering, 2021, [Online; accessed 23-April-2021].
  30. Y. Zhao, Z. Nasrullah, and Z. Li, “Pyod: A python toolbox for scalable outlier detection,” Journal of Machine Learning Research, vol. 20, no. 96, pp. 1–7, 2019. [Online]. Available: http://jmlr.org/papers/v20/19-011.html
  31. J. Lu, A. Liu, F. Dong, F. Gu, J. Gama, and G. Zhang, “Learning under concept drift: A review,” IEEE Trans. Knowl. Data Eng., vol. 31, no. 12, pp. 2346–2363, 2019.
  32. B. Hermann, S. Winter, and J. Siegmund, “Community expectations for research artifacts and evaluation processes,” in ESEC/FSE ’20: 28th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, Virtual Event, USA, November 8-13, 2020, P. Devanbu, M. B. Cohen, and T. Zimmermann, Eds.   ACM, 2020, pp. 469–480.
  33. A. P. Mathur and N. O. Tippenhauer, “Swat: a water treatment testbed for research and training on ICS security,” in 2016 International Workshop on Cyber-physical Systems for Smart Water Networks, CySWater@CPSWeek 2016, Vienna, Austria, April 11, 2016.   IEEE Computer Society, 2016, pp. 31–36.
  34. J. Goh, S. Adepu, K. N. Junejo, and A. Mathur, “A dataset to support research in the design of secure water treatment systems,” in Critical Information Infrastructures Security - 11th International Conference, CRITIS 2016, Paris, France, October 10-12, 2016, Revised Selected Papers, ser. Lecture Notes in Computer Science.   Springer, 2016, pp. 88–99.
  35. P. Malhotra, A. Ramakrishnan, G. Anand, L. Vig, P. Agarwal, and G. Shroff, “Lstm-based encoder-decoder for multi-sensor anomaly detection,” CoRR, vol. abs/1607.00148, 2016.
  36. I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. C. Courville, and Y. Bengio, “Generative adversarial nets,” in Proceedings of the 27th Conference on Neural Information Processing Systems 2014, (NeurIPS), 2014, pp. 2672–2680.
  37. H. Wang and D. Yeung, “Towards bayesian deep learning: A framework and some existing methods,” IEEE Trans. Knowl. Data Eng., vol. 28, no. 12, pp. 3395–3408, 2016.
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets