Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
91 tokens/sec
GPT-4o
12 tokens/sec
Gemini 2.5 Pro Pro
o3 Pro
5 tokens/sec
GPT-4.1 Pro
37 tokens/sec
DeepSeek R1 via Azure Pro
33 tokens/sec
Gemini 2.5 Flash Deprecated
12 tokens/sec
2000 character limit reached

Sub-Adjacent Transformer: Improving Time Series Anomaly Detection with Reconstruction Error from Sub-Adjacent Neighborhoods (2404.18948v1)

Published 27 Apr 2024 in cs.LG

Abstract: In this paper, we present the Sub-Adjacent Transformer with a novel attention mechanism for unsupervised time series anomaly detection. Unlike previous approaches that rely on all the points within some neighborhood for time point reconstruction, our method restricts the attention to regions not immediately adjacent to the target points, termed sub-adjacent neighborhoods. Our key observation is that owing to the rarity of anomalies, they typically exhibit more pronounced differences from their sub-adjacent neighborhoods than from their immediate vicinities. By focusing the attention on the sub-adjacent areas, we make the reconstruction of anomalies more challenging, thereby enhancing their detectability. Technically, our approach concentrates attention on the non-diagonal areas of the attention matrix by enlarging the corresponding elements in the training stage. To facilitate the implementation of the desired attention matrix pattern, we adopt linear attention because of its flexibility and adaptability. Moreover, a learnable mapping function is proposed to improve the performance of linear attention. Empirically, the Sub-Adjacent Transformer achieves state-of-the-art performance across six real-world anomaly detection benchmarks, covering diverse fields such as server monitoring, space exploration, and water treatment.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (49)
  1. Practical approach to asynchronous multivariate time series anomaly detection and localization. In SIGKDD, pages 2485–2494, 2021.
  2. Wadi: a water distribution testbed for research in the design of secure cyber physical systems. In Proceedings of the 3rd international workshop on cyber-physical systems for smart water networks, pages 25–28, 2017.
  3. Usad: Unsupervised anomaly detection on multivariate time series. In SIGKDD, pages 3395–3404, 2020.
  4. A review on outlier/anomaly detection in time series data. ACM Computing Surveys, 54(3):1–33, April 2021.
  5. Hydra attention: Efficient attention with many heads. In ECCV, pages 35–49. Springer, 2022.
  6. Series2graph: Graph-based subsequence anomaly detection for time series. arXiv preprint arXiv:2207.12208, 2022.
  7. Lof: identifying density-based local outliers. In SIGMOD, pages 93–104, 2000.
  8. Language models are few-shot learners. NIPS, 33:1877–1901, 2020.
  9. Efficientvit: Lightweight multi-scale attention for high-resolution dense prediction. In ICCV, pages 17302–17313, 2023.
  10. Learning graph structures with transformer for multivariate time-series anomaly detection in iot. IEEE Internet of Things Journal, 9(12):9179–9189, 2021.
  11. Tiankai Chen. Anomaly detection in semiconductor manufacturing through time series forecasting using neural networks. PhD thesis, Massachusetts Institute of Technology, 2018.
  12. A robust graph-based algorithm for detection and characterization of anomalies in noisy multivariate time series. In ICDM workshops, pages 349–358. IEEE, 2008.
  13. Graph neural network-based anomaly detection in multivariate time series. In AAAI, volume 35, pages 4027–4035, 2021.
  14. Real-time anomaly detection based on long short-term memory and gaussian mixture model. Computers & Electrical Engineering, 79:106458, 2019.
  15. Anthony J Fox. Outliers in time series. Journal of the Royal Statistical Society Series B: Statistical Methodology, 34(3):350–363, 1972.
  16. An evaluation of anomaly detection and diagnosis in multivariate time series. IEEE Transactions on Neural Networks and Learning Systems, 33(6):2508–2517, 2022.
  17. Outlier detection for temporal data: A survey. IEEE Transactions on Knowledge and data Engineering, 26(9):2250–2267, 2013.
  18. Flatten transformer: Vision transformer using focused linear attention. In ICCV, pages 5961–5971, 2023.
  19. Denoising diffusion probabilistic models. NIPS, 33:6840–6851, 2020.
  20. Detecting spacecraft anomalies using lstms and nonparametric dynamic thresholding. In SIGKDD, pages 387–395, 2018.
  21. Transformers are rnns: Fast autoregressive transformers with linear attention. In ICML, pages 5156–5165. PMLR, 2020.
  22. Outlier detection for time series with recurrent autoencoder ensembles. In IJCAI, pages 2725–2732, 2019.
  23. Adam: A method for stochastic optimization. In ICLR, 2015.
  24. Revisiting time series outlier detection: Definitions and benchmarks. In NeurIPS Dataset and Benchmark Track, 2021.
  25. Nominality score conditioned time series anomaly detection by point/sequential reconstruction. arXiv preprint arXiv:2310.15416, 2023.
  26. Mad-gan: Multivariate anomaly detection for time series data with generative adversarial networks. In International conference on artificial neural networks, pages 703–716. Springer, 2019.
  27. Multivariate time series anomaly detection and interpretation using hierarchical inter-metric and temporal embedding. In SIGKDD, pages 3220–3230, 2021.
  28. Svdd-based outlier detection on uncertain data. Knowledge and information systems, 34:597–618, 2013.
  29. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, pages 10012–10022, 2021.
  30. Swat: a water treatment testbed for research and training on ics security. In CySWATER, pages 31–36. IEEE, 2016.
  31. A multimodal anomaly detector for robot-assisted feeding using an lstm-based variational autoencoder. IEEE Robotics and Automation Letters, 3(3):1544–1551, 2018.
  32. Learning representations from healthcare time series data for unsupervised anomaly detection. In 2019 IEEE international conference on big data and smart computing (BigComp), pages 1–7. IEEE, 2019.
  33. Time series anomaly detection using diffusion-based models. arXiv preprint arXiv:2311.01452, 2023.
  34. Anomaly detection in time series: a comprehensive evaluation. Proceedings of the VLDB Endowment, 15(9):1779–1797, 2022.
  35. Estimating the support of a high-dimensional distribution. Neural computation, 13(7):1443–1471, 2001.
  36. Timeseries anomaly detection using temporal hierarchical one-class network. NIPS, 33:13016–13026, 2020.
  37. Efficient attention: Attention with linear complexities. In WACV, pages 3531–3539, 2021.
  38. Robust anomaly detection for multivariate time series through stochastic recurrent neural network. In SIGKDD, pages 2828–2837, 2019.
  39. Adjusting for autocorrelated errors in neural networks for time series. NIPS, 34:29806–29819, 2021.
  40. Support vector data description. Machine learning, 54:45–66, 2004.
  41. Tranad: Deep transformer networks for anomaly detection in multivariate time series data. arXiv preprint arXiv:2201.07284, 2022.
  42. Attention is all you need. NIPS, 30, 2017.
  43. Timesnet: Temporal 2d-variation modeling for general time series analysis. ICLR, 2023.
  44. Anomaly transformer: Time series anomaly detection with association discrepancy. In ICLR, 2022.
  45. A deep neural network for unsupervised anomaly detection and diagnosis in multivariate time series data. In AAAI, volume 33, pages 1409–1416, 2019.
  46. Multivariate time-series anomaly detection via graph attention network. In ICDM, pages 841–850. IEEE, 2020.
  47. A comparative study on unsupervised anomaly detection for time series: Experiments and analysis. arXiv preprint arXiv:2209.04635, 2022.
  48. Beatgan: Anomalous rhythm detection using adversarially generated time series. In IJCAI, volume 2019, pages 4433–4439, 2019.
  49. Deep autoencoding gaussian mixture model for unsupervised anomaly detection. In ICLR, 2018.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets