TranAD: Deep Transformer Networks for Anomaly Detection in Multivariate Time Series Data

Abstract: Efficient anomaly detection and diagnosis in multivariate time-series data is of great importance for modern industrial applications. However, building a system that is able to quickly and accurately pinpoint anomalous observations is a challenging problem. This is due to the lack of anomaly labels, high data volatility and the demands of ultra-low inference times in modern applications. Despite the recent developments of deep learning approaches for anomaly detection, only a few of them can address all of these challenges. In this paper, we propose TranAD, a deep transformer network based anomaly detection and diagnosis model which uses attention-based sequence encoders to swiftly perform inference with the knowledge of the broader temporal trends in the data. TranAD uses focus score-based self-conditioning to enable robust multi-modal feature extraction and adversarial training to gain stability. Additionally, model-agnostic meta learning (MAML) allows us to train the model using limited data. Extensive empirical studies on six publicly available datasets demonstrate that TranAD can outperform state-of-the-art baseline methods in detection and diagnosis performance with data and time-efficient training. Specifically, TranAD increases F1 scores by up to 17%, reducing training times by up to 99% compared to the baselines.

• The paper introduces TranAD, a deep transformer model with a focus score self-conditioning mechanism that enhances anomaly detection in multivariate time series.
• TranAD uses a two-phase reconstruction with adversarial training to achieve up to a 17% improvement in F1 scores while reducing training times by up to 99%.
• The approach leverages attention mechanisms to effectively identify and diagnose anomalies, offering robust performance on complex datasets.

TranAD: Deep Transformer Networks for Anomaly Detection in Multivariate Time Series Data

The paper "TranAD: Deep Transformer Networks for Anomaly Detection in Multivariate Time Series Data" proposes a novel approach for anomaly detection using a transformer-based architecture. TranAD is designed to address the challenges of anomaly detection in multivariate time series, characterized by high data volatility and the lack of anomaly labels, while requiring ultra-low inference times for real-world applications.

TranAD Architecture

TranAD leverages transformer networks, renowned for their capacity to model long-range dependencies using self-attention mechanisms. The TranAD model consists of encoder-decoder structures that utilize attention-based transformations to encode temporal trends effectively.

Figure 1: The TranAD Model.

The model is further enhanced with a focus score self-conditioning mechanism and adversarial training. This allows robust feature extraction and enhances model stability. The focus score, generated from initial reconstruction errors, acts as a prior to modify attention weights for subsequent inferences, effectively highlighting regions of the input that are likely to cause anomalies.

Anomaly Detection and Diagnosis

TranAD is implemented to handle two key tasks efficiently: anomaly detection and diagnosis. The model predicts anomaly scores for test sequences, identifying whether entire timestamps or specific data modes within a timestamp are anomalous. The prediction process involves a two-phase reconstruction strategy that not only identifies anomalies but also pinpoints their causes with high precision.

Figure 2: Visualization of anomaly prediction.

Figure 3: Visualization of focus and attention scores.

The use of self-conditioning in TranAD aids in amplifying deviations, minimizing false positives, and enabling the detection of subtle anomalies that might be overlooked by traditional methods. Furthermore, model-agnostic meta-learning (MAML) allows TranAD to function effectively even with limited training data.

Performance Evaluation

Extensive empirical studies conducted using several multivariate time-series datasets showcase TranAD's superiority over existing methods. It achieves up to a 17% improvement in F1 scores and reduces training times by up to 99% compared to baseline methods. A critical difference analysis shows TranAD's significant statistical improvement over prior models across these datasets.

Figure 4: Critical difference diagrams for F1 and AUC scores using the Wilkoxon pairwise signed rank test (with alpha=0.05) on all datasets. Rightmost methods are ranked higher.

TranAD demonstrates enhanced capability in both detection and diagnosis performance, evidenced by its higher HitRate and NDCG scores in pinpointing root causes of anomalies in datasets like the Multi-Source Distributed System (MSDS) dataset.

Figure 5: Predicted and Ground Truth labels for the MSDS test set using the TranAD model.

Implementation Considerations and Trade-offs

The transformer architecture and attention mechanisms enable TranAD to process large datasets with complex temporal patterns efficiently, maintaining low computational overheads. The choice of a two-phase adversarial training regime is designed to maintain training stability and improve the model's generalizability across diverse datasets.

However, the model performance may vary with dataset-specific parameters like window size, where smaller windows favor faster anomaly detection but might miss longer temporal dependencies. Hyperparameter optimization remains critical for achieving optimal performance across varied applications.

Figure 6: F1 score, ROC/AUC score and training times with dataset size.

Figure 7: F1 score, ROC/AUC score and training times with window size.

Conclusion

TranAD offers a compelling approach for detecting and diagnosing anomalies in multivariate time-series data through its innovative use of transformer networks. It effectively balances between quick, accurate anomaly detection and low computational overheads, making it suitable for modern industrial applications. Future work could explore enhancing model scalability and integrating diverse attention-based architectures for even broader applicability in diverse temporal settings.