Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
169 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

LogFormer: A Pre-train and Tuning Pipeline for Log Anomaly Detection (2401.04749v1)

Published 9 Jan 2024 in cs.LG, cs.AI, and cs.SE

Abstract: Log anomaly detection is a key component in the field of artificial intelligence for IT operations (AIOps). Considering log data of variant domains, retraining the whole network for unknown domains is inefficient in real industrial scenarios. However, previous deep models merely focused on extracting the semantics of log sequences in the same domain, leading to poor generalization on multi-domain logs. To alleviate this issue, we propose a unified Transformer-based framework for Log anomaly detection (LogFormer) to improve the generalization ability across different domains, where we establish a two-stage process including the pre-training and adapter-based tuning stage. Specifically, our model is first pre-trained on the source domain to obtain shared semantic knowledge of log data. Then, we transfer such knowledge to the target domain via shared parameters. Besides, the Log-Attention module is proposed to supplement the information ignored by the log-paring. The proposed method is evaluated on three public and one real-world datasets. Experimental results on multiple benchmarks demonstrate the effectiveness of our LogFormer with fewer trainable parameters and lower training costs.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (31)
  1. GripRank: Bridging the Gap between Retrieval and Generation via the Generative Knowledge Improved Passage Ranking. In CIKM 2023, 36–46. ACM.
  2. Anomaly detection from log files using data mining techniques. In Information Science and Applications. Springer.
  3. Bert: Pre-training of deep bidirectional transformers for language understanding. NAACL 2019.
  4. Spell: Streaming parsing of system event logs. In ICDM 2016.
  5. Deeplog: Anomaly detection and diagnosis from system logs through deep learning. In CCS 2017.
  6. LogLG: Weakly Supervised Log Anomaly Detection via Log-Event Graph Construction. In DASFAA 2023, volume 13946 of Lecture Notes in Computer Science, 490–501. Springer.
  7. LVP-M3: Language-aware Visual Prompt for Multilingual Multimodal Machine Translation. In EMNLP 2022, 2862–2872. Association for Computational Linguistics.
  8. OWL: A Large Language Model for IT Operations. CoRR, abs/2309.09298.
  9. Drain: An online log parsing approach with fixed depth tree. In ICWS 2017, 33–40.
  10. Loghub: A Large Collection of System Log Datasets towards Automated Log Analytics. CoRR, abs/2008.06448.
  11. Long short-term memory. Neural computation, 1735–1780.
  12. Parameter-efficient transfer learning for NLP. In ICML 2019.
  13. LoRA: Low-Rank Adaptation of Large Language Models. In ICLR 2022.
  14. HitAnomaly: Hierarchical Transformers for Anomaly Detection in System Log. TNSM, 17(4): 2064–2076.
  15. Abstracting Execution Logs to Execution Events for Enterprise Applications (Short Paper). In QSIC 2008, 181–186.
  16. Detecting Anomaly in Big Data System Logs Using Convolutional Neural Network. In DASC 2018, 151–158.
  17. Clustering event logs using iterative partitioning. In KDD 2009, 1255–1264.
  18. LogAnomaly: Unsupervised Detection of Sequential and Quantitative Anomalies in Unstructured Logs. In IJCAI 2019.
  19. What Supercomputers Say: A Study of Five System Logs. In DSN 2007, 575–584.
  20. Training language models to follow instructions with human feedback. In NeurIPS.
  21. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. In EMNLP 2019, 3980–3990.
  22. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. In NeurIPS.
  23. Detecting large-scale system problems by mining console logs. In ICML 2010.
  24. UM4: Unified Multilingual Multiple Teacher-Student Model for Zero-Resource Neural Machine Translation. In IJCAI 2022, 4454–4460. ijcai.org.
  25. PLELog: Semi-Supervised Log-Based Anomaly Detection via Probabilistic Label Estimation. In ICSE 2021, 230–231.
  26. Semi-supervised Log-based Anomaly Detection via Probabilistic Label Estimation. In ICSE 2021, 1448–1460.
  27. A study of the performance of general compressors on log files. ESE, 25(5): 3043–3085.
  28. Automated IT system failure prediction: A deep learning approach. In BigData 2016.
  29. Rapidand robust impact assessment of software changes in large internet-based services. In ENET 2015.
  30. Robust log-based anomaly detection on unstable log data. In FSE 2019.
  31. Serial or Parallel? Plug-able Adapter for multilingual machine translation. CoRR.
Citations (10)
View on Semantic Scholar

Summary

  • The paper introduces LogFormer, a two-stage method that pretrains on source log data and adapts to diverse domains using an adapter module.
  • The experimental results demonstrate superior performance with high precision, recall, and F1 scores on benchmark and real-world datasets.
  • The paper's approach effectively handles partially structured logs and preserves semantic context, reducing retraining overhead for anomaly detection.

Overview

Anomaly detection in log data is critical for maintaining the health and security of IT operations. With the ever-increasing volume and complexity of log data in different domains, finding an efficient and generalizable solution for identifying anomalies across various sources becomes essential. The model introduced in the discussed paper, known as LogFormer, provides a substantial step forward in this area, utilizing a Transformer-based architecture to detect log anomalies.

The Challenge of Log Anomaly Detection

Traditional log anomaly detection methods face limitations, especially when encountering logs from new or multiple domains. The approaches that preprocess logs usually lose valuable semantic information, and retraining existing models to accommodate new log data can be resource-intensive. Identifying anomalies can further be complicated by logs that are only partially structured and contain elements similar to natural language.

LogFormer Architecture

The proposed structure of LogFormer tackles these challenges through a two-stage process: pre-training and adapter-based tuning. Initially, the model is pre-trained on logs from a source domain to capture the underlying semantic patterns common in log data. Post pre-training, an Adapter component maps this knowledge to target domains with varying log characteristics. This approach enables the LogFormer to generalize across different log sources effectively. An added component, the Log-Attention module, is specifically designed to address the loss of parameter information, which is typically a byproduct of log parsing processes.

Experimental Results

LogFormer was rigorously tested against several benchmark datasets and a real-world dataset from a cloud service company. The results are promising, showcasing the model's superior performance in terms of accuracy and efficiency, as evidenced by its high precision, recall, and F1 scores. LogFormer not only demonstrated improvements over existing state-of-the-art models but also achieved this with fewer trainable parameters and reduced training costs, underlining its practical viability for industrial applications.

Conclusion

In summary, LogFormer serves as an innovative and effective solution for log anomaly detection across different domains without the need to extensively retrain the network. Its two-stage process and unique Log-Attention mechanism equip it to handle the intricacies of log data semantics, making it a robust tool for AI operations in IT environments.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown