Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
156 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Exploring LLM-based Agents for Root Cause Analysis (2403.04123v1)

Published 7 Mar 2024 in cs.SE, cs.CL, and cs.LG

Abstract: The growing complexity of cloud based software systems has resulted in incident management becoming an integral part of the software development lifecycle. Root cause analysis (RCA), a critical part of the incident management process, is a demanding task for on-call engineers, requiring deep domain knowledge and extensive experience with a team's specific services. Automation of RCA can result in significant savings of time, and ease the burden of incident management on on-call engineers. Recently, researchers have utilized LLMs to perform RCA, and have demonstrated promising results. However, these approaches are not able to dynamically collect additional diagnostic information such as incident related logs, metrics or databases, severely restricting their ability to diagnose root causes. In this work, we explore the use of LLM based agents for RCA to address this limitation. We present a thorough empirical evaluation of a ReAct agent equipped with retrieval tools, on an out-of-distribution dataset of production incidents collected at Microsoft. Results show that ReAct performs competitively with strong retrieval and reasoning baselines, but with highly increased factual accuracy. We then extend this evaluation by incorporating discussions associated with incident reports as additional inputs for the models, which surprisingly does not yield significant performance improvements. Lastly, we conduct a case study with a team at Microsoft to equip the ReAct agent with tools that give it access to external diagnostic services that are used by the team for manual RCA. Our results show how agents can overcome the limitations of prior work, and practical considerations for implementing such a system in practice.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (43)
  1. Recommending Root-Cause and mitigation steps for cloud incidents using large language models.
  2. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization (Ann Arbor, Michigan, June 2005), Association for Computational Linguistics, pp. 65–72.
  3. Decaf: Diagnosing and triaging performance issues in large-scale cloud services. CoRR abs/1910.05339 (2019).
  4. The use of MMR, diversity-based reranking for reordering documents and producing summaries. In Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval (New York, NY, USA, Aug. 1998), SIGIR ’98, Association for Computing Machinery, pp. 335–336.
  5. Chase, H. LangChain. Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, HP d. O (2022).
  6. Teaching large language models to self-debug. arXiv preprint arXiv:2304. 05128 (2023).
  7. {{\{{Push-Button}}\}} reliability testing for {{\{{Cloud-Backed}}\}} applications with rainmaker. In 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23) (2023), pp. 1701–1716.
  8. Empowering practical root cause analysis by large language models for cloud incidents.
  9. PAL: Program-aided language models. 10764–10799.
  10. A systematic review on anomaly detection for cloud computing environments. In Proceedings of the 2020 3rd Artificial Intelligence and Cloud Computing Conference (New York, NY, USA, 2021), AICCC ’20, Association for Computing Machinery, p. 83–96.
  11. How to mitigate the incident? an effective troubleshooting guide recommendation technique for online service systems. In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (New York, NY, USA, 2020), ESEC/FSE 2020, Association for Computing Machinery, p. 1410–1420.
  12. BART: Denoising Sequence-to-Sequence pre-training for natural language generation, translation, and comprehension.
  13. Retrieval-augmented generation for knowledge-intensive nlp tasks. Adv. Neural Inf. Process. Syst. 33 (2020), 9459–9474.
  14. Lin, C.-Y. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out (Barcelona, Spain, July 2004), Association for Computational Linguistics, pp. 74–81.
  15. {RESIN}: A holistic service for dealing with memory leaks in production cloud infrastructure. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22) (2022), pp. 109–125.
  16. Correlating events with time series for incident diagnosis. In Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (New York, NY, USA, 2014), KDD ’14, Association for Computing Machinery, p. 1583–1592.
  17. Diagnosing root causes of intermittent slow queries in cloud databases. Proceedings VLDB Endowment 13, 8 (Apr. 2020), 1176–1189.
  18. Self-Refine: Iterative refinement with Self-Feedback.
  19. Tangled up in BLEU: Reevaluating the evaluation of automatic machine translation evaluation metrics.
  20. Augmented language models: A survey.
  21. Learning a hierarchical monitoring system for detecting and diagnosing service issues. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (New York, NY, USA, 2015), KDD ’15, Association for Computing Machinery, p. 2029–2038.
  22. OpenAI. GPT-4 technical report.
  23. BLEU: A method for automatic evaluation of machine translation. https://aclanthology.org/P02-1040.pdf, 2002. Accessed: 2023-9-27.
  24. Sentence-BERT: Sentence embeddings using siamese BERT-Networks.
  25. The probabilistic relevance framework: BM25 and beyond. Foundations and Trends® in Information Retrieval 3, 4 (2009), 333–389.
  26. Reassessing automatic evaluation metrics for code summarization tasks. In Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (New York, NY, USA, Aug. 2021), ESEC/FSE 2021, Association for Computing Machinery, pp. 1105–1116.
  27. Toolformer: Language models can teach themselves to use tools.
  28. Autotsg: Learning and synthesis for incident troubleshooting. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (New York, NY, USA, 2022), ESEC/FSE 2022, Association for Computing Machinery, p. 1477–1488.
  29. Shinn, N. reflexion: Reflexion: Language agents with verbal reinforcement learning.
  30. Reflexion: an autonomous agent with dynamic memory and self-reflection.
  31. ALFWorld: Aligning text and embodied environments for interactive learning.
  32. Anomaly detection and failure root cause analysis in (micro) service-based cloud applications: A survey. ACM Comput. Surv. 55, 3 (feb 2022).
  33. LLM-Planner: Few-Shot grounded planning for embodied agents with large language models.
  34. Automated traces-based anomaly detection and root cause analysis in cloud platforms. In 2022 IEEE International Conference on Cloud Engineering (IC2E) (2022), pp. 253–260.
  35. Interleaving retrieval with Chain-of-Thought reasoning for Knowledge-Intensive Multi-Step questions.
  36. Chain of thought prompting elicits reasoning in large language models.
  37. Chain-of-thought prompting elicits reasoning in large language models. 24824–24837.
  38. WebShop: Towards scalable real-world web interaction with grounded language agents. 20744–20757.
  39. ReAct: Synergizing reasoning and acting in language models.
  40. TraceArk: Towards actionable performance anomaly alerting for online service systems. In 2023 IEEE/ACM 45th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) (May 2023), pp. 258–269.
  41. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv (2019).
  42. ExpeL: LLM agents are experiential learners.
  43. WebArena: A realistic web environment for building autonomous agents.
Citations (8)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now