Tools and Benchmarks for Automated Log Parsing (1811.03509v2)

Abstract: Logs are imperative in the development and maintenance process of many software systems. They record detailed runtime information that allows developers and support engineers to monitor their systems and dissect anomalous behaviors and errors. The increasing scale and complexity of modern software systems, however, make the volume of logs explodes. In many cases, the traditional way of manual log inspection becomes impractical. Many recent studies, as well as industrial tools, resort to powerful text search and machine learning-based analytics solutions. Due to the unstructured nature of logs, a first crucial step is to parse log messages into structured data for subsequent analysis. In recent years, automated log parsing has been widely studied in both academia and industry, producing a series of log parsers by different techniques. To better understand the characteristics of these log parsers, in this paper, we present a comprehensive evaluation study on automated log parsing and further release the tools and benchmarks for easy reuse. More specifically, we evaluate 13 log parsers on a total of 16 log datasets spanning distributed systems, supercomputers, operating systems, mobile systems, server applications, and standalone software. We report the benchmarking results in terms of accuracy, robustness, and efficiency, which are of practical importance when deploying automated log parsing in production. We also share the success stories and lessons learned in an industrial application at Huawei. We believe that our work could serve as the basis and provide valuable guidance to future research and deployment of automated log parsing.

Tools and Benchmarks for Automated Log Parsing

The paper "Tools and Benchmarks for Automated Log Parsing," presents a comprehensive evaluation of various methods for automated log parsing, analyzing their effectiveness across multiple datasets. This paper serves to provide foundational tools and data benchmarks that can ease future research and practical deployment of log parsing methods in industry. It encompasses a diverse set of log parsers, tools, and datasets that aim to enhance log parsing research by implementing a clear framework for evaluation.

Overview of Automated Log Parsing

The paper highlights the critical role of logs in monitoring software systems, where logs record runtime system information invaluable for diagnostics. With the exponential growth of log volumes in modern distributed, supercomputer, and mobile systems, manual log inspection is non-feasible. Thus, the paper discusses automated log parsing as the key step to convert unstructured log text into structured data, facilitating subsequent analysis activities. Various automated log parsers have been developed utilizing techniques such as frequent pattern mining, clustering, and iterative partitioning.

Methodology

Thirteen log parsers were rigorously evaluated on sixteen log datasets, encompassing multiple domains: distributed systems, supercomputers, operating systems, mobile systems, server applications, and standalone software. These datasets totaled over 440 million log messages, offering a substantial evaluation scale. The evaluation metrics focused on three primary qualities of log parsers:

1. Accuracy: How well a parser distinguishes event templates and parameters in log messages.
2. Robustness: Consistency in performance across various log sizes and types.
3. Efficiency: Parsing speed and resource utilization in processing differing log volumes.

Findings

The results indicated that no single parser overwhelmingly excels across all datasets in all metrics. However, certain parsers like Drain demonstrated significantly higher average accuracy and robustness across diverse datasets. Specific systems like Hadoop and Apache, with simpler log structure, were parsed with near-perfect accuracy by multiple parsers. Despite this, complex systems such as Android and Mac logs posed challenges due to varied and frequent event template changes.

Industrial Application and Implications

The research found significant industrial relevance, as demonstrated by the deployment in Huawei's System X product line. Automated log parsing drastically reduced laborious manual log analysis efforts needed for dynamic systems with rapidly evolving logging statements. The industry's need for refined log parsing techniques was evident, particularly in handling log messages with variable lengths and automating parameter tuning processes.

Future Directions

The paper suggests paths for improvement in state recognition of log messages, handling variability in message lengths, and automating parameter adjustments for different environments. The provision of an open-source toolkit and high-quality benchmark datasets aims to bridge the gap between research innovations and industry applications in log parsing, likely fostering further technological and methodological advancements. The potential for these tools to be integrated within broader AIOps (Artificial Intelligence for IT Operations) strategies remains a compelling trajectory for future exploration.

This paper contributes substantial insights into automated log parsing capabilities and challenges, setting a foundation for forward-looking innovations in automated system monitoring and management.