LogPilot: Intent-Aware Log Diagnosis

• LogPilot is an intent-aware, scalable LLM-powered framework for automated log-based alert diagnosis in large-scale online services.
• It reconstructs spatiotemporal log chains per request and uses hierarchical clustering to extract recurring execution patterns for precise root cause localization.
• Empirical evaluations on Volcano Engine Cloud alerts show significant improvements in diagnostic usefulness and localization accuracy with reduced latency and cost.

Searching arXiv for the LogPilot paper and closely related RCA/log-diagnosis work to ground citations. to=arxiv_search.search 天天中彩票一等奖? to=arxiv_search.search 玩大发快三 েমনjson {"query":"LogPilot intent-aware and scalable alert diagnosis for large-scale online service systems", "max_results": 5} to=arxiv_search.search 招商总代json {"query":"RCACopilot OpenRCA RCA Agent arXiv root cause analysis logs LLM", "max_results": 10} LogPilot is an intent-aware and scalable, LLM-powered framework for automated log-based alert diagnosis in large-scale online service systems. It addresses two persistent gaps observed in production: alert-agnostic log scoping and ineffective organization of massive, interleaved logs for reasoning. Its core design is to interpret the logic in alert definitions such as PromQL, precisely identify causally related logs and requests, reconstruct each request’s execution into a spatiotemporal log chain, cluster similar chains to identify recurring execution patterns, and provide representative samples to the LLMs for diagnosis. Evaluated on real-world alerts from Volcano Engine Cloud, it improves the usefulness of root cause summarization by 50.34% and exact localization accuracy by 54.79% over state-of-the-art methods, with end-to-end diagnosis in 58.6 seconds per alert on average and a cost of approximately $0.074 per alert (Jiang et al., 30 Sep 2025).

1. Problem setting and formalization

Large-scale online services instrument per-request metrics and aggregate them into SLIs. Alerts, for example via Prometheus evaluating PromQL rules, indicate SLO violations but rarely reveal root causes. The operational burden is therefore shifted to on-call engineers, who must inspect vast, distributed logs, correlate requests, reconstruct execution flows, and reason about causes. Manual inspection is slow and burdensome, and automated tools often scope logs without alert intent, producing irrelevant or incomplete evidence, or feed interleaved, oversized logs to LLMs, exceeding context windows and impairing reasoning (Jiang et al., 30 Sep 2025).

The framework formalizes an alert rule as

$a = (E, \Theta, \Delta),$</p> <p>where$E$$is a PromQL expression over label-filtered time series,$$\Theta$is a threshold condition, and$\Delta$is the required duration for firing. Let$L$$be the log corpus during the alert window, and let</p> <p>$$G = (V, E_G)$$</p> <p>denote the service topology or request <a href="https://www.emergentmind.com/topics/metric-graph-bundles" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">graph</a> if available, although LogPilot primarily uses component IDs from logs. Let</p> <p>$$C = \{C_i\}$$</p> <p>be reconstructed spatiotemporal chains for alert-related requests, and let</p> <p>$$D$$</p> <p>be the diagnosis consisting of a root cause summary and an exact localization, namely a faulty component set.</p> <p>Its objective is explicit: given$$a$,$L$, and component metadata, produce$E$0 with high usefulness and localization accuracy under constraints of LLM context window, low latency of less than one minute, and low cost of approximately $E$1 per alert. Inputs include the alert definition, sampled log examples for schema, DSL grammar for log query, full logs around alert time, component identifiers, request IDs, and an optional SOP knowledge base. Outputs include a diagnosis

$E$2

along with developer-facing logging quality feedback.

A common misconception is to treat the framework as a generic “LLM over logs.” The architecture instead constrains diagnosis by alert semantics, request-centric structure, and bounded representative sampling. This suggests that its central contribution is not merely language-model application, but a reorganization of observability data so that LLM reasoning remains causally aligned and context-window feasible.

2. Intent-aware alert interpretation

LogPilot introduces an alert-log correlation agent, abbreviated ALC, that reads the alert’s PromQL definition to infer diagnostic intent and scope, then generates a lightweight, executable filtering tool (Jiang et al., 30 Sep 2025). The agent parses $E$$3 to extract metric names, aggregation operators, label filters, and temporal windowing such as <code>rate()</code>, <code>increase()</code>, and <code>avg_over_time</code>. It then heuristically aligns label semantics to log fields using a small sample of logs and DSL grammar. The motivating example is that fields in <code>http_requests_total{gateway_id=…, account_id=…, [code](https://www.emergentmind.com/topics/karpathy-agent-code)=…}</code> can be mapped to semantically similar log fields even if names differ.</p> <p>The paper expresses this stage through intent derivation and scoped retrieval. Given$$E$4 and auxiliary knowledge $E$5 consisting of DSL grammar and sampled logs, the system derives intent $E$6 and a candidate component set $E$7:

$E$8

Here, $E$9 encodes label constraints, error conditions such as code =~ 5.., time windows, and aggregation scope. Based on this intent, the log scoping set is

$\Theta$0

where $\Theta$1 checks that a log entry semantically satisfies the label constraints in $\Theta$2 and falls within the alert’s firing window.

Tool generation is central to the ALC design. Given the alert definition, randomly sampled logs from relevant services, and DSL grammar for querying the log store, the agent generates a Python script embedding DSL queries that retrieve $\Theta$3 and extract request IDs linked to the failure condition. The script is executed, and both query results and runtime errors are used as feedback to iteratively refine logic for up to three iterations. Validated tools are cached keyed by PromQL definitions to avoid regeneration. Runtime cost is dominated by the log store query over the alert window, while the refinement bound of at most three iterations constrains tool-generation overhead.

This design is technically important because it converts an alert definition into executable log-scope semantics rather than relying on keyword search or alert-agnostic anomaly filtering. A plausible implication is that the ALC stage serves as the principal interface between metrics-space symptom detection and logs-space causal evidence.

3. Request-centric spatiotemporal log chains

After scoping, LogPilot organizes logs per request into coherent chains that preserve temporal ordering and cross-component invocation structure. A spatiotemporal log chain for a request is defined as

$\Theta$4

where $\Theta$5 is the component or service node, $\Theta$6 an operation or event represented as a log template, $\Theta$7 the timestamp, $\Theta$8 the message text, and $\Theta$9 attributes such as level, code, and path. The paper also notes an alternative representation as a temporal multigraph $\Delta$0, but LogPilot uses the sequence representation for LLM input (Jiang et al., 30 Sep 2025).

Chain construction relies on a two-tier parsing strategy. The first tier performs coarse clustering by logging path, understood as source code location found in logs; entries without paths are grouped separately. The second tier parses messages within each group using Drain to extract structured templates and variables, yielding log events with a template ID. This is intended to handle dynamic variable content while exploiting path-level regularity in production logs.

For each request ID, the framework then performs a fixed sequence of operations. It collects all raw logs across components, deduplicates by retaining only the first occurrence of each unique log event template, sorts the resulting set by timestamp, partitions sorted events by component to form segments, orders segments by the earliest timestamp in each segment, and assembles the chain as an ordered sequence of per-component segments. If trace or span IDs exist, they are used first; otherwise, request IDs together with component IDs and timestamps correlate events. Missing logs or inconsistent levels are not ignored: they are flagged by the RCA agent as logging-quality issues.

The complexity characterization is explicit. Drain is near-linear in logs per request group, two-tier parsing adds a coarse clustering pass, deduplication and sorting per request are $\Delta$1, and partitioning by component is $\Delta$2. Space is proportional to the number of unique templates per request after deduplication.

The significance of the chain representation is methodological as much as operational. Rather than present the LLM with raw, interleaved logs, LogPilot reconstructs end-to-end execution semantics at request granularity. This suggests that the framework treats observability not as a flat text corpus but as a structured causal sequence.

4. Clustering, representative selection, and scalable compression

To ensure that the LLM input remains “rich yet compact,” LogPilot clusters requests by their log pattern similarity and selects one representative per cluster (Jiang et al., 30 Sep 2025). Each request chain is embedded as a bag-of-events vector

$\Delta$3

where $\Delta$4 is the count of the $\Delta$5-th distinct log event template across all requests. It then applies log-scaling,

$\Delta$6

and computes cosine similarity:

$\Delta$7

Clustering uses Hierarchical Agglomerative Clustering. Each request begins as its own cluster, and the algorithm iteratively merges the pair with highest similarity until the best pair’s similarity falls below $\Delta$8, with default $\Delta$9. The paper characterizes this as minimizing intra-cluster variance or maximizing intra-cluster similarity without predefining the number of clusters. For a cluster with $L$0 requests, the centroid is

$L$1

and the representative is the request whose embedding is closest to $L$2, for example by highest cosine similarity.

This stage is the main scalability mechanism. Prefiltering by the ALC agent limits requests to those causally linked to the alert. Deduplication reduces per-request token load. Clustering reduces LLM invocations from $L$3 requests to $L$4 clusters, where $L$5 and is typically less than $L$6. The complexity is stated as $L$7 for embedding construction, $L$8 for HAC pairwise similarities and merges, and $L$9 for representative selection within a cluster. With $G = (V, E_G)$0 in hundreds and $G = (V, E_G)$1 bounded by unique templates in the alert window, the method is described as tractable.

A likely misconception is that clustering is used merely for efficiency. In the LogPilot design, clustering also functions as a denoising and pattern-extraction mechanism: representative requests stand in for recurring execution patterns, so compression is intended to preserve diagnostically salient structure rather than discard it.

5. Multi-agent diagnosis workflow and output structure

The diagnosis pipeline orchestrates three agents: the ALC agent for intent-aware scoping, an RCA agent for cluster-wise request diagnosis, and a summary agent for aggregation and retrieval-augmented generation over SOP documents (Jiang et al., 30 Sep 2025). The RCA agent receives alert context, including the PromQL rule, labels, firing interval, and SLI/SLO semantics; a representative request’s spatiotemporal chain, including ordered segments with component IDs, timestamps, event templates, and key attributes; and service component metadata such as role names.

Its output is a structured analysis comprising root cause hypotheses, faulty component or components, key log evidence, and step-by-step inference. When logs are insufficient or inconsistent, the RCA agent explicitly reports logging-quality issues, including silent failure and unpropagated errors. This behavior is notable because logging defects are treated as diagnostic findings rather than merely missing data.

The summary agent aggregates per-cluster findings, reconciles overlaps, and uses RAG over SOP documents to provide actionable remediation steps. Parallel cluster analyses reduce latency, while caching of ALC tools further reduces turnaround time. The paper does not formalize confidence scoring; instead, aggregation and explicit logging-quality reporting are presented as mechanisms that provide transparency into evidence strength.

The paper also provides a concrete alert example. For

$G = (V, E_G)$2

the intent is a surge in 5xx responses for gateway gw-123 and account acct-42 in the last minute. The ALC agent maps labels to log fields through sampled logs, retrieves logs satisfying the error condition within the firing window, and extracts request IDs. For each request, the system reconstructs a chain, deduplicates by template, orders by timestamp, groups by component, embeds the request, performs HAC clustering, selects one representative per cluster, and runs RCA analysis before summary aggregation.

This example clarifies an important architectural point: LogPilot does not ask an LLM to infer everything from raw observability exhaust. It first externalizes scope, structure, and compression into executable stages, then reserves LLM reasoning for representative, causally organized evidence.

6. Empirical results, deployment, limitations, and place in the RCA landscape

The empirical study uses alerts from four large-scale Volcano Engine Cloud services over the period from June 15 to July 15, 2025. The evaluation set contains 202 alerts, distributed as 35, 40, 52, and 75 across services A, B, C, and D, with 161 unique root causes. Complete logs for alert windows were available, and many alerts had human-authored failure review reports (Jiang et al., 30 Sep 2025).

Root cause summarization is evaluated with ROUGE-1, METEOR, and embedding-based Semantic Similarity using Doubao-1.5-Embedding, along with human usefulness scores from 0 to 1. Root cause localization is evaluated with Exact Match,

$G = (V, E_G)$3

and Top-3 Accuracy, defined as whether all ground-truth components are contained within the top-3 proposed components. Against state-of-the-art baselines, including LLM with sampling, RCA Agent, and RCACopilot, LogPilot improves usefulness by 50.34% on average and Exact Match localization by 54.79% on average. It also reports consistent gains in ROUGE-1, METEOR, and Semantic Similarity; for example, relative to RCA Agent, the reported gains are +20.75% ROUGE-1, +27.37% METEOR, and +8.66% semantic. Efficiency figures are equally specific: average end-to-end diagnosis time is 58.6 seconds per alert, with stage-wise averages of 17.2 seconds for ALC tool generation, 21.3 seconds per representative for RCA, and 4.9 seconds for the summary agent. The mean token usage is 69.73K prompt tokens and 8.08K response tokens per alert, corresponding to approximately $0.074 per alert. Clustering reduces LLM invocations by about 98.71%, with mean clusters 2.56 versus up to thousands of failed requests.

Robustness analyses further isolate important design choices. With log examples and refinement, ALC tool quality reaches an average score of 0.984; without examples the tools are largely unusable, and without refinement the average is 0.727 due to semantic mismatches. For the HAC threshold, Top-3 accuracy stays high, approximately 94%, for $G = (V, E_G)$4 in the range $G = (V, E_G)$5, while lowering $G = (V, E_G)$6 reduces clusters and can lower accuracy, for example to approximately 89.1% at $G = (V, E_G)$7. The default $G = (V, E_G)$8 is therefore presented as a balance of cost and performance.

In production, the system is integrated with Volcano Engine monitoring. Upon alert firing, it runs autonomously and posts reports to on-call channels such as Lark. It has been deployed across 12 services and had analyzed more than 3,500 alerts by the end of July 2025. Reported acceptance by engineers is 84.21%, with 60.53% exact matches and 23.68% partial matches; the remaining 15.79% still provided useful clues for manual follow-up.

The limitations are operationally concrete. Incomplete instrumentation or missing or low-quality logs, including silent failures and info-level logs for errors, can hinder RCA. Semantic alignment between PromQL labels and log fields relies on LLM heuristics, and rare schema idiosyncrasies may require human intervention in less than or equal to 3% of cases. Complex cross-service causality beyond request-centric logs, such as batch jobs and asynchronous pipelines, may not be fully captured. Failure modes include over-aggregation or under-aggregation in HAC if $G = (V, E_G)$9 is mis-tuned, and log parsing ambiguity for highly dynamic templates even after two-tier parsing.

Within the RCA landscape, LogPilot is distinguished by intent-aware scoping from PromQL, request-centric spatiotemporal chains, clustering-driven representative selection that keeps LLM inputs both rich and compact, production-grade tool generation, refinement, and caching, and explicit logging-quality feedback. Prior log-based scoping often uses keyword search or anomaly detection and ignores the alert’s intent, while LLM-based RCA systems can struggle with massive, interleaved logs exceeding context windows or with fragmented inputs. Pattern-learning RCA systems, as described in the paper, require labeled failures and scale poorly. LogPilot instead avoids task-specific training and emphasizes in-context LLM reasoning over structured, representative evidence (Jiang et al., 30 Sep 2025).