OpenRCA 2.0: From Outcome Labels to Causal Process Supervision

Abstract: Root cause analysis (RCA) poses a holistic test of LLM agentic capabilities, such as long-context understanding, multi-step reasoning, and tool use. However, existing datasets suffer from a fundamental gap: they label only the root cause, not the propagation path connecting it to the observed symptom, which largely simplifies the task to naive pattern matching. To support rigorous evaluation, we introduce PAVE, a step-wise labeling protocol that leverages known interventions from fault injection to reconstruct causal propagation paths. The mechanism is forward verification: reasoning from cause to effect rather than inferring backward from symptoms. Applying PAVE yields OpenRCA 2.0 (500 instances), the first cross-system RCA benchmark with step-wise causal annotations for LLM agents. Across 11 frontier LLMs, recovering the exact root-cause set succeeds in only 20.7% of cases on average. To locate where this difficulty lies, we relax the criterion and find what we call the ungrounded diagnosis: agents identify at least one correct root-cause service in 76.0% of cases, but ground that service in a verified causal propagation path to the observed symptom in only 61.5%. Outcome-only evaluation hides this failure mode; step-wise causal ground truth is the missing piece for trustworthy LLM-based RCA agents.

• The paper introduces PAVE, a protocol that shifts RCA evaluation from outcome-only to step-by-step causal process supervision.
• It employs structural, statistical, and temporal verification gates to extract and validate causal chains in microservice systems.
• The benchmark highlights common RCA reasoning failures, paving the way for improved LLM-based diagnostic methods in distributed systems.

OpenRCA 2.0: Advancing Root Cause Analysis through Step-Wise Causal Process Supervision

Motivation: Beyond Outcome-Only RCA Evaluation

Root Cause Analysis (RCA) in microservice-based distributed systems presents a formidable testbed for agentic LLMs, demanding integration of long-context reasoning, multi-step causal inference, and tool-augmented investigation. Prevailing RCA benchmarks rely on outcome-only supervision, labeling merely the root-cause component affected by injected faults, and therefore reduce RCA challenge to pattern matching rather than robust causal reasoning. This abstraction readily admits the “ungrounded diagnosis”: agents label the right component but fail to substantiate causal chains from root cause to downstream symptom. The paper "OpenRCA 2.0: From Outcome Labels to Causal Process Supervision" (2606.27154) introduces a new paradigm—PAVE, an annotation protocol for step-wise causal ground truth—enabling process-level evaluation and illuminating reasoning failures invisible under outcome-centric scoring.

PAVE Protocol: Verified Causal Path Extraction

The core contribution is PAVE (Path Annotation via Verified Effects), an annotation pipeline that leverages information asymmetry between forward verification (given the intervention) and backward RCA (from telemetry). PAVE exploits knowledge of the intervention $do(v_{root})$—precisely the injected fault, its window and parameters—shifting the annotation task from ill-posed inverse reasoning to a tractable verification problem: confirming which downstream effects actually propagated from the root.

PAVE enforces three conjunctive verification gates:

• Mechanism Conformance (Structural Pruning): Candidate paths must be topologically consistent with the microservice dependency graph and permitted propagation rule set, eliminating infeasible edges.
• Statistical Evidence: Each node along the candidate path must exhibit statistically significant deviation from a pre-injection baseline, using tailored Z-score or percentile detectors.
• Temporal Alignment: Propagation must have downstream anomalies onset after upstream causes, respecting causality.

This coarse-to-fine approach prunes cluttered observation graphs into verified causal chains, discarding coincidental anomalies (Figure 1).

Figure 1: Step-wise verification pipeline refines observed telemetry into a unique causal chain, filtering non-causal paths and substantiating each link through structural, statistical, and temporal gates.

OpenRCA 2.0 Benchmark: Step-Wise Supervision at Scale

Applying PAVE to three architecturally distinct microservice systems (TrainTicket [44 services, deep REST chains], OpenTelemetry Demo [15 polyglot services, gRPC/Kafka], and Hotel Reservation [9 Go/gRPC services]) and 27 primary fault types yields OpenRCA 2.0—500 instances with process-level annotations averaging 7.5 verified causal edges per cascade. The benchmark uniquely exposes not just which service is at fault, but the propagation path connecting cause to observed symptoms.

Figure 2: RCA setting with injective faults cascading horizontally along call chains or vertically across shared infrastructure, observed via traces, metrics, and logs.

Process-Level Evaluation: Quantifying RCA Reasoning Failures

Standard outcome metrics (e.g., Exact Match, F1 on root-cause set) camouflage critical failure modes. On OpenRCA 2.0, across 11 frontier LLMs, outcome-level F1 averages only 34.1%, Exact Match 20.7%. However, when relaxing to identifying at least one correct service (AnySvc), the rate goes up to 76.0%. Critically, process-level Path Reachability (PR)—requiring an agent to substantiate a causal chain from root cause to observed symptom—drops to 61.5%. Thus, nearly 20% of cases classified as outcome-success are ungrounded diagnoses, missing verifiable causal links (Figure 3).

Figure 3: Ungrounded diagnosis—agent names the correct root cause but produces a propagation graph missing actual causal intermediates, masked by outcome-only evaluation.

Edge F1 (43.4%) consistently trails Node F1 (62.2%), confirming agents are better at enumerating relevant services than ordering them into causal propagation structures. No model unambiguously dominates; optimal selection depends on recall/precision trade-off and downstream requirements.

Failure Mode Analysis: Characterizing Ungrounded Diagnoses

The benchmark enables categorization of dominant reasoning failures:

• Premature Commitment: Agents halt investigation after finding one plausible root cause, skipping verification of propagation chains or alternate attributions.
• Presence Bias: Silence (e.g., killed pods) is misinterpreted as absence of error, overlooking that the absence itself may be the key fault signature.
• Salience Capture: Agents focus on the loudest anomalous signal (e.g., high-traffic hubs, prominent log lines), misattributing faults to downstream amplifiers rather than actual origin.

This is exemplified in case studies where LLMs hallucinate direct edges, omitting intermediate nodes traversed by the fault cascade (Edge F1 = 44.4%).

Annotation Reliability and Assumptions

A rigorous two-annotator audit on 100 instances shows 94% agreement with disagreements solely due to omitted weakly supported edges (precision-favoring). PAVE’s reliability is underwritten by the strict conjunction of structural, statistical, and temporal verification, with exponentially decaying false-positive probability for longer propagation paths.

Key assumptions include availability of controlled interventions, comprehensive dependency graphs, clean baselines, and telemetry covering fault signatures. Numbers are therefore best interpreted within the controlled testbed envelope; transfer to production deployments must be explicitly calibrated.

Comparison to Prior Work and Implications

OpenRCA 2.0 introduces step-wise process supervision missing from previous RCA benchmarks, which either limit to outcome labels or have only partial propagation tags. This enables process reward models (PRMs) for RCA, paralleling recent advances in mathematical reasoning and code generation, and opening avenues for training LLMs with step-by-step verification signals in reliability domains.

Theoretical implications: raising the evaluation standard from fault localization to causal reasoning accentuates the gap between memorization and robust generalization, calling for new architectures and training regimes that reward substantiated causal chains.

Practical ramifications: RCA agents evaluated with OpenRCA 2.0 must not just point at the culprit but justify the diagnosis with explicit, verifiable propagation reasoning, suitable for integration into high-stakes reliability engineering workflows.

Conclusion

OpenRCA 2.0, built upon PAVE’s step-wise causal process supervision, defines the state-of-the-art in RCA benchmark design by exposing reasoning failures concealed under outcome-only evaluation, and sets a new bar for trustworthy LLM-based RCA agents. It establishes the necessary ground truth for process reward modeling and motivates development of diagnostic agents capable not just of localization, but rigorous causal explanation—essential for deployment in critical distributed systems.