LEMMA-RCA: A Multi-Modal Root Cause Analysis Benchmark

• The paper introduces LEMMA-RCA as a pioneering benchmark that unifies RCA evaluation by combining metrics, textual logs, and KPI data across IT and OT systems.
• It provides a detailed data collection and preprocessing pipeline, including stationarity checks, log parsing, and KPI construction for consistent multi-modal input.
• Baseline methods like PC, DYNOTEARS, and CORAL are evaluated, highlighting the benefits of multi-modal fusion and online learning for dynamic system environments.

LEMMA-RCA is a large-scale, multi-modal, multi-domain dataset and benchmark specifically designed for root cause analysis (RCA) in complex IT and OT systems. It addresses the shortage of extensive, open-source RCA benchmarks by providing diverse data across IT microservices and operational technology testbeds, combining real-world and systematically injected fault scenarios with detailed system observability data streams. LEMMA-RCA supports standardized comparative evaluation of RCA algorithms across multiple settings, modalities, and domains, and is publicly available for download and benchmarking (Zheng et al., 2024).

1. Dataset Structure and Multi-Domain Scope

LEMMA-RCA comprises four sub-datasets, capturing fault-rich scenarios in both information technology (IT) and operational technology (OT) environments:

• IT-AIOps (microservices):
  • ProductReview: 4 fault scenarios, 216 pods/6 nodes
  • CloudComputing: 6 fault scenarios, ~168 pods/11 nodes
• OT-ICS (industrial control):
  • SWaT (Secure Water Treatment): 16 fault events, 51 sensors/actuators
  • WADI (Water Distribution): 9 fault events, 123 sensors/actuators

Across all sub-datasets, 35 real-fault instances are included, each involving hundreds of system entities per scenario.

Modalities and Data Types:

• Time-series metrics (node- and pod-level, 1 Hz, CSV/JSON via Prometheus/CloudWatch)
• Textual logs (unstructured, JSON, via ElasticSearch/CloudWatch Logs)
• Key Performance Indicators (KPIs): Derived series such as latency, throughput, error-rate, or anomaly scores

This composition supports evaluation of RCA methods in both IT (microservices, cloud) and OT (critical infrastructure) contexts, ensuring broad coverage and heterogeneity (Zheng et al., 2024).

2. Data Collection, Preprocessing, and Representation

Data Collection

IT Domain: Simulated real-fault injections (e.g., out-of-memory, DDoS, cryptojacking, storage-full) on running microservice clusters. Metrics and logs are collected at 1 Hz, with KPIs tracked via synthetic workload tools (e.g., JMeter).

OT Domain: Data is generated from the iTrust SWaT and WADI testbeds. Each includes long periods of normal operation followed by sequences under attack, with finely timestamped control/sensor data.

Preprocessing Pipeline

1. Stationarity Check: Only stationary entity metric streams are retained.
2. Log Feature Extraction:
   • Parse raw logs to templates (Drain parser).
   • Slide a 10-minute window (30s stride); compute:
     • $X^L_1(t)$: per-template frequencies
     • $X^L_2(t)$: “golden-signal” error/exception template counts
     • $X^L_3(t)$: PCA top TF–IDF log token features
   • Log windows are concatenated to obtain $X^L \in \mathbb{R}^{3 \times T}$.
3. KPI Construction: Discrete label columns are transformed into continuous anomaly scores (using SVDD or IsolationForest for SWaT/WADI), yielding KPI series $Y \in \mathbb{R}^T$.

This ensures standardized, multi-modal input representations for all scenarios and facilitates fair evaluation across diverse RCA models.

3. Formal RCA Task Definition

The RCA task is defined as follows. For each fault scenario $a$:

• Input:
  • Metrics $X^M \in \mathbb{R}^{n \times T}$ (n entities x time points)
  • Log features $X^L \in \mathbb{R}^{m \times T}$
  • KPI series $Y \in \mathbb{R}^T$
• Model Output:
  • A ranking $R_a$ of the n entities (most likely root causes first)

The ground-truth set of root causes is $X^L_2(t)$0. The core objective is to maximize the overlap between the top-K ranked entities and the ground-truth set.

Formal mapping:

$X^L_2(t)$1

This formalization supports direct comparison of methods that exploit metrics, logs, and KPIs jointly or separately (Zheng et al., 2024).

4. Baseline Methods and Benchmark Results

LEMMA-RCA provides a suite of baseline RCA methods, all centered on causal-graph frameworks:

#	Model	Brief Description
1	PC	Constraint-based independence tests
2	DYNOTEARS	Dynamic VAR + continuous DAG constraint
3	C-LSTM	LSTM-based Granger causality
4	GOLEM	Score-based DAG learning
5	REASON	Multi-level causal interdependency
6	NeZha	Multi-modal abnormality detection
7	MULAN	Multimodal structure + root-cause co-learning
8	CORAL	Online incremental causal DAG learning

Online variants (NOTEARS$X^L_2(t)$2, GOLEM$X^L_2(t)$3) are included for streaming settings.

Evaluation Protocols and Metrics:

• Offline: Full historical data to time-of-fault.
• Online: Incrementally updated, windowed streaming batches.
• Modalities: Single (metrics/logs) or fused multi-modal.

Primary Metrics:

• Precision@K ($X^L_2(t)$4), Mean Average Precision@K (MAP@K), Mean Reciprocal Rank (MRR).

Key empirical results:

• Offline, ProductReview/CloudComputing (Multi-modal):
  • MULAN, REASON: near-perfect PR@K and MAP@K
  • C-LSTM: high performance (MAP@10 = 70%)
  • Classical PC, DYNOTEARS: perform poorly without multimodal fusion
• Offline, SWaT/WADI (metrics only):
  • REASON yields best performance, though absolute MAP@10 is lower due to attack fleetingness
• Online:
  • CORAL outperforms online NOTEARS and GOLEM in streaming metrics-only setting

Model	PR@1	PR@5	MAP@10 (ProductReview, multi-modal, offline)
MULAN	100%	100%	100%
REASON	75%	100%	97.5%
C-LSTM	50%	75%	70%
CORAL (online, ProductReview)	75%	100%	97.5%

This establishes concrete methodological baselines and exposes strengths and current limitations under varying conditions (Zheng et al., 2024).

5. Evaluation, Metrics, and Protocols

LEMMA-RCA enforces standardized, reproducible evaluation:

• Splitting Strategy:
  • Offline: Use normal operation period for causal graph fitting; fault period for RCA evaluation.
  • Online: Warm-start causal model, update in batch as new data is ingested.
• Workflow:

1. Load $X^L_2(t)$5, $X^L_2(t)$6, $X^L_2(t)$7
2. Learn/update causal structure (offline/online)
3. Rank entities; compare against ground-truth root causes using PR@K, MAP@K, and MRR.

• Extensibility:
  • All new methods must utilize the same canonical inputs and protocols, ensuring comparability.

Metrics are formally defined as: $X^L_2(t)$8

$X^L_2(t)$9

$X^L_3(t)$0

Rigorous adherence to these protocols underpins meaningful head-to-head benchmarking.

6. Usage Recommendations, Insights, and Extensions

Usage:

• Download the full dataset from https://lemma-rca.github.io.
• For offline, split and process as prescribed; for online, update models incrementally.
• To benchmark a new method, process the original metrics, logs, and KPI series identically and report all standard metrics across modalities and modes.

Key insights:

• Multi-modal fusion is critical for high RCA accuracy in IT domains.
• Offline RCA is less effective when attack characteristics are ephemeral (noted in OT scenarios).
• Streaming/online learning (as in CORAL) is advantageous in dynamic environments.

Recommended Research Practice:

• Precisely match input structure and evaluation protocols for fair comparisons.
• Use provided LaTeX metric definitions to ensure reporting consistency.

LEMMA-RCA sets a new standard for evaluating RCA methods, providing a challenging and diverse testbed with multi-modal, temporal, and multi-domain data. Its extensibility, standardized evaluation, and domain breadth make it a foundational benchmark for RCA research in both operational and enterprise contexts (Zheng et al., 2024).