---
title: 'LEMMA-RCA: A Multi-Modal Root Cause Analysis Benchmark'
url: https://www.emergentmind.com/topics/lemma-rca-benchmark
type: topic
---

# LEMMA-RCA: A Multi-Modal Root Cause Analysis Benchmark

LEMMA-RCA is a large-scale, multi-modal, multi-domain dataset and benchmark specifically designed for root cause analysis (RCA) in complex IT and OT systems. It addresses the shortage of extensive, open-source RCA benchmarks by providing diverse data across IT microservices and operational technology testbeds, combining real-world and systematically injected fault scenarios with detailed system observability data streams. LEMMA-RCA supports standardized comparative evaluation of RCA algorithms across multiple settings, modalities, and domains, and is publicly available for download and benchmarking [2406.05375].

## 1. Dataset Structure and Multi-Domain Scope

LEMMA-RCA comprises four sub-datasets, capturing fault-rich scenarios in both information technology (IT) and operational technology (OT) environments:

- **IT-AIOps (microservices):**
  - ProductReview: 4 fault scenarios, 216 pods/6 nodes
  - CloudComputing: 6 fault scenarios, ~168 pods/11 nodes
- **OT-ICS (industrial control):**
  - SWaT (Secure Water Treatment): 16 fault events, 51 sensors/actuators
  - WADI (Water Distribution): 9 fault events, 123 sensors/actuators

Across all sub-datasets, 35 real-fault instances are included, each involving hundreds of system entities per scenario.

**Modalities and Data Types:**
- **Time-series metrics** (node- and pod-level, 1 Hz, CSV/JSON via Prometheus/CloudWatch)
- **Textual logs** (unstructured, JSON, via ElasticSearch/CloudWatch Logs)
- **Key Performance Indicators (KPIs):** Derived series such as latency, throughput, error-rate, or anomaly scores

This composition supports evaluation of RCA methods in both IT (microservices, cloud) and OT (critical infrastructure) contexts, ensuring broad coverage and heterogeneity [2406.05375].

## 2. Data Collection, Preprocessing, and Representation

### Data Collection

**IT Domain:** Simulated real-fault injections (e.g., out-of-memory, DDoS, cryptojacking, storage-full) on running microservice clusters. Metrics and logs are collected at 1 Hz, with KPIs tracked via synthetic workload tools (e.g., JMeter).

**OT Domain:** Data is generated from the iTrust SWaT and WADI testbeds. Each includes long periods of normal operation followed by sequences under attack, with finely timestamped control/sensor data.

### Preprocessing Pipeline

1. **Stationarity Check:** Only stationary entity metric streams are retained.
2. **Log Feature Extraction:**
   - Parse raw logs to templates (Drain parser).
   - Slide a 10-minute window (30s stride); compute:
     - \(X^L_1(t)\): per-template frequencies
     - \(X^L_2(t)\): “golden-signal” error/exception template counts
     - \(X^L_3(t)\): PCA top TF–IDF log token features
   - Log windows are concatenated to obtain \(X^L \in \mathbb{R}^{3 \times T}\).
3. **KPI Construction:** Discrete label columns are transformed into continuous anomaly scores (using SVDD or IsolationForest for SWaT/WADI), yielding KPI series \(Y \in \mathbb{R}^T\).

*This ensures standardized, multi-modal input representations for all scenarios and facilitates fair evaluation across diverse RCA models.*

## 3. Formal RCA Task Definition

The RCA task is defined as follows. For each fault scenario \(a\):

- **Input:** 
  - Metrics \(X^M \in \mathbb{R}^{n \times T}\) (n entities x time points)
  - Log features \(X^L \in \mathbb{R}^{m \times T}\)
  - KPI series \(Y \in \mathbb{R}^T\)
- **Model Output:** 
  - A ranking \(R_a\) of the n entities (most likely root causes first)

The ground-truth set of root causes is \(V_a \subseteq \{1,\ldots,n\}\). The core objective is to maximize the overlap between the top-K ranked entities and the ground-truth set.

**Formal mapping:** 
\[
\mathcal{M}: (X^M, X^L, Y) \mapsto R_a,\quad \text{select top-}K \text{ entities }R_a(1:K)
\]

This formalization supports direct comparison of methods that exploit metrics, logs, and KPIs jointly or separately [2406.05375].

## 4. Baseline Methods and Benchmark Results

LEMMA-RCA provides a suite of baseline RCA methods, all centered on causal-graph frameworks:

| # | Model      | Brief Description                        |
|---|------------|------------------------------------------|
| 1 | PC         | Constraint-based independence tests      |
| 2 | DYNOTEARS  | Dynamic VAR + continuous DAG constraint  |
| 3 | C-LSTM     | LSTM-based Granger causality             |
| 4 | GOLEM      | Score-based DAG learning                 |
| 5 | REASON     | Multi-level causal interdependency       |
| 6 | NeZha      | Multi-modal abnormality detection        |
| 7 | MULAN      | Multimodal structure + root-cause co-learning |
| 8 | CORAL      | Online incremental causal DAG learning   |

*Online variants* (NOTEARS\(^*\), GOLEM\(^*\)) are included for streaming settings.

**Evaluation Protocols and Metrics:**
- **Offline:** Full historical data to time-of-fault.
- **Online:** Incrementally updated, windowed streaming batches.
- **Modalities:** Single (metrics/logs) or fused multi-modal.

**Primary Metrics:**
- Precision@K (\(\mathrm{PR@K}\)), Mean Average Precision@K (MAP@K), Mean Reciprocal Rank (MRR).

**Key empirical results:**

- **Offline, ProductReview/CloudComputing (Multi-modal):**
  - MULAN, REASON: near-perfect PR@K and MAP@K
  - C-LSTM: high performance (MAP@10 = 70%)
  - Classical PC, DYNOTEARS: perform poorly without multimodal fusion
- **Offline, SWaT/WADI (metrics only):**
  - REASON yields best performance, though absolute MAP@10 is lower due to attack fleetingness
- **Online:**
  - CORAL outperforms online NOTEARS and GOLEM in streaming metrics-only setting

| Model        | PR@1 | PR@5 | MAP@10 (ProductReview, multi-modal, offline) |
|--------------|------|------|----------------------------------------------|
| MULAN        | 100% | 100% | 100%                                         |
| REASON       | 75%  | 100% | 97.5%                                        |
| C-LSTM       | 50%  | 75%  | 70%                                          |
| CORAL (online, ProductReview) | 75% | 100% | 97.5%                         |

This establishes concrete methodological baselines and exposes strengths and current limitations under varying conditions [2406.05375].

## 5. Evaluation, Metrics, and Protocols

LEMMA-RCA enforces standardized, reproducible evaluation:

- **Splitting Strategy:** 
  - *Offline*: Use normal operation period for causal graph fitting; fault period for RCA evaluation.
  - *Online*: Warm-start causal model, update in batch as new data is ingested.
- **Workflow:**
  1. Load \(X^M\), \(X^L\), \(Y\)
  2. Learn/update causal structure (offline/online)
  3. Rank entities; compare against ground-truth root causes using PR@K, MAP@K, and MRR.
- **Extensibility:** 
  - All new methods must utilize the same canonical inputs and protocols, ensuring comparability.

**Metrics** are formally defined as:
\[
\mathrm{PR@K} = \frac{1}{|\mathbb{A}|}\sum_{a\in\mathbb{A}} 
\frac{\left|\{i \le K : R_a(i) \in V_a\}\right|}{\min(K, |V_a|)}
\]
\[
\mathrm{MAP@K} = 
\frac{1}{|\mathbb{A}|}\sum_{a\in\mathbb{A}} 
\left( \frac{1}{K} \sum_{j=1}^K \mathrm{PR@j}_a \right)
\]
\[
\mathrm{MRR} = 
\frac{1}{|\mathbb{A}|}\sum_{a\in\mathbb{A}} 
\frac{1}{\min \{ i : R_a(i) \in V_a \} }
\]

Rigorous adherence to these protocols underpins meaningful head-to-head benchmarking.

## 6. Usage Recommendations, Insights, and Extensions

**Usage:**
- Download the full dataset from https://lemma-rca.github.io.
- For *offline*, split and process as prescribed; for *online*, update models incrementally.
- To benchmark a new method, process the original metrics, logs, and KPI series identically and report all standard metrics across modalities and modes.

**Key insights:**
- Multi-modal fusion is critical for high RCA accuracy in IT domains.
- Offline RCA is less effective when attack characteristics are ephemeral (noted in OT scenarios).
- Streaming/online learning (as in CORAL) is advantageous in dynamic environments.

**Recommended Research Practice:**
- Precisely match input structure and evaluation protocols for fair comparisons.
- Use provided LaTeX metric definitions to ensure reporting consistency.

LEMMA-RCA sets a new standard for evaluating RCA methods, providing a challenging and diverse testbed with multi-modal, temporal, and multi-domain data. Its extensibility, standardized evaluation, and domain breadth make it a foundational benchmark for RCA research in both operational and enterprise contexts [2406.05375].

Source: https://www.emergentmind.com/topics/lemma-rca-benchmark