---
title: 'KylinRCA: Cloud-native RCA Systems'
url: https://www.emergentmind.com/topics/kylinrca
type: topic
---

# KylinRCA: Cloud-native RCA Systems

Searching arXiv for KylinRCA and closely related RCA papers to ground the article.
KylinRCA is a name used in recent arXiv literature for end-to-end root cause analysis systems in cloud-native and microservice environments. In one usage, it denotes a full-stack observability framework that combines dynamic causal discovery with multimodal, cross-level graph reasoning over metrics, logs, and traces to localize root-cause entities, identify fault types, and produce auditable explanations [2509.12231]. In a second usage, a later paper refers to KRCA as KylinRCA/KRCA and describes a hyper-scale microservice RCA system that progressively narrows the search space through API-level drilldown, skeleton-based causal graph instantiation, and memory-augmented multi-agent reasoning [2607.01788]. Across both uses, the name is associated with production-oriented RCA under large-scale, dynamic, and noisy observability conditions rather than with a single canonical algorithm.

## 1. Terminology, provenance, and scope

The term **KylinRCA** does not denote a single universally fixed architecture in the current arXiv record. Instead, it appears in at least two technically distinct but thematically related RCA systems.

| Usage of the name | Core setting | Defining mechanism |
|---|---|---|
| KylinRCA | Full-stack observability in cloud-native AIOps | Dynamic causality + multimodal cross-level graph reasoning |
| KylinRCA/KRCA | Hyper-scale microservice systems | API drilldown + causal skeleton + multi-agent verification |

The 2025 formulation presents KylinRCA as a response to three persistent RCA gaps in cloud-native AIOps: single-modal methods miss important signals, multimodal methods often ignore temporal causality, and causal discovery methods usually fail to handle cross-layer architectures such as host–Pod–service [2509.12231]. The 2026 formulation addresses a different operational regime: more than 200,000 microservices, hundreds of metrics per service, deep dependency chains, fast topology change, and context explosion for LLM-based diagnosis [2607.01788].

A common misconception is that KylinRCA names a single mature product or a single published algorithmic lineage. The available sources instead support a narrower statement: the label is used for RCA systems that share the aim of accurate localization, failure-type identification, and operator-facing diagnosis reports, but they implement that aim through substantially different pipelines.

The term should also not be conflated with an unrelated 2020 extra-large MIMO receiver paper whose summary explicitly notes that the paper itself does **not** use the name “KylinRCA” [2012.05409].

## 2. The 2025 full-stack observability framework

The 2025 KylinRCA framework is an end-to-end RCA design for observability streams composed of metrics, logs, and traces. It is organized as a five-module pipeline: **multimodal preprocessing**, **cross-modal representation learning**, **cross-level dynamic causality discovery**, **cross-hierarchical heterogeneous graph reasoning**, and **explainable output** [2509.12231].

Its workflow is split into an **offline training stage** and an **online reasoning stage**. Offline, it trains modality encoders, the temporal causal model, and the type-aware graph reasoning model using historical fault data and operational data, while also building a static topology database of deployment and invocation relationships. Online, it ingests real-time observability data, aligns and preprocesses them, learns unified entity representations, discovers local and cross-level causal edges in real time, fuses those edges with static topology into a global heterogeneous graph, and outputs root-cause locations, fault types, causal chains, and a structured diagnostic report to the O&M platform [2509.12231].

Conceptually, this KylinRCA is framed as a synthesis of two representative prior methods discussed in the source paper. **FaultInsight** contributes the temporal causal discovery idea: metric-only time series, a TCN-based semantic decoupling autoencoder, and perturbation experiments for causal-strength estimation and root-cause ranking. **HolisticRCA** contributes multimodal and cross-level modeling: dedicated encoders for metrics, logs, and traces; a heterogeneous graph over hosts, Pods, and services; GAT-based reasoning; and mask-based explanation. KylinRCA preserves FaultInsight’s emphasis on temporal causality but extends it beyond metrics and beyond a single layer, and preserves HolisticRCA’s multimodal fusion and graph reasoning while enriching them with causal edges, temporal modeling, type-aware attention, and evidence-chain explanations.

The target environment is explicitly cloud-native host–Pod–service architecture. This is significant because the framework is not limited to service-layer RCA: it is meant to trace fault spread from resource layer to container layer to service layer. The paper also claims that average inference latency is kept within about 2 seconds to satisfy real-time production requirements, indicating that the design is intended for operational deployment rather than offline analysis alone [2509.12231].

## 3. Temporal causality, multimodal encoding, and graph reasoning

The temporal causal discovery component is the primary mechanism for establishing propagation direction. After preprocessing and cross-modal fusion, each entity at each hierarchy level is represented by a multimodal feature vector. At the service tier, the paper describes a **Temporal Causal Network semantic decoupling autoencoder** whose encoder uses multiple kernel sizes—3, 5, and 7—to extract temporal semantics from multimodal composite features, while the decoder reconstructs the inputs and uses a VAR model in latent space to capture temporal dependencies [2509.12231].

The causal intensity from feature \(i\) to feature \(j\) at time \(t\) is defined as
\[
C (i->j,t) = (Loss (j,t|perturb (i)) - Loss (j,t|original)) Loss (j,t|original)
\]
and is estimated through an enhanced chronological perturbation experiment called **chronological perturbation++**. The intended change relative to global shuffling is segmented perturbation of one feature’s temporal order so that local temporal effects are better reflected. Causal edges are then filtered in two stages: a threshold retaining edges with \(C > 0.3\), followed by a mutual information test to remove false causation such as indirect effects due to common causes. The output is described as a three-dimensional relationship matrix over entity, time, and causal intensity [2509.12231].

To extend causal discovery across layers, KylinRCA introduces a causal transmission mechanism:
\[
Ccross Cintra (11,e1->12,e2,t)= (11,e1,t) * R (11,e1->12,e2)
\]
where \(l_1\) and \(l_2\) denote different levels, \(e_1\) and \(e_2\) are entities at those levels, and \(R\) is the entity association degree in \([0,1]\). In the appendix, this is implemented as matrix multiplication over time followed by min-max normalization to \([0,1]\). The stated effect is to convert local temporal causal edges into cross-tier causal edges by weighting them with topology or deployment affinity.

The multimodal front end is highly structured. Metrics are denoised with **wavelet transform**, normalized by **Z-score**, segmented with a **5-second sliding window**, and enriched with **12 time-series features** such as growth rate, short-term volatility variance, and cross-window change rate. Logs are processed with an enhanced **Drain++** template miner and semantic encoding via **BERT**, producing **768-dimensional** vectors and additional structured features such as template frequency, log level, and related entity IDs. Traces are converted into call topology graphs and span-derived performance features, and when sampling is below **5%**, similarity-based interpolation is used to fill missing spans. A time alignment mechanism then maps second-level metric timestamps, millisecond-level log events, and minute-level trace statistics into aligned time windows, yielding a three-dimensional tensor of time-entity-modal features [2509.12231].

For representation learning, metrics are encoded by a time-aware Transformer with positional information, multi-head attention, and a **512-dimensional** output; logs by a **CNN-BiLSTM** hybrid with a **512-dimensional** output; and traces by a **GCN-based** topological encoder combined with performance features, again yielding **512-dimensional** representations. A modality attention network computes weights over the three modalities, with weights summing to 1. The paper’s example is that error logs may receive greater weight when they are highly indicative.

The graph reasoning stage constructs a heterogeneous graph \(G = (V, E, T)\), where \(V\) contains hosts, Pods, and services, \(E\) includes static topology edges plus dynamic causal edges, and \(T\) encodes node and edge types. Its key modification over HolisticRCA is a **type-aware GAT**: node type and relation type influence attention weights, causal edges are weighted more heavily for root cause localization, and topology edges more heavily for entity association. The model serves two tasks. For localization, a sigmoid head outputs a fault probability for each node, and nodes with probability greater than **0.5** are deemed fault entities. For classification, separate heads are used for different entity types, with examples including host classes such as **CPU spike** and **memory leak**, and service classes such as **interface timeout** and **call failure** [2509.12231].

## 4. Explainability, evaluation, and deployment characteristics

The explainability layer is designed to produce auditable evidence chains rather than only labels. It combines **mask learning** with visualization. The mask module freezes the inference model and applies learnable masks, or random combination masking, to multimodal features, edges, or nodes to measure how outputs change when specific inputs are hidden. This is inherited from HolisticRCA but extended with a temporal weight distribution to reduce computation and better incorporate time. The resulting explanation is explicitly organized into three levels: **micro-level key features**, **meso-level propagation chain**, and **macro-level report** [2509.12231].

The automatic diagnostic report contains four sections: **fault overview**, **root cause location**, **propagation path**, and **repair suggestions**. Causal propagation is visualized from the three-dimensional causal matrix, with time on the horizontal axis, hierarchy on the vertical axis, and arrow thickness encoding causal strength. This is notable because the framework treats interpretability as a first-class output rather than as an auxiliary visualization.

The empirical evaluation uses three datasets: **Alibaba Cluster Trace Dataset V2**, **AIOps Challenge 2023**, and an **enterprise production dataset from a major e-commerce platform during the Double 11 festival**. The enterprise dataset uses a three-tier annotation system—entity-level, feature-level, and causal-level labels—with annotations performed by **five senior operations engineers** and quality checked using **Krippendorff’s alpha of 0.89**. Baselines include **FaultInsight**, **LogBERT**, **TraceRCA**, **HolisticRCA**, and **CausalGNN**. Evaluation is grouped into localization, classification, interpretability, and efficiency, using **entity accuracy, recall, F1**, **Macro-F1 and Micro-F1**, **causal chain accuracy and feature importance accuracy**, and **inference latency and throughput** [2509.12231].

On the enterprise production dataset, the paper reports **entity localization F1 of 92.3%**, **fault-type Macro-F1 of 89.7%**, and **causal chain accuracy of 88.1%**, together with **average latency of 1.8 seconds** and **throughput of 56 cases per minute**. It states that this corresponds to an **8.5% improvement over HolisticRCA** and a **15.2% improvement over FaultInsight** in localization F1. Ablation studies remove cross-level causal transmission, type attention, modality attention, and the mask module. The largest degradation occurs when cross-level causal modeling is removed, reducing entity localization F1 by **10.2%** and causal chain accuracy by **15.3%**; removing type attention reduces fault-type Macro-F1 by **8.7%**; removing modality attention reduces localization F1 by **6.5%**; and removing the mask module does not affect localization or classification but eliminates feature-importance interpretation [2509.12231].

Deployment guidance is concrete. The framework is containerized and deployed on **Kubernetes**, with **Prometheus** collecting metrics, **Fluentd** collecting logs, and **Jaeger** collecting traces. It is packaged into **Docker** images, managed with **Helm** charts, monitored with **Grafana**, and uses **OpenTelemetry APIs** for standardized data access and format conversion. The paper further notes distributed and horizontally scalable deployment, GPU assignment to key modules, and **HPA-based autoscaling**. It also discusses engineering challenges: small-sample cold start, efficient **PB-scale** processing, balancing robustness with interpretability, and standardizing integration across observability platforms.

## 5. KRCA as KylinRCA in hyper-scale microservice systems

A later paper uses **KRCA**, referred to throughout as **KylinRCA/KRCA**, for an RCA system built specifically for hyper-scale microservice environments [2607.01788]. The motivating setting is substantially larger and more dynamic than the 2025 framework’s presentation: more than **200,000 microservices**, hundreds of metrics per service, rapid topological and behavioral change, and failure blast radii that can exceed **10,000 services**.

Its architecture is a three-stage funnel designed to reduce search space while preserving recall. The first stage is **API-level drilldown**, which starts from the alerting API and recursively traverses the service dependency graph using API-level failure-rate and latency signals. The overall drilldown score is
\[
\text{Score}(P,C) = \max (\text{Score}_{f}(P,C), \text{Score}_{l}(P,C))
\]
with \(\text{Score}_{f}\) defined by time-lagged correlation over the failure-rate series of upstream and downstream APIs, and \(\text{Score}_{l}\) combining anomaly degree, fluctuation contribution, and correlation degree. The implementation uses propagation threshold **0.8**, maximum lag \(L=5\), and keeps the **top-3 suspicious services** for later stages [2607.01788].

The second stage is **skeleton-based causal graph instantiation**. Rather than performing expensive causal discovery over all raw anomalous metrics, KRCA maps anomalies into a generic causal skeleton \(G_s\) defined by four meta-metric categories: **External (E)**, **Internal (I)**, **Dependency (D)**, and **KPI (K)**. Anomalous metrics are detected using **Isolation Forest** together with **PatternMatcher**, then instantiated into an initial graph \(G_o\). The paper is explicit that this is a **high-recall structural prior**, not a full causal reconstruction.

The third stage is a **memory-augmented multi-agent framework**. A **Main Agent** orchestrates the process, while **nine domain-specific Sub Agents** cover specialized operational domains such as traffic, CPU, GPU, and other classic failure types. The system uses a three-tier memory hierarchy: **Working Memory**, **Factual Memory**, and **Experiential Memory**. Retrieval is RAG-based and uses a scoring function
\[
\text{Score}(Q, H) = \alpha \cdot S(Q_{s}, H_{s}) + \beta \cdot S(Q_{m}, H_{m}) + \gamma \cdot e^{-\lambda (T_Q - T_H)}
\]
where \(S(\cdot,\cdot)\) is BM25-based lexical similarity over service names and metric names, and historical entries are retrieved in **top-3** fashion, prioritizing Factual Memory and then falling back to Experiential Memory [2607.01788].

The evaluation uses a real-world dataset from **Kuaishou’s production environment**, containing **300 failures** sampled from **October 2025 to March 2026** across business lines including **Short Video**, **Recommendation**, and **E-commerce**. Failures cover **nine common classes** such as abnormal traffic, CPU/GPU saturation, and MySQL slow queries, with ground truth annotated by **five experienced SREs**. The principal metrics are **AC@k** and **Avg@k** for root-cause service localization and failure-type classification, together with end-to-end latency and human evaluation of report quality [2607.01788].

The headline result is **AC@1 of 0.88** for root-cause service localization and **0.79** for failure-type classification. Against the strongest baseline, **RCA-Agent**, these are reported as absolute gains of **31%** and **32%**. The same table gives **AC@3 of 0.96** for root-cause service and **0.98** for failure type, with average latency of **146.59 seconds per case** in the primary configuration. Ablations show that removing API-level drilldown reduces root-cause AC@1 from **0.88 to 0.64**; removing the skeleton graph reduces root-cause AC@1 to **0.72** and failure-type AC@1 to **0.61**; removing the multi-agent design reduces performance to **0.75** and **0.67**; and replacing structured retrieval with naive flat RAG degrades both tasks [2607.01788].

Deployment results are also reported. KRCA has been running in Kuaishou’s internal monitoring platform, **Tianwen**, from October 2025 to March 2026, supports more than **2,000 alerts per day** across **six business lines**, and in **483 emergency incidents** correctly identified both the root cause service and failure type in **82%** of cases. Manual diagnosis previously took about **52 minutes** on average; after deployment, average time to localize the root cause fell to **11.8 minutes**, a **77.3% reduction**. The paper also reports report-quality feedback from five SREs: **4.21** for readability and **4.10** for usefulness on a 5-point Likert scale [2607.01788].

## 6. Relation to adjacent RCA research, misconceptions, and limitations

The two KylinRCA usages occupy different positions in the RCA design space. The 2025 framework is primarily a **causal-discovery-plus-heterogeneous-graph** system built around full-stack observability. The 2026 KRCA system is primarily a **progressive search-space reduction plus agentic verification** system for hyper-scale microservices. A frequent misconception is therefore that KylinRCA is intrinsically an LLM-based method. That is not supported by the sources: the 2025 system is centered on temporal causal discovery, multimodal encoding, type-aware graph reasoning, and mask-based explanation, whereas the 2026 system introduces LLMs only after API-level and metric-level structure has narrowed the candidate space [2509.12231].

Another misconception is that KylinRCA is simply a multimodal fusion model. The 2025 source explicitly positions it against both **single-modal** RCA and multimodal methods that ignore temporal causality, and its main architectural claim is the joint treatment of **time direction**, **cross-level propagation**, and **multimodal evidence**. Conversely, the 2026 KRCA source is explicit that naive LLM reasoning over large telemetry contexts leads to “Lost in the Middle” effects, hallucinated causal relations, and large latency, motivating staged narrowing before agentic inference [2607.01788].

A useful nearby comparison is **RC-LLM**, which proposes a residual-connection-based RCA pipeline using an LLM as the final reasoning engine after hierarchical extraction and fusion of traces, metrics, and logs. On the **CCF AIOPS 2025 Root Cause Analysis Challenge** dataset, its final method reports **48.25% accuracy** and **3.4 average reasoning steps**, improving over raw telemetry prompting but remaining less explicitly graph- and causality-centric than the 2025 KylinRCA formulation [2602.08804]. This suggests a broader taxonomy within recent RCA research: structured causal-graph systems, staged agentic systems, and LLM-centric evidence-fusion systems.

Both KylinRCA usages also state limitations. The 2025 framework acknowledges bias from **hidden or unmonitored variables**, especially unseen dependencies and resources, and notes the difficulty of **cold start** under weakly labeled or newly emerging failures. The 2026 KRCA system depends on sufficiently observable **API-level signals** and degrades in incidents with **multiple simultaneous cascading anomalies**, where correlated noise complicates separation of the true propagation path. A plausible implication is that current KylinRCA-style research treats RCA less as a single prediction problem than as a constrained inference workflow in which observability alignment, structural priors, causal directionality, and explanation are all operationally significant.

In that sense, KylinRCA is best understood not as a monolithic algorithm, but as a name attached to a recent class of production-oriented RCA systems whose central concern is turning noisy full-stack telemetry into localized, typed, and auditable diagnoses under cloud-native scale and dynamism.

Source: https://www.emergentmind.com/topics/kylinrca