KylinRCA: Cloud-native RCA Systems

• KylinRCA is a set of production-oriented root cause analysis systems that integrate full-stack observability signals with dynamic causal discovery for accurate fault localization.
• It utilizes temporal autoencoders, cross-level causal transmission, and type-aware heterogeneous graph reasoning to classify failure types and generate actionable diagnostic reports.
• The KRCA variant extends the framework to hyper-scale microservices through API drilldown, causal skeleton graphs, and memory-augmented multi-agent reasoning to reduce diagnostic time and improve precision.

Searching arXiv for KylinRCA and closely related RCA papers to ground the article. KylinRCA is a name used in recent arXiv literature for end-to-end root cause analysis systems in cloud-native and microservice environments. In one usage, it denotes a full-stack observability framework that combines dynamic causal discovery with multimodal, cross-level graph reasoning over metrics, logs, and traces to localize root-cause entities, identify fault types, and produce auditable explanations (Hou, 8 Sep 2025). In a second usage, a later paper refers to KRCA as KylinRCA/KRCA and describes a hyper-scale microservice RCA system that progressively narrows the search space through API-level drilldown, skeleton-based causal graph instantiation, and memory-augmented multi-agent reasoning (Jiang et al., 2 Jul 2026). Across both uses, the name is associated with production-oriented RCA under large-scale, dynamic, and noisy observability conditions rather than with a single canonical algorithm.

1. Terminology, provenance, and scope

The term KylinRCA does not denote a single universally fixed architecture in the current arXiv record. Instead, it appears in at least two technically distinct but thematically related RCA systems.

Usage of the name	Core setting	Defining mechanism
KylinRCA	Full-stack observability in cloud-native AIOps	Dynamic causality + multimodal cross-level graph reasoning
KylinRCA/KRCA	Hyper-scale microservice systems	API drilldown + causal skeleton + multi-agent verification

The 2025 formulation presents KylinRCA as a response to three persistent RCA gaps in cloud-native AIOps: single-modal methods miss important signals, multimodal methods often ignore temporal causality, and causal discovery methods usually fail to handle cross-layer architectures such as host–Pod–service (Hou, 8 Sep 2025). The 2026 formulation addresses a different operational regime: more than 200,000 microservices, hundreds of metrics per service, deep dependency chains, fast topology change, and context explosion for LLM-based diagnosis (Jiang et al., 2 Jul 2026).

A common misconception is that KylinRCA names a single mature product or a single published algorithmic lineage. The available sources instead support a narrower statement: the label is used for RCA systems that share the aim of accurate localization, failure-type identification, and operator-facing diagnosis reports, but they implement that aim through substantially different pipelines.

The term should also not be conflated with an unrelated 2020 extra-large MIMO receiver paper whose summary explicitly notes that the paper itself does not use the name “KylinRCA” (Rodrigues et al., 2020).

2. The 2025 full-stack observability framework

The 2025 KylinRCA framework is an end-to-end RCA design for observability streams composed of metrics, logs, and traces. It is organized as a five-module pipeline: multimodal preprocessing, cross-modal representation learning, cross-level dynamic causality discovery, cross-hierarchical heterogeneous graph reasoning, and explainable output (Hou, 8 Sep 2025).

Its workflow is split into an offline training stage and an online reasoning stage. Offline, it trains modality encoders, the temporal causal model, and the type-aware graph reasoning model using historical fault data and operational data, while also building a static topology database of deployment and invocation relationships. Online, it ingests real-time observability data, aligns and preprocesses them, learns unified entity representations, discovers local and cross-level causal edges in real time, fuses those edges with static topology into a global heterogeneous graph, and outputs root-cause locations, fault types, causal chains, and a structured diagnostic report to the O&M platform (Hou, 8 Sep 2025).

Conceptually, this KylinRCA is framed as an overview of two representative prior methods discussed in the source paper. FaultInsight contributes the temporal causal discovery idea: metric-only time series, a TCN-based semantic decoupling autoencoder, and perturbation experiments for causal-strength estimation and root-cause ranking. HolisticRCA contributes multimodal and cross-level modeling: dedicated encoders for metrics, logs, and traces; a heterogeneous graph over hosts, Pods, and services; GAT-based reasoning; and mask-based explanation. KylinRCA preserves FaultInsight’s emphasis on temporal causality but extends it beyond metrics and beyond a single layer, and preserves HolisticRCA’s multimodal fusion and graph reasoning while enriching them with causal edges, temporal modeling, type-aware attention, and evidence-chain explanations.

The target environment is explicitly cloud-native host–Pod–service architecture. This is significant because the framework is not limited to service-layer RCA: it is meant to trace fault spread from resource layer to container layer to service layer. The paper also claims that average inference latency is kept within about 2 seconds to satisfy real-time production requirements, indicating that the design is intended for operational deployment rather than offline analysis alone (Hou, 8 Sep 2025).

3. Temporal causality, multimodal encoding, and graph reasoning

The temporal causal discovery component is the primary mechanism for establishing propagation direction. After preprocessing and cross-modal fusion, each entity at each hierarchy level is represented by a multimodal feature vector. At the service tier, the paper describes a Temporal Causal Network semantic decoupling autoencoder whose encoder uses multiple kernel sizes—3, 5, and 7—to extract temporal semantics from multimodal composite features, while the decoder reconstructs the inputs and uses a VAR model in latent space to capture temporal dependencies (Hou, 8 Sep 2025).

The causal intensity from feature $i$ to feature $j$ at time $t$ is defined as

$$C (i->j,t) = (Loss (j,t|perturb (i)) - Loss (j,t|original)) Loss (j,t|original)$$

and is estimated through an enhanced chronological perturbation experiment called chronological perturbation++. The intended change relative to global shuffling is segmented perturbation of one feature’s temporal order so that local temporal effects are better reflected. Causal edges are then filtered in two stages: a threshold retaining edges with $C > 0.3$, followed by a mutual information test to remove false causation such as indirect effects due to common causes. The output is described as a three-dimensional relationship matrix over entity, time, and causal intensity (Hou, 8 Sep 2025).

To extend causal discovery across layers, KylinRCA introduces a causal transmission mechanism: $$Ccross Cintra (11,e1->12,e2,t)= (11,e1,t) * R (11,e1->12,e2)$$ where $l_1$ and $l_2$ denote different levels, $e_1$ and $e_2$ are entities at those levels, and $j$0 is the entity association degree in $j$1. In the appendix, this is implemented as matrix multiplication over time followed by min-max normalization to $j$2. The stated effect is to convert local temporal causal edges into cross-tier causal edges by weighting them with topology or deployment affinity.

The multimodal front end is highly structured. Metrics are denoised with wavelet transform, normalized by Z-score, segmented with a 5-second sliding window, and enriched with 12 time-series features such as growth rate, short-term volatility variance, and cross-window change rate. Logs are processed with an enhanced Drain++ template miner and semantic encoding via BERT, producing 768-dimensional vectors and additional structured features such as template frequency, log level, and related entity IDs. Traces are converted into call topology graphs and span-derived performance features, and when sampling is below 5%, similarity-based interpolation is used to fill missing spans. A time alignment mechanism then maps second-level metric timestamps, millisecond-level log events, and minute-level trace statistics into aligned time windows, yielding a three-dimensional tensor of time-entity-modal features (Hou, 8 Sep 2025).

For representation learning, metrics are encoded by a time-aware Transformer with positional information, multi-head attention, and a 512-dimensional output; logs by a CNN-BiLSTM hybrid with a 512-dimensional output; and traces by a GCN-based topological encoder combined with performance features, again yielding 512-dimensional representations. A modality attention network computes weights over the three modalities, with weights summing to 1. The paper’s example is that error logs may receive greater weight when they are highly indicative.

The graph reasoning stage constructs a heterogeneous graph $j$3, where $j$4 contains hosts, Pods, and services, $j$5 includes static topology edges plus dynamic causal edges, and $j$6 encodes node and edge types. Its key modification over HolisticRCA is a type-aware GAT: node type and relation type influence attention weights, causal edges are weighted more heavily for root cause localization, and topology edges more heavily for entity association. The model serves two tasks. For localization, a sigmoid head outputs a fault probability for each node, and nodes with probability greater than 0.5 are deemed fault entities. For classification, separate heads are used for different entity types, with examples including host classes such as CPU spike and memory leak, and service classes such as interface timeout and call failure (Hou, 8 Sep 2025).

4. Explainability, evaluation, and deployment characteristics

The explainability layer is designed to produce auditable evidence chains rather than only labels. It combines mask learning with visualization. The mask module freezes the inference model and applies learnable masks, or random combination masking, to multimodal features, edges, or nodes to measure how outputs change when specific inputs are hidden. This is inherited from HolisticRCA but extended with a temporal weight distribution to reduce computation and better incorporate time. The resulting explanation is explicitly organized into three levels: micro-level key features, meso-level propagation chain, and macro-level report (Hou, 8 Sep 2025).

The automatic diagnostic report contains four sections: fault overview, root cause location, propagation path, and repair suggestions. Causal propagation is visualized from the three-dimensional causal matrix, with time on the horizontal axis, hierarchy on the vertical axis, and arrow thickness encoding causal strength. This is notable because the framework treats interpretability as a first-class output rather than as an auxiliary visualization.

The empirical evaluation uses three datasets: Alibaba Cluster Trace Dataset V2, AIOps Challenge 2023, and an enterprise production dataset from a major e-commerce platform during the Double 11 festival. The enterprise dataset uses a three-tier annotation system—entity-level, feature-level, and causal-level labels—with annotations performed by five senior operations engineers and quality checked using Krippendorff’s alpha of 0.89. Baselines include FaultInsight, LogBERT, TraceRCA, HolisticRCA, and CausalGNN. Evaluation is grouped into localization, classification, interpretability, and efficiency, using entity accuracy, recall, F1, Macro-F1 and Micro-F1, causal chain accuracy and feature importance accuracy, and inference latency and throughput (Hou, 8 Sep 2025).

On the enterprise production dataset, the paper reports entity localization F1 of 92.3%, fault-type Macro-F1 of 89.7%, and causal chain accuracy of 88.1%, together with average latency of 1.8 seconds and throughput of 56 cases per minute. It states that this corresponds to an 8.5% improvement over HolisticRCA and a 15.2% improvement over FaultInsight in localization F1. Ablation studies remove cross-level causal transmission, type attention, modality attention, and the mask module. The largest degradation occurs when cross-level causal modeling is removed, reducing entity localization F1 by 10.2% and causal chain accuracy by 15.3%; removing type attention reduces fault-type Macro-F1 by 8.7%; removing modality attention reduces localization F1 by 6.5%; and removing the mask module does not affect localization or classification but eliminates feature-importance interpretation (Hou, 8 Sep 2025).

Deployment guidance is concrete. The framework is containerized and deployed on Kubernetes, with Prometheus collecting metrics, Fluentd collecting logs, and Jaeger collecting traces. It is packaged into Docker images, managed with Helm charts, monitored with Grafana, and uses OpenTelemetry APIs for standardized data access and format conversion. The paper further notes distributed and horizontally scalable deployment, GPU assignment to key modules, and HPA-based autoscaling. It also discusses engineering challenges: small-sample cold start, efficient PB-scale processing, balancing robustness with interpretability, and standardizing integration across observability platforms.

5. KRCA as KylinRCA in hyper-scale microservice systems

A later paper uses KRCA, referred to throughout as KylinRCA/KRCA, for an RCA system built specifically for hyper-scale microservice environments (Jiang et al., 2 Jul 2026). The motivating setting is substantially larger and more dynamic than the 2025 framework’s presentation: more than 200,000 microservices, hundreds of metrics per service, rapid topological and behavioral change, and failure blast radii that can exceed 10,000 services.

Its architecture is a three-stage funnel designed to reduce search space while preserving recall. The first stage is API-level drilldown, which starts from the alerting API and recursively traverses the service dependency graph using API-level failure-rate and latency signals. The overall drilldown score is

$j$7

with $j$8 defined by time-lagged correlation over the failure-rate series of upstream and downstream APIs, and $j$9 combining anomaly degree, fluctuation contribution, and correlation degree. The implementation uses propagation threshold 0.8, maximum lag $t$0, and keeps the top-3 suspicious services for later stages (Jiang et al., 2 Jul 2026).

The second stage is skeleton-based causal graph instantiation. Rather than performing expensive causal discovery over all raw anomalous metrics, KRCA maps anomalies into a generic causal skeleton $t$1 defined by four meta-metric categories: External (E), Internal (I), Dependency (D), and KPI (K). Anomalous metrics are detected using Isolation Forest together with PatternMatcher, then instantiated into an initial graph $t$2. The paper is explicit that this is a high-recall structural prior, not a full causal reconstruction.

The third stage is a memory-augmented multi-agent framework. A Main Agent orchestrates the process, while nine domain-specific Sub Agents cover specialized operational domains such as traffic, CPU, GPU, and other classic failure types. The system uses a three-tier memory hierarchy: Working Memory, Factual Memory, and Experiential Memory. Retrieval is RAG-based and uses a scoring function

$t$3

where $t$4 is BM25-based lexical similarity over service names and metric names, and historical entries are retrieved in top-3 fashion, prioritizing Factual Memory and then falling back to Experiential Memory (Jiang et al., 2 Jul 2026).

The evaluation uses a real-world dataset from Kuaishou’s production environment, containing 300 failures sampled from October 2025 to March 2026 across business lines including Short Video, Recommendation, and E-commerce. Failures cover nine common classes such as abnormal traffic, CPU/GPU saturation, and MySQL slow queries, with ground truth annotated by five experienced SREs. The principal metrics are AC@k and Avg@k for root-cause service localization and failure-type classification, together with end-to-end latency and human evaluation of report quality (Jiang et al., 2 Jul 2026).

The headline result is AC@1 of 0.88 for root-cause service localization and 0.79 for failure-type classification. Against the strongest baseline, RCA-Agent, these are reported as absolute gains of 31% and 32%. The same table gives AC@3 of 0.96 for root-cause service and 0.98 for failure type, with average latency of 146.59 seconds per case in the primary configuration. Ablations show that removing API-level drilldown reduces root-cause AC@1 from 0.88 to 0.64; removing the skeleton graph reduces root-cause AC@1 to 0.72 and failure-type AC@1 to 0.61; removing the multi-agent design reduces performance to 0.75 and 0.67; and replacing structured retrieval with naive flat RAG degrades both tasks (Jiang et al., 2 Jul 2026).

Deployment results are also reported. KRCA has been running in Kuaishou’s internal monitoring platform, Tianwen, from October 2025 to March 2026, supports more than 2,000 alerts per day across six business lines, and in 483 emergency incidents correctly identified both the root cause service and failure type in 82% of cases. Manual diagnosis previously took about 52 minutes on average; after deployment, average time to localize the root cause fell to 11.8 minutes, a 77.3% reduction. The paper also reports report-quality feedback from five SREs: 4.21 for readability and 4.10 for usefulness on a 5-point Likert scale (Jiang et al., 2 Jul 2026).

6. Relation to adjacent RCA research, misconceptions, and limitations

The two KylinRCA usages occupy different positions in the RCA design space. The 2025 framework is primarily a causal-discovery-plus-heterogeneous-graph system built around full-stack observability. The 2026 KRCA system is primarily a progressive search-space reduction plus agentic verification system for hyper-scale microservices. A frequent misconception is therefore that KylinRCA is intrinsically an LLM-based method. That is not supported by the sources: the 2025 system is centered on temporal causal discovery, multimodal encoding, type-aware graph reasoning, and mask-based explanation, whereas the 2026 system introduces LLMs only after API-level and metric-level structure has narrowed the candidate space (Hou, 8 Sep 2025).

Another misconception is that KylinRCA is simply a multimodal fusion model. The 2025 source explicitly positions it against both single-modal RCA and multimodal methods that ignore temporal causality, and its main architectural claim is the joint treatment of time direction, cross-level propagation, and multimodal evidence. Conversely, the 2026 KRCA source is explicit that naive LLM reasoning over large telemetry contexts leads to “Lost in the Middle” effects, hallucinated causal relations, and large latency, motivating staged narrowing before agentic inference (Jiang et al., 2 Jul 2026).

A useful nearby comparison is RC-LLM, which proposes a residual-connection-based RCA pipeline using an LLM as the final reasoning engine after hierarchical extraction and fusion of traces, metrics, and logs. On the CCF AIOPS 2025 Root Cause Analysis Challenge dataset, its final method reports 48.25% accuracy and 3.4 average reasoning steps, improving over raw telemetry prompting but remaining less explicitly graph- and causality-centric than the 2025 KylinRCA formulation (Zhou et al., 9 Feb 2026). This suggests a broader taxonomy within recent RCA research: structured causal-graph systems, staged agentic systems, and LLM-centric evidence-fusion systems.

Both KylinRCA usages also state limitations. The 2025 framework acknowledges bias from hidden or unmonitored variables, especially unseen dependencies and resources, and notes the difficulty of cold start under weakly labeled or newly emerging failures. The 2026 KRCA system depends on sufficiently observable API-level signals and degrades in incidents with multiple simultaneous cascading anomalies, where correlated noise complicates separation of the true propagation path. A plausible implication is that current KylinRCA-style research treats RCA less as a single prediction problem than as a constrained inference workflow in which observability alignment, structural priors, causal directionality, and explanation are all operationally significant.

In that sense, KylinRCA is best understood not as a monolithic algorithm, but as a name attached to a recent class of production-oriented RCA systems whose central concern is turning noisy full-stack telemetry into localized, typed, and auditable diagnoses under cloud-native scale and dynamism.