---
title: 'Cloud-OpsBench: Benchmark for Kubernetes RCA'
url: https://www.emergentmind.com/topics/cloud-opsbench
type: topic
---

# Cloud-OpsBench: Benchmark for Kubernetes RCA

Searching arXiv for Cloud-OpsBench and related benchmarks to ground the article in current literature.
Cloud-OpsBench is a full-stack, deterministic digital twin of a Kubernetes-based cloud environment designed to evaluate and train agentic Root Cause Analysis (RCA) systems: agents that behave like site reliability engineers (SREs), not like classifiers. Introduced in "Cloud-OpsBench: A Reproducible Benchmark for Agentic Root Cause Analysis in Cloud Systems" [2603.00468], it couples realistic incidents, a tool-using environment, and process-level ground truth, so that evaluation targets not only whether an agent finds the right diagnosis, but also how the investigation unfolds. The benchmark contains 452 distinct fault cases across 40 root cause types spanning the full Kubernetes stack, and is positioned simultaneously as a data engine for supervised fine-tuning (SFT), a reinforcement learning (RL) environment, and a diagnostic standard for process-centric assessment [2603.00468].

## 1. Problem setting and benchmark rationale

Cloud-OpsBench is motivated by a mismatch between conventional RCA benchmarks and operational reality. Much prior AIOps and RCA work treats diagnosis as a discriminative pattern-recognition problem over static telemetry dumps such as KPI time series, logs, and traces. Benchmarks such as LogHub, TimeSeriesBench, Nezha, RCAEval, LogEval, OpenRCA, and the AIOps challenges are therefore well suited to anomaly detection and label prediction, but they do not model interactive investigation. In particular, there is no analogue of running `kubectl get pods`, examining logs, checking node state, and revising hypotheses across multiple steps [2603.00468].

Dynamic environments such as AIOpsLab, AIOpsArena, and ITBench move closer to operational practice by deploying real microservices and injecting faults live, but they introduce stochasticity, poor reproducibility, and high operational cost and latency. Cloud-OpsBench is explicitly framed as an attempt to bridge this trade-off: preserve interaction and realism while converting the environment into a deterministic, zero-latency digital twin that can be replayed indefinitely [2603.00468].

The benchmark also responds to a metric-level blind spot. Outcome-only scores such as Accuracy, F1, and Top-$k$ cannot distinguish between a correct diagnosis reached through valid evidence gathering and a lucky guess driven by prior bias. For agentic systems, especially LLM-based systems prone to shortcut exploitation and hallucination, Cloud-OpsBench treats the diagnostic process itself as a first-class object of evaluation [2603.00468].

In that sense, Cloud-OpsBench belongs to a broader line of cloud benchmarking that emphasizes operational realism rather than isolated prediction or throughput measurements. Related directions include cross-layer monitoring and benchmarking frameworks such as CLAMBS [1502.00206], decision-centric cloud benchmarks such as CloudCons [2606.13513], and metric frameworks for elasticity, isolation, availability, and operational risk [1604.03470]. Cloud-OpsBench specializes this general concern to Kubernetes RCA.

## 2. Deterministic digital twin and incident construction

The core architectural concept is the **State Snapshot Paradigm**. For each fault case, a real fault is injected into a live Kubernetes cluster, the fault is allowed to manifest and be verified, and the relevant system state is then frozen into a snapshot. That snapshot includes control-plane objects such as Deployments, Pods, Services, PVCs, and events; observability data including Prometheus metrics, Istio traces, and logs; and cluster context such as node states, taints, labels, and control-plane component status [2603.00468].

After capture, state and interaction are decoupled. The snapshot becomes a static, immutable JSON repository, and a mocked interface serves standard operational queries such as `kubectl get`, log retrieval, and node health checks. Tool calls are answered from the snapshot with zero latency and deterministic outputs. For a given case, every agent therefore sees the same environment on every run [2603.00468].

The live environment used to generate these snapshots is a Kubernetes v1.31 cluster on Huawei Cloud ECS with 4 instances. Prometheus provides metrics, Istio provides traffic and service-mesh telemetry, and the native Kubernetes API provides events, objects, and logs. The application workload is Google Online Boutique, consisting of 11 microservices, with Locust used as the load generator. Fault injection is performed with ChaosBlade for infrastructure perturbations such as network delay, network loss, and disk pressure, together with direct `kubectl`-style operations for configuration errors [2603.00468].

Fault coverage spans 452 distinct fault cases and 40 root cause types organized into 7 categories: Admission Control, Scheduling, Startup, Runtime, Service Routing, Performance, and Infrastructure. The examples listed in the benchmark include namespace quotas exceeded, missing service account, node cordoned, affinity or selector mismatches, taints, PVC issues, image pull errors, volume permission issues, OOMKilled, liveness and readiness probe misconfiguration, selector mismatch, port or protocol mismatch, environment variable misconfiguration, pod-level CPU overload, network delay, and failures of kubelet, kube-proxy, scheduler, or containerd [2603.00468].

A common misconception is that the term “snapshot” implies the loss of temporal structure. In Cloud-OpsBench, time is not removed but sealed. Each snapshot reflects a particular incident window, and event timestamps, metric timestamps, and log timestamps remain available for reasoning about temporal order. What is absent is state evolution during the episode: no healing, no new logs, and no intervention-induced transitions. The benchmark therefore evaluates post-mortem RCA rather than online control or remediation [2603.00468].

## 3. Task formulation, tool interface, and trajectory model

Cloud-OpsBench formalizes agentic RCA as a trajectory-based decision process. For each case, the ground-truth diagnosis is a tuple
$$
\mathcal{R}^* = \langle S, C, R \rangle,
$$
where $S$ is the lifecycle stage, $C$ is the component, and $R$ is the root cause type. The agent receives a natural-language alert $A_{alert}$ and tool access to the snapshot environment $\mathcal{E}_{snapshot}$, and implements
$$
f: \langle A_{alert}, \mathcal{E}_{snapshot} \rangle \rightarrow \langle \mathcal{T}, \hat{\mathcal{R}} \rangle,
$$
where $\mathcal{T}$ is the diagnostic trajectory and $\hat{\mathcal{R}}$ is the predicted diagnosis [2603.00468].

The trajectory is represented as
$$
\mathcal{T} = [(t_1, a_1, o_1), \dots, (t_n, a_n, o_n)],
$$
with $t_i$ denoting the agent’s natural-language thought, $a_i$ the tool invocation, and $o_i$ the resulting observation. Gold supervision is provided not only for $\mathcal{R}^*$ but also for a canonical expert trajectory $\mathcal{T}^*$, synthesized by inverting the injection logic; for example, if a node was tainted during fault creation, the expert confirmation path includes a node description step that exposes taints [2603.00468].

Interaction is organized around ten tool APIs. Resource inspection is handled by `T1.GetResources`, `T2.DescribeResource`, and `T3.GetAppYAML`. Service interaction and topology use `T4.GetServiceDependencies` and `T6.CheckServiceConnectivity`. Telemetry analysis uses `T5.GetRecentLogs`, `T8.GetAlerts`, and `T9.GetErrorLogs`. Infrastructure diagnostics use `T7.GetClusterConfiguration` and `T10.CheckNodeServiceStatus` [2603.00468].

These tools are exposed as structured APIs with Pydantic schemas. The action space is textual but schema-constrained in execution: the agent must generate a valid tool name and valid arguments such as service names and namespaces. Invalid queries produce realistic errors and contribute to robustness metrics. Although the environment is finite because only precomputed queries have entries, the agent does not receive the complete menu and must infer valid interactions from context [2603.00468].

This design shifts RCA from passive pattern recognition to active perception, knowledge grounding, and deductive reasoning. The agent selects what to inspect, integrates telemetry with documentation-like or configuration-like context, and verifies hypotheses through multi-step evidence gathering rather than one-shot prediction [2603.00468].

## 4. Evaluation protocol and empirical behavior

Cloud-OpsBench evaluates both outcome and process. For diagnostic accuracy, the benchmark uses Top-$k$ Accuracy,
$$
A@k = \frac{1}{|I|} \sum_{i=1}^{|I|} \mathbb{I}(rc_i \subseteq E_i^k),
$$
together with Task Completion Rate (TCR), defined as the fraction of episodes in which the agent produces a well-formed diagnosis. Process-based evaluation includes trajectory alignment, tool-usage efficiency, and operational robustness [2603.00468].

Trajectory alignment is measured in three forms: Exact Match, Any-Order Match, and In-Order Match. Tool-usage efficiency is quantified with Relevance, Coverage, total Steps, and Mean Time To Identify (MTTI). If $\mathcal{S}_{agent}$ is the set of distinct tools used by the agent and $\mathcal{S}_{gold}$ is the set of distinct tools in the gold trajectory, then
$$
\text{Relevance} = \frac{|\mathcal{S}_{agent} \cap \mathcal{S}_{gold}|}{|\mathcal{S}_{agent}|},
\qquad
\text{Coverage} = \frac{|\mathcal{S}_{agent} \cap \mathcal{S}_{gold}|}{|\mathcal{S}_{gold}|}.
$$
Operational robustness is assessed with Invalid Action Count (IAC), Redundant Action Rate (RAR), and Zero-Tool Diagnosis Rate (ZTDR) [2603.00468].

The reported A@1 results for seven models are: DeepSeek-V3.2 at 0.73, GPT-5 at 0.67, GPT-4o at 0.49, Qwen3-235B at 0.50, Claude-4-Sonnet at 0.50, Qwen3-14B at 0.34, and Qwen3-8B at 0.21. DeepSeek-V3.2 records 10.0 steps on average, Coverage 0.88, Relevance 0.43, RAR 0.11, and ZTDR 0. GPT-5 records 5.57 steps, Relevance 0.65, Coverage 0.77, and RAR 0.05. Qwen3-14B records IAC 0.40, Relevance 0.63, and A@1 0.34 [2603.00468].

Several empirical patterns are emphasized. Exploration depth correlates with accuracy: more steps and higher expert-tool coverage are associated with higher A@1. Any-Order Match exceeds In-Order Match for all models, whereas Exact Match remains low, around 0 to 0.16, indicating that agents often access the correct evidence sources but interleave them with exploratory noise. Explicit faults such as Startup and Runtime cases are easier because they expose clear Kubernetes status and events, whereas cross-layer faults such as Admission and Performance are harder because they require correlating application symptoms with underlying infrastructure or configuration conditions [2603.00468].

Prompting and knowledge strategies also produce marked shifts. For GPT-4o, A@1 is 0.49 under Base and CoT, 0.70 under ICL, and 0.61 under RAG. For Qwen3-14B, A@1 is 0.34 under Base, 0.44 under CoT, 0.71 under ICL, and 0.50 under RAG. For Qwen3-235B, A@1 is 0.50 under Base, 0.47 under CoT, 0.59 under ICL, and 0.60 under RAG. The benchmark interprets these results as evidence that in-context procedural traces are more effective than CoT alone for agentic RCA [2603.00468].

## 5. Data engine, RL environment, and diagnostic standard

Cloud-OpsBench is presented not only as a benchmark but also as enabling infrastructure for next-generation SRE research. As a **Data Engine**, it harvests high-quality reasoning trajectories that can be used for SFT of small language models. As an **RL environment**, it turns high-risk operations into a safe, low-latency sandbox for policy optimization. As a **Diagnostic Standard**, it exposes architectural bottlenecks through process-centric evaluation rather than outcome-only scoring [2603.00468].

The data-engine role rests on the claim that procedural demonstrations are more useful than declarative knowledge for SRE agents. The benchmark logs alert text, trajectories of thought, tool calls, and tool outputs, final diagnosis structures, correctness labels, and metadata such as steps and invalid actions. CrewAI is used as the orchestrator for multi-step tool use, and Langfuse collects structured traces of prompts, outputs, and tool invocations. These logs are directly usable as SFT data or as ICL exemplars [2603.00468].

The RL interpretation is episodic and finite-horizon. An episode begins with the alert and optional initial context, proceeds through repeated tool invocations and observations, and ends when the agent emits a final diagnosis or exhausts its step or invalid-action budget. The paper does not instantiate a fixed reward equation, but it explicitly proposes a multi-term reward over diagnosis accuracy, action cost, latency, process alignment, and robustness, with negative contributions from Steps, IAC, and RAR, and positive contributions from Tool Relevance, Coverage, and trajectory alignment. This suggests direct compatibility with methods such as PPO or DPO over sequence-level policies [2603.00468].

As a diagnostic standard, Cloud-OpsBench identifies two named phenomena. The first is **Syntactic Fragility (SLMs)**: Qwen3-14B exhibits high tool relevance but many invalid tool invocations, indicating that schema correctness and tool syntax, rather than raw semantic reasoning, are the main bottlenecks. The second is the **Redundancy Paradox (DeepSeek-V3.2 vs GPT-4o)**: DeepSeek-V3.2 achieves higher A@1 by taking more steps and tolerating more redundancy, whereas GPT-4o is faster and more concise but substantially less accurate. The benchmark interprets this as evidence that extra verification steps can function as a reliability mechanism rather than a defect [2603.00468].

A further failure mode concerns zero-tool diagnoses. Some models, notably Claude-4-Sonnet, display a high ZTDR and speculate from parametric priors rather than grounded inspection. Within the benchmark’s process-centric framing, this behavior is unacceptable in real SRE contexts because it bypasses evidential verification [2603.00468].

## 6. Reproducibility, implementation, and limitations

Cloud-OpsBench’s reproducibility claim depends on a concrete generation pipeline. Fault knowledge is assembled from Kubernetes documentation and tutorials, Stack Overflow, KodeKloud notes, and academic papers on Kubernetes reliability, then converted by Gemini 3 Pro into a structured model $\langle \mathcal{P}, \mathcal{A}, \mathcal{S} \rangle$ of prerequisites, fault artifact, and activation sequence. Human review and LLM-as-a-judge are used to validate semantics and reproducibility. Automatic fault-case generation then proceeds through a Generator Agent, Executor Agent, and Verifier Agent. Once a case is verified, all diagnostic tools are run in a parameter sweep; on average, approximately 487 distinct tool invocations are stored per case in JSON form [2603.00468].

This precomputation is central to determinism. Any reasonable query is either present in the snapshot or yields a realistic not-found error. Benchmark execution therefore requires only enough CPU or GPU for model inference, not a live cluster. Code and dataset are provided at `https://github.com/LLM4Ops/Cloud-OpsBench`, including scripts for replaying cases, running agents, collecting traces, and accessing state snapshots and fault definitions [2603.00468].

The benchmark nonetheless has explicit scope limitations. It is focused on Kubernetes microservices; it does not model monoliths, broader data platforms, or vendor-specific APM systems. Because state does not evolve during an episode, it does not evaluate remediation, control policies, autoscaling, or self-healing dynamics. Its observability stack is deliberately vanilla, relying on Prometheus, Istio, and Kubernetes standard APIs. Extending it also requires familiarity with Python and LLM tooling [2603.00468].

Within the wider cloud benchmarking landscape, this places Cloud-OpsBench in a distinctive position. Reusable cloud datasets can characterize heterogeneous compute services across workload classes [2501.06128], and continuous ecosystems can store and curate benchmark results for application-specific comparisons over time [1812.05257]. Cloud-OpsBench instead concentrates on deterministic, tool-centric digital twins with white-box process evaluation for RCA. The paper’s formulation suggests a broader implication: cloud-operations benchmarks need not choose between realism and reproducibility if operational interaction is reconstructed from verified snapshots rather than replayed exclusively in live infrastructure [2603.00468].

Source: https://www.emergentmind.com/topics/cloud-opsbench