Cloud-OpsBench: Benchmark for Kubernetes RCA

• Cloud-OpsBench is a deterministic digital twin of a Kubernetes environment featuring 452 fault cases across 40 root causes, enabling robust agentic RCA evaluation.
• It integrates realistic incident simulation, tool-based interactions, and immutable state snapshots to preserve operational context and reproducibility.
• The benchmark supports both supervised fine-tuning and reinforcement learning by evaluating diagnostic trajectories alongside traditional accuracy metrics.

Searching arXiv for Cloud-OpsBench and related benchmarks to ground the article in current literature. Cloud-OpsBench is a full-stack, deterministic digital twin of a Kubernetes-based cloud environment designed to evaluate and train agentic Root Cause Analysis (RCA) systems: agents that behave like site reliability engineers (SREs), not like classifiers. Introduced in "Cloud-OpsBench: A Reproducible Benchmark for Agentic Root Cause Analysis in Cloud Systems" (Wang et al., 28 Feb 2026), it couples realistic incidents, a tool-using environment, and process-level ground truth, so that evaluation targets not only whether an agent finds the right diagnosis, but also how the investigation unfolds. The benchmark contains 452 distinct fault cases across 40 root cause types spanning the full Kubernetes stack, and is positioned simultaneously as a data engine for supervised fine-tuning (SFT), a reinforcement learning (RL) environment, and a diagnostic standard for process-centric assessment (Wang et al., 28 Feb 2026).

1. Problem setting and benchmark rationale

Cloud-OpsBench is motivated by a mismatch between conventional RCA benchmarks and operational reality. Much prior AIOps and RCA work treats diagnosis as a discriminative pattern-recognition problem over static telemetry dumps such as KPI time series, logs, and traces. Benchmarks such as LogHub, TimeSeriesBench, Nezha, RCAEval, LogEval, OpenRCA, and the AIOps challenges are therefore well suited to anomaly detection and label prediction, but they do not model interactive investigation. In particular, there is no analogue of running kubectl get pods, examining logs, checking node state, and revising hypotheses across multiple steps (Wang et al., 28 Feb 2026).

Dynamic environments such as AIOpsLab, AIOpsArena, and ITBench move closer to operational practice by deploying real microservices and injecting faults live, but they introduce stochasticity, poor reproducibility, and high operational cost and latency. Cloud-OpsBench is explicitly framed as an attempt to bridge this trade-off: preserve interaction and realism while converting the environment into a deterministic, zero-latency digital twin that can be replayed indefinitely (Wang et al., 28 Feb 2026).

The benchmark also responds to a metric-level blind spot. Outcome-only scores such as Accuracy, F1, and Top-$k$ cannot distinguish between a correct diagnosis reached through valid evidence gathering and a lucky guess driven by prior bias. For agentic systems, especially LLM-based systems prone to shortcut exploitation and hallucination, Cloud-OpsBench treats the diagnostic process itself as a first-class object of evaluation (Wang et al., 28 Feb 2026).

In that sense, Cloud-OpsBench belongs to a broader line of cloud benchmarking that emphasizes operational realism rather than isolated prediction or throughput measurements. Related directions include cross-layer monitoring and benchmarking frameworks such as CLAMBS (Alhamazani et al., 2015), decision-centric cloud benchmarks such as CloudCons (Zhang et al., 11 Jun 2026), and metric frameworks for elasticity, isolation, availability, and operational risk (Herbst et al., 2016). Cloud-OpsBench specializes this general concern to Kubernetes RCA.

2. Deterministic digital twin and incident construction

The core architectural concept is the State Snapshot Paradigm. For each fault case, a real fault is injected into a live Kubernetes cluster, the fault is allowed to manifest and be verified, and the relevant system state is then frozen into a snapshot. That snapshot includes control-plane objects such as Deployments, Pods, Services, PVCs, and events; observability data including Prometheus metrics, Istio traces, and logs; and cluster context such as node states, taints, labels, and control-plane component status (Wang et al., 28 Feb 2026).

After capture, state and interaction are decoupled. The snapshot becomes a static, immutable JSON repository, and a mocked interface serves standard operational queries such as kubectl get, log retrieval, and node health checks. Tool calls are answered from the snapshot with zero latency and deterministic outputs. For a given case, every agent therefore sees the same environment on every run (Wang et al., 28 Feb 2026).

The live environment used to generate these snapshots is a Kubernetes v1.31 cluster on Huawei Cloud ECS with 4 instances. Prometheus provides metrics, Istio provides traffic and service-mesh telemetry, and the native Kubernetes API provides events, objects, and logs. The application workload is Google Online Boutique, consisting of 11 microservices, with Locust used as the load generator. Fault injection is performed with ChaosBlade for infrastructure perturbations such as network delay, network loss, and disk pressure, together with direct kubectl-style operations for configuration errors (Wang et al., 28 Feb 2026).

Fault coverage spans 452 distinct fault cases and 40 root cause types organized into 7 categories: Admission Control, Scheduling, Startup, Runtime, Service Routing, Performance, and Infrastructure. The examples listed in the benchmark include namespace quotas exceeded, missing service account, node cordoned, affinity or selector mismatches, taints, PVC issues, image pull errors, volume permission issues, OOMKilled, liveness and readiness probe misconfiguration, selector mismatch, port or protocol mismatch, environment variable misconfiguration, pod-level CPU overload, network delay, and failures of kubelet, kube-proxy, scheduler, or containerd (Wang et al., 28 Feb 2026).

A common misconception is that the term “snapshot” implies the loss of temporal structure. In Cloud-OpsBench, time is not removed but sealed. Each snapshot reflects a particular incident window, and event timestamps, metric timestamps, and log timestamps remain available for reasoning about temporal order. What is absent is state evolution during the episode: no healing, no new logs, and no intervention-induced transitions. The benchmark therefore evaluates post-mortem RCA rather than online control or remediation (Wang et al., 28 Feb 2026).

3. Task formulation, tool interface, and trajectory model

Cloud-OpsBench formalizes agentic RCA as a trajectory-based decision process. For each case, the ground-truth diagnosis is a tuple

$\mathcal{R}^* = \langle S, C, R \rangle,$

where $S$ is the lifecycle stage, $C$ is the component, and $R$ is the root cause type. The agent receives a natural-language alert $A_{alert}$ and tool access to the snapshot environment $\mathcal{E}_{snapshot}$, and implements

$$f: \langle A_{alert}, \mathcal{E}_{snapshot} \rangle \rightarrow \langle \mathcal{T}, \hat{\mathcal{R}} \rangle,$$

where $\mathcal{T}$ is the diagnostic trajectory and $\hat{\mathcal{R}}$ is the predicted diagnosis (Wang et al., 28 Feb 2026).

The trajectory is represented as

$\mathcal{R}^* = \langle S, C, R \rangle,$0

with $\mathcal{R}^* = \langle S, C, R \rangle,$1 denoting the agent’s natural-language thought, $\mathcal{R}^* = \langle S, C, R \rangle,$2 the tool invocation, and $\mathcal{R}^* = \langle S, C, R \rangle,$3 the resulting observation. Gold supervision is provided not only for $\mathcal{R}^* = \langle S, C, R \rangle,$4 but also for a canonical expert trajectory $\mathcal{R}^* = \langle S, C, R \rangle,$5, synthesized by inverting the injection logic; for example, if a node was tainted during fault creation, the expert confirmation path includes a node description step that exposes taints (Wang et al., 28 Feb 2026).

Interaction is organized around ten tool APIs. Resource inspection is handled by T1.GetResources, T2.DescribeResource, and T3.GetAppYAML. Service interaction and topology use T4.GetServiceDependencies and T6.CheckServiceConnectivity. Telemetry analysis uses T5.GetRecentLogs, T8.GetAlerts, and T9.GetErrorLogs. Infrastructure diagnostics use T7.GetClusterConfiguration and T10.CheckNodeServiceStatus (Wang et al., 28 Feb 2026).

These tools are exposed as structured APIs with Pydantic schemas. The action space is textual but schema-constrained in execution: the agent must generate a valid tool name and valid arguments such as service names and namespaces. Invalid queries produce realistic errors and contribute to robustness metrics. Although the environment is finite because only precomputed queries have entries, the agent does not receive the complete menu and must infer valid interactions from context (Wang et al., 28 Feb 2026).

This design shifts RCA from passive pattern recognition to active perception, knowledge grounding, and deductive reasoning. The agent selects what to inspect, integrates telemetry with documentation-like or configuration-like context, and verifies hypotheses through multi-step evidence gathering rather than one-shot prediction (Wang et al., 28 Feb 2026).

4. Evaluation protocol and empirical behavior

Cloud-OpsBench evaluates both outcome and process. For diagnostic accuracy, the benchmark uses Top-$\mathcal{R}^* = \langle S, C, R \rangle,$6 Accuracy,

$\mathcal{R}^* = \langle S, C, R \rangle,$7

together with Task Completion Rate (TCR), defined as the fraction of episodes in which the agent produces a well-formed diagnosis. Process-based evaluation includes trajectory alignment, tool-usage efficiency, and operational robustness (Wang et al., 28 Feb 2026).

Trajectory alignment is measured in three forms: Exact Match, Any-Order Match, and In-Order Match. Tool-usage efficiency is quantified with Relevance, Coverage, total Steps, and Mean Time To Identify (MTTI). If $\mathcal{R}^* = \langle S, C, R \rangle,$8 is the set of distinct tools used by the agent and $\mathcal{R}^* = \langle S, C, R \rangle,$9 is the set of distinct tools in the gold trajectory, then

$S$0

Operational robustness is assessed with Invalid Action Count (IAC), Redundant Action Rate (RAR), and Zero-Tool Diagnosis Rate (ZTDR) (Wang et al., 28 Feb 2026).

The reported A@1 results for seven models are: DeepSeek-V3.2 at 0.73, GPT-5 at 0.67, GPT-4o at 0.49, Qwen3-235B at 0.50, Claude-4-Sonnet at 0.50, Qwen3-14B at 0.34, and Qwen3-8B at 0.21. DeepSeek-V3.2 records 10.0 steps on average, Coverage 0.88, Relevance 0.43, RAR 0.11, and ZTDR 0. GPT-5 records 5.57 steps, Relevance 0.65, Coverage 0.77, and RAR 0.05. Qwen3-14B records IAC 0.40, Relevance 0.63, and A@1 0.34 (Wang et al., 28 Feb 2026).

Several empirical patterns are emphasized. Exploration depth correlates with accuracy: more steps and higher expert-tool coverage are associated with higher A@1. Any-Order Match exceeds In-Order Match for all models, whereas Exact Match remains low, around 0 to 0.16, indicating that agents often access the correct evidence sources but interleave them with exploratory noise. Explicit faults such as Startup and Runtime cases are easier because they expose clear Kubernetes status and events, whereas cross-layer faults such as Admission and Performance are harder because they require correlating application symptoms with underlying infrastructure or configuration conditions (Wang et al., 28 Feb 2026).

Prompting and knowledge strategies also produce marked shifts. For GPT-4o, A@1 is 0.49 under Base and CoT, 0.70 under ICL, and 0.61 under RAG. For Qwen3-14B, A@1 is 0.34 under Base, 0.44 under CoT, 0.71 under ICL, and 0.50 under RAG. For Qwen3-235B, A@1 is 0.50 under Base, 0.47 under CoT, 0.59 under ICL, and 0.60 under RAG. The benchmark interprets these results as evidence that in-context procedural traces are more effective than CoT alone for agentic RCA (Wang et al., 28 Feb 2026).

5. Data engine, RL environment, and diagnostic standard

Cloud-OpsBench is presented not only as a benchmark but also as enabling infrastructure for next-generation SRE research. As a Data Engine, it harvests high-quality reasoning trajectories that can be used for SFT of small LLMs. As an RL environment, it turns high-risk operations into a safe, low-latency sandbox for policy optimization. As a Diagnostic Standard, it exposes architectural bottlenecks through process-centric evaluation rather than outcome-only scoring (Wang et al., 28 Feb 2026).

The data-engine role rests on the claim that procedural demonstrations are more useful than declarative knowledge for SRE agents. The benchmark logs alert text, trajectories of thought, tool calls, and tool outputs, final diagnosis structures, correctness labels, and metadata such as steps and invalid actions. CrewAI is used as the orchestrator for multi-step tool use, and Langfuse collects structured traces of prompts, outputs, and tool invocations. These logs are directly usable as SFT data or as ICL exemplars (Wang et al., 28 Feb 2026).

The RL interpretation is episodic and finite-horizon. An episode begins with the alert and optional initial context, proceeds through repeated tool invocations and observations, and ends when the agent emits a final diagnosis or exhausts its step or invalid-action budget. The paper does not instantiate a fixed reward equation, but it explicitly proposes a multi-term reward over diagnosis accuracy, action cost, latency, process alignment, and robustness, with negative contributions from Steps, IAC, and RAR, and positive contributions from Tool Relevance, Coverage, and trajectory alignment. This suggests direct compatibility with methods such as PPO or DPO over sequence-level policies (Wang et al., 28 Feb 2026).

As a diagnostic standard, Cloud-OpsBench identifies two named phenomena. The first is Syntactic Fragility (SLMs): Qwen3-14B exhibits high tool relevance but many invalid tool invocations, indicating that schema correctness and tool syntax, rather than raw semantic reasoning, are the main bottlenecks. The second is the Redundancy Paradox (DeepSeek-V3.2 vs GPT-4o): DeepSeek-V3.2 achieves higher A@1 by taking more steps and tolerating more redundancy, whereas GPT-4o is faster and more concise but substantially less accurate. The benchmark interprets this as evidence that extra verification steps can function as a reliability mechanism rather than a defect (Wang et al., 28 Feb 2026).

A further failure mode concerns zero-tool diagnoses. Some models, notably Claude-4-Sonnet, display a high ZTDR and speculate from parametric priors rather than grounded inspection. Within the benchmark’s process-centric framing, this behavior is unacceptable in real SRE contexts because it bypasses evidential verification (Wang et al., 28 Feb 2026).

6. Reproducibility, implementation, and limitations

Cloud-OpsBench’s reproducibility claim depends on a concrete generation pipeline. Fault knowledge is assembled from Kubernetes documentation and tutorials, Stack Overflow, KodeKloud notes, and academic papers on Kubernetes reliability, then converted by Gemini 3 Pro into a structured model $S$1 of prerequisites, fault artifact, and activation sequence. Human review and LLM-as-a-judge are used to validate semantics and reproducibility. Automatic fault-case generation then proceeds through a Generator Agent, Executor Agent, and Verifier Agent. Once a case is verified, all diagnostic tools are run in a parameter sweep; on average, approximately 487 distinct tool invocations are stored per case in JSON form (Wang et al., 28 Feb 2026).

This precomputation is central to determinism. Any reasonable query is either present in the snapshot or yields a realistic not-found error. Benchmark execution therefore requires only enough CPU or GPU for model inference, not a live cluster. Code and dataset are provided at https://github.com/LLM4Ops/Cloud-OpsBench, including scripts for replaying cases, running agents, collecting traces, and accessing state snapshots and fault definitions (Wang et al., 28 Feb 2026).

The benchmark nonetheless has explicit scope limitations. It is focused on Kubernetes microservices; it does not model monoliths, broader data platforms, or vendor-specific APM systems. Because state does not evolve during an episode, it does not evaluate remediation, control policies, autoscaling, or self-healing dynamics. Its observability stack is deliberately vanilla, relying on Prometheus, Istio, and Kubernetes standard APIs. Extending it also requires familiarity with Python and LLM tooling (Wang et al., 28 Feb 2026).

Within the wider cloud benchmarking landscape, this places Cloud-OpsBench in a distinctive position. Reusable cloud datasets can characterize heterogeneous compute services across workload classes (Duggi et al., 10 Jan 2025), and continuous ecosystems can store and curate benchmark results for application-specific comparisons over time (Mohammadi et al., 2018). Cloud-OpsBench instead concentrates on deterministic, tool-centric digital twins with white-box process evaluation for RCA. The paper’s formulation suggests a broader implication: cloud-operations benchmarks need not choose between realism and reproducibility if operational interaction is reconstructed from verified snapshots rather than replayed exclusively in live infrastructure (Wang et al., 28 Feb 2026).