---
title: 'SREGym: Live SRE Benchmark'
url: https://www.emergentmind.com/topics/sregym
type: topic
---

# SREGym: Live SRE Benchmark

SREGym is a live benchmark for evaluating AI agents on site reliability engineering tasks in realistic cloud environments. It is designed for **agentic SRE**, in which an agent investigates a running system, infers the root cause of failures from live observability data, and attempts mitigations whose correctness is only visible at runtime. The benchmark is built atop real-world cloud-native system stacks, simulates high-fidelity failure scenarios through fault injectors, and currently includes **90 realistic, challenging SRE problems** [2605.07161].

## 1. Definition and formal problem model

SREGym is defined around a live system rather than a static dataset. A benchmark problem is formalized as
\[
P = (\mathcal{E}, \mathcal{I}, \mathcal{F}, \mathcal{O}),
\]
where \(\mathcal{E}\) is the live system environment, \(\mathcal{I}\) is the agent interface, \(\mathcal{F}\) is a set of injected faults and noises, and \(\mathcal{O} = (\mathcal{O}_d, \mathcal{O}_m)\) are diagnosis and mitigation oracles [2605.07161]. In this formulation, the agent observes \(\mathcal{E}\) through \(\mathcal{I}\), while \(\mathcal{F}\) creates the failure scenario; the agent must submit a natural-language diagnosis and a mitigation completion signal.

The benchmark is motivated by the claim that production operations differ fundamentally from coding or static software engineering benchmarks. An SRE agent must reason over configuration, metrics, logs, and traces; use operational tools such as Kubernetes control-plane commands; and execute interventions whose consequences unfold dynamically in a live environment [2605.07161]. This distinguishes SREGym from static QA datasets, anomaly-detection datasets, and root-cause analysis corpora that evaluate analysis over fixed traces or telemetry but do not test whether systems can actually be recovered.

A recurrent misconception is that any benchmark with incident descriptions or observability data suffices for SRE evaluation. SREGym explicitly rejects that equivalence. It is not a static knowledge benchmark, and it is not primarily a trace-classification benchmark. Its object of evaluation is end-to-end diagnosis and mitigation in a live system, with runtime state transitions and oracle-based verification [2605.07161].

## 2. Environment architecture and agent interface

SREGym uses a **cloud-native and Kubernetes-based** environment in which applications are deployed in **Docker containers** on Kubernetes [2605.07161]. The supported software stack includes applications from **DeathStarBench**, **Train Ticket**, and **Astronomy Shop / OpenTelemetry demo**, together with in-house applications including a **satellite orbit simulator** and a **flight booking service**. Backend systems include **MongoDB**, **TiDB**, **Kafka**, and **MySQL**; applications, backend systems, and management services are handled by **Kubernetes operators**, and deployment uses **Helm** [2605.07161].

Observability is provided through **Prometheus** for time-series metrics, **Loki** for logs, and **Jaeger** for distributed traces [2605.07161]. Kubernetes state and control are available through `kubectl` and Kubernetes APIs. This environment is intended to approximate the “de facto” cloud-native production stack rather than a simplified toy deployment.

SREGym deliberately avoids binding evaluation to a single agent architecture. It exposes the system through **Model Context Protocol (MCP) servers**, providing metrics access via Prometheus, logs via Loki, traces via Jaeger, cluster control via `kubectl`, and submission interfaces for diagnosis and mitigation results [2605.07161]. It also exposes raw observability endpoints and Kubernetes API endpoints for custom tools. This separation between benchmark environment and agent implementation is central to its extensibility.

The framework is modular. Problems are authored in Python by combining application deployment logic, oracle definitions, and fault injection logic; noise injection is handled separately by the benchmark runtime [2605.07161]. The current framework provides **90 curated problems**, **50 fault primitives**, **139 deployable services** across **5 applications**, and **3,623 viable fault-component pairs** before accounting for noise and multi-fault composition [2605.07161]. The paper emphasizes that the 90 problems are only a validated subset of a larger compositional space.

## 3. Failure modeling, ambient noise, and realism claims

SREGym’s realism is organized around three dimensions: **fault diversity across the stack**, **ambient noise**, and **complex failure modes** [2605.07161]. Fault injectors create underlying root-cause defects rather than merely externally visible symptoms. The mechanisms include killing a process or pod, stressing hardware, failing syscalls via eBPF, corrupting a disk sector via `dm-dust`, mutating `deploy.yaml`, mutating application or Kubernetes configuration, using buggy application code, using buggy operators, increasing client loads, and injecting network delay or packet drops [2605.07161].

The benchmark spans multiple layers: application, Kubernetes/platform, OS/kernel, hardware/storage, network, and resource/performance [2605.07161]. This cross-layer coverage is a major distinction from earlier live SRE benchmarks that the paper characterizes as concentrating on application-layer issues.

Noise is defined as transient, self-recovering disturbances that are not the root cause of the target incident. Examples include pausing or restarting unrelated pods, transient packet loss or latency, and resource stress on nodes to emulate noisy neighbors [2605.07161]. In the evaluation setup, SREGym injects **two noise patterns every five minutes**, each lasting **two minutes** [2605.07161]. The paper’s argument is that real production systems are noisy and eventful, and that clean environments make diagnosis unrealistically easy.

The benchmark also includes named higher-level failure modes. **Metastable failures** are self-sustaining degraded states that persist after the original trigger disappears; the paper’s example combines aggressive gRPC timeout/retry settings, a high-load state at **3000 requests/second**, and transient CPU stress to trigger synchronized timeouts and retries [2605.07161]. **Concurrent failures** involve multiple independent failures occurring simultaneously, while **correlated failures** involve multiple components failing together because they share a dependency or common cause [2605.07161]. This suggests that SREGym is not only a fault-injection framework but also a structured benchmark of interaction effects across layers and services.

## 4. Scenario suite and benchmark tasks

SREGym currently includes **90 SRE problems** divided into three coarse groups used in evaluation: **34 Ported** problems directly ported from AIOpsLab/ITBench, **43 Similar Failures** that share failure patterns with prior work but use different applications or components, and **13 New Failures** unique to SREGym, especially low-level or compound failures [2605.07161]. The benchmark does not attach fixed easy/medium/hard labels to every scenario, but it does vary incident structure along multiple axes.

These axes include fault category, affected component, stack layer, failure mode, observability profile, and remediation expectation [2605.07161]. Scenarios include missing environment variables in frontend services, network port misconfiguration in the Social Network application, an observability service made unschedulable by scheduler misconfiguration, latent disk sector errors causing intermittent `pread()` failures and MongoDB crashes, aggressive retry settings plus CPU stress causing metastable retry storms, invalid Valkey passwords causing authentication disruption, image-pull misconfiguration for a TiDB operator, and noisy-neighbor resource stress [2605.07161].

An SRE agent is expected to perform an end-to-end incident workflow: investigate using metrics, logs, traces, and cluster state; diagnose the root cause in natural language; mitigate by executing commands that change system state; verify recovery; and submit diagnosis and mitigation completion for oracle evaluation [2605.07161]. The benchmark explicitly rejects decomposing evaluation into isolated subtasks such as detection, localization, and mitigation scored independently.

The scenario suite is intended to challenge common shortcut strategies. In particular, the benchmark differentiates root causes from downstream victims, target faults from unrelated noise, and genuine mitigation from superficial alert suppression or restarts [2605.07161]. A plausible implication is that SREGym operationalizes SRE competence as intervention quality under ambiguity rather than as observability interpretation in isolation.

## 5. Evaluation methodology and reported results

SREGym evaluates agents on **diagnosis success rate**, **mitigation success rate**, **end-to-end (E2E) success rate**, **TTD** (Time-To-Diagnose), **TTM** (Time-To-Mitigate), and **mean token usage per run** [2605.07161]. Diagnosis is scored by a checklist-based LLM judge. Given ground-truth root cause description \(g\) and agent diagnosis \(d\), the diagnosis oracle constructs a prompt with \(g\), \(d\), and \(N=9\) yes/no questions grouped into \(K=3\) dimensions. For dimension \(k\),
\[
s_k = \frac{1}{|Q_k|}\sum_{q\in Q_k} y_q,
\]
and the aggregated score is
\[
S = \sum_{k=1}^{K} w_k\, s_k,
\]
with equal weights
\[
w_k=\frac{1}{3}.
\]
The final verdict is
\[
\hat{v}=\mathbb{1}[S \ge \tau],
\]
with default threshold
\[
\tau=\frac{7}{9}.
\]
The three diagnosis dimensions are **Fault localization**, **Fault characterization**, and **Failure scope** [2605.07161].

The diagnosis oracle was validated on a stratified sample of **100** agent diagnoses, achieving **0.95 raw agreement** and **Cohen’s \(\kappa = 0.90\)** with a human expert for the default Sonnet-4.6 judge [2605.07161]. Mitigation is scored by a problem-specific oracle that checks both whether the target fault is resolved and whether the system has recovered to a healthy state, using client-side observability and system-side state [2605.07161].

Each agent-model pair is evaluated with **three runs per problem** across the 90 problems [2605.07161]. TTD and TTM are capped at **1800 seconds**, and timed-out runs contribute the cap value to the average [2605.07161]. The paper evaluates **Stratus** with **Claude Sonnet-4.6** and **Kimi K2.5**, **Claude Code** with **Sonnet-4.6**, and **Codex** with **GPT-5.4** [2605.07161].

The main reported overall results are summarized below.

| Agent-model | Condition | Diagnosis / Mitigation / E2E |
|---|---|---|
| Stratus + Sonnet-4.6 | no noise | 61.5% / 78.5% / 54.8% |
| Stratus + Sonnet-4.6 | with noise | 51.5% / 65.5% / 40.2% |
| Claude Code + Sonnet-4.6 | no noise | 72.6% / 75.6% / 60.7% |
| Claude Code + Sonnet-4.6 | with noise | 62.6% / 76.3% / 53.7% |
| Codex + GPT-5.4 | no noise | 70.0% / 65.2% / 53.3% |
| Codex + GPT-5.4 | with noise | 59.3% / 64.0% / 45.9% |

Across these configurations, diagnosis ranges from **38.9% to 72.6%**, mitigation from **57.3% to 78.5%**, and E2E from **30.4% to 60.7%** [2605.07161]. The headline result is that capabilities vary strongly across failure types, with **up to 40% differences in end-to-end results** [2605.07161]. For example, with Stratus + Sonnet-4.6 and no noise, **Ported** scenarios achieve **63.7%** E2E success, whereas **New Failures** achieve **17.9%** [2605.07161].

The benchmark also reports conditional mitigation success. For Stratus + Sonnet-4.6 without noise,
\[
P(\text{M} \mid \text{D}) = 0.880,\qquad P(\text{M} \mid \neg \text{D}) = 0.588,
\]
while for Claude Code,
\[
P(\text{M} \mid \text{D}) = 0.798,\qquad P(\text{M} \mid \neg \text{D}) = 0.500,
\]
and for Codex,
\[
P(\text{M} \mid \text{D}) = 0.734,\qquad P(\text{M} \mid \neg \text{D}) = 0.435
\]
[2605.07161]. Diagnosis therefore improves mitigation substantially, but the benchmark also reveals nontrivial rates of successful mitigation despite incorrect diagnosis.

## 6. Failure patterns, limitations, and future directions

The paper identifies several characteristic weaknesses of current agents on SREGym. They are relatively strong on application-layer issues, simpler single-fault scenarios, and scenarios similar to previously published live benchmarks, but substantially weaker on low-level hardware faults, OS/kernel-rooted problems, compound failures, metastable failures, correlated failures, and noisy environments [2605.07161]. In the `latent_sector_error` problem, no run of Stratus, Claude Code, or Codex without noise produced a diagnosis score above **0.22**, and fault characterization scored **0** in every run [2605.07161]. In metastable incidents, no agent identified both the application trigger and the infrastructure constraint together [2605.07161].

Tool-use analysis also exposes structural limitations. **60%–72%** of tool calls are read-only `kubectl` commands, about **87%** of all read operations are `kubectl get` and `kubectl logs`, and agents issue around **19–28** read commands before their first write [2605.07161]. The paper describes a recurring **greedy diagnosis strategy** in which agents latch onto the first plausible anomaly and stop exploring alternatives [2605.07161]. This suggests that current frontier agents are often better at local anomaly explanation than at systematic hypothesis management under live, noisy conditions.

The benchmark itself has stated limitations. Diagnosis relies on an LLM judge, which introduces oracle variance even though agreement with humans is strong [2605.07161]. Current noise modeling does not yet include high-variance traffic anomalies, partial network partitions, or slow degradation from gradual exhaustion [2605.07161]. System scale is modest relative to hyperscale production systems: the largest application is **Train Ticket with 40 microservices** [2605.07161]. The environment scope is limited to cloud-native Kubernetes deployments and does not cover monolithic systems or edge deployments [2605.07161]. The reported evaluation covers only three agents and a small set of frontier models [2605.07161].

Future work includes richer noise models, more diverse fault simulation and failure modes, new system environments including edge presence, and turning SREGym into an **RL-style training ground** beyond a fixed benchmark suite [2605.07161]. The framework is **open source**, actively maintained at **https://github.com/SREGym/SREGym**, and has been used by researchers and practitioners [2605.07161]. This suggests a dual role for SREGym: it is both a benchmark for current agentic SRE systems and an extensible infrastructure for studying how diagnosis, control, and recovery strategies behave in live cloud environments.

Source: https://www.emergentmind.com/topics/sregym