SREGym: Live SRE Benchmark

• SREGym is a live benchmark for agentic SRE that evaluates end-to-end diagnosis and mitigation in realistic cloud environments.
• It uses a modular, Kubernetes-based architecture, combining observability tools and dynamic fault injection to simulate high-fidelity failures.
• The benchmark challenges AI agents with 90 curated SRE problems, revealing insights on performance variations in diagnosis and mitigation strategies.

SREGym is a live benchmark for evaluating AI agents on site reliability engineering tasks in realistic cloud environments. It is designed for agentic SRE, in which an agent investigates a running system, infers the root cause of failures from live observability data, and attempts mitigations whose correctness is only visible at runtime. The benchmark is built atop real-world cloud-native system stacks, simulates high-fidelity failure scenarios through fault injectors, and currently includes 90 realistic, challenging SRE problems (Clark et al., 8 May 2026).

1. Definition and formal problem model

SREGym is defined around a live system rather than a static dataset. A benchmark problem is formalized as

$$P = (\mathcal{E}, \mathcal{I}, \mathcal{F}, \mathcal{O}),$$

where $\mathcal{E}$ is the live system environment, $\mathcal{I}$ is the agent interface, $\mathcal{F}$ is a set of injected faults and noises, and $\mathcal{O} = (\mathcal{O}_d, \mathcal{O}_m)$ are diagnosis and mitigation oracles (Clark et al., 8 May 2026). In this formulation, the agent observes $\mathcal{E}$ through $\mathcal{I}$, while $\mathcal{F}$ creates the failure scenario; the agent must submit a natural-language diagnosis and a mitigation completion signal.

The benchmark is motivated by the claim that production operations differ fundamentally from coding or static software engineering benchmarks. An SRE agent must reason over configuration, metrics, logs, and traces; use operational tools such as Kubernetes control-plane commands; and execute interventions whose consequences unfold dynamically in a live environment (Clark et al., 8 May 2026). This distinguishes SREGym from static QA datasets, anomaly-detection datasets, and root-cause analysis corpora that evaluate analysis over fixed traces or telemetry but do not test whether systems can actually be recovered.

A recurrent misconception is that any benchmark with incident descriptions or observability data suffices for SRE evaluation. SREGym explicitly rejects that equivalence. It is not a static knowledge benchmark, and it is not primarily a trace-classification benchmark. Its object of evaluation is end-to-end diagnosis and mitigation in a live system, with runtime state transitions and oracle-based verification (Clark et al., 8 May 2026).

2. Environment architecture and agent interface

SREGym uses a cloud-native and Kubernetes-based environment in which applications are deployed in Docker containers on Kubernetes (Clark et al., 8 May 2026). The supported software stack includes applications from DeathStarBench, Train Ticket, and Astronomy Shop / OpenTelemetry demo, together with in-house applications including a satellite orbit simulator and a flight booking service. Backend systems include MongoDB, TiDB, Kafka, and MySQL; applications, backend systems, and management services are handled by Kubernetes operators, and deployment uses Helm (Clark et al., 8 May 2026).

Observability is provided through Prometheus for time-series metrics, Loki for logs, and Jaeger for distributed traces (Clark et al., 8 May 2026). Kubernetes state and control are available through kubectl and Kubernetes APIs. This environment is intended to approximate the “de facto” cloud-native production stack rather than a simplified toy deployment.

SREGym deliberately avoids binding evaluation to a single agent architecture. It exposes the system through Model Context Protocol (MCP) servers, providing metrics access via Prometheus, logs via Loki, traces via Jaeger, cluster control via kubectl, and submission interfaces for diagnosis and mitigation results (Clark et al., 8 May 2026). It also exposes raw observability endpoints and Kubernetes API endpoints for custom tools. This separation between benchmark environment and agent implementation is central to its extensibility.

The framework is modular. Problems are authored in Python by combining application deployment logic, oracle definitions, and fault injection logic; noise injection is handled separately by the benchmark runtime (Clark et al., 8 May 2026). The current framework provides 90 curated problems, 50 fault primitives, 139 deployable services across 5 applications, and 3,623 viable fault-component pairs before accounting for noise and multi-fault composition (Clark et al., 8 May 2026). The paper emphasizes that the 90 problems are only a validated subset of a larger compositional space.

3. Failure modeling, ambient noise, and realism claims

SREGym’s realism is organized around three dimensions: fault diversity across the stack, ambient noise, and complex failure modes (Clark et al., 8 May 2026). Fault injectors create underlying root-cause defects rather than merely externally visible symptoms. The mechanisms include killing a process or pod, stressing hardware, failing syscalls via eBPF, corrupting a disk sector via dm-dust, mutating deploy.yaml, mutating application or Kubernetes configuration, using buggy application code, using buggy operators, increasing client loads, and injecting network delay or packet drops (Clark et al., 8 May 2026).

The benchmark spans multiple layers: application, Kubernetes/platform, OS/kernel, hardware/storage, network, and resource/performance (Clark et al., 8 May 2026). This cross-layer coverage is a major distinction from earlier live SRE benchmarks that the paper characterizes as concentrating on application-layer issues.

Noise is defined as transient, self-recovering disturbances that are not the root cause of the target incident. Examples include pausing or restarting unrelated pods, transient packet loss or latency, and resource stress on nodes to emulate noisy neighbors (Clark et al., 8 May 2026). In the evaluation setup, SREGym injects two noise patterns every five minutes, each lasting two minutes (Clark et al., 8 May 2026). The paper’s argument is that real production systems are noisy and eventful, and that clean environments make diagnosis unrealistically easy.

The benchmark also includes named higher-level failure modes. Metastable failures are self-sustaining degraded states that persist after the original trigger disappears; the paper’s example combines aggressive gRPC timeout/retry settings, a high-load state at 3000 requests/second, and transient CPU stress to trigger synchronized timeouts and retries (Clark et al., 8 May 2026). Concurrent failures involve multiple independent failures occurring simultaneously, while correlated failures involve multiple components failing together because they share a dependency or common cause (Clark et al., 8 May 2026). This suggests that SREGym is not only a fault-injection framework but also a structured benchmark of interaction effects across layers and services.

4. Scenario suite and benchmark tasks

SREGym currently includes 90 SRE problems divided into three coarse groups used in evaluation: 34 Ported problems directly ported from AIOpsLab/ITBench, 43 Similar Failures that share failure patterns with prior work but use different applications or components, and 13 New Failures unique to SREGym, especially low-level or compound failures (Clark et al., 8 May 2026). The benchmark does not attach fixed easy/medium/hard labels to every scenario, but it does vary incident structure along multiple axes.

These axes include fault category, affected component, stack layer, failure mode, observability profile, and remediation expectation (Clark et al., 8 May 2026). Scenarios include missing environment variables in frontend services, network port misconfiguration in the Social Network application, an observability service made unschedulable by scheduler misconfiguration, latent disk sector errors causing intermittent pread() failures and MongoDB crashes, aggressive retry settings plus CPU stress causing metastable retry storms, invalid Valkey passwords causing authentication disruption, image-pull misconfiguration for a TiDB operator, and noisy-neighbor resource stress (Clark et al., 8 May 2026).

An SRE agent is expected to perform an end-to-end incident workflow: investigate using metrics, logs, traces, and cluster state; diagnose the root cause in natural language; mitigate by executing commands that change system state; verify recovery; and submit diagnosis and mitigation completion for oracle evaluation (Clark et al., 8 May 2026). The benchmark explicitly rejects decomposing evaluation into isolated subtasks such as detection, localization, and mitigation scored independently.

The scenario suite is intended to challenge common shortcut strategies. In particular, the benchmark differentiates root causes from downstream victims, target faults from unrelated noise, and genuine mitigation from superficial alert suppression or restarts (Clark et al., 8 May 2026). A plausible implication is that SREGym operationalizes SRE competence as intervention quality under ambiguity rather than as observability interpretation in isolation.

5. Evaluation methodology and reported results

SREGym evaluates agents on diagnosis success rate, mitigation success rate, end-to-end (E2E) success rate, TTD (Time-To-Diagnose), TTM (Time-To-Mitigate), and mean token usage per run (Clark et al., 8 May 2026). Diagnosis is scored by a checklist-based LLM judge. Given ground-truth root cause description $g$ and agent diagnosis $d$, the diagnosis oracle constructs a prompt with $\mathcal{E}$0, $\mathcal{E}$1, and $\mathcal{E}$2 yes/no questions grouped into $\mathcal{E}$3 dimensions. For dimension $\mathcal{E}$4,

$\mathcal{E}$5

and the aggregated score is

$\mathcal{E}$6

with equal weights

$\mathcal{E}$7

The final verdict is

$\mathcal{E}$8

with default threshold

$\mathcal{E}$9

The three diagnosis dimensions are Fault localization, Fault characterization, and Failure scope (Clark et al., 8 May 2026).

The diagnosis oracle was validated on a stratified sample of 100 agent diagnoses, achieving 0.95 raw agreement and Cohen’s $\mathcal{I}$0 with a human expert for the default Sonnet-4.6 judge (Clark et al., 8 May 2026). Mitigation is scored by a problem-specific oracle that checks both whether the target fault is resolved and whether the system has recovered to a healthy state, using client-side observability and system-side state (Clark et al., 8 May 2026).

Each agent-model pair is evaluated with three runs per problem across the 90 problems (Clark et al., 8 May 2026). TTD and TTM are capped at 1800 seconds, and timed-out runs contribute the cap value to the average (Clark et al., 8 May 2026). The paper evaluates Stratus with Claude Sonnet-4.6 and Kimi K2.5, Claude Code with Sonnet-4.6, and Codex with GPT-5.4 (Clark et al., 8 May 2026).

The main reported overall results are summarized below.

Agent-model	Condition	Diagnosis / Mitigation / E2E
Stratus + Sonnet-4.6	no noise	61.5% / 78.5% / 54.8%
Stratus + Sonnet-4.6	with noise	51.5% / 65.5% / 40.2%
Claude Code + Sonnet-4.6	no noise	72.6% / 75.6% / 60.7%
Claude Code + Sonnet-4.6	with noise	62.6% / 76.3% / 53.7%
Codex + GPT-5.4	no noise	70.0% / 65.2% / 53.3%
Codex + GPT-5.4	with noise	59.3% / 64.0% / 45.9%

Across these configurations, diagnosis ranges from 38.9% to 72.6%, mitigation from 57.3% to 78.5%, and E2E from 30.4% to 60.7% (Clark et al., 8 May 2026). The headline result is that capabilities vary strongly across failure types, with up to 40% differences in end-to-end results (Clark et al., 8 May 2026). For example, with Stratus + Sonnet-4.6 and no noise, Ported scenarios achieve 63.7% E2E success, whereas New Failures achieve 17.9% (Clark et al., 8 May 2026).

The benchmark also reports conditional mitigation success. For Stratus + Sonnet-4.6 without noise,

$\mathcal{I}$1

while for Claude Code,

$\mathcal{I}$2

and for Codex,

$\mathcal{I}$3

(Clark et al., 8 May 2026). Diagnosis therefore improves mitigation substantially, but the benchmark also reveals nontrivial rates of successful mitigation despite incorrect diagnosis.

6. Failure patterns, limitations, and future directions

The paper identifies several characteristic weaknesses of current agents on SREGym. They are relatively strong on application-layer issues, simpler single-fault scenarios, and scenarios similar to previously published live benchmarks, but substantially weaker on low-level hardware faults, OS/kernel-rooted problems, compound failures, metastable failures, correlated failures, and noisy environments (Clark et al., 8 May 2026). In the latent_sector_error problem, no run of Stratus, Claude Code, or Codex without noise produced a diagnosis score above 0.22, and fault characterization scored 0 in every run (Clark et al., 8 May 2026). In metastable incidents, no agent identified both the application trigger and the infrastructure constraint together (Clark et al., 8 May 2026).

Tool-use analysis also exposes structural limitations. 60%–72% of tool calls are read-only kubectl commands, about 87% of all read operations are kubectl get and kubectl logs, and agents issue around 19–28 read commands before their first write (Clark et al., 8 May 2026). The paper describes a recurring greedy diagnosis strategy in which agents latch onto the first plausible anomaly and stop exploring alternatives (Clark et al., 8 May 2026). This suggests that current frontier agents are often better at local anomaly explanation than at systematic hypothesis management under live, noisy conditions.

The benchmark itself has stated limitations. Diagnosis relies on an LLM judge, which introduces oracle variance even though agreement with humans is strong (Clark et al., 8 May 2026). Current noise modeling does not yet include high-variance traffic anomalies, partial network partitions, or slow degradation from gradual exhaustion (Clark et al., 8 May 2026). System scale is modest relative to hyperscale production systems: the largest application is Train Ticket with 40 microservices (Clark et al., 8 May 2026). The environment scope is limited to cloud-native Kubernetes deployments and does not cover monolithic systems or edge deployments (Clark et al., 8 May 2026). The reported evaluation covers only three agents and a small set of frontier models (Clark et al., 8 May 2026).

Future work includes richer noise models, more diverse fault simulation and failure modes, new system environments including edge presence, and turning SREGym into an RL-style training ground beyond a fixed benchmark suite (Clark et al., 8 May 2026). The framework is open source, actively maintained at https://github.com/SREGym/SREGym, and has been used by researchers and practitioners (Clark et al., 8 May 2026). This suggests a dual role for SREGym: it is both a benchmark for current agentic SRE systems and an extensible infrastructure for studying how diagnosis, control, and recovery strategies behave in live cloud environments.