SREGym: A Live Benchmark for AI SRE Agents with High-Fidelity Failure Scenarios

Abstract: AI agents are increasingly used to diagnose and mitigate failures in production systems, known as agentic Site Reliability Engineering (SRE). Current SRE benchmarks are limited to oversimplistic SRE tasks and are unfortunately hard to extend due to bespoke designs. We present SREGym, a high-fidelity benchmark for SRE agents. SREGym exposes a live system environment built atop real-world cloud-native system stacks, where high-fidelity failure scenarios are simulated through fault injectors. SREGym models the complexity of production environments by simulating (1) a wide range of faults at different layers, (2) various ambient noises, and (3) diverse failure modes such as metastable failures and correlated failures. SREGym is architected as a modular, extensible framework that orchestrates fault and noise injectors across stacks. SREGym currently includes 90 realistic, challenging SRE problems. We use SREGym to evaluate frontier agents and show that their capabilities varies significantly in addressing different kinds of failures, with up to 40% differences in end-to-end results. SREGym is actively maintained as an open-source project and has been used by researchers and practitioners.

• The paper introduces SREGym, a live benchmark that simulates realistic multi-layer failures and ambient noise for evaluating AI SRE agents.
• It employs a four-tuple scenario design, combining system environment, agent interface, faults/noises, and oracles to ensure rigorous diagnosis and mitigation evaluation.
• Evaluation results highlight significant performance discrepancies across agents, emphasizing challenges in cross-layer reasoning and tool usage efficiency.

SREGym: A Live Benchmark for AI SRE Agents with High-Fidelity Failure Scenarios

Motivation and Limitations of Existing Benchmarks

The proliferation of AI-generated code in production environments has introduced new reliability challenges, with AI-authored changes frequently responsible for defects and outages. Site Reliability Engineering (SRE) tasks are distinguished from general Software Engineering by their requirement for multi-modal reasoning across observability data (metrics, logs, traces), domain-specific tool use, and operational interaction across different layers (application, OS, hardware). Prior benchmarks for agentic SRE fail to accurately simulate the complexity of real-world failures: most focus exclusively on application-layer issues, lack ambient noise, do not cover multi-layer faults, and are restricted by hardcoded, non-extensible scenario design.

SREGym Architecture and Features

SREGym addresses these deficiencies with a modular, extensible framework for evaluating AI agents against high-fidelity SRE scenarios. The benchmark orchestrates fault and noise injection across distributed system stacks, simulating diverse failure modes—including correlated failures, metastable behavior, and faults at hardware, OS, and application layers.

Figure 1: Overview of the SREGym framework and benchmark suites.

A scenario in SREGym is parameterized as a four-tuple: (i) the system environment ($\mathcal{E}$); (ii) agent interface ($\mathcal{I}$), exposing standardized observability and control APIs; (iii) faults/noises ($\mathcal{F}$); and (iv) oracles ($\mathcal{O}$), programmatic and checklist-based for diagnosis and mitigation evaluation. Environment deployment leverages real-world cloud-native stacks (Kubernetes, Helm, Docker) and allows for injection of realistic faults via mechanisms such as eBPF, hardware stressors, misconfiguration, and operator misoperation.

Figure 2: A failure scenario where a microservice misses an environment variable it needs to send requests to.

Noise injectors create transient, ambiguous disturbances, forcing the agents to disambiguate root causes from unrelated events. Failure modes extend to metastable collapses—persistent degraded states induced by temporally coordinated triggers and constraints (e.g., retry storms and resource quotas)—and compound or correlated failures cascading across multiple services.

Figure 3: Metastable failures resulting from persistent self-sustaining degradations.

Figure 4: Bad sectors in hard disk drives, exemplifying hardware fault injection.

Evaluation Protocol and Oracle Design

Diagnosis evaluation is structured via a checklist-based rubric administered by LLMs, decomposed into dimensions of fault localization, characterization, and scope precision. The mitigation oracle employs state-based verification, checking both system health and the explicit resolution of injected faults, thus avoiding reward hacking/exploitability common in chaos-engineering-based benchmarks. Empirical validation demonstrates high inter-evaluator agreement ($\kappa=0.90$ human vs. LLM), supporting reliability of the protocol.

Benchmarking Frontier Agents and Results

SREGym was used to evaluate three agent architectures: Stratus (multi-agent SRE-centric), Claude Code, and Codex (general-purpose coding agents), each paired with frontier LLMs (Claude Sonnet-4.6, Kimi K2.5, GPT-5.4).

• Diagnosis success rates: 38.9–72.6%
• Mitigation success rates: 57.3–78.5%
• End-to-end success rates (correct diagnosis and mitigation): 30.4–60.7%
• Token consumption: Claude Code and Codex require 3–3.6x more tokens per run than Stratus due to less efficient data preprocessing.

Results indicate that agent performance varies drastically across failure modes. Application-layer failures imported from legacy benchmarks are readily solved (>80% mitigation success), whereas high-fidelity, compound, or hardware/OS-level failures unique to SREGym reduce end-to-end success rates by over 40% across all agents. Agents consistently exhibit a greedy anomaly-targeting behavior and lack comprehensive cross-layer reasoning, failing to diagnose induced hardware faults (Figure 4) or metastable collapses (Figure 3).

Impact of Ambient Noise and Multi-Fault Composition

Ambient noise simulation consistently reduces diagnosis rates for all agents, confirming that distractor events in production significantly challenge agent robustness. When diagnosis is incorrect, agents often succeed in mitigation via repeated hypothesis formation and system tuning but incur higher tool invocation counts and longer time-to-mitigate. Conditional probability analysis establishes that correct diagnosis is strongly correlated with successful mitigation; nonetheless, approximately 35–59% mitigation occurs even after an incorrect diagnosis, reflecting the utility of iterative operational strategies.

Figure 5: Conditional mitigation probabilities with diagnosis outcomes and average tool usage by problem category.

Analysis of Tool Usage and Operational Strategies

Agents rely heavily on cluster-control commands (kubectl get/logs/describe), with Stratus leveraging specialized instruments for metric and trace analysis, contrasting with coding agents' inefficient raw data ingestion. Tool usage patterns reveal that most agents require substantial read operations (19–28) prior to the first mutation, and mitigation actions differ—with Stratus favoring patch/set commands, Claude Code and Codex employing rollout/run/delete.

Practical and Theoretical Implications

SREGym exposes critical deficiencies in contemporary AI SRE agents: inability to generalize reasoning across system stack layers, anchoring on superficial symptoms, and limited robustness to production noise. The combinatorial scope of SREGym—fault/noise injectors and multi-service environments—positions it as a foundation for reinforcement learning-based agentic SRE training and evaluation. Practically, empirical evidence from SREGym should inform safety and deployment policy for commercial AI SRE solutions, ensuring they are evaluated on production-relevant incident spectra.

Conclusion

SREGym constitutes a step-change benchmark for Site Reliability Engineering, delivering high-fidelity, composable scenarios covering realistic multi-layer failures, noise, and compound events. Results demonstrate substantial gaps in agent reasoning and mitigation capability. The modular architecture and rigorous oracles enable extensibility and reproducibility, while combinatorial task diversity suggests a clear path for future RL-style agent training. Benchmark adoption and further agent development are required for production-readiness in autonomous SRE tooling.