DevOps-Gym: End-to-End DevOps Benchmark
- DevOps-Gym is a benchmark that evaluates AI agents in dynamic DevOps environments, spanning build, monitoring, issue resolution, and test generation.
- It uses over 700 tasks from 30+ Java and Go projects, leveraging real repositories, executable environments, and sequential command interactions.
- Empirical results reveal challenges in multi-step reasoning and domain-specific automation, highlighting gaps in current LLM capabilities for end-to-end DevOps workflows.
DevOps-Gym is introduced as, to the authors’ knowledge, the first benchmark explicitly designed to evaluate AI agents across an end-to-end slice of the software DevOps lifecycle on real repositories, with executable environments and tool-using interfaces rather than isolated static coding prompts. It targets four core workflows—build and configuration, monitoring, issue resolving, and test generation—and reports 700+ real-world tasks collected from 30+ projects in Java and Go. Its stated purpose is to measure whether code-capable agents can handle the broader DevOps cycle, where success depends on analyzing large repositories, understanding dynamic program behaviors, leveraging domain-specific tools, and making sequential decisions (Tang et al., 27 Jan 2026).
1. Benchmark concept and lifecycle scope
The paper defines DevOps as the end-to-end process of developing, delivering, deploying, and managing software projects, then narrows that broad space to four “critical stages” that together form what it calls the “minimum viable pipeline”: build and configuration, monitoring, issue resolving, and test generation, plus a smaller set of end-to-end pipeline tasks that chain these stages sequentially. The benchmark is motivated by two gaps. First, there is a lifecycle gap: existing benchmarks such as HumanEval, SWE-bench, and SWT-bench focus primarily on static generation or bug fixing, while broader operational workflows such as build repair, deployment-oriented configuration, and runtime monitoring are underrepresented. Second, there is an agent gap: prior software benchmarks emphasize reasoning or generation quality more than autonomous tool use over dynamic execution environments (Tang et al., 27 Jan 2026).
Within this framing, DevOps-Gym is deliberately agentic. It is built around terminal tools, live environments, command execution, and nontrivial multi-step trajectories. The benchmark therefore treats DevOps automation not as one-shot text generation, but as sequential interaction with repositories, build systems, test harnesses, and runtime observability surfaces. This suggests a conception of DevOps benchmarking in which the decisive variable is not only whether an agent can propose code, but whether it can operate under the same execution constraints that structure real software delivery.
2. Task families and benchmark semantics
The four task families are differentiated by input modality, tool access, and executable success criteria. In all cases, grading is operational rather than purely textual.
| Task family | Benchmark semantics | Success criterion |
|---|---|---|
| Build and configuration | Repair failing build setups or create/transform build configurations | Successful build execution; dedicated tests for implementation tasks |
| Monitoring | Diagnose anomalies in a running application with terminal monitoring tools | Diagnosis file exists and anomaly label matches ground truth |
| Issue resolving | Fix a buggy repository from a natural-language bug description | Fail-to-pass transitions and no new failures |
| Test generation | Produce regression tests from the buggy repository and bug description | At least one generated test fails on buggy code and passes on patched code, with no spurious failures on the patched code |
| End-to-end pipeline | Repair build configuration, monitor the running system, fix the issue, then generate tests | Full four-stage completion; failure at any stage terminates the pipeline |
For build and configuration, the paper separates repair tasks from implementation tasks. Repair tasks target dependency version conflicts, build misconfiguration, compilation errors, tool-chain mismatches, and dependency resource unavailability. Implementation tasks ask the agent to create or transform build configurations for scenarios such as build system migration, target release setup, plugin integration, dependency version upgrades, and configuration initialization. For monitoring, the agent is given a running application inside a container and terminal access to tools such as top, free, ps, and netstat, but no access to source code, config files, or trigger scripts. The monitored anomaly classes include memory leaks, disk leaks or exhaustion, file or socket handle exhaustion, CPU spikes, I/O bottlenecks, and inefficient SQL query handling; healthy-system cases are also included to test false-positive control (Tang et al., 27 Jan 2026).
Issue resolving follows a SWE-bench-style setup, but in Java and Go. The agent receives a buggy repository, a natural-language bug description, and development and test tools, and must output a unified diff patch. Test generation is stricter than coverage-based evaluation: a generated test succeeds only if it fails on the buggy code, passes on the patched code, and does not introduce spurious failures on the patched version. The end-to-end tasks convert these isolated capabilities into a cascade: repair injected build configuration errors, deploy and monitor a running system to detect a latent anomaly, fix the underlying issue in code, then rebuild and generate regression tests. The paper characterizes these pipeline tasks as intentionally all-or-nothing (Tang et al., 27 Jan 2026).
3. Construction pipeline, reproducibility, and contamination control
DevOps-Gym is described as labor-intensive and semi-automated rather than automatically mined. Project selection begins with Java and Go repositories, chosen because these languages are representative of enterprise software engineering and come with standardized but nontrivial build systems, richer dependency structures, and mature monitoring ecosystems. The authors manually analyzed more than 1,000 GitHub issues from repositories cited in DevOps technical reports, especially the 2024 DORA State of DevOps report, to identify representative issue types for monitoring and build and configuration. Build repair tasks follow BugSwarm-style mining of recent CI failure-success pairs from logs, then filter to configuration-level fixes. Build implementation tasks are synthesized by three domain experts. Monitoring tasks combine real closed GitHub issues with expert-injected synthetic anomalies. Issue-resolving and test-generation tasks adapt the Multi-SWE-bench and SWT-bench collection pipelines (Tang et al., 27 Jan 2026).
The paper emphasizes the cost of reproducibility. One expert software engineering researcher often needed more than 10 hours to fully reproduce and validate a single monitoring or build issue because issue reports often omit critical environment, dependency, configuration, or triggering-input details. Around 40% of initially selected build issues required multiple iterations to reach consistent reproducibility. For synthetic monitoring anomalies, three senior DevOps engineers independently validated that each injected problem was reliably detectable using the provided monitoring tools and without source access.
Contamination prevention is formalized. For each candidate repository, the authors sample 20 code snippets from CI/CD configurations, build files, and test files. For each snippet, they choose a random start point, extract a 50-token prefix, and prompt a model to continue it. Let denote the model’s predicted 50-token continuation and the ground-truth suffix. They compute five similarity measures: normalized Levenshtein distance ratio , consecutive matching token count from the prefix , position of the first token mismatch , longest common substring length in tokens , and sentence-level BLEU score . A snippet is classified as high-risk if any of the following holds: The repository-level contamination rate is the proportion of high-risk snippets among the 20 samples, and repositories with are excluded. Git metadata is also removed from all repositories so agents cannot inspect future commits, branches, or solution-bearing history (Tang et al., 27 Jan 2026).
4. Execution environment and agent interface
All tasks run in isolated, reproducible Docker containers, and the benchmark is converted into Terminal-Bench format, giving agents a standardized terminal-oriented interface. The concrete tool surface depends on task type. Build and configuration tasks expose repositories plus terminal access with tools and utilities including Maven, Gradle, npm, text utilities, and package managers. Monitoring tasks place the agent in a containerized environment with an already running application containing an anomaly; the agent has terminal access to standard system-observation tools such as top, free, ps, netstat, watch, curl, and ls, but not to source code, configuration files, or trigger scripts. Issue-resolving tasks expose common development shell tools such as git, grep, sed, and awk, plus language-specific test runners. Test-generation tasks include the buggy repository, bug description, file manipulation tools such as touch, echo, and cat, and the relevant test frameworks (Tang et al., 27 Jan 2026).
Executable grading is central. Build tasks use successful build execution plus tests validating artifact functionality. Issue resolving and test generation are evaluated entirely through fail-to-pass behavior. Monitoring uses pytest scripts that check for the diagnosis file and label correctness. End-to-end tasks require completion of all four stages sequentially.
The paper also specifies the evaluation protocol. Models are run with temperature 0, top-p 1, and a maximum context length of 256K tokens. Qwen3-Coder-30B is hosted with vLLM locally; other models are accessed through official APIs. Inference has a 60-second timeout with up to 3 retry attempts. The benchmark does not provide a formal benchmark-wide action-observation-loop equation; instead, stage-wise success rate, pass rate, and monitoring accuracy serve as the aggregate metrics.
5. Empirical results and diagnostic findings
The reported evaluation spans OpenHands, mini-SWE-Agent, Aider, and Claude Code, with multiple backbone models. The main results are stark: even the strongest systems remain weak on monitoring, issue resolving in Java and Go, and especially test generation (Tang et al., 27 Jan 2026).
| Agent | Build/Config | Monitoring | Issue Resolving | Test Generation |
|---|---|---|---|---|
| OpenHands + Qwen3-Coder-30B | 20.37% | 5.89% | 13.22% | 6.13% |
| OpenHands + o4-mini | 24.07% | 8.82% | 10.32% | 8.70% |
| OpenHands + DeepSeek-V3.1 | 11.11% | 0.00% | 14.20% | 3.22% |
| OpenHands + Gemini-2.5-Pro | 16.66% | 11.76% | 10.96% | 2.90% |
| OpenHands + Claude-4-Sonnet | 42.59% | 14.70% | 23.87% | 11.61% |
| mini-SWE-Agent + Claude-4-Sonnet | 29.62% | 2.91% | 5.16% | 0.98% |
| Aider + Claude-4-Sonnet | 5.55% | 0.00% | 9.67% | 2.25% |
| Claude Code + Claude-4-Sonnet | 51.85% | 20.56% | 23.87% | 13.87% |
Claude Code with Claude-4-Sonnet is the best overall system, but even that result leaves half of build and configuration tasks unsolved and the large majority of monitoring, issue-resolving, and test-generation tasks unsolved. Monitoring is especially severe: the top result is only 20.56%, and several settings are at or near 0%. End-to-end tasks are harsher still: full four-stage success is 0% for all tested agents.
The paper uses these results to argue that build and configuration and monitoring are especially challenging because they are out-of-distribution relative to current coding-focused LLM training and benchmark exposure. The appendix’s build implementation table makes this more precise. Across implementation subtasks there are 8 migration tasks, 5 target release tasks, 2 plugin integration tasks, and 3 version upgrade tasks, and even the best agents do very poorly on them. The issue-resolving results also expose a cross-benchmark gap: OpenHands + Claude-4-Sonnet achieves 70.4% on SWE-bench-Verified, yet only 23.87% on DevOps-Gym’s Java and Go issue-resolving tasks.
The error analysis is unusually explicit. For build and configuration, the authors identify three main failure categories after examining run logs: toolchain and environment instrumentation limitations (33%), multi-step reasoning and sequential planning failures (23%), and domain-specific knowledge gaps (37%). They additionally state that 17% of all failures fall into an “inherently difficult” category. For monitoring, the common failure types are inadequate monitoring methodology (37%), premature conclusion (26%), insufficient temporal granularity (11%), and interpretation failure (26%). A small stability study over five independent runs on a random subset of 50 tasks reports 16%, 16%, 20%, 20%, and 22% for Claude Code + Claude-4-Sonnet, yielding a mean of 18.8%, standard deviation 2.40, and Pass@5 of 26.0%; OpenHands + o4-mini scores 14%, 16%, 18%, 20%, and 16%, for a mean of 16.8%, standard deviation 2.04, and Pass@5 of 20.0% (Tang et al., 27 Jan 2026).
6. Boundaries, internal tensions, and position among gym-style systems
The benchmark is intentionally not the full DevOps universe. The paper explicitly omits some important DevOps areas such as CI/CD automation and infrastructure management because those depend on mutable external systems like cloud APIs and Kubernetes clusters, making reproducible benchmark construction and evaluation difficult. Monitoring tasks are constrained to anomaly classification rather than richer root-cause localization or remediation planning. Language coverage is limited to Java and Go. Synthetic task construction, especially for build implementation and monitoring injection, necessarily involves expert judgment (Tang et al., 27 Jan 2026).
The paper also contains a notable internal tension in its reported counts. The abstract says DevOps-Gym includes “700+” real-world tasks. The introduction says “704 tasks collected from 30+ real-world projects in Java and Go” and mentions 14 end-to-end pipeline tasks. The construction overview, however, states 54 build and configuration tasks, 34 monitoring tasks, 310 issue-resolving tasks, 310 test-generation tasks, and 18 end-to-end tasks. Later appendix tables report 48 build tasks and 30 monitoring tasks, and the end-to-end results table contains 14 task names. The paper itself therefore supports only the cautious conclusion that DevOps-Gym contains roughly 700 core tasks over 30+ repositories or projects in Java and Go.
Relative to adjacent “gym” environments, DevOps-Gym occupies a distinct position. CLI-Gym presents 1,655 environment-intensive tasks for terminal-facing repair work in reproducible containers, but the paper explicitly states that it “does not present a full ‘DevOps-Gym’ in the broad sense of CI/CD pipelines, distributed systems, Kubernetes, cloud infrastructure, or observability workflows” (Lin et al., 11 Feb 2026). CybORG, by contrast, is a work-in-progress gym for Autonomous Cyber Operations that pairs simulation and emulation under a common interface; it is relevant as an architectural precedent for gym-style environments with executable backends, but its domain is adversarial cyber operations rather than DevOps workflows (Baillie et al., 2020). This suggests that DevOps-Gym is narrower than the full operational scope implied by “DevOps,” yet substantially closer to realistic DevOps automation than static code benchmarks.
The future directions are correspondingly direct. The authors plan to expand the number of tasks within each stage, cover more error types and more projects, extend DevOps-Gym to more programming languages and more DevOps stages, and encourage community contributions in scenarios, metrics, and agent development (Tang et al., 27 Jan 2026).