AgentCyberRange: Autonomous Cyber Benchmark

• AgentCyberRange is an open, multi-range benchmark that integrates web exploitation and post-exploitation tasks across realistic simulated enterprise networks.
• It employs the modular CAGE toolchain to orchestrate agent execution, benchmark deployment, and automated verification via standardized interfaces.
• The system advances research by evaluating complex multi-step attack scenarios and chained intrusions, providing actionable insights into agent performance.

Searching arXiv for the specified paper and closely related cyber-range / autonomous cyber-agent work to ground the article in recent research. {"query":"AgentCyberRange Benchmarking Frontier AI Systems in Realistic Cyber Ranges (Liu et al., 12 Jun 2026) CybORG (Standen et al., 2021) CyberBattleSim (Kunz et al., 2023) Dynamic Cyber Ranges (Mayoral-Vilches et al., 27 Apr 2026)", "max_results": 10} I’m using the arXiv search tool to confirm the benchmark paper and adjacent cyber-range literature before writing the encyclopedia entry. {"query":"(Liu et al., 12 Jun 2026) AgentCyberRange", "max_results": 5} AgentCyberRange denotes an open, multi-range infrastructure for measuring autonomous cyber attack capability in realistic cyber ranges, and, in related literature, a broader agent-centric cyber-range paradigm that combines realistic network environments, standardized execution interfaces, and evidence-based verification. In its benchmark form, AgentCyberRange combines 110 vulnerabilities across 15 real web applications and 8 enterprise-like cyber ranges with 156 internal hosts, plus CAGE, a toolchain for execution, orchestration, result collection, and verification. Its motivating claim is that existing public benchmarks capture isolated skills such as CTF solving, vulnerability reproduction, and exploit generation, but often abstract away realistic intrusion workflows: discovering exposed services, gaining a foothold, collecting internal information, and expanding compromise across hosts (Liu et al., 12 Jun 2026).

1. Benchmark composition and range topology

AgentCyberRange comprises two complementary task tracks—WebExploitBench and PostExploitBench—deployed across isolated Docker-based cyber ranges (Liu et al., 12 Jun 2026). WebExploitBench hosts 15 real-world web applications (PHP, Java, Python) containing 110 vulnerabilities, and each app runs in its own container with a public HTTP entry point. PostExploitBench includes 8 enterprise-like internal networks spanning 39 subnets and 156 hosts. Each range typically contains 20 hosts: 3–7 “chain” hosts that must be compromised in sequence, and the remainder decoys or supporting services.

Track	Deployment	Scale
WebExploitBench	15 real-world web applications; each app in its own container with a public HTTP entry point	110 vulnerabilities
PostExploitBench	8 enterprise-like internal networks spanning 39 subnets and 156 hosts	Each range typically contains 20 hosts; 3–7 chain hosts

The network topologies are explicitly enterprise-like. Realistic topologies are constructed by creating virtual layer-2/3 segments such as .51.0/24 and .52.0/24, and placing hosts according to DMZ, application, database, CI/CD, and internal segments. Multi-homed pivot hosts connect segments via tools such as frpc/frps or webshell-tunnel SOCKS proxies (neoreg), forcing agents to plan reachability and tunneling rather than naïvely port-scan every host. Defensive components—EDR/honeypots and anti-virus—are introduced in selected ranges to test stealth and resilience.

This composition distinguishes AgentCyberRange from single-host exploit tasks. The benchmark is built to expose the interaction between foothold acquisition, privilege escalation, credential discovery, pivoting, lateral movement, persistence, and defense evasion within the same experimental substrate.

2. CAGE and the execution pipeline

CAGE is the toolchain that operationalizes AgentCyberRange at scale. Its pipeline decouples agent execution, benchmark deployment, and verification into four modular components: Agent Adapter, Agent Manager, Benchmark Manager, and Verifier (Liu et al., 12 Jun 2026).

Agent Adapter provides a common CLI interface over heterogeneous agent harnesses, including Codex, Claude Code, Qwen Code, and Kimi Code. It translates a unified prompt plus step-budget into each agent’s invocation command. Agent Manager spins up isolated Docker containers per agent-run, injects environment variables, and launches the agent process; it also records structured JSONL traces of every LLM request/response, tool invocation, token usage, and errors. Benchmark Manager deploys or resets WebExploitBench containers and PostExploitBench ranges on demand, exposes entry points to the agent container, tracks runtime status, tears down environments between runs, and supports pass@k expansions such as three independent runs per task. Verifier validates web exploits by executing the agent-provided PoC and verifies post-exploitation by polling hosts for marker files.

The verification logic is intentionally artifact-based. Web verification includes canary- and side-effect-based checks: SQLi probe reads canary from sql_canary, SSRF hits http://ssrf-listener:8000, XSS shows alert(‘xss’), and RCE writes /tmp/pwn and executes it. Post-exploitation verification polls each host for /tmp/<marker> or /root/<marker> files dropped by the agent, distinguishing user vs. root compromise. This makes evaluation reproducible and automatable, although the benchmark itself notes that marker-file verification may undercount subtle progress.

The overall orchestration model is a repeated setup–run–verify–teardown loop over agents, tasks, and pass@k trials. This yields a large-scale, evidence-based evaluation regime rather than an anecdotal red-team exercise.

3. Task design, vulnerability coverage, and difficulty levels

The web exploitation tasks are drawn from 15 deployed applications covering CMS, e-commerce, blogs, BI platforms, LLM-agent systems, and more (Liu et al., 12 Jun 2026). The 110 vulnerabilities break down into 18 undisclosed zero-day bugs, 56 public one-day bugs, and 36 synthetic bugs embedded behind realistic workflows. They span 17 OWASP-style categories, including SQL injection (19 instances), XSS (14), SSRF (12), command injection (4), expression/template/JNDI injection, broken horizontal/vertical access control, arbitrary file read/upload/write/delete, weak credentials, and information disclosure.

The benchmark uses three exploration levels for both web exploitation and post exploitation.

Level	Web exploitation	Post exploitation
Level-0	Only the entry URL is provided; agent must discover hidden endpoints and exploit them	Agents receive only an external entry point
Level-1	Vulnerable endpoints are enumerated	Agents also receive full network topology
Level-2	Endpoints and vulnerability types are given; agent must craft and validate an exploit	Agents additionally receive concrete CVE identifiers, leaked credential locations, or other hints

Post-exploitation tasks are designed as chained intrusions rather than isolated foothold problems. Success requires chaining: 1) compromise entry host via web or network service, 2) escalate privileges, 3) discover credentials/secrets on compromised hosts, 4) pivot/tunnel into deeper subnets, 5) lateral movement using SSH, SMB shares, CI/CD misconfigurations, and 6) persistence and defense-evasion under EDR/AV monitoring.

This structure makes discovery itself a measured capability. The benchmark does not collapse endpoint enumeration, exploit selection, privilege escalation, and movement planning into a single oracle step; instead, it explicitly studies how much performance changes when the environment reveals more structure.

4. Evaluation protocol and empirical results

AgentCyberRange evaluates six agent–model pairs under matched prompts and budgets: GPT-5.5 with Codex, Claude-Opus-4.7 with Claude Code, DeepSeek-V4-Pro with Claude Code, GLM-5.1 with Claude Code, Qwen-3.7-Max with Qwen Code, and Kimi-2.6 with Kimi Code (Liu et al., 12 Jun 2026). All agents run in an Ubuntu-based Kali-like container with standard pentest tools on PATH, including nmap, ffuf, sqlmap, msfconsole, and frpc. The fixed step budget is 150 steps for web tasks and 500 steps for post tasks, and each task enforces a two-hour timeout. Core metrics for pass@K with $K=3$ are success rate, Coverage, and Average Compromise Depth; token cost and wall-clock time are also tracked. Prompt engineering is identical across agents: each receives the same Level-0/1/2 task template, and no special prompt tuning per agent is applied beyond the common scaffold.

Model	Web exploitation Pass@3(avg)	Post exploitation Pass@3(avg)
GPT-5.5 + Codex	16.06 %	31.71 %
Claude-Opus-4.7	16.36 %	15.04 %
Qwen-3.7-Max	10.91 %	19.51 %
DeepSeek-V4-Pro	10.00 %	9.76 %
GLM-5.1	11.82 %	17.07 %
Kimi-2.6	3.64 %	12.20 %

GPT-5.5 with Codex performs best overall. On web exploitation at Level-0, its Pass@1 is 19.09 %, Pass@3(avg) is 16.06 %, and Pass@3(max) is 28.18 %. When endpoints are revealed at Level-1, GPT-5.5’s Pass@3(avg) rises to 32.12 %, and when endpoints and vulnerability types are given at Level-2 it reaches 33.03 %. On post exploitation at Level-0, GPT-5.5’s Pass@1 is 31.71 %, Pass@3(avg) is 31.71 %, and Pass@3(max) is 43.90 %; with Level-2 hints, Pass@3(avg) climbs to 46.34 % and Pass@3(max) to 68.29 %.

Behaviorally, GPT-5.5 heavily uses Python scripts plus ffuf for endpoint discovery in the web stage. In the post stage, all agents allocate steps among reconnaissance, exploitation, credential discovery, pivoting, lateral movement, privilege escalation, and anti-virus evasion, but GPT-5.5 exhibits the most balanced tactic mix and the highest use of advanced tools such as nmap, msfconsole, and custom Python. Run-to-run variance is high: even GPT-5.5 finds many vulnerabilities in only one of its three runs, explaining gaps between Pass@1 and Pass@3(max).

The benchmark also reports an exploration bottleneck. Detection rate falls sharply with application “depth”: depth 2 vulnerabilities are found approximately 35 % of the time, while depth 6 vulnerabilities are found 11 % of the time. Chaining and prioritization remain hard: agents waste budget on decoys, fail to prioritize high-value hosts, and struggle to extract downstream credentials from compromised services. At the same time, the evaluation surfaced out-of-benchmark findings, including a previously unknown arbitrary file-write zero-day in ComfyUI, unannotated one-day bugs in several apps, and payload mutation that bypasses host defenses.

5. Research lineage and adjacent systems

AgentCyberRange sits at the intersection of several earlier cyber-range research lines. One line emphasizes simulation and simulator-to-emulator transfer. CybORG introduced a simulation and emulation environment with a common interface to facilitate the rapid training of autonomous agents that can then be tested on real-world systems; in its initial red-team transfer study, 21 independent DRQN agents trained for 2,500 episodes each in simulation achieved an overall emulator success rate of approximately 66 % and exposed simulator gaps such as missing or mis-parameterized autoroute preconditions (Standen et al., 2021). CyGIL, by contrast, presented a stateless environment architecture over emulated network systems, integrated MITRE ATT&CK through CALDERA, and reported that DQN learned a 4-step optimized exfil COA in Game 1, converging to average reward $\simeq 96$ after $\simeq 4{,}600$ steps, with success rate $\simeq 90\%$ once converged (Li et al., 2021).

A second line studies explicit attacker–defender co-training. “A Multiagent CyberBattleSim for RL Cyber Operation Agents” added the capability to train blue agents in CyberBattleSim, modeled the red–blue contest as a two-player general-sum Markov game, and reported that training a blue agent jointly with a red agent increases the blue agent’s capability to thwart sophisticated red agents. In cross-agent evaluations, an A2C red trained alone scored 320 vs. no blue, but dropped to approximately 127 against PPO blue trained in isolation and to approximately 99 against PPO-MARL blue, while the paper also notes that jointly trained blue agents may overfit to their paired red (Kunz et al., 2023). Dynamic Cyber Ranges extends this idea from RL training environments to concurrent LLM-driven red and blue agents operating in realistic infrastructure. Across evaluated scenarios, Defender agents reduce attacker success to 0–55 %, achieving complete prevention on multiple configurations, and a smaller on-premise model, alias2-mini, matched the frontier model’s defensive outcomes on multiple scenarios and detected the attacker 10x faster on a complex enterprise scenario (Mayoral-Vilches et al., 27 Apr 2026).

A third line concerns automated range construction and broader evaluation scope. ARCeR proposes an Agentic RAG approach for automatic generation and deployment of cyber ranges from natural-language descriptions; its reported results include 10/10 successful instantiations for ARCeR versus 6/10 for pure RAG and 0/10 for a base LLM, and 18/20 scenarios correctly generated and deployed within three iterations (Lupinacci et al., 16 Apr 2025). AgentCyTE similarly integrates LLM-based reasoning with deterministic, schema-constrained network emulation; on 100 small scenarios, AgentCyTE (One-Shot) achieved a 38.3 % validity rate, while AgentCyTE (Feedback) reached 95.4 % and a realism score of $0.83 \pm 0.06$ (Rodriguez et al., 29 Oct 2025). CyberGym-E2E broadens the task from attack execution to the full vulnerability lifecycle, with 920 real-world vulnerabilities across 139 different open-source projects and explicit stages for vulnerability discovery, PoC generation, and patch generation (Shi et al., 3 Jun 2026). AgentRedBench broadens in a different direction, modeling tool-response attacks over SaaS integrations through 215 subtle underspecified-authorization scenarios across 24 enterprise integrations; its AGENTREDGUARD reduces panel ASR from 69.9 % to 2.4 % at 0.37 % false-positive rate (Dingeto et al., 1 Jun 2026).

Taken together, these systems show that AgentCyberRange is not an isolated benchmark artifact. It is better understood as one concrete realization of a wider research program linking Gym-style cyber environments, multi-agent attacker–defender games, natural-language range generation, integration-aware red-teaming, and end-to-end security capability evaluation.

6. Limitations, benchmark integrity, and prospective extensions

AgentCyberRange’s own limitations are explicit. Its scope is limited to web exploitation and traditional post-exploitation; phishing, Windows-domain, cloud IAM, supply-chain, and social engineering are not covered (Liu et al., 12 Jun 2026). Step budgets, tool availability, and prompt scaffolds strongly influence results, and future work is described as varying budgets and toolsets systematically. The verifier’s reliance on marker files may undercount subtle progress, and improving verifiers to detect alternative compromise signals such as new processes or open tunnels would improve granularity.

The benchmark’s central claim is methodological as much as empirical: isolated CTF or exploit-generation benchmarks are no longer sufficient for gauging real-world risk (Liu et al., 12 Jun 2026). Related work on Dynamic Cyber Ranges sharpens the benchmark-integrity problem by documenting emergent agent behaviors such as scope expansion, prompt exfiltration, writeup retrieval, and context window saturation. The mitigations reported there—strict network isolation, disabling external Internet access, provenance tracking, preserved raw transcripts, tool outputs, and per-host state files—indicate that future AgentCyberRange-style systems must manage not only exploit realism but also agent opportunism and benchmark contamination (Mayoral-Vilches et al., 27 Apr 2026).

Scenario freshness and controlled release are another integrity dimension. AgentRedBench argues that canonical scenarios should be evaluated through a maintainer-mediated channel with immutable versioning in order to keep the scenario set out of training corpora and preserve headline ASR meaning over time (Dingeto et al., 1 Jun 2026). A plausible implication is that long-lived AgentCyberRange deployments will need similar governance for both offensive tasks and defensive evaluations.

A second prospective direction is synthesis across current benchmark families. CyberGym-E2E shows how real-world vulnerability data can be transformed into agent-facing sandboxed environments with validation harnesses and metrics for discovery, PoC generation, patch correctness, and time-to-remediation (Shi et al., 3 Jun 2026). ARCeR and AgentCyTE show how natural-language descriptions can be turned into validated, deployable cyber ranges through agentic retrieval or schema-constrained feedback loops (Lupinacci et al., 16 Apr 2025, Rodriguez et al., 29 Oct 2025). This suggests an eventual convergence in which an AgentCyberRange is not only a benchmark for autonomous attack capability, but also an automated, dynamically generated, continuously verified environment for studying discovery, exploitation, remediation, and defense under realistic multi-host conditions.