AgentCyberRange: Benchmarking Frontier AI Systems in Realistic Cyber Ranges

Abstract: Frontier AI systems are increasingly capable of cybersecurity tasks, including codebase inspection, vulnerability detection, and exploitation. However, evaluating their offensive capabilities remains constrained by limited access to open, reproducible, multi-host cyber ranges. Existing public benchmarks capture isolated skills such as CTF solving, vulnerability reproduction, and exploit generation, but often abstract away realistic intrusion workflows: discovering exposed services, gaining a foothold, collecting internal information, and expanding compromise across hosts. This gap makes it difficult to observe emerging risks early, because frontier AI systems are rarely evaluated under realistic attack conditions. We introduce AgentCyberRange, the first open, multi-range infrastructure for measuring autonomous cyber attack capability in realistic cyber ranges. It combines 110 vulnerabilities across 15 real web applications and 8 enterprise-like cyber ranges with 156 internal hosts, plus Cage, a toolchain for execution, orchestration, result collection, and verification. The benchmark covers two core stages: web exploitation, where agents explore exposed applications and validate vulnerabilities, and post exploitation, where agents turn an initial foothold into broader internal compromise. We evaluate six frontier AI systems under matched prompts and budgets. GPT-5.5 with Codex performs best, solving 16.1% of web exploitation tasks and 31.7% of post-exploitation tasks; with more concrete hints, these rates increase to 33.0% and 46.3%. We also observe out-of-benchmark findings, including unknown vulnerabilities in popular projects, and payload mutation that bypasses host defenses. These results show that open cyber-range evaluation is necessary for observing emerging offensive capabilities under realistic and reproducible conditions.

• The paper introduces an open and reproducible benchmark that evaluates autonomous cyberattack capabilities of state-of-the-art AI models.
• It systematically assesses both web exploitation and post-exploitation phases with detailed real-world vulnerabilities and multi-host dynamics, reporting metrics like Pass@1 and Pass@3.
• The experimental results reveal that while advanced AI agents can execute multi-stage operations, they still struggle with reliable end-to-end compromises due to instability and complex environmental challenges.

AgentCyberRange: A Comprehensive Benchmark for Evaluating Autonomous Cyberattack Capabilities of Frontier AI Systems

Motivation and Background

The increasing offensive capabilities demonstrated by frontier AI models necessitate robust, reproducible, and realistic evaluation of their potential for autonomous cyber operations. While prior benchmarks have measured isolated skills such as CTF solving, vulnerability reproduction, and exploit generation, these tasks abstract away the operational complexity and workflow structure inherent in real multi-stage cyber attacks. The lack of open, scalable, multi-host infrastructure limits the community's ability to observe, quantify, and anticipate emergent offensive behaviors in AI systems under end-to-end adversarial conditions.

AgentCyberRange addresses this critical gap by introducing the first open benchmark suite and execution pipeline designed for evaluating autonomous cyberattack capabilities of state-of-the-art AI agents in realistic web and multi-host enterprise environments. The benchmark models the operational workflow of real-world pentesting, spanning discovery, initial foothold, escalation, lateral movement, and defense evasion. It enables systematic, fine-grained measurement of AI systems' capacity to detect and exploit diverse vulnerabilities and to progress through complex internal attack paths.

Benchmark Structure and Design

AgentCyberRange comprises two core tracks, each designed with operational fidelity and diversity:

Web Exploitation (WebExploitBench):

This track includes 15 real-world web applications covering 110 vulnerabilities (18 zero-day, 56 one-day, 36 synthetic). Vulnerability classes span SQLi, SSRF, XSS, broken access control, RCE, and more. The agent must discover endpoints in a black-box manner and demonstrate valid exploitation—mere generation of PoCs is insufficient. The tasks are constructed to require extensive exploration and to prevent overfitting to public exploit patterns.

Post-Exploitation (PostExploitBench):

Eight enterprise-like, Dockerized cyber ranges totaling 156 internal hosts evaluate agents’ capacity for lateral movement, privilege escalation, credential use/discovery, tunneling, defense evasion, persistence, and multi-host chained exploitation. Each range includes realistic topology with DMZs, segmented internal networks, layered defenses, honeypots, and decoys, intentionally reflecting the complexity and nondeterminism observed in real enterprise intrusions.

The benchmark stratifies task difficulty across three levels by manipulating the granularity of environment and vulnerability hints, facilitating probe of both open-ended exploration and exploit synthesis in highly informed regimes.

Figure 1: The archetypal cyberattack workflow captured by AgentCyberRange, spanning reconnaissance, web exploitation, post exploitation, and reporting, with concrete examples of endpoint discovery, exploitation, lateral movement, and privilege escalation.

Evaluation Pipeline and Methodology

AgentCyberRange introduces CAGE, a modular execution pipeline supporting heterogeneous agents and scalable, isolated testbed instantiation. Key architectural elements include:

• Agent Adapters: Standardized interfaces abstracting invocation, environment configuration, and execution details for a diverse spectrum of coding and security-specific agents (e.g., Codex, Claude Code, Qwen, Kimi Code).
• Benchmark Manager: Automated deployment and reset of complex web applications and cyber ranges, ensuring reproducibility and independence of test trials.
• Verifier: Task- and vulnerability-specific verification of agent submissions through observable security effects (e.g., canary read for SQLi, marker files for host compromise), ensuring precise measurement and minimizing over/under-counting of partial progress.
• Comprehensive Logging and Inspector: Unified, fine-grained logging of every action, tool use, environment state, and decision point, enabling deep behavioral analysis, reproducibility, and manual audit.

  Figure 2: Overview of AgentCyberRange benchmark tasks and scalable execution pipeline (CAGE), supporting heterogeneous agent evaluation with modular deployment and automated verification.

Experimental Results

Aggregate Performance Across Tracks

Six representative state-of-the-art coding and pentest agents were evaluated under matched prompts and budgets. In aggregate:

• Web Exploitation:

GPT-5.5 with Codex achieved 19.1% Pass@1, 16.1% Pass@3 (Avg.), and 28.2% Pass@3 (Max) under the most challenging, hintless setting (Level-0). Performance increased to 33.0% Pass@3 (Avg.) when detailed vulnerability hints were supplied (Level-2).

• Post-Exploitation:

GPT-5.5 reached 31.7% Pass@1/Avg. and 43.9% Pass@3 (Max) under Level-0, rising to 46.3% under Level-2. Substantial run-to-run variance and sensitivity to environmental complexity was observed for all agents.

Figure 3: Overall success rates (Pass@3, average and max) for all systems across web exploitation and post-exploitation tasks, showing distinct performance separation but broad margins for improvement; GPT-5.5 with Codex leads, particularly under increased hint granularity.

Behavioral Analysis and Bottlenecks

• Exploration vs. Exploitation: Agents primarily used general-purpose tools (curl, python3) for probing; only the most capable (GPT-5.5) consistently employed specialized endpoint-discovery utilities (e.g., ffuf), supporting improved coverage.
• Multi-Step Reasoning: Agents showed the ability to perform chained actions—compromising a perimeter service, extracting credentials, pivoting internally—but often failed to fully synthesize complex multi-stage chains, especially under defensive perturbations.
• Variance and Failure Modes: High run-to-run variance reflected non-deterministic exploration. Failure to reach deep or obfuscated endpoints and non-systematic post-compromise knowledge extraction (e.g., not retrieving downstream credentials) were the predominant barriers.
• Out-of-Benchmark Discoveries: Agents occasionally identified previously unknown vulnerabilities ("zero-days"), confirming that the benchmark tasks provide unconstrained scope for emergent attack behaviors from frontier models.

  Figure 4: Behavioral breakdown of web exploitation actions for major agents; GPT-5.5 uniquely leverages advanced endpoint-discovery tooling, correlating with higher unique vulnerability discovery rates.

  

  Figure 5: Mapping of post-exploitation actions into MITRE ATT&CK tactics; most effort is expended on reconnaissance and exploitation, but advanced agents execute credential and anti-virus evasion steps.

Difficulty Stratification

In web exploitation, endpoint discovery emerges as the dominant bottleneck; explicit URL hints increase Pass@3 by over 20 percentage points. In post-exploitation, detailed vulnerability hints (Level-2) give the largest performance jump, evidencing substantial search space reduction.

Figure 6: Success rates as a function of difficulty level (hint granularity), emphasizing the criticality of attack-surface discovery and multi-host reasoning in agent performance.

Case Study: Multi-Stage Internal Compromise

GPT-5.5 demonstrated the ability to compromise segmented networks by chaining privilege escalation, credential recovery, tunneling, and escalation across multiple services and hosts.

Figure 7: Example trajectory of a successful multi-stage post-exploitation attack by GPT-5.5, combining exploitation, lateral movement, credential reuse, and defense evasion, but failing to achieve full environment compromise.

Implications and Limitations

The results indicate that current frontier AI systems can autonomously execute realistic cyber operations beyond trivial vulnerability exploitation—progressing through discovery, exploitation, lateral network traversal, privilege escalation, and (occasionally) defense evasion. However, reliable end-to-end compromise remains elusive: agents miss hidden surfaces, demonstrate instability across runs, and lack disciplined, stealth-oriented tradecraft as observed in expert penetration testers.

These findings imply:

• Evaluation of offensive AI capability must include complex workflow benchmarks; single-bug or flag-based scoring substantially underestimates the operational potential and limitations of current models.
• Defense planning and policy must begin accounting for AI-accelerated adversaries capable of traversing web-to-internal boundaries and adapting dynamically to observed defenses.
• Open benchmarks with high environmental fidelity are essential for meaningful capability tracking, safety evaluation, and responsible model release governance.

Notably, AgentCyberRange does not address all potential cyberattack modalities (e.g., cloud abuse, social engineering, supply chain), nor does it measure defender-aware or truly adaptive adversarial learning under blue-team pressure. Evaluated results are bounded by toolkits, prompts, and execution budgets.

Conclusion

AgentCyberRange establishes a comprehensive, open, and reproducible infrastructure for benchmarking the autonomous cyberattack capabilities of frontier AI models. By capturing both web exploitation and post-exploitation stages in realistic, modular cyber ranges, the benchmark makes it possible to systematically evaluate, compare, and audit the progression of offensive AI capabilities. Experimental results highlight both tangible risk—realistic exploitation, multi-host compromise, adaptive payload mutation—and significant unsolved challenges—instability, incomplete attack chaining, and suboptimal exploration.

Figure 8: Example topology and attack chain for a post-exploitation task, illustrating the multi-host, multi-segment structure that agents must navigate.

AgentCyberRange enables continuous, fine-grained tracking of emergent AI-enabled threats and provides a critical foundation for proactive defense, policy analysis, and future research. As AI-driven offensive capabilities mature, benchmarks of this sophistication will remain essential for responsible development, adoption, and red-teaming of advanced AI systems.