ExploitGym: AI-Driven Exploitation Benchmark

• ExploitGym is a comprehensive benchmark that defines and measures AI agents' ability to convert software vulnerabilities into functional exploits.
• It features 898 containerized instances from userspace, browser engines, and kernel domains to assess multi-step reasoning and low-level programming challenges.
• The benchmark evaluates exploit success under toggled security defenses, offering actionable insights for advancing both automated exploitation and defense strategies.

ExploitGym is a comprehensive, large-scale benchmark designed to rigorously evaluate the exploitation capabilities of AI agents—specifically, their ability to transform latent software vulnerabilities into concrete, impactful security breaches. It targets end-to-end exploitation: given a program input that triggers a vulnerability, agents must synthesize and iteratively refine a working exploit that achieves unauthorized code execution or file access. ExploitGym is motivated by the diagnostic and dual-use nature of this task. While previous benchmarks have addressed vulnerability discovery and patch generation, practical exploitation requires advanced, low-level program reasoning, runtime adaptability, and sustained problem-solving across multi-step temporal horizons (Wang et al., 11 May 2026).

1. Benchmark Design and Domain Coverage

At the core of ExploitGym is a corpus of 898 containerized task instances, selected to span major attack surfaces relevant to real-world software exploitation. Each instance contains:

• Vulnerable source code and reproducible build scripts,
• A proof-of-vulnerability (PoV) input (e.g., a crashing fuzzer trace or minimal C reproducer) and a textual vulnerability description,
• A remote, restricted execution environment that exposes only the target interface, isolating the task and mitigating environmental noise.

Agents interact within these environments, iteratively compiling alternate binaries, probing inputs, and resetting targets, until either retrieving a secret flag (demonstrating a successful exploit) or exhausting a strict wall-clock budget. The benchmark is divided into three domains:

• Userspace binaries (520 instances): Drawn from OSS-Fuzz via the CyberGym corpus and OSV, featuring memory-safety bugs in widely used C/C++ applications (e.g., FFmpeg, OpenSSL).
• Browser engines (185 instances): Targeting Google’s V8 JavaScript engine in Chromium via standalone “d8” shells stripped of convenience APIs; thus, successful exploitation requires genuine sandbox escapes.
• Linux kernel privilege escalations (193 instances): Served via QEMU/KVM per-connection VMs, with the agent running under an nsjail sandbox; the secret flag resides on a raw block device inaccessible to user-namespace root.

Each domain exposes two independent mitigation toggles controlling the presence of contemporary security defenses.

2. Security Protections and Instance Variants

To measure defence efficacy against automated exploitation, ExploitGym systematically toggles mainstream mitigations. The benchmark records rates of successful exploitation under various combinations of protections:

Domain	Defense 1	Defense 2
Userspace	Stack canaries	ASLR + PIE
V8 Engine	OS-level ASLR	V8 heap sandbox
Kernel	KASLR	User-namespace creation

Vulnerability types include buffer overflows, use-after-free flaws, out-of-bounds accesses, type confusion, and data races. Each protection can be enabled or disabled at container/build level, facilitating controlled ablation experiments. The exploit success rate $S$ for a configuration is:

$$S = \frac{\#\{\text{instances where agent captures flag and passes judge}\}}{898}$$

This standardization isolates the impact of mitigations and enables direct cross-agent and cross-model comparisons (Wang et al., 11 May 2026).

3. Evaluation Protocol and Metrics

Agents are provided with two hours per instance and access to PoV, vulnerability description, and source/build scripts (excluding the upstream patch). Interaction is mediated via scaffolding appropriate to the LLM vendor (e.g., Claude Code, Codex CLI), and agents may invoke any strategy confined to the container—compilation, binary rewriting, dynamic analysis, or arbitrary input probing.

Exploit success is determined in two stages:

1. The agent must capture the secret flag, proving unauthorized execution.
2. An automatic secondary judge (Codex CLI with GPT-5.5 or Claude Code with Opus 4.6) must certify—by analyzing the full transcript and artifacts—that the exploit genuinely leverages the intended vulnerability. Exploits that exploit environmental shortcuts or unrelated bugs do not count.

Auxiliary metrics tracked include:

• Number of model API invocations,
• Wall-clock time taken,
• API cost in dollars.

For a subset of instances, human reviewers adjudicate in cases of judge disagreement to ensure evaluation fidelity. Success is only credited to exploits meeting both criteria.

4. Model Performance and Agent Complementarity

Empirical results show nontrivial exploitability of real-world vulnerabilities by frontier LLM-based agents when all defenses are disabled:

• Claude Mythos Preview with Claude Code: 157/898 (17.5%)
  • Breakdown: 107 userspace, 38 V8, 12 kernel
• GPT-5.5 with Codex CLI: 120/898 (13.4%)
  • Breakdown: 71 userspace, 27 V8, 22 kernel
• GPT-5.4: 54/898 (below 6.1%) Other configurations remain below 15 successes.

When mitigations are enabled, success rates decrease but do not disappear:

• Claude Mythos Preview achieves 25 userspace, 17 V8, 3 kernel bypasses.
• GPT-5.5 records 10 userspace, 3 V8, and 8 kernel successes.
• GPT-5.4 is largely neutralized: 2 userspace, 0 V8, 1 kernel (Wang et al., 11 May 2026).

A significant fraction of exploits solve alternative or previously unknown bugs: ~70% of Claude's flag captures and 56.7% of GPT-5.5’s match the intended PoV, while the remainder correspond to alternative bugs in the corpus. The intersection between the two top-performing models’ solved instances is modest (91 out of 157 and 120, respectively), suggesting potential for agent ensembles to yield higher overall coverage.

5. Analysis of Mitigation Bypass and Defense Efficacy

Results robustly demonstrate that traditional defenses—ASLR, stack canaries, heap sandboxing, KASLR, user-namespace restrictions—reduce, but do not eliminate, exploitation by present-day AI agents:

• While overall $S$ drops sharply with mitigations set, the persistence of successes, especially against heap sandboxing and KASLR, signals residual risk even under widely adopted protections.
• The revealed complementarity between models (i.e., differing success sets) implies that no single model currently dominates exploit engineering across all instances or defense configurations.

A plausible implication is that AI-driven attackers equipped with ensembles of LLM agents may be able to bypass a larger swath of current mitigations than any individual configuration’s success rate would suggest.

6. Broader Implications, Dual-Use Concerns, and Future Directions

ExploitGym findings underscore critical dual-use dimensions in automated exploitation research. For defenders, such capabilities promise acceleration in severity triage, mitigation validation, and red-team automation. For offensive actors, they lower the expertise barrier and lend scale to exploit engineering. The demonstration that ASLR, heap sandboxing, and user-namespace boundaries are not fully impregnable to state-of-the-art AI agents exposes a pressing need for defense-in-depth mechanisms purpose-built to thwart automated exploitation.

Looking forward, priorities for ExploitGym and the broader community include:

• Extension to additional platforms (e.g., Windows, mobile),
• Evaluation of alternate success criteria (arbitrary read/write, sandbox escape without code execution),
• Integration of advanced agents leveraging symbolic execution or fuzzing,
• Research into next-generation runtime monitoring and fine-grained control flow integrity tailored to adversarial LLM tactics,
• Exploration of rapid patch deployment pipelines guided by AI.

As AI exploitation capabilities are expected to improve further, rigorous benchmarks like ExploitGym will remain central to defensive and policy strategies—providing empirical evidence of both mitigation effectiveness and evolving risk frontiers (Wang et al., 11 May 2026).