NYU CTF Benchmark for Cybersecurity Evaluation
- NYU CTF Benchmark is a standardized, open-source dataset aggregating Capture-The-Flag challenges across cryptography, forensics, pwn, reverse engineering, web, and misc security tasks.
- It organizes tasks by competition, year, and category, and employs controlled, containerized environments with metrics such as pass@1 and budget-constrained attempts.
- Recent studies utilize the benchmark to assess autonomous LM agents’ interactive tool use, multi-step planning, and overall cybersecurity reasoning capabilities.
The NYU CTF Dataset, commonly referred to as NYU CTF Bench or the NYU CTF Benchmark, is a scalable open-source benchmark for evaluating language-model agents on offensive-security tasks framed as Capture-The-Flag challenges. Introduced to provide a standardized, automated, and reproducible testbed for end-to-end cybersecurity reasoning, it aggregates CSAW CTF challenges across cryptography, forensics, binary exploitation, reverse engineering, web, and miscellaneous categories, and evaluates success by exact flag recovery inside controlled containerized environments (Shao et al., 2024). Subsequent work has treated it as a primary benchmark for autonomous LM agents in cybersecurity, especially for interactive tool use, long-horizon planning, and verified exploitation (Abramovich et al., 2024).
1. Origin and benchmark role
NYU CTF Bench was created to address the absence of a standardized, open, scalable benchmark for offensive-security evaluation of LLMs. Its motivating premise is that CTF challenges are unusually well suited to agentic evaluation: they are action-centric, require multi-step procedural reasoning, often depend on tool use and interaction with dynamic services, and yield an exact success signal through flag capture rather than subjective grading (Shao et al., 2024). The benchmark is drawn from CSAW CTF competitions and is repeatedly described as a realistic testbed for autonomous cybersecurity agents rather than a synthetic prompt collection (Shao et al., 2024, Abramovich et al., 2024).
Later papers position NYU CTF Bench as the university-level benchmark in a three-way evaluation spectrum alongside InterCode-CTF and Cybench. In that comparative framing, InterCode-CTF is described as high-school difficulty, Cybench as professional difficulty, and NYU CTF Bench as intermediate in both difficulty and scale (Zhuo et al., 25 Aug 2025, Zhuo et al., 29 Jul 2025). This comparative role has made it a common held-out benchmark for studies of single-agent tool use, multi-agent decomposition, retrieval-augmented execution, and fine-tuning from synthetic or execution-grounded trajectories (Abramovich et al., 2024, Udeshi et al., 15 Feb 2025, Shao et al., 21 May 2025, Zhuo et al., 29 Jul 2025, Zhuo et al., 25 Aug 2025).
2. Corpus composition and task taxonomy
The original release reports 200 validated challenges curated from CSAW competitions, with validated tasks spanning 2017–2023 and sourced from a larger pool of 568 CSAW CTF problems (2011–2023) (Shao et al., 2024). The benchmark covers six canonical jeopardy-style CTF categories and includes both static and dynamic tasks.
| Category | Count |
|---|---|
| Crypto | 53 |
| Forensics | 15 |
| Pwn | 38 |
| Reverse engineering | 51 |
| Web | 19 |
| Misc | 24 |
In the original release, challenge point values range from 1 to 500, and qualifying problems are reported to skew easier than finals (Shao et al., 2024). EnIGMA characterizes the benchmark difficulty as ranging from “very easy” to “hard”, while later cross-benchmark papers describe it as university-level (Abramovich et al., 2024, Zhuo et al., 25 Aug 2025).
The benchmark’s breadth is visible in representative tasks described across the literature. The pwn challenge “puffin” is presented as a buffer-overflow problem with a binary, a readme, and a reachable service; the web challenge “smug-dino” is described as an HTTP smuggling scenario behind NGINX; the reverse-engineering challenge “rap” is solved through decompilation and inversion of arithmetic checks; the forensics challenge “1black0white” reconstructs a QR code from numeric data; and the crypto challenge “jackpot” involves an ECC-based lottery service accessed over a remote connection (Shao et al., 2024, Abramovich et al., 2024).
A notable complication appears in later work: some EnIGMA+-based papers report 192 evaluated NYU CTF tasks while reproducing the same six-category counts shown above and also mentioning 200 elsewhere. Those papers do not fully reconcile the discrepancy, although one attributes the reduced evaluation suite to repairs and exclusions for reproducibility (Zhuo et al., 29 Jul 2025, Zhuo et al., 25 Aug 2025).
3. Data model, metadata, and execution environment
The original benchmark organizes challenges by directory as Year/Competition/Event/Category/Challenge_Name. Each challenge includes a challenge.json metadata file, optional docker-compose.yml for service-backed tasks, and challenge artifacts such as source code, binaries, configurations, Dockerfiles, media, or other task-specific files (Shao et al., 2024). Only the files listed in the metadata are revealed to the model, typically under a mounted challenge directory such as ~/ctf_files (Shao et al., 2024, Shao et al., 21 May 2025).
The agent-visible metadata includes fields such as name, category, description, files, box, internal_ports, and points; later prompt templates also expose flag_format and, when applicable, a server_description indicating how to reach a remote service (Shao et al., 2024, Shao et al., 21 May 2025). The ground-truth flag remains hidden and is used for automated verification.
The original automation framework places the agent inside an Ubuntu 22.04 container with purpose-built tools for command execution, file creation, Ghidra-backed disassembly and decompilation, flag checking, and early termination (Shao et al., 2024). EnIGMA later re-expressed benchmark interaction through an Agent-Computer Interface layered on SWE-agent, with a structured DISCUSSION + COMMAND protocol, environment annotations in each observation, and new Interactive Agent Tools for persistent debugging and remote server interaction. These additions include debug_start, debug_add_breakpoint, debug_step, debug_continue, debug_exec, connect_start, and connect_sendline, along with summarizers to compress long observations and a large preinstalled security toolset including pwntools, radare2, wine, pycryptodome, sagemath, gmpy2, RsaCtfTool, tshark, binwalk, sqlmap, and nikto (Abramovich et al., 2024).
Subsequent frameworks reuse the benchmark through their own scaffolds. D-CIPHER evaluates each task in a shared Linux container with network access to the challenge server and the internet, using function-calling tools such as RunCommand, CreateFile, Disassemble, Decompile, SubmitFlag, and Giveup (Udeshi et al., 15 Feb 2025). CRAKEN inherits a similar planner–executor scaffold, mounts challenge files into a Docker environment, and injects retrieval-derived knowledge hints into executor contexts (Shao et al., 21 May 2025). EnIGMA+ later standardizes isolated Docker execution and parallel rollout evaluation across benchmarks, including NYU CTF Bench (Zhuo et al., 29 Jul 2025, Zhuo et al., 25 Aug 2025).
4. Evaluation protocols and metrics
The benchmark’s core semantics are stable: a challenge is solved by recovering the correct flag, typically of a known format such as flag{...} or csawctf{...}. The original benchmark paper reports an evaluation procedure with up to 5 attempts per challenge and a 48-hour wall-clock budget per model across the dataset (Shao et al., 2024). Later work shifts toward one-trajectory evaluation and reports Pass@1 or percentage solved under explicit budget constraints (Abramovich et al., 2024, Zhuo et al., 29 Jul 2025).
Several formal metrics appear in the associated literature. The original paper defines overall success rate as
with analogous per-category and normalized-score variants (Shao et al., 2024). EnIGMA reports pass@1 as
and also tracks average model API cost over solved instances,
Its default NYU CTF protocol uses one trajectory per challenge and a \$3 per-instance budget** [2409.16165]. D-CIPHER and CRAKEN retain percentage solved and average cost per solved challenge as primary metrics, likewise with a **\$3 per-challenge budget, but use framework-specific multi-agent round limits and allow success to count if the correct flag appears in conversation logs even when bookkeeping fails to route it through the final submission tool (Udeshi et al., 15 Feb 2025, Shao et al., 21 May 2025).
EnIGMA+-based training papers adopt a different budget model: they cap each rollout at 40 interaction turns, evaluate in isolated Docker containers, and report Pass@1 with a single sampled attempt per task (Zhuo et al., 29 Jul 2025, Zhuo et al., 25 Aug 2025). For these papers, the general definition is
with all reported NYU CTF results using (Zhuo et al., 25 Aug 2025).
Because the benchmark has been used under distinct protocols—five-attempt evaluation in the original release, single-trajectory pass@1 in EnIGMA, multi-agent solve-rate reporting in D-CIPHER and CRAKEN, and 40-turn capped evaluation in EnIGMA+—reported numbers are not interchangeable without attention to scaffold, budget, and task count (Shao et al., 2024, Abramovich et al., 2024, Udeshi et al., 15 Feb 2025, Zhuo et al., 29 Jul 2025).
5. Empirical use and benchmark trajectory
The original automated benchmark study reported low category-level solve rates for then-current models, with open-source systems such as Mixtral and LLaMA 3 recording all-zero category solves, while GPT-3.5, GPT-4, and Claude-3 achieved only limited success concentrated in reverse engineering, pwn, forensics, and misc (Shao et al., 2024). NYU CTF Bench subsequently became the principal testbed for more capable agentic frameworks.
Direct comparison across later papers requires caution because the protocols differ, but the benchmark has supported a visible performance progression.
| System | NYU CTF result | Notes |
|---|---|---|
| EnIGMA with Claude 3.5 Sonnet | 13.5% pass@1 | 200 challenges, \$3 budget (Abramovich et al., 2024) |
| D-CIPHER with Claude 3.5 Sonnet | 19.0% solved | 200 challenges, \$3 budget (Udeshi et al., 15 Feb 2025) |
| D-CIPHER without Auto-prompter | 22.0% solved | Best D-CIPHER NYU result (Udeshi et al., 15 Feb 2025) |
| CRAKEN with Self-RAG + classic RAG | 21.0% solved | 200 challenges, \$3 budget (Shao et al., 21 May 2025) |
| CRAKEN + Graph-RAG | 22.0% solved | Reported SOTA in that paper (Shao et al., 21 May 2025) |
| Cyber-Zero-32B | 13.5% Pass@1 | 192-task evaluated suite (Zhuo et al., 29 Jul 2025) |
| CTF-Dojo-32B | 10.4% Pass@1 | CTF-Dojo-trained, NYU held out (Zhuo et al., 25 Aug 2025) |
Category-level findings are also recurrent. EnIGMA reported 0% web on NYU CTF, while showing stronger performance in forensics, pwn, reverse engineering, and misc (Abramovich et al., 2024). D-CIPHER improved substantially over EnIGMA in crypto, reverse engineering, and misc, but EnIGMA remained stronger on pwn in one Claude 3.5 Sonnet comparison (Udeshi et al., 15 Feb 2025). CRAKEN reported gains from knowledge-based execution, with its Graph-RAG configuration increasing crypto, forensics, pwn, web, and misc over its default retrieval setting, though reverse engineering decreased relative to the default variant (Shao et al., 21 May 2025). For open-weight models, both Cyber-Zero and CTF-Dojo reported sizeable improvements over zero-shot Qwen-family baselines on NYU CTF Bench, while still trailing frontier proprietary systems on this benchmark (Zhuo et al., 29 Jul 2025, Zhuo et al., 25 Aug 2025).
6. Development sets, distilled subsets, and surrounding ecosystem
Although NYU CTF Bench is primarily treated as a held-out benchmark, later work created adjacent resources to support principled tuning without contaminating test evaluation. EnIGMA released a 55-challenge development set sampled from the same CSAW competitions and spanning the same six categories: Crypto 10, Forensics 10, Pwn 10, Reverse 9, Misc 6, Web 10 (Abramovich et al., 2024). This dev set was used for agent design, demonstrations, and guidelines rather than final benchmarking.
A later derivative benchmark, CTFTiny, is a curated 50-challenge subset distilled from NYU CTF Bench for rapid evaluation and hyperparameter sweeps. CTFTiny preserves category diversity—12 crypto, 2 forensics, 11 pwn, 16 rev, 3 web, 6 misc—and pairs each challenge with a single expert write-up used by CTFJudge, an LLM-as-a-judge framework that scores agent trajectories across six competency dimensions: vulnerability understanding, reconnaissance thoroughness, exploitation methodology, technical accuracy, efficiency of approach, and adaptability (Shao et al., 5 Aug 2025).
The benchmark also anchors a broader software ecosystem. The original release includes an automated playground framework for challenge orchestration and tool calling (Shao et al., 2024). EnIGMA builds on SWE-agent and releases code plus the NYU development dataset (Abramovich et al., 2024). D-CIPHER and CRAKEN provide open-source multi-agent implementations targeting NYU CTF Bench (Udeshi et al., 15 Feb 2025, Shao et al., 21 May 2025). Training-oriented work such as Cyber-Zero and CTF-Dojo treats NYU CTF Bench as a benchmark-only target and explicitly excludes overlapping material from training data or writeup corpora to preserve evaluation validity (Zhuo et al., 29 Jul 2025, Zhuo et al., 25 Aug 2025).
7. Validity, contamination control, and limitations
Several papers foreground benchmark validity. EnIGMA analyzes a phenomenon termed soliloquizing, in which the model emits hallucinated environment-like observations without actual interaction. On NYU CTF with Claude 3.5 Sonnet, it reports 48.4% of trajectories affected by soliloquy and 2.7% flagged as leakage, with a negative correlation between soliloquy frequency and success; GPT-4 Turbo and GPT-4o did not exhibit soliloquies in that setup (Abramovich et al., 2024). This establishes that leakage and protocol circumvention are benchmark-level concerns even when the measured leak rate on NYU CTF is lower than on older datasets.
Contamination control is correspondingly strict in later training papers. Cyber-Zero excludes writeups from competitions covered by NYU CTF Bench and other evaluation suites when synthesizing training trajectories (Zhuo et al., 29 Jul 2025). CRAKEN excludes CSAW CTF writeups from its external knowledge base because the benchmark itself uses CSAW challenges (Shao et al., 21 May 2025). CTF-Dojo decontaminates its training pool by removing all evaluation benchmark tasks, including NYU CTF Bench, and uses NYU strictly for held-out evaluation (Zhuo et al., 25 Aug 2025).
The benchmark also has persistent practical limitations. The original release notes category imbalance, single-source bias from CSAW-only coverage, and the exclusion of Incident Response-style tasks because of validation complexity (Shao et al., 2024). Later papers document reproducibility problems involving missing Docker settings, absent server aliases, missing docker-compose.yml, or missing artifacts, leading one runtime study to repair some NYU tasks and exclude others (Zhuo et al., 29 Jul 2025). Runtime specifics are not consistently documented across papers: some later evaluations do not specify OS/image details, memory limits, or per-challenge service constraints for NYU tasks (Zhuo et al., 25 Aug 2025). The web category remains a recurrent weak point, particularly under CLI-only interfaces without dedicated browser or GUI tooling (Abramovich et al., 2024, Udeshi et al., 15 Feb 2025).
A final unresolved issue is the benchmark’s reported size in later literature. The original release and several agent papers consistently state 200 challenges (Shao et al., 2024, Abramovich et al., 2024, Udeshi et al., 15 Feb 2025, Shao et al., 21 May 2025). Later EnIGMA+-based papers report 192 evaluated tasks after repairs or exclusions, yet reproduce the original six category counts and also mention 200 elsewhere, leaving the exact evaluated subset insufficiently reconciled in the published record (Zhuo et al., 29 Jul 2025, Zhuo et al., 25 Aug 2025). This suggests that NYU CTF Bench functions simultaneously as an official 200-challenge benchmark and, in some later experimental settings, as a repaired executable subset whose boundaries are paper-specific.