---
title: NYU CTF Benchmark for Cybersecurity Evaluation
url: https://www.emergentmind.com/topics/nyu-ctf-dataset
type: topic
---

# NYU CTF Benchmark for Cybersecurity Evaluation

The NYU CTF Dataset, commonly referred to as **NYU CTF Bench** or the **NYU CTF Benchmark**, is a scalable open-source benchmark for evaluating language-model agents on offensive-security tasks framed as Capture-The-Flag challenges. Introduced to provide a standardized, automated, and reproducible testbed for end-to-end cybersecurity reasoning, it aggregates CSAW CTF challenges across cryptography, forensics, binary exploitation, reverse engineering, web, and miscellaneous categories, and evaluates success by exact flag recovery inside controlled containerized environments [2406.05590]. Subsequent work has treated it as a primary benchmark for autonomous LM agents in cybersecurity, especially for interactive tool use, long-horizon planning, and verified exploitation [2409.16165].

## 1. Origin and benchmark role

NYU CTF Bench was created to address the absence of a standardized, open, scalable benchmark for offensive-security evaluation of LLMs. Its motivating premise is that CTF challenges are unusually well suited to agentic evaluation: they are action-centric, require multi-step procedural reasoning, often depend on tool use and interaction with dynamic services, and yield an exact success signal through flag capture rather than subjective grading [2406.05590]. The benchmark is drawn from CSAW CTF competitions and is repeatedly described as a realistic testbed for autonomous cybersecurity agents rather than a synthetic prompt collection [2406.05590][2409.16165].

Later papers position NYU CTF Bench as the **university-level** benchmark in a three-way evaluation spectrum alongside InterCode-CTF and Cybench. In that comparative framing, InterCode-CTF is described as high-school difficulty, Cybench as professional difficulty, and NYU CTF Bench as intermediate in both difficulty and scale [2508.18370][2508.00910]. This comparative role has made it a common held-out benchmark for studies of single-agent tool use, multi-agent decomposition, retrieval-augmented execution, and fine-tuning from synthetic or execution-grounded trajectories [2409.16165][2502.10931][2505.17107][2508.00910][2508.18370].

## 2. Corpus composition and task taxonomy

The original release reports **200 validated challenges** curated from CSAW competitions, with validated tasks spanning **2017–2023** and sourced from a larger pool of **568 CSAW CTF problems (2011–2023)** [2406.05590]. The benchmark covers six canonical jeopardy-style CTF categories and includes both static and dynamic tasks.

| Category | Count |
|---|---:|
| Crypto | 53 |
| Forensics | 15 |
| Pwn | 38 |
| Reverse engineering | 51 |
| Web | 19 |
| Misc | 24 |

In the original release, challenge point values range from **1 to 500**, and qualifying problems are reported to skew easier than finals [2406.05590]. EnIGMA characterizes the benchmark difficulty as ranging from **“very easy” to “hard”**, while later cross-benchmark papers describe it as **university-level** [2409.16165][2508.18370].

The benchmark’s breadth is visible in representative tasks described across the literature. The pwn challenge **“puffin”** is presented as a buffer-overflow problem with a binary, a readme, and a reachable service; the web challenge **“smug-dino”** is described as an HTTP smuggling scenario behind NGINX; the reverse-engineering challenge **“rap”** is solved through decompilation and inversion of arithmetic checks; the forensics challenge **“1black0white”** reconstructs a QR code from numeric data; and the crypto challenge **“jackpot”** involves an ECC-based lottery service accessed over a remote connection [2406.05590][2409.16165].

A notable complication appears in later work: some EnIGMA+-based papers report **192 evaluated NYU CTF tasks** while reproducing the same six-category counts shown above and also mentioning **200** elsewhere. Those papers do not fully reconcile the discrepancy, although one attributes the reduced evaluation suite to repairs and exclusions for reproducibility [2508.00910][2508.18370].

## 3. Data model, metadata, and execution environment

The original benchmark organizes challenges by directory as **Year/Competition/Event/Category/Challenge_Name**. Each challenge includes a **challenge.json** metadata file, optional **docker-compose.yml** for service-backed tasks, and challenge artifacts such as source code, binaries, configurations, Dockerfiles, media, or other task-specific files [2406.05590]. Only the files listed in the metadata are revealed to the model, typically under a mounted challenge directory such as `~/ctf_files` [2406.05590][2505.17107].

The agent-visible metadata includes fields such as **name**, **category**, **description**, **files**, **box**, **internal_ports**, and **points**; later prompt templates also expose **flag_format** and, when applicable, a **server_description** indicating how to reach a remote service [2406.05590][2505.17107]. The ground-truth flag remains hidden and is used for automated verification.

The original automation framework places the agent inside an **Ubuntu 22.04** container with purpose-built tools for command execution, file creation, Ghidra-backed disassembly and decompilation, flag checking, and early termination [2406.05590]. EnIGMA later re-expressed benchmark interaction through an Agent-Computer Interface layered on SWE-agent, with a structured **DISCUSSION + COMMAND** protocol, environment annotations in each observation, and new **Interactive Agent Tools** for persistent debugging and remote server interaction. These additions include `debug_start`, `debug_add_breakpoint`, `debug_step`, `debug_continue`, `debug_exec`, `connect_start`, and `connect_sendline`, along with summarizers to compress long observations and a large preinstalled security toolset including pwntools, radare2, wine, pycryptodome, sagemath, gmpy2, RsaCtfTool, tshark, binwalk, sqlmap, and nikto [2409.16165].

Subsequent frameworks reuse the benchmark through their own scaffolds. D-CIPHER evaluates each task in a shared Linux container with network access to the challenge server and the internet, using function-calling tools such as `RunCommand`, `CreateFile`, `Disassemble`, `Decompile`, `SubmitFlag`, and `Giveup` [2502.10931]. CRAKEN inherits a similar planner–executor scaffold, mounts challenge files into a Docker environment, and injects retrieval-derived knowledge hints into executor contexts [2505.17107]. EnIGMA+ later standardizes isolated Docker execution and parallel rollout evaluation across benchmarks, including NYU CTF Bench [2508.00910][2508.18370].

## 4. Evaluation protocols and metrics

The benchmark’s core semantics are stable: a challenge is solved by recovering the correct flag, typically of a known format such as `flag{...}` or `csawctf{...}`. The original benchmark paper reports an evaluation procedure with **up to 5 attempts per challenge** and a **48-hour wall-clock budget per model** across the dataset [2406.05590]. Later work shifts toward one-trajectory evaluation and reports **Pass@1** or percentage solved under explicit budget constraints [2409.16165][2508.00910].

Several formal metrics appear in the associated literature. The original paper defines overall success rate as
$$
S = \frac{1}{N}\sum_{i=1}^{N}\mathbf{1}[\mathrm{solve}_i],
$$
with analogous per-category and normalized-score variants [2406.05590]. EnIGMA reports **pass@1** as
$$
R = \frac{N_{\mathrm{solved}}}{N_{\mathrm{total}}},
$$
and also tracks average model API cost over solved instances,
$$
C_{\mathrm{avg}} = \frac{\sum_{i \in \mathrm{solved}} \mathrm{cost}_i}{N_{\mathrm{solved}}}.
$$
Its default NYU CTF protocol uses **one trajectory per challenge** and a **\$3 per-instance budget** [2409.16165]. D-CIPHER and CRAKEN retain percentage solved and average cost per solved challenge as primary metrics, likewise with a **\$3** per-challenge budget, but use framework-specific multi-agent round limits and allow success to count if the correct flag appears in conversation logs even when bookkeeping fails to route it through the final submission tool [2502.10931][2505.17107].

EnIGMA+-based training papers adopt a different budget model: they cap each rollout at **40 interaction turns**, evaluate in isolated Docker containers, and report **Pass@1** with a single sampled attempt per task [2508.00910][2508.18370]. For these papers, the general definition is
$$
\mathrm{Pass@k} = \frac{1}{N}\sum_{i=1}^{N}\mathbf{1}\{\text{any of top-}k\text{ attempts solve task }i\},
$$
with all reported NYU CTF results using $k=1$ [2508.18370].

Because the benchmark has been used under distinct protocols—five-attempt evaluation in the original release, single-trajectory pass@1 in EnIGMA, multi-agent solve-rate reporting in D-CIPHER and CRAKEN, and 40-turn capped evaluation in EnIGMA+—reported numbers are not interchangeable without attention to scaffold, budget, and task count [2406.05590][2409.16165][2502.10931][2508.00910].

## 5. Empirical use and benchmark trajectory

The original automated benchmark study reported low category-level solve rates for then-current models, with open-source systems such as Mixtral and LLaMA 3 recording all-zero category solves, while GPT-3.5, GPT-4, and Claude-3 achieved only limited success concentrated in reverse engineering, pwn, forensics, and misc [2406.05590]. NYU CTF Bench subsequently became the principal testbed for more capable agentic frameworks.

Direct comparison across later papers requires caution because the protocols differ, but the benchmark has supported a visible performance progression.

| System | NYU CTF result | Notes |
|---|---:|---|
| EnIGMA with Claude 3.5 Sonnet | 13.5% pass@1 | 200 challenges, \$3 budget [2409.16165] |
| D-CIPHER with Claude 3.5 Sonnet | 19.0% solved | 200 challenges, \$3 budget [2502.10931] |
| D-CIPHER without Auto-prompter | 22.0% solved | Best D-CIPHER NYU result [2502.10931] |
| CRAKEN with Self-RAG + classic RAG | 21.0% solved | 200 challenges, \$3 budget [2505.17107] |
| CRAKEN + Graph-RAG | 22.0% solved | Reported SOTA in that paper [2505.17107] |
| Cyber-Zero-32B | 13.5% Pass@1 | 192-task evaluated suite [2508.00910] |
| CTF-Dojo-32B | 10.4% Pass@1 | CTF-Dojo-trained, NYU held out [2508.18370] |

Category-level findings are also recurrent. EnIGMA reported **0% web** on NYU CTF, while showing stronger performance in forensics, pwn, reverse engineering, and misc [2409.16165]. D-CIPHER improved substantially over EnIGMA in crypto, reverse engineering, and misc, but EnIGMA remained stronger on pwn in one Claude 3.5 Sonnet comparison [2502.10931]. CRAKEN reported gains from knowledge-based execution, with its Graph-RAG configuration increasing crypto, forensics, pwn, web, and misc over its default retrieval setting, though reverse engineering decreased relative to the default variant [2505.17107]. For open-weight models, both Cyber-Zero and CTF-Dojo reported sizeable improvements over zero-shot Qwen-family baselines on NYU CTF Bench, while still trailing frontier proprietary systems on this benchmark [2508.00910][2508.18370].

## 6. Development sets, distilled subsets, and surrounding ecosystem

Although NYU CTF Bench is primarily treated as a held-out benchmark, later work created adjacent resources to support principled tuning without contaminating test evaluation. EnIGMA released a **55-challenge development set** sampled from the same CSAW competitions and spanning the same six categories: **Crypto 10, Forensics 10, Pwn 10, Reverse 9, Misc 6, Web 10** [2409.16165]. This dev set was used for agent design, demonstrations, and guidelines rather than final benchmarking.

A later derivative benchmark, **CTFTiny**, is a curated **50-challenge** subset distilled from NYU CTF Bench for rapid evaluation and hyperparameter sweeps. CTFTiny preserves category diversity—**12 crypto, 2 forensics, 11 pwn, 16 rev, 3 web, 6 misc**—and pairs each challenge with a single expert write-up used by **CTFJudge**, an LLM-as-a-judge framework that scores agent trajectories across six competency dimensions: vulnerability understanding, reconnaissance thoroughness, exploitation methodology, technical accuracy, efficiency of approach, and adaptability [2508.05674].

The benchmark also anchors a broader software ecosystem. The original release includes an automated playground framework for challenge orchestration and tool calling [2406.05590]. EnIGMA builds on SWE-agent and releases code plus the NYU development dataset [2409.16165]. D-CIPHER and CRAKEN provide open-source multi-agent implementations targeting NYU CTF Bench [2502.10931][2505.17107]. Training-oriented work such as Cyber-Zero and CTF-Dojo treats NYU CTF Bench as a benchmark-only target and explicitly excludes overlapping material from training data or writeup corpora to preserve evaluation validity [2508.00910][2508.18370].

## 7. Validity, contamination control, and limitations

Several papers foreground benchmark validity. EnIGMA analyzes a phenomenon termed **soliloquizing**, in which the model emits hallucinated environment-like observations without actual interaction. On NYU CTF with Claude 3.5 Sonnet, it reports **48.4%** of trajectories affected by soliloquy and **2.7%** flagged as leakage, with a negative correlation between soliloquy frequency and success; GPT-4 Turbo and GPT-4o did not exhibit soliloquies in that setup [2409.16165]. This establishes that leakage and protocol circumvention are benchmark-level concerns even when the measured leak rate on NYU CTF is lower than on older datasets.

Contamination control is correspondingly strict in later training papers. Cyber-Zero excludes writeups from competitions covered by NYU CTF Bench and other evaluation suites when synthesizing training trajectories [2508.00910]. CRAKEN excludes **CSAW CTF writeups** from its external knowledge base because the benchmark itself uses CSAW challenges [2505.17107]. CTF-Dojo decontaminates its training pool by removing all evaluation benchmark tasks, including NYU CTF Bench, and uses NYU strictly for held-out evaluation [2508.18370].

The benchmark also has persistent practical limitations. The original release notes **category imbalance**, **single-source bias** from CSAW-only coverage, and the exclusion of **Incident Response-style** tasks because of validation complexity [2406.05590]. Later papers document reproducibility problems involving missing Docker settings, absent server aliases, missing `docker-compose.yml`, or missing artifacts, leading one runtime study to repair some NYU tasks and exclude others [2508.00910]. Runtime specifics are not consistently documented across papers: some later evaluations do not specify OS/image details, memory limits, or per-challenge service constraints for NYU tasks [2508.18370]. The web category remains a recurrent weak point, particularly under CLI-only interfaces without dedicated browser or GUI tooling [2409.16165][2502.10931].

A final unresolved issue is the benchmark’s reported size in later literature. The original release and several agent papers consistently state **200 challenges** [2406.05590][2409.16165][2502.10931][2505.17107]. Later EnIGMA+-based papers report **192 evaluated tasks** after repairs or exclusions, yet reproduce the original six category counts and also mention 200 elsewhere, leaving the exact evaluated subset insufficiently reconciled in the published record [2508.00910][2508.18370]. This suggests that NYU CTF Bench functions simultaneously as an official 200-challenge benchmark and, in some later experimental settings, as a repaired executable subset whose boundaries are paper-specific.

Source: https://www.emergentmind.com/topics/nyu-ctf-dataset