CTFTiny: Lightweight CTF Benchmark

• CTFTiny is a compact benchmark designed for rapid evaluation of offensive-security LLM agents with 50 curated CTF challenges reflecting diverse difficulty levels.
• The benchmark integrates with CTFJudge and the CTF Competency Index to provide detailed trajectory-level scoring that measures agents’ reasoning, planning, and exploitation methodology.
• Empirical difficulty modeling, balanced challenge selection, and sensitivity studies on hyperparameters make CTFTiny practical for iterative development and comparative evaluation.

Searching arXiv for the specified paper and closely related context. arXiv search query: (Shao et al., 5 Aug 2025) CTFTiny is a lightweight, reproducible Capture the Flag benchmark designed for rapid evaluation of offensive-security LLM agents without the computational burden of running the full NYU CTF Bench. It was introduced to support large-scale agent experiments—especially hyperparameter sweeps, model ablations, and sensitivity studies—that are too expensive and slow on the full benchmark. In the paper’s evaluation stack, CTFTiny supplies the benchmark tasks, D-CIPHER is the multi-agent solver architecture being evaluated, CTFJudge provides trajectory-level judging, and the CTF Competency Index (CCI) converts those judgments into a fine-grained score of offensive-security competence (Shao et al., 5 Aug 2025).

1. Design Objective and Position in the Evaluation Stack

CTFTiny was created to solve a practical benchmarking problem for offensive-security agents. Full benchmarks are suitable for final reporting, but poor for fast iteration during development. The benchmark therefore functions as a compact testbed that can be used frequently while still preserving the diversity and difficulty structure of real CTFs. The paper frames it as a low-cost proxy that exposes whether an agent can perform meaningful decomposition, reconnaissance, exploit construction, and adaptation across a representative CTF mix (Shao et al., 5 Aug 2025).

This role is central to the paper’s broader methodology. CTFTiny is not presented as a standalone scoring artifact, but as one layer in a larger instrumentation stack. The benchmark provides the tasks; CTFJudge evaluates the trajectory against expert writeups; and CCI turns those trajectory judgments into a scalar measure of partial correctness. This organization reflects the paper’s position that offensive-security evaluation should not collapse agent behavior into final flag extraction alone.

The benchmark is also explicitly open source. The paper states that CTFTiny is made public at https://github.com/NYU-LLM-CTF/CTFTiny, alongside CTFJudge at https://github.com/NYU-LLM-CTF/CTFJudge.

2. Benchmark Construction and Empirical Difficulty Modeling

CTFTiny consists of 50 real CTF challenges curated from NYU CTF Bench. The paper states that the subset is not random. Instead, challenge selection is based on an empirical difficulty measure derived from results on NYU CTF Bench across 12 configurations from prior systems, specifically D-CIPHER and CRAKEN (Shao et al., 5 Aug 2025).

For each challenge, the authors count how many of those 12 configurations solved it. That solve-count acts as a proxy for ease or hardness. The benchmark is then balanced to include tasks that are neither trivial nor impossibly hard. The paper defines four difficulty bands by the number of successful configurations:

• Hard: 0–3 solves
• Moderate: 4–6 solves
• Easy: 6–9 solves
• Very easy: 9–12 solves

Higher solve count therefore means easier. The paper says CTFTiny has a healthy spread across these difficulty levels. This selection rule is significant because it ensures that the benchmark is discriminative: harder tasks are deliberately included so that the benchmark is not dominated by easy wins.

A recurring misconception would be to treat CTFTiny as merely a smaller benchmark. The paper’s construction procedure argues against that reading. Because the subset is difficulty-balanced using prior solve frequencies, the compactness is paired with deliberate preservation of discriminatory power. A plausible implication is that the benchmark is intended to approximate the diagnostic value of the full benchmark while making repeated experimentation feasible.

3. Domain Coverage and Challenge Composition

The paper explicitly states that CTFTiny spans six domains: cryptography, forensics, binary exploitation (pwn), reverse engineering, web exploitation, and miscellaneous (misc). This mix is intentional: the benchmark is meant to represent the broad spectrum of offensive-security reasoning and tooling that an LLM agent needs, rather than focusing on one narrow skill (Shao et al., 5 Aug 2025).

Category	Tasks
Cryptography	12
Forensics	2
Pwn	11
Reverse engineering	16
Web exploitation	3
Misc	6

The appendix lists the included challenges and their difficulty labels. Examples named in the paper include ecxor, lupin, babycrypto, super_curve, hybrid2, perfect_secrecy, whyos, pilot, baby_boi, puffin, tablez, maze, poem_collection, shreeramquest, and showdown.

The category distribution matters methodologically. Reverse engineering and cryptography are heavily represented, while forensics and web exploitation are small subsets. The paper does not present this as a flaw; rather, the benchmark is described as a curated subset of real tasks chosen to preserve breadth and difficulty structure under a strict budget of 50 challenges.

4. Evaluation Protocols: pass@1, CTFJudge, and CCI

CTFTiny is evaluated in two ways. First, the paper uses standard solve-rate metrics, primarily pass@1, defined as the proportion of challenges solved on the first attempt. It also reports average computational cost. Second, it uses CTFJudge for trajectory-level, LLM-based evaluation (Shao et al., 5 Aug 2025).

CTFJudge compares the agent’s solution trajectory to an expert writeup, not just the final flag. The expert writeups are transformed into stepwise summaries, and the agent traces are summarized in parallel, enabling reference-guided comparison. CTFJudge scores performance across six dimensions:

• vulnerability understanding
• reconnaissance thoroughness
• exploitation methodology
• technical accuracy
• efficiency of approach
• adaptability

The paper describes CTFJudge as a three-agent pipeline. One summarizer turns the expert writeup into structured steps, another summarizes the agent trajectory, and a third qualitative-evaluation agent compares them and returns a structured report plus the six-dimensional score matrix. The prompts are constrained to encourage stepwise, JSON-formatted outputs and to record detailed comparisons, key insights, and failure analysis. For grading, CTFJudge uses Claude 3 Sonnet at temperature 0.1.

The core metric produced by CTFJudge is the CTF Competency Index:

$$\mathrm{CCI}(T, G) \;=\; \sum_{i=1}^{n} w_i\,F_i(T, G), \quad \sum_{i=1}^{n} w_i = 1$$

Here, $T$ is the agent’s trajectory summary, $G$ is the human-curated gold solution, $F_i(T,G)$ are the factor scores, and $w_i$ are the weights. In the default setup, $n=6$, corresponding to the six criteria above, and the weights are equal. Because the weights sum to 1, the score lies in $[0,1]$. In prose, the paper characterizes CCI as measuring how closely the agent’s process matches a competent human-style offensive workflow, balancing strategic insight, technical correctness, and operational efficiency.

This metric is important because the paper stresses that some agents may solve a challenge by brute force or chance while still exhibiting poor reasoning quality. CCI is intended to expose that distinction.

5. Quantitative Results on CTFTiny

The benchmark supports both baseline comparison and hyperparameter study. In the paper’s model comparison, Claude 4 Sonnet is the strongest model on CTFTiny, solving 38 out of 50 tasks for 76% success. Other reported results are as follows (Shao et al., 5 Aug 2025).

Model	Solved	Success rate
Claude 4 Sonnet	38/50	76%
Gemini 2.5 Flash	32/50	64%
Gemini 2.5 Pro	24/50	48%
GPT-4.1	20/50	40%
Qwen 3	14/50	28%
DeepSeek V3	11/50	22%
LLaMA 4 Maverick	4/50	8%

The paper also reports category-level performance. Claude 4 Sonnet leads in reverse engineering at 81.3%, cryptography at 75%, and miscellaneous tasks at 83.3%. Gemini 2.5 Flash performs particularly well in binary exploitation at 72.7%, and both Gemini models achieve perfect scores on the tiny forensics subset. In web exploitation, Claude 4 Sonnet and Qwen 3 both reach 100%.

Task-level distributions reveal substantial heterogeneity. The appendix shows that poem_collection and unvirtualization are solved by all or most models, whereas lupin, perfect_secrecy, and algebra remain unsolved by all models. This supports the paper’s claim that the benchmark includes both accessible and discriminative tasks.

CCI analysis provides a second axis of differentiation. The paper states that Claude 4 Sonnet has the strongest and most consistent CCI, with scores roughly in the 77.5–84.5 range across dimensions. Gemini 2.5 Pro can have a higher average CCI than Gemini 2.5 Flash even when Flash solves more tasks, suggesting that Pro’s trajectories are more structured and human-aligned. The largest separation between success and failure appears in exploitation methodology, efficiency, and adaptability, while reconnaissance and vulnerability understanding vary less. The paper also notes that most models are weak on efficiency of approach, often relying on brute-force or redundant exploration.

CTFTiny is especially important for the paper’s hyperparameter study because its size makes sweeps practical. The authors vary temperature, top-p, and maximum token length. Performance is reported as sensitive to all three. For Claude 4 Sonnet, the best accuracy occurs at high temperature ($T=1.0$) and full top-p ($p=1.0$). GPT-4.1 peaks at more moderate values, with temperature around 0.6 and top-p near 0.9, and is described as less sensitive overall. For max tokens, the best setting is often 4096 rather than 8192, which the paper interprets as a “Goldilocks zone” where the context is large enough for reasoning but not so large that it causes distraction or verbosity.

6. Case Studies, Interpretive Value, and Limits

The paper uses CTFJudge case studies to illustrate why CTFTiny is more informative when paired with trajectory-level evaluation than when used with pass/fail alone (Shao et al., 5 Aug 2025).

In pwn-slithery, the agent’s reasoning aligns well with the expert’s encoding-based sandbox bypass, yielding perfect scores across all six CCI dimensions. In rev-maze, the agent fails to recognize the self-modifying, knight’s-tour-style structure and receives zero across the board. In for-1black0white, the agent partially succeeds by identifying the QR-code generation path but is blocked from final decoding by environment limits, yielding a middling score rather than a simple fail.

These examples clarify the benchmark’s intended use. CTFTiny is not only a set of 50 tasks; it is a compact substrate for measuring how well offensive-security agents reason, plan, execute, and adapt. The paper’s larger methodological claim is that solve rate alone is insufficient for characterizing offensive-security competence. CTFTiny matters because it makes repeated experiments practical, while CTFJudge and CCI make those experiments diagnostically rich.

The benchmark’s compactness is therefore a design choice rather than a concession. It preserves the diversity and difficulty structure of real CTFs, supports rapid iteration, and remains capable of discriminating among models and agent configurations. At the same time, the paper does not present CTFTiny as a replacement for the full NYU CTF Bench in all settings. Its stated role is complementary: fast evaluation during development, with richer interpretation than pass/fail alone, and with enough breadth to expose differences in planning quality, technical execution, and strategic adaptation.