---
title: InterCode-CTF Benchmark
url: https://www.emergentmind.com/topics/intercode-ctf-benchmark
type: topic
---

# InterCode-CTF Benchmark

InterCode-CTF is a widely adopted benchmark designed to evaluate the capability of automated agents and language models to solve multi-step, interactive Capture-The-Flag (CTF) cybersecurity challenges within a reproducible, execution-driven environment. Derived from picoCTF—a high-school and undergraduate-level CTF competition—InterCode-CTF offers a standardized, container-based suite of tasks, enabling rigorous assessment of agent planning, tool use, vulnerability exploitation, and iterative problem solving. The benchmark has become a de facto yardstick for offensive security agents and is integral to numerous recent advances in agent-driven code intelligence and security research.

## 1. Benchmark Composition and Task Structure

InterCode-CTF consists of a curated set of CTF challenges sourced from the picoCTF archive, with the selection designed to balance category coverage and reproducibility. The canonical instantiations comprise between 85 and 100 tasks, after filtering out instances requiring vision, external internet access, or with broken containers [2412.02776].

Category distribution across the main variants:

| Category             | # Tasks (typical)   | Description                                     |
|----------------------|---------------------|-------------------------------------------------|
| Crypto               | 16–19               | Cryptography, including RSA, group theory       |
| Forensics            | 13–15               | File carving, steganography, network analysis   |
| Binary Exploitation  | 2–4                 | "Pwn" tasks—buffer overflows, stack exploits    |
| Reverse Engineering  | 27                  | ELF analysis, static/dynamic inspection         |
| Web Exploitation     | 2                   | Simple webserver flaws, network endpoints       |
| Miscellaneous        | 31–33               | Scripting, general skills, logic puzzles        |

Each challenge is packaged as an isolated Docker container, including the following components [2306.14898, 2409.16165, 2508.18370]:

- **Problem Statement**: Natural-language instructions describing the task and desired goal (typically “find and submit the flag”).
- **Artifacts**: Binaries, scripts, data files (images, PCAPs), or remote access endpoints.
- **Execution Environment**: Pre-installed Linux utilities (e.g., gdb, binwalk, tshark), scripting languages, and occasionally category-specific tools (e.g., RsaCtfTool for Crypto).
- **Flag**: A hidden “golden” string; static or dynamically generated, submission required for task completion.
- **Interaction Scaffold**: Agents interact via pre-defined commands (e.g., `ls`, `decompile`, `debug_start`, `connect_start`), a controlled action interface (bash/Python shell), and can submit the extracted flag for verification.

Tasks are capped at a fixed number of interaction turns (commonly 30–40), and category assignments follow the original picoCTF taxonomy [2412.02776, 2306.14898].

## 2. Formal Environment Specification

InterCode-CTF models each challenge as a partially observable Markov Decision Process (POMDP) [2306.14898]:

- **State Space** ($\mathcal{S}$): Full container filesystem and process state, plus a flag-discovery indicator.
- **Action Space** ($\mathcal{A}$): Admissible shell or Python commands and the flag submission action; each action must be syntactically valid.
- **Observation Space** ($\mathcal{O}$): Pair $(\texttt{stdout},\,\Delta\texttt{fs})$—command output and record of filesystem mutations.
- **Transition Function** ($\mathcal{T}$): Deterministic application of agent commands in the containerized OS context.
- **Reward Function** ($\mathcal{R}$): Sparse (+1 for correct flag submission), with optional negative rewards for invalid commands and shaped rewards for uncovering subflags.
- **Episode Termination**: On correct flag submission or after exceeding the turn budget.

This formalization supports reinforcement learning (RL), imitation learning, as well as scripted and prompt-based agent strategies [2306.14898].

## 3. Evaluation Protocols and Metrics

The primary metric is Pass@1 (success rate on the first attempt), where a challenge is considered solved if and only if the agent submits exactly the golden flag during its trajectory [2409.16165, 2508.18370, 2412.02776]. Extensions include Pass@k for multiple attempts with environment resets.

Mathematically, for $N$ challenges:

$$
\mathrm{Pass@1} = \frac{1}{N}\sum_{i=1}^N s_i
$$

where $s_i=1$ if flag successfully submitted on challenge $i$, $0$ otherwise. For $k$ independent attempts, the per-task probability is $P_i(k) = 1 - [1 - P_i(1)]^k$.

Secondary metrics include:

- **Average Steps to Flag**: Mean number of actions until flag submission.
- **Error Rate**: Fraction of non-admissible (syntax-error) commands.
- **Category Breakdown**: Per-category Pass@1, revealing agent strengths/weaknesses.
- **Granular Status Codes**: Success, budget exhausted, context overflow, forfeit, error [2409.16165].

Constraints are imposed via strict generation budgets ($\leq\$3$ per instance in some studies), fixed turn limits, and tool usage restrictions. Reproducibility is ensured by deterministic Docker images and orchestrated evaluation scripts [2306.14898, 2409.16165].

## 4. Baseline Agent Architectures and Comparative Results

Multiple agent paradigms have been evaluated on InterCode-CTF:

- **Zero-shot Prompting**: Single-pass, static prompt generation; poor performance (25–47% Pass@1) [2306.14898, 2412.02776].
- **Iterative/Chain-of-Thought (ReAct)**: Alternating “thought” and “action” steps, using agent-environment stepwise feedback. Significant improvement up to 83% Pass@1 [2412.02776].
- **Plan-and-Solve**: High-level planning followed by sequential execution; intermediate improvement (65% Pass@1) [2412.02776].
- **ReAct→Plan**: Interleaving ReAct with planning step, further enhanced with strong LLMs for replanning, peaks at 89% Pass@1 on one attempt and 95% across five attempts [2412.02776].
- **EnIGMA**: Custom interactive tools (debugger, connect) and LM-driven summarization modules; performances up to 72% Pass@1 on full 100-task suite [2409.16165].
- **CTF-Dojo/Cyber-Zero**: LLM-based agents fine-tuned on execution-grounded or synthetic trajectory datasets, reaching 83.5% (CTF-Dojo-32B) and 82.4% (Cyber-Zero-32B) [2508.18370, 2508.00910].

Notable aggregate scores:

| Agent                            | Pass@1 (%) | Attempt Spec. | Reference       |
|-----------------------------------|------------|---------------|-----------------|
| InterCode (2023, zero-shot)       | 25–40      | @1            | [2306.14898, 2412.02776] |
| EnIGMA (GPT-4o)                   | 69         | @1            | [2409.16165]    |
| EnIGMA (GPT-4 Turbo)              | 72         | @1            | [2409.16165]    |
| ReActPlan (GPT-4o, o1-preview)    | 89         | @1            | [2412.02776]    |
| ReActPlan                         | 95         | @5            | [2412.02776]    |
| CTF-Dojo-32B                      | 83.5       | @1            | [2508.18370]    |
| Cyber-Zero-32B                    | 82.4       | @1            | [2508.00910]    |
| DeepSeek-V3-0324 (zero-shot)      | 82.5       | @1            | [2508.18370]    |

Category-wise performance reveals strong agents (ReActPlan@5) attain 100% on general skills and web, ~96% on reverse engineering, and >90% on cryptography and forensics, with only vision-based or internet-dependent tasks forming persistent failures [2412.02776].

## 5. Technical Insights and Observed Failure Modes

Empirical studies highlight several key determinants of performance:

- **Active Tool Use**: Integrating category-specific binaries and debuggers is critical; omitting these tools drops solve rate by ~2.5 percentage points overall, with cryptography and binary exploitation most impacted [2409.16165].
- **Summarization**: LM-driven output summarizers outperform naive or no summarization, preventing context overflow and increasing success [2409.16165].
- **Trajectory Length and Recovery**: Long-horizon, multi-turn interactions (64.8% Pass@1) outperform single-turn demonstrations (25.3%), primarily by decreasing stuck-in-loop rates (11.1% vs. 73.5%) [2508.00910].
- **Multiple Independent Attempts**: Allowing $k>1$ attempts (with resets) enables near-saturation (95%) by correcting for action mis-ranking and exploration variance.

However, InterCode-CTF exposes several open limitations:

- **Data leakage**: A nontrivial proportion of flags appear to be directly memorized by some foundation models (e.g., 14% of Claude 3.5 Sonnet runs), undermining benchmark validity [2409.16165, 2412.02776].
- **Soliloquizing**: Models sometimes fabricate non-existent observations in absence of environmental cues [2409.16165].
- **Vision and Networking Gaps**: Tasks requiring image analysis or browser/HTTP APIs remain unsolved by most agents [2412.02776].
- **Population Memorization**: Simple “blind submission” strategies achieved 10% solves, suggesting benchmark contamination or overfitting [2412.02776].

These findings motivate the creation of future, harder benchmarks with private or obfuscated challenge sets, integrated web/vision interfaces, and stricter data curation.

## 6. Impact and Research Significance

InterCode-CTF has become a central fixture for agent-based cybersecurity research, AI4Sec competitions, and LLM evaluations. Key impacts include:

- **Standardization**: Provides a reproducible, extensible RL-style environment with support for new CTF challenges, tool augmentations, and reward shaping [2306.14898].
- **Innovation Driver**: Enabled the development and validation of execution-grounded agent learning methodologies, fine-tuning strategies such as CTF-Dojo and Cyber-Zero, and detailed analysis of tool/plan chaining effects [2508.18370, 2508.00910].
- **Curriculum and Sensitivity Studies**: The structure inspired benchmarks like CTF-Code, which targets sensitivity to problem detail via counterfactual perturbations, and CTF-Instruct, which enhances LLM generalization and robustness [2505.14597].

Limitations include its “high school” challenge level—now saturated by plain LLM agents using modest prompting and tool selection—necessitating more sophisticated future benchmarks (e.g., NYU CTF Bench, HackTheBox) to track continued advances [2412.02776]. Nevertheless, InterCode-CTF remains the reference suite for diagnostic, ablation, and transfer learning studies on interactive exploit discovery and agent robustness.

## 7. Extensibility, Best Practices, and Future Directions

The architecture of InterCode-CTF supports easy addition of new challenges via Docker image and dataset extension, reward/observation augmentation, and custom agent-computer interfaces [2306.14898]. Recommendations for benchmark evolution, drawn from empirical studies, include:

- Broadening ACI coverage (browser, HTTP, database tools) for expanded challenge domains [2409.16165].
- Strengthening privacy/obfuscation measures to prevent model contamination and leakage.
- Structuring multi-stage dependencies and cross-challenge memory to test long-horizon reasoning and generalization.
- Deploying benchmarking infrastructure to support continuous scoreboard updates and per-category analyses as new models and strategies emerge.

In sum, InterCode-CTF continues to influence both methodological advances in security-oriented LLMs and the broader design of reproducible, execution-driven agent benchmarks in applied machine learning and cybersecurity research [2306.14898, 2409.16165, 2412.02776, 2508.18370].

Source: https://www.emergentmind.com/topics/intercode-ctf-benchmark