CTF-Dojo-32B: Open-Weight CTF Security Agent

• CTF-Dojo-32B is a 32-billion-parameter model trained on verified CTF trajectories to identify vulnerabilities in containerized challenges.
• The model employs the Qwen3-32B decoder-only architecture and the CTF-Forge pipeline for automated Dockerization of over 650 reproducible CTF tasks.
• Empirical results reveal that CTF-Dojo-32B outperforms its zero-shot baseline and competes with proprietary models, achieving up to 31.9% Pass@1 on diverse benchmarks.

CTF-Dojo-32B is a 32-billion-parameter open-weight LLM agent developed for vulnerability discovery and automated offensive security reasoning within large-scale, executable Capture-The-Flag (CTF) environments. Architecturally founded on the Qwen3-32B decoder-only transformer, CTF-Dojo-32B is fine-tuned via lightweight supervised learning on execution-verified trajectories. The system is trained and evaluated within “CTF-Dojo,” an environment comprising 658 containerized, real-world CTF challenges with guaranteed reproducibility. Empirical results establish CTF-Dojo-32B as a new open-weight state-of-the-art for automated CTF solving, with capabilities competitive against leading proprietary models such as DeepSeek-V3-0324 and Gemini-2.5-Flash, despite relying on a relatively small number of trajectories and limited computational resources (Zhuo et al., 25 Aug 2025).

1. Model Architecture and Fine-Tuning Environment

CTF-Dojo-32B is derived from the Qwen3-32B backbone (Yang et al., 2025), itself a decoder-only transformer with 32 billion parameters. No modifications were made to the original transformer architecture; specialization was achieved through supervised fine-tuning using execution-grounded data.

The training and evaluation proceeded within the CTF-Dojo framework, comprising 658 CTF-style tasks containerized through Docker. Each challenge operates in an isolated environment with dedicated filesystems, network ports (e.g., 1337 for "pwn" tasks, 80/8080 for web), and verifiable flag-checking endpoints (via “/flag” or “flagCheck” binaries). The supporting “CTF-Forge” pipeline automates environment generation from artifacts, producing Dockerfiles, orchestrating via docker-compose, and managing challenge metadata for agent interaction. Manual validation demonstrated that 650 out of 658 (98%) environments built and ran successfully without human intervention.

2. Data Sources and Supervised Fine-Tuning Regime

The fine-tuning dataset comprises only successful trajectories—full sequences of agent actions that culminate in capturing the true flag. These trajectories were harvested from teacher models (DeepSeek-V3-0324 and Qwen3-Coder-32B) through a generative rollout process, where models were sampled with temperature 0.6 and top-p 0.95, augmented by bag-of-tricks from public CTF writeups with redacted content and runtime perturbations (port shuffling, distractor files). The full rollout corpus yielded 1,006 raw successful sequences; sample capping and de-duplication reduced this to 486 high-quality, execution-verified trajectories, distributed over 274 unique challenges.

The supervised fine-tuning leveraged the NVIDIA NeMo framework with the following hyperparameters: global batch size 16, learning rate $5 \times 10^{-6}$, 2 epochs, and sequence truncation at 32,768 tokens. Training adopted rejection-sampling supervised fine-tuning, whereby only outputs verified to solve the challenge (i.e., capture the flag) were included, optimizing

$$L(\theta) = -\sum_{(x, y) \in D} \log p_\theta(y|x).$$

All command reasoning, as well as tool output (stdout, stderr), was recorded during agent-environment interactions, supporting verifiable data lineage.

3. Evaluation Metrics and Empirical Results

Task performance was measured using Pass@k, following the definition from Chen et al. (2021):

$\text{Pass@k} = 1 - \frac{C(n - s, k)}{C(n, k)},$

where $n$ is the number of outputs and $s$ the number of correct solutions; for single-sample evaluation ($n = 1$), Pass@1 reduces to the fraction of tasks solved by the first rollout.

Evaluation proceeded on three external benchmark suites: InterCode-CTF (91 tasks), NYU CTF Bench (192 tasks), and Cybench (40 tasks). Results are summarized below:

Model	InterCode-CTF	NYU CTF	Cybench	Avg
Qwen3-32B (zero-shot)	60.0%	4.7%	5.0%	20.3%
CTF-Dojo-32B	83.5%	10.4%	17.5%	31.9%

CTF-Dojo-32B outperforms its zero-shot Qwen3-32B baseline by absolute margins of +23.5 percentage points on InterCode-CTF, +5.7 pp on NYU, and +12.5 pp on Cybench. The average improvement is +11.6 pp. Notably, CTF-Dojo-32B achieves a 31.9% Pass@1 average with only 486 trajectories, closing the gap to leading proprietary models (DeepSeek-V3-0324 at 30.3%, Claude-3.5 at 37.2%) trained on orders-of-magnitude more data (Zhuo et al., 25 Aug 2025).

4. Qualitative Analysis, Failure Modes, and Scalability

CTF-Dojo-32B demonstrates the capacity for advanced vulnerability reasoning, as illustrated by various challenge domains:

• In ECTF 2014 “Lowkey” (Crypto), the agent performed integer factorization, computed RSA private keys, decrypted sequences, and identified ASCII-encoded flags, incidentally revealing a flag error in the source repository.
• In HSCTF 2019 “Hidden Flag” (Misc), the agent correctly identified steganographic encoding in a PNG-named file, applied XOR decryption, reconstructed and detected the presence of a flag through OCR.

Observed failure modes include difficulties with brittle gadget-style exploits, failures due to incorrect interpretation of environment cues, and overfitting to static challenge templates. Augmentation (runtime perturbations, distractors) improved generalizability, yielding up to +24.9% relatively more solved tasks over four evaluation rollouts.

The finite size of the CTF-Dojo corpus (658 challenges) constrains exposure to novel exploit types, limiting transfer to unencountered vulnerability patterns. However, the CTF-Forge infrastructure supports organic extensibility, continuously integrating new artifacts from the pwn.college archive and supporting near real-time Docker environment synthesis at sub-second per-task latency.

5. Security Applications and Practical Impact

The CTF-Dojo-32B system extends the role of LLMs into executable-agent learning for offensive security, transcending static code analysis or synthetic penetration test scenarios. By training on execution-verified feedback within realistic, containerized challenges, the agent aligns more closely with the practical workflows of human security researchers.

A practical implication is the agent’s applicability to automated penetration testing: CTF-Dojo-32B can be deployed to discover buffer overflows, weak cryptographic implementations, and injection vulnerabilities in binary and web applications. The reproducibility and scaling enabled by CTF-Forge further position the broader CTF-Dojo framework as a foundation for high-throughput vulnerability research.

6. Key Contributions and Future Directions

CTF-Dojo-32B’s primary contributions include (1) the CTF-Forge pipeline for rapid, automated Dockerization of CTF tasks; (2) a high-fidelity trajectory collection protocol based on strict execution-grounding; and (3) ablation studies isolating the effects of writeup augmentation, inference-time perturbations, and diversity in teacher model selection.

Looking forward, extension directions include deploying a live “CTF in the Wild” benchmark continuously updated from new competition artifacts, exploring reinforcement-learning protocols with partial rewards (e.g., memory-safety signal integration), scaling to larger model backbones, and investigation into multi-agent red-team/blue-team setups. These advances are expected to progress towards fully autonomous, reliable, high-performance security-analysis agents (Zhuo et al., 25 Aug 2025).