CTF-Forge: Automated CTF Environment Pipeline

• CTF-Forge is an automated pipeline that transforms public CTF artifacts into Dockerized, execution-grounded environments for scalable LLM training and security benchmarks.
• It streamlines configuration by leveraging LLM-driven dependency analysis, automatically generating Dockerfiles and ensuring 98% reproducible builds.
• CTF-Forge accelerates agent training by reducing setup time from hours to seconds, enabling consistent benchmarking across a variety of security challenges.

CTF-Forge is an automated pipeline for transforming public Capture-The-Flag (CTF) artifacts into fully containerized, execution-grounded environments, supporting the scalable training and evaluation of LLM agents and facilitating reproducible security challenge benchmarks. Designed to address the bottleneck of manual challenge environment preparation, CTF-Forge generates Dockerized CTF environments from structured challenge descriptions and artifacts, enabling efficient construction, validation, and deployment for agent training within frameworks such as CTF-Dojo (Zhuo et al., 25 Aug 2025).

1. Architecture and Pipeline Design

CTF-Forge employs a multistage pipeline orchestrated by LLM-driven configuration analysis. The pipeline ingests artifacts—such as challenge descriptions, file manifests, and reproduction instructions—from public sources (e.g., pwn.college repositories). For each challenge, the system automatically:

• Determines challenge type (Web, Binary/Pwn, Crypto, Reverse, Forensics)
• Selects the appropriate base image (e.g., ubuntu:20.04)
• Identifies and installs required runtime and library dependencies (including support for 32-bit binaries on 64-bit systems)
• Declares service setup (creation of Dockerfiles and docker-compose files, with network/service aliasing)
• Generates challenge metadata (e.g., challenge.json describing exposed ports, file access, and flag verification logic)

The containerization process is done fully automatically, using tailored prompts for LLMs to interpret challenge requirements and synthesize configuration scripts. Continuous reliability checks are integrated by running automated validation scripts; the system reported a 98% success rate in environment construction across 658 challenges (Zhuo et al., 25 Aug 2025). The time to build each environment averages 0.5 seconds, transforming weeks of configuration into minutes.

2. Technical Implementation Details

CTF-Forge leverages both deterministic and adaptive LLM inference to generate environment configurations. The generation process includes:

• Dependency Management: Automated installation scripts (apt-get/Yum/Pip/Conda) based on LLM classification of challenge requirements.
• Networked Service Identification: Challenge artifacts are parsed to detect if a network server is required; resulting Docker Compose scripts are produced with DNS-compliant aliases.
• Metadata Extraction: Files required for the challenge are enumerated and exposed. Port assignments, allowed/expected input/output formats, and verification logic are programmatically written into container metadata.
• Reproducibility Assurance: Each generated environment undergoes three independent validation runs; failures trigger prompts to regenerate or human review. The process ensures mathematical determinism for the vast majority of challenges (successful builds: $N_\text{success} = 0.98 \times N_\text{total}$).

This automation extends to complex dependency logic (e.g., dynamic binary linking or compilation), with LLMs adapting scripts as new challenge types emerge.

3. Comparative Performance and Benchmarking

CTF-Forge’s integration with CTF-Dojo yields notable improvements in data-driven agent training:

• Efficiency: Manual configuration for 658 CTF challenges traditionally takes up to an hour each (cumulatively weeks); CTF-Forge compresses this to minutes for batch setup, with only outlier cases requiring expert intervention.
• Quality and Scale: Rigorous build validation yields reproducible, execution-grounded environments suitable for automated feedback-driven agent learning.
• Results: Models fine-tuned on 486 execution-verified message trajectories within CTF-Dojo achieved up to 11.6% absolute gains over strong baselines on three competitive benchmarks (InterCode-CTF, NYU CTF Bench, Cybench). The best open-weight model (32B) reached 31.9% Pass@1, matching proprietary systems (DeepSeek-V3-0324, Gemini-2.5-Flash), despite training on far fewer trajectories.

These outcomes highlight the pivotal role of execution-grounded feedback—enabled by CTF-Forge’s environment construction—in efficient and generalizable agent training.

4. Advantages over Traditional Challenge Preparation

CTF-Forge delivers several key advances:

Aspect	Traditional Setup	CTF-Forge Automated Pipeline
Expert time per env.	~1 hour	0.5 seconds
Scaling	Linear, manual	Massively parallel/automated
Error frequency	Manual config errors	98% reproducible configs
Reproducibility	Variable	Deterministic w/ validation

Automation removes human bottlenecks, minimizing inconsistencies caused by manual scripting of containers and dependency installation. Environment standardization ensures that all supported agents train and evaluate under identical execution conditions, improving benchmarking fairness.

5. Role in Execution-Grounded Agent Learning

CTF-Forge catalyzes a shift toward execution-grounded machine learning in cybersecurity:

• Executable feedback loops: Agents trained within CTF-Dojo receive direct, verifiable interaction signals based on environment runtime behavior rather than static simulation or synthetic scoring. This paradigm is critical for developing practical vulnerability discovery skills and reliable exploit automation.
• Scalable trajectory collection: The platform enables large-scale data acquisition—rehabilitating the scarcity of reproducible training data—which is then leveraged for reinforcement, imitation, or curriculum learning in LLM architectures.
• Benchmarking and reproducibility: Rapid environment provisioning lowers the barrier for live CTF benchmarking, continuous agent evaluation, and rigorous comparisons across security models.

6. Future Implications and Extensions

CTF-Forge’s architecture is extensible beyond CTF training:

• Dynamic live benchmarking: On-the-fly containerization for newly published CTF tasks allows continuous agent evaluation and online model updating.
• Integration with reinforcement learning: Agents can interactively receive partial and dynamic rewards from environment execution, fostering reinforcement learning strategies suited for complex security tasks.
• Cross-domain portability: The modular approach, leveraging public artifact ingestion and LLM-driven environment synthesis, can be adapted to domains such as automated vulnerability testing, rapid security prototyping, and secure system simulation.

A plausible implication is that CTF-Forge will facilitate broader democratization of execution-grounded training resources, supporting open-weight models and collaborative research efforts that are not dependent on proprietary, closed-source infrastructures.

7. Summary

CTF-Forge automates the translation of public CTF artifacts into reproducible, containerized environments at unprecedented scale and speed, enabling execution-grounded learning and benchmarking for advanced LLM cybersecurity agents (Zhuo et al., 25 Aug 2025). This pipeline not only dramatically reduces manual configuration time but also supports efficient agent training and evaluation within platforms like CTF-Dojo, setting new performance standards for open-weight models. Its scalable architecture is poised to advance both research and practice in cybersecurity automation, with extensibility to dynamic benchmarking, reinforcement learning integration, and cross-domain prototyping.