picoCTF: Open-Source Cybersecurity Platform
- picoCTF is an open-source educational cybersecurity platform offering varied challenges in areas like cryptography, forensics, web exploitation, and binary exploitation.
- It functions both as a pedagogical tool and a research benchmark, featuring curated challenges with metadata, dynamic flag resolution, and difficulty tiers.
- Recent studies leverage picoCTF to analyze keystroke dynamics, cognitive biases, and autonomous agent performance, providing insights into adversarial decision-making.
Searching arXiv for recent picoCTF-related papers to ground the article.
picoCTF is an open-source educational cybersecurity platform that hosts annual competitions and a practice environment called picoGym. Within that environment, participants solve hands-on challenges and submit flags, typically in the picoCTF{...} format, across categories such as Web Exploitation, Cryptography, Reverse Engineering, Forensics, Binary Exploitation, and General Skills. In recent research, picoCTF has served simultaneously as an instructional platform, a benchmark source for autonomous agents, and a large-scale naturalistic corpus for studying human cyber behavior under uncertainty and time pressure (Carreira et al., 7 Oct 2025).
1. Platform structure and challenge ecology
picoCTF is used in research as a heterogeneous challenge environment rather than as a single fixed benchmark. One line of work treats it as a platform with annual events and the picoGym practice setting; another constructs curated subsets with explicit metadata, solver scripts, and evaluation harnesses. A standardized benchmark derived from picoCTF selected 120 challenges from a platform with over 300 challenges, spanning six categories—General Skills, Cryptography, Web Exploitation, Forensics, Reverse Engineering, and Binary Exploitation—and three difficulty levels: Easy, Medium, and Hard. For those tasks, the benchmark included the challenge description, hints, file download paths, category, difficulty, and a solver function; 104 of the 120 challenges have hints, with 184 total hints. A separate evaluation used 86 published picoCTF challenges across seven categories—Web, Cryptography, Binary Exploitation, Forensics, Reverse Engineering, General Skills, and Blockchain—and three difficulty tiers of 27 Easy, 31 Medium, and 28 Hard. Intercode-CTF, another commonly used end-to-end benchmark, is described as comprising 100 CTF challenges collected from picoCTF (Muzsai et al., 2024, Abdulzada, 14 Jul 2025, Gerard et al., 3 Jul 2026, Ji et al., 21 Jun 2025).
These constructions expose a central feature of picoCTF: it supports both narrowly controlled benchmarking and broader ecological study. The challenge mix ranges from simple command-line reasoning to file analysis, web tasks, reverse engineering, and binary exploitation. In one benchmark design, dynamic flag resolution was treated as especially important because PicoCTF flags can vary by user and over time; solver functions were therefore written to retrieve flags dynamically rather than rely on static stored answers. This suggests that picoCTF is valued not only for topic diversity, but also for benchmark properties that reduce trivial memorization and make live evaluation more meaningful (Muzsai et al., 2024).
2. Instrumenting picoCTF-style play as human behavioral data
Methodological work on picoCTF-style competitions has emphasized that challenge-solving behavior is richer than final scores or flag counts. In an exploratory human-subject experiment with 10 participants from one U.S. university’s CTF team, participants were grouped by years of CTF participation into Beginner (0–1 years), Intermediate (1–3 years), and Advanced (3+ years), with the experience survey explicitly referencing ecosystems such as picoCTF and Hack The Box. The study used a live offense/defense CTF and collected command history from bash with timestamps, keystrokes with logkeys, and screen recordings with recordmydesktop / recordmyscreen. Raw keypresses were then normalized into intended shell commands by simulating shell-buffer behavior, and commands were labeled with MITRE ATT&CK using an interactive tool called Pathfinder, whose hierarchy represented Categories / Tactics, Techniques, and Sub-techniques. The central metric was keystroke accuracy, defined as
where is the total number of keystroke events leading up to a command submission and is the length of the final submitted command buffer (Savin et al., 2023).
The dataset contained 2,994 commands from the 10 participants; across participants, the average number of commands was 299 and the average keystroke accuracy was 69.1%. The paper reported that P1 had the fewest commands (62) and the lowest mean accuracy (52.6%), while P4 had the highest keystroke accuracy (80.1%). By experience group, mean commands increased from 247.2 for Beginners to 324 for Intermediate and 461 for Advanced participants, while mean accuracy rose from 0.6552 to 0.7350 and 0.7625, respectively. The authors argued that, for picoCTF-like competitions, instrumentation at the level of keystrokes, normalized commands, command timing, and ATT&CK labels can reveal how participants search, correct errors, and sequence actions while solving tasks, rather than merely whether they solved them (Savin et al., 2023).
The same study also cautioned against reducing performance to typing efficiency. In the reported team comparison, the higher-scoring team did not simply correspond to higher accuracy or fewer commands: Team 2 scored 3,950 versus 3,146 for Team 1, yet the authors concluded that score outcome is not a simple function of typing accuracy or command count. A plausible implication for picoCTF analysis is that behavioral telemetry may be most informative when combined with strategy, team composition, and temporal patterns rather than used as a standalone proxy for expertise (Savin et al., 2023).
3. picoCTF as a large-scale observatory of attacker cognition
A distinct body of work treats picoCTF not merely as an educational contest, but as a natural laboratory for attacker cognition. Using the picoGym environment, one study analyzed 525,771 submission records from 26,716 unique participants across 378 challenges, spanning September 25, 2020, to April 8, 2024. Each record included fields such as anonymized user ID, timestamp, challenge ID, category, correctness (is_correct), submission text, and demographic metadata where available. The study used a mixed-methods design: expert qualitative coding of challenge instructions, descriptive statistics on submission behavior, a binomial GLM with logit link for misformatted submissions, and a generalized linear mixed-effects logistic model with random intercepts for both user and challenge to model success after repeated failures (Carreira et al., 7 Oct 2025).
The first behavioral signature was interpreted as availability bias. The authors operationalized this as submissions that were semantically correct but syntactically invalid—for example, correct flag content submitted without the required picoCTF{} wrapper. Of the 378 challenges, 60 contained explicit formatting instructions; 57 of those explicitly mentioned the wrapper and 28 required additional transformation. After excluding dynamic-flag cases and challenges with fewer than 50 incorrect submissions, the analysis retained 321 challenges. Within the full set of 378, 165 challenges (44%) had at least one misformatted-but-correct submission. The mean rate of such submissions was about 2%, the median was 0.63%, and 75% of challenges had a rate below about 1.8%. The most extreme rates appeared in Binary Exploitation, where the top two challenges had 38% and 27% misformatted-but-correct submissions. The challenge “2Warm” was used as a concrete illustration: the decoded answer is 101010, the correct submission is picoCTF{101010}, and yet 14% of incorrect submissions were misformatted-but-correct (Carreira et al., 7 Oct 2025).
The second signature was interpreted as the sunk cost fallacy. Descriptively, 48.5% of all submissions were correct on the first attempt; the median number of incorrect attempts per user–challenge pair was 3; and some users exceeded 20 incorrect attempts on the same challenge. In the mixed-effects logistic model, the fixed effect for prior incorrect attempts was , with SE = 0.003, , and p < .001, corresponding to an odds ratio of approximately 0.54 for each additional incorrect attempt. The intercept was , SE = 0.052, p < .001, corresponding to an estimated 76.5% probability of solving a challenge when there had been no prior failures. Random effects showed user ability SD = 0.495 and challenge difficulty SD = 0.974, indicating larger variation across challenges than across users (Carreira et al., 7 Oct 2025).
These results locate picoCTF within a cognitive-security research program. Rather than viewing incorrect submissions as mere noise, the study argues that they encode systematic behavioral tendencies with potential defensive implications. The proposed framework for bias-informed adaptive defenses is organized around Triggers, Sensors, and Defensive mechanisms; examples include decoy files or directories for availability bias and honeypots or appealing but unproductive targets for sunk-cost tendencies. The broader claim is not that picoCTF replicates every aspect of real intrusion behavior, but that its scale, repeated interactions, and structured outcomes make it unusually useful for observing adversarial decision-making at scale (Carreira et al., 7 Oct 2025).
4. picoCTF as a benchmark for autonomous agents
Recent work has made picoCTF a central benchmark for LLM-driven cybersecurity agents, both in live competition settings and in curated benchmark suites.
| Study | picoCTF role | Headline result |
|---|---|---|
| HackSynth (Muzsai et al., 2024) | 120-task benchmark from PicoCTF | GPT-4o solved 41/120 |
| VMS (Abdulzada, 14 Jul 2025) | PicoCTF-based benchmark of 120 challenges | GPT-4o solved 41/120 |
| CTFAgent (Ji et al., 21 Jun 2025) | Intercode-CTF and live picoCTF2024 | 19 challenges, 1,875 points, top 23.6% |
| HexStrike-AI (Gerard et al., 3 Jul 2026) | 86 published picoCTF challenges | 55.4% to 72.0% overall solve rate |
| Random-Crypto (Muzsai et al., 1 Jun 2025) | External transfer benchmark | best Pass@8 = 0.18 vs 0.10 baseline |
In benchmark-style autonomous penetration testing, both HackSynth and VMS use a two-module Planner + Summarizer architecture, execute commands in a containerized Kali Linux environment, and limit each challenge to 20 iterative steps. On a PicoCTF benchmark of 120 challenges, the best-performing model was GPT-4o, which solved 41 challenges; reported runners-up included Llama-3.1-70B with 27 and GPT-4o-mini with 26 in one evaluation. The reported best operating point included temperature = 1.0 and top-p = 0.9, with an observation window of 250 characters for PicoCTF. Binary exploitation remained especially difficult: in one category breakdown, every evaluated model solved 0 out of 4 binary tasks (Muzsai et al., 2024, Abdulzada, 14 Jul 2025).
In a live-competition setting, CTFAgent was evaluated on picoCTF2024, which had 46 challenges and 6,957 valid participants with point values ranging from 25 to 500. CTFAgent solved 19 challenges for 1,875 points, achieving top 23.6%. The NYU CTF framework baseline solved 12 challenges for 925 points, ranking top 47.2%. The category breakdown reported 575 points (7 solved) in Misc, 100 points (1 solved) in Pwn, 200 points (3 solved) in Web, 400 points (4 solved) in Forensics, 300 points (2 solved) in Crypto, and 300 points (2 solved) in Reverse (Ji et al., 21 Jun 2025).
HexStrike-AI used picoCTF in a different way: as a controlled benchmark for diagnosing the determinants of LLM security-tool orchestration. Across 774 trials on 86 published picoCTF challenges, baseline performance was 55.4% overall, with per-configuration solve rates of 78.7% for Claude/Sonnet 4.6 via Claude Code, 59.7% for DeepSeek (deepseek-chat) via RooCode, and 27.9% for DeepSeek (deepseek-chat) via 5ire. After corrections to existing tools, agent-behavior changes, and 11 new tools, the overall solve rate rose to 72.0%, with every configuration improving significantly and the remaining failures characterized as reasoning- or environment-bound rather than missing-tool-bound (Gerard et al., 3 Jul 2026).
5. Knowledge, reasoning, and tool use in picoCTF studies
A recurring theme across picoCTF research is that end-to-end challenge solving decomposes into distinct competencies. One study explicitly framed CTF solving as
To measure the knowledge component, it built CTFKnow from 1,996 final questions derived from 1,084 high-quality CTF write-ups, themselves filtered from over 10,000 write-ups collected from CTFtime. The study found that models often perform well on single-choice questions yet degrade sharply on open-ended ones, and then used picoCTF-linked benchmarks to test whether better knowledge retrieval and environment interaction improve actual solving (Ji et al., 21 Jun 2025).
The resulting CTFAgent addresses those deficiencies with two-stage RAG and Interactive Environmental Augmentation (EA). RAG-Understanding uses vulnerable code snippets as retrieval keys to identify likely vulnerabilities, while RAG-Exploiting retrieves technical knowledge trunks to refine exploit strategy. EA adds interactive commands, tool-use hints for known failure cases, dynamic netcat interaction through start_nc_session(ip, port), nc_send_line(...), and close_nc_session(...), and upgrades the decompiler from Ghidra to IDA Pro 9.1. On Intercode-CTF, the baseline solved 39/100 challenges, while CTFAgent solved 73/100; with o1-preview as the backbone, CTFAgent-o1-preview reached 84/100 (Ji et al., 21 Jun 2025).
A separate line of work studies whether specialized RL on synthetic security tasks transfers to picoCTF. Random-Crypto is a procedurally generated cryptographic CTF dataset used to fine-tune a Python tool-augmented Llama-3.1-8B via GRPO in a secure execution environment. In picoCTF evaluation, the agent used an MCP-based Python REPL exposing execute_python, list_variables, and install_package, with the same tool-augmented prompt format as training and no special prompt adaptation for picoCTF. Each task was attempted 8 times with temperature = 0.7 and nucleus sampling , and performance was reported with Pass@8 and Maj@8. The reported baseline Pass@8 was 0.10; after training, the w/ Hints and Curriculum variants both reached Pass@8 = 0.18, while w/o Hints reached 0.08. Crucially, the paper notes that 95 of the 120 picoCTF challenges include auxiliary artifacts such as ELF binaries, PCAP traces, and PNGs, which the current agent “cannot yet process efficiently” (Muzsai et al., 1 Jun 2025).
HexStrike-AI, by contrast, emphasizes orchestration rather than specialized training. It exposes 150+ external Kali utilities, about 169 MCP tools, and 176 Flask routes through a Flask backend, a FastMCP interface, and execution via MCP, HTTP, and shell. Its main finding in picoCTF evaluation is that the client/harness materially affects performance: holding the DeepSeek model constant, solve rate differed by a reported 2.1× between RooCode (59.7%) and 5ire (27.9%), with two-proportion z-test significance of (Gerard et al., 3 Jul 2026).
6. Scope, interpretive limits, and research significance
The research record uses picoCTF in at least three distinct senses: as a pedagogical platform, as a benchmark source, and as a behavioral observatory. Those uses overlap, but they are not interchangeable. Several papers are explicit that picoCTF is a proxy for offensive-security work rather than a direct measure of real-world end-to-end performance. One study states that picoCTF is a sanctioned educational platform, so its results do not establish end-to-end performance on real engagements; another benchmark excludes challenges requiring personalized instances because creating them requires substantial resources and involves CAPTCHAs; still another shows that “HexStrike-only” tool constraints were only soft prompt constraints, with substantial native tool leakage of 2,908 native execute_command calls and 717 Bash calls (Gerard et al., 3 Jul 2026, Muzsai et al., 2024).
The limitations of human-behavior studies are also clear. The keystroke-instrumentation study was a methodological pilot with only 10 participants, no baseline typing profile outside the CTF, no think-aloud or interview-based validation of intent, incomplete ATT&CK labeling, and no MITRE D3FEND labeling. The cognitive-bias study did not compute inter-rater reliability in its coding stage, relying instead on expert consensus. These are not incidental caveats: they define the evidentiary status of current claims about cyber-operator expertise, frustration, fatigue, or bias in picoCTF-style settings (Savin et al., 2023, Carreira et al., 7 Oct 2025).
Within those limits, a consistent picture emerges. picoCTF supports structured correctness checks, repeated interaction, diverse task formats, and difficulty variation, making it unusually suitable for both measurement and intervention. Human-centered studies use it to analyze keystrokes, commands, flag-formatting errors, and persistence across failures. Agent-centered studies use it to isolate the effects of retrieval, tool use, planning loops, harness design, and RL fine-tuning. The combined literature suggests that picoCTF is valuable not because it collapses cybersecurity skill into a single score, but because it exposes the intermediate processes—reasoning, tool selection, error correction, and strategic persistence—through which both humans and autonomous systems approach cyber tasks (Ji et al., 21 Jun 2025, Muzsai et al., 1 Jun 2025, Gerard et al., 3 Jul 2026).