CTF Competency Index (CCI) Overview

• CTF Competency Index (CCI) is a structured measure that evaluates performance in CTF challenges through trajectory analysis and role-based scoring.
• It integrates multiple factors such as vulnerability understanding, exploitation methodology, and adaptability to provide a partial-credit, multidimensional score.
• Methodologies from psychometric testing, process telemetry, and live benchmarking inform its design, ensuring robustness and sensitivity to diverse CTF formats.

The CTF Competency Index (CCI) is a research construct for measuring competence in Capture-the-Flag environments, but current literature does not present it as a single universally standardized instrument. Instead, it appears in at least two closely related forms: as an explicit trajectory-based partial-credit metric for LLM agents solving CTF challenges, and as a broader assessment objective for profiling human or machine performance across challenge categories, reasoning stages, tool use, and operational behavior (Shao et al., 5 Aug 2025). This usage should be distinguished from the unrelated Cybersecurity Concept Inventory (CCI) of the CATS project, which is a concept inventory for a first cybersecurity course and is explicitly “not a certification exam, not a practical skills test, and not a broad workforce-readiness index” (Sherman et al., 2017).

1. Definition and terminological scope

The clearest formal definition of a CTF Competency Index appears in “Towards Effective Offensive Security LLM Agents: Hyperparameter Tuning, LLM as a Judge, and a Lightweight CTF Benchmark,” where the index is defined for an agent trajectory summary $T$ and a human-curated gold solution $G$ as

$$\mathrm{CCI}(T, G)=\sum_{i=1}^{n} w_i F_i(T, G), \qquad \sum_{i=1}^{n} w_i = 1.$$

In the paper’s initial setup, $n=6$, and the six default factors are vulnerability understanding, reconnaissance thoroughness, exploitation methodology, technical accuracy, efficiency of approach, and adaptability; the resulting score lies in $[0,1]$ (Shao et al., 5 Aug 2025). In that formulation, CCI is not a flag-only score: it is a reference-guided competency score comparing an agent’s summarized trajectory against an expert write-up.

Outside that explicit formula, the phrase “CTF Competency Index” is also used more generically to mean a structured profile of CTF-derived competence rather than a single scalar. “CTF for education” argues that CTF performance should not be reduced to one generic score because attack-based, defense-based, Jeopardy, and gamified/wargame formats emphasize different learning objectives and skill profiles; the paper’s strongest conclusion is that combining these formats can help participants build a more holistic cybersecurity knowledge base (Lyu et al., 24 Jan 2026). A plausible implication is that a defensible CCI is usually multidimensional and format-aware, not merely a rank or total points measure.

The acronym requires terminological care. In the CATS literature, CCI means Cybersecurity Concept Inventory, aimed at students “who have recently completed any first course in cybersecurity,” with a focus on conceptual understanding and “especially adversarial thinking” (Sherman et al., 2020). That instrument is methodologically relevant to CTF assessment, but it is not itself a CTF Competency Index.

2. Assessment lineage and methodological foundations

The strongest methodological precursors for a CTF Competency Index come from the CATS project, even though those papers do not define a CTF-specific index. “Creating a Cybersecurity Concept Inventory: A Status Report on the CATS Project” describes a staged assessment-development process: Delphi studies to identify core concepts, think-aloud student interviews to uncover misconceptions, scenario-based multiple-choice drafting, and validation through cognitive interviews, expert reviews, and psychometric testing (Sherman et al., 2017). The top five reconciled concepts were “Identify vulnerabilities and failures,” “Identify attacks against CIA triad and authentication,” “Devise a defense,” “Identify the security goals,” and “Identify potential targets and attackers,” with adversarial thinking identified as “a vital core of cybersecurity” (Sherman et al., 2017).

“Experiences and Lessons Learned Creating and Validating Concept Inventories for Cybersecurity” extends that design logic and reports a 25-item CCI, five items per concept, approximately one hour in length, built for students after a first course in cybersecurity (Sherman et al., 2020). The paper explicitly frames validation as a chain of evidence and reports pilot testing with 142 students from 6 schools, 12 experts, and approximately 7 cognitive interviews, together with early classical test theory evidence that the CCI was “sufficiently reliable” but “too difficult as a whole” (Sherman et al., 2020). For CTF competency measurement, this work contributes a rigorous template: define scope, identify constructs, elicit misconceptions, build scenario-centered items, and validate iteratively.

“The CATS Hackathon: Creating and Refining Test Items for Cybersecurity Concept Inventories” makes the item-construction process more operational. Each test item has three parts—a scenario, a stem, and five answer choices—and the teams used misconception-based distractors, iterative review, and category coverage tracking (Sherman et al., 2019). The paper emphasizes that assessment items should be “easy for experts to answer but hard for students with poor or incomplete conceptual understanding” (Sherman et al., 2019). For a CTF Competency Index, this suggests that competence measurement should not begin from challenge categories alone; it should begin from a construct model, a target population, and observable novice–expert differences.

A plausible implication is that a rigorous CTF Competency Index inherits two principles from this lineage. First, it should separate construct definition from scoring convenience. Second, it should distinguish between conceptual understanding, operational execution, and misconception-driven failure, rather than treating all failed solves as equivalent.

3. Competency dimensions across CTF formats and roles

“CTF for education” provides the clearest direct account of how CTF format shapes the meaning of competence. It divides CTFs into four categories—attack-based CTFs, defense-based CTFs, Jeopardy CTFs, and gamified / wargame CTFs—and explicitly argues that these formats are not interchangeable (Lyu et al., 24 Jan 2026). Attack-based events emphasize workflows such as reconnaissance, scanning and enumeration, selecting an attack path, gaining access, escalating privileges, and stealing or submitting flags. Defense-based events emphasize securing services, maintaining functionality, detecting and responding to outside threats, and balancing security with business or service needs. Jeopardy events emphasize breadth of knowledge and discrete challenge-by-challenge progress. Gamified and wargame formats emphasize realism, exploration, time for information gathering, and contextualized multi-step attack reasoning (Lyu et al., 24 Jan 2026).

CTF format	Direct emphasis	CCI-relevant signal
Attack-based	reconnaissance, exploitation, privilege escalation	offensive discovery and attack efficiency
Defense-based	service hardening, response, availability	operational defense and service continuity
Jeopardy	category breadth, discrete tasks	foundational competence and measurable progress
Gamified / Wargame	realism, exploration, multi-step context	deep understanding and realistic operational skill

This format-sensitive view is reinforced by “Capture the Flag for Team Construction in Cybersecurity,” which operationalizes CTF performance as a participant-by-role score matrix $S$ of size $p \times r$, where each entry $S_{ij}$ is the $i$-th participant’s score in the $j$-th role category (Chang et al., 2022). The role taxonomy comprises Investigation, Design, Analysis, Implementation, Testing and Evaluation, and Coordination. The paper does not define a “CTF Competency Index” explicitly, but it provides a concrete prototype: per-category CTF scores function as sub-index values, participant capacity is the sum of category scores, and team capacity is the sum of participant scores within a team (Chang et al., 2022). Role assignment is then solved with the Hungarian algorithm after transforming scores into costs.

A plausible implication is that some CCI designs should be role-aware rather than merely challenge-aware. In such settings, the index is less a single number than a structured profile of strengths across offensive, defensive, analytical, implementation, and coordination dimensions.

4. Formalizations, signals, and scoring logic

The most explicit multidimensional scoring logic remains the six-factor agent CCI of CTFJudge (Shao et al., 5 Aug 2025). In the reported experiments, the six factors are equally weighted, so the operational CCI is effectively the mean of six judge-assigned scores. The paper’s case studies show the intended dynamic range: the “slithery” case scores 1.0 on every factor, the “maze” case scores 0.0 on every factor, and the “for-1black0white” case receives 0.75 on all six criteria despite failing to retrieve the flag because the trajectory still demonstrates substantial competence (Shao et al., 5 Aug 2025). This is the clearest current example of partial-credit CTF competency measurement.

Other work expands the set of measurable signals beyond final solves. “Battle Ground: Data Collection and Labeling of CTF Games to Understand Human Cyber Operators” introduces keystroke accuracy,

$G$0

where $G$1 is the length of the final submitted command buffer and $G$2 is the total number of keystroke events counted until submission (Savin et al., 2023). The same paper reconstructs commands from raw keystrokes, labels them with MITRE ATT&CK using Pathfinder, and reports that more experienced players issued more commands and had higher mean keystroke accuracy, while team score itself did not show an obvious relation to those behavioral measures (Savin et al., 2023). A plausible implication is that process telemetry can reveal competence dimensions that raw scoreboard outcomes miss.

“A Human Study of Cognitive Biases in Web Application Security” provides a complementary caution against equating flag count with competence. In its Satisfaction of Search study, 6 participants, or 35.3% of the valid sample, were classified into the satisfied group; those exhibiting Satisfaction of Search found 25% fewer flags on average, with a linear mixed-effects result of $G$3, estimated reduction $G$4 flags, and $G$5 (Yang et al., 17 May 2025). The paper’s central warning is that open-ended multi-target CTF score can be materially depressed by stopping behavior and bias, not only by lack of technical skill (Yang et al., 17 May 2025).

“Measuring and Augmenting LLMs for Solving Capture-the-Flag Challenges” isolates a different pair of sub-competencies: technical knowledge and technical knowledge matching/application. It defines two phases of CTF solving—Understanding and Exploiting—and builds CTFKnow with 3,992 questions, comprising 1,996 single-choice items and 1,996 open-ended items (Ji et al., 21 Jun 2025). The paper reports that large models have high recognition-level technical knowledge but much lower scenario-matched application accuracy. For example, GPT-4o reaches 87.83% on single-choice questions, while GPT-4-Turbo reaches 51.90% on the open-ended questions, and the paper’s central conclusion is that models “master the vast majority of technical knowledge encountered in most CTF contexts” yet “exhibit poor capability in matching technical knowledge to specific CTF scenarios” (Ji et al., 21 Jun 2025). For a CCI, this supports separate subscores for knowledge recognition and knowledge application.

A further extension appears in “Towards Improving IDS Using CTF Events,” where the challenge score depends jointly on flag submission and IDS alert burden. The paper defines $G$6 as max points, $G$7 as min points, and $G$8 as steepness, then scores alert burden with

$G$9

In the proof-of-concept event, a perfect stealthy solution corresponded to 500 points, minimum points were 100, and the text describes steepness as 0.2 (Kern et al., 20 Jan 2025). This demonstrates a different CCI design principle: competence may be evaluated not only by whether a challenge is solved, but by how it is solved.

5. Platforms, archives, and benchmark substrates

Persistent platforms and executable benchmarks supply much of the infrastructure on which a CTF Competency Index can be built. “CTF Archive: Capture, Curate, Learn Forever” addresses the educational weakness of ephemeral CTFs by preserving challenges on pwn.college with a VS Code interface, in-browser terminal, optional GUI desktop, and persistent storage (Gupta et al., 1 Dec 2025). Its preservation workflow collects metadata such as event, year, category, and point value; reconstructs original environments; documents deviations in per-challenge REHOST.md; and deploys challenges as containerized services with uniform entry points (Gupta et al., 1 Dec 2025). The paper reports 650+ challenges in narrative sections, 700 available in Table 2, and 7,395 solves across categories including Cryptography, Binary Exploitation, Reverse Engineering, Web Exploitation, Forensics, OSINT, Blockchain, Radio Frequency, Social Engineering, Steganography, and MISC (Gupta et al., 1 Dec 2025). A plausible implication is that a persistent archive with stable metadata and user-linked solve data is a strong substrate for longitudinal CCI modeling.

For AI-agent evaluation, “CTF-Dojo” provides a large executable runtime rather than a persistent archive. It reports 658 CTF-style challenges after decontamination, spanning 50 competitions from 2011–2025, with category counts of 228 Crypto, 38 Forensics, 163 Pwn, 123 Rev, 21 Web, and 85 Misc (Zhuo et al., 25 Aug 2025). CTF-Forge generates Dockerfile, docker-compose.yml, and challenge.json, and the authors report 650 stable and reproducible environments, with the pipeline run 3 independent times and 98% (650) consistently passing validation (Zhuo et al., 25 Aug 2025). This work supports an execution-grounded CCI in which competence is measured in live containers with verifiable feedback.

“CTFusion: A CTF-based Benchmark for LLM Agent Evaluation” pushes the same idea toward contamination-resistant live evaluation. It evaluates agents on LIVE CTFs through a control panel, isolated agent runners, an MCP server, a submission proxy, and CTFd integration (Lee et al., 12 May 2026). Its core technical contribution is preserving per-agent independence under a single team account while forwarding only the first correct flag per challenge to the competition server (Lee et al., 12 May 2026). The paper’s main empirical warning is that static benchmarks can inflate capability: average performance is 14.4% on NYU CTF Bench but 6.3% on Live CTFs (Lee et al., 12 May 2026). For a CCI, this implies that freshness and contamination resistance are validity properties, not implementation details.

A plausible synthesis is that three benchmark substrates now coexist. Archives such as CTF Archive favor persistent practice and longitudinal learner measurement. Executable suites such as CTF-Dojo favor reproducible training and testbed construction. Live systems such as CTFusion favor novelty and contamination resistance. A mature CCI may need all three.

6. Validity, limitations, and emerging directions

The literature is unusually clear that CTF performance is not a transparent measure of underlying competence. The strongest human-side warning is cognitive bias: Satisfaction of Search can reduce observed output without reducing technical capacity (Yang et al., 17 May 2025). The strongest agent-side warning is contamination and benchmark leakage: static reused tasks can overstate capability, especially when agents are given web access or benchmark packaging details permit shortcut retrieval (Lee et al., 12 May 2026). These findings argue against interpreting a single solve-rate number as a complete or context-free CCI.

Challenge design itself also affects validity. “Towards Improving IDS Using CTF Events” shows that an evasion-oriented CTF can reveal real IDS weaknesses, including default configurations that miss an exploit and static detection rules that are bypassed through alternative vulnerabilities such as CVE-2024-45195 rather than the intended CVE-2024-38856 route (Kern et al., 20 Jan 2025). At the same time, the same paper documents measurement threats from challenge opacity, infrastructure inconsistencies, logging gaps, and format unfamiliarity (Kern et al., 20 Jan 2025). A plausible implication is that CCI results should always be interpreted together with challenge-design metadata: what the challenge was intended to elicit, what infrastructure actually exposed, and how much of performance reflects the target competency rather than incidental friction.

A further direction comes from “Beyond the Flag,” which proposes the Ethical-Cognitive Apprenticeship in Cybersecurity (ECAC) framework with five phases—Foundational Modeling, Scaffolding the Arena, Coaching and Articulation, Ethical Dilemma Injections, and Reflective Exploration—for integrating CTFs into K–12 education (Le et al., 18 Feb 2026). That paper does not define a numerical CCI, but it explicitly argues that CTF competence in educational settings includes not only technical skill but also higher-order thinking, collaboration, ethics, reflection, and autonomy/self-efficacy (Le et al., 18 Feb 2026). This suggests that future CCIs, especially for learners rather than autonomous agents, may become more developmental, ethically integrated, and equity-aware.

In current research, then, the CTF Competency Index is best understood not as a settled standard but as an emerging family of assessment models. Its most mature formalization is the six-factor trajectory metric of CTFJudge (Shao et al., 5 Aug 2025). Its strongest methodological roots come from concept-inventory construction and misconception-based educational assessment (Sherman et al., 2017). Its most informative behavioral signals come from telemetry, ATT&CK labeling, keystroke reconstruction, and bias-sensitive process analysis (Savin et al., 2023). Its most credible benchmark substrates are executable and increasingly live (Zhuo et al., 25 Aug 2025, Lee et al., 12 May 2026). Across these lines of work, the common conclusion is consistent: a serious CCI must measure more than whether a flag was captured. It must also measure the competence revealed by reasoning, action, adaptation, context, and the conditions under which success or failure occurred.