---
title: CTF Competency Index (CCI) Overview
url: https://www.emergentmind.com/topics/ctf-competency-index-cci
type: topic
---

# CTF Competency Index (CCI) Overview

The **CTF Competency Index (CCI)** is a research construct for measuring competence in Capture-the-Flag environments, but current literature does not present it as a single universally standardized instrument. Instead, it appears in at least two closely related forms: as an explicit **trajectory-based partial-credit metric** for LLM agents solving CTF challenges, and as a broader assessment objective for profiling human or machine performance across challenge categories, reasoning stages, tool use, and operational behavior [2508.05674]. This usage should be distinguished from the unrelated **Cybersecurity Concept Inventory (CCI)** of the CATS project, which is a concept inventory for a first cybersecurity course and is explicitly “not a certification exam, not a practical skills test, and not a broad workforce-readiness index” [1706.05092].

## 1. Definition and terminological scope

The clearest formal definition of a CTF Competency Index appears in “Towards Effective Offensive Security LLM Agents: Hyperparameter Tuning, LLM as a Judge, and a Lightweight CTF Benchmark,” where the index is defined for an agent trajectory summary \(T\) and a human-curated gold solution \(G\) as
\[
\mathrm{CCI}(T, G)=\sum_{i=1}^{n} w_i F_i(T, G), \qquad \sum_{i=1}^{n} w_i = 1.
\]
In the paper’s initial setup, \(n=6\), and the six default factors are **vulnerability understanding**, **reconnaissance thoroughness**, **exploitation methodology**, **technical accuracy**, **efficiency of approach**, and **adaptability**; the resulting score lies in \([0,1]\) [2508.05674]. In that formulation, CCI is not a flag-only score: it is a **reference-guided competency score** comparing an agent’s summarized trajectory against an expert write-up.

Outside that explicit formula, the phrase “CTF Competency Index” is also used more generically to mean a structured profile of CTF-derived competence rather than a single scalar. “CTF for education” argues that CTF performance should not be reduced to one generic score because **attack-based**, **defense-based**, **Jeopardy**, and **gamified/wargame** formats emphasize different learning objectives and skill profiles; the paper’s strongest conclusion is that combining these formats can help participants build a more holistic cybersecurity knowledge base [2601.17543]. A plausible implication is that a defensible CCI is usually **multidimensional and format-aware**, not merely a rank or total points measure.

The acronym requires terminological care. In the CATS literature, **CCI** means **Cybersecurity Concept Inventory**, aimed at students “who have recently completed any first course in cybersecurity,” with a focus on conceptual understanding and “especially adversarial thinking” [2004.05248]. That instrument is methodologically relevant to CTF assessment, but it is not itself a CTF Competency Index.

## 2. Assessment lineage and methodological foundations

The strongest methodological precursors for a CTF Competency Index come from the CATS project, even though those papers do not define a CTF-specific index. “Creating a Cybersecurity Concept Inventory: A Status Report on the CATS Project” describes a staged assessment-development process: Delphi studies to identify core concepts, think-aloud student interviews to uncover misconceptions, scenario-based multiple-choice drafting, and validation through **cognitive interviews, expert reviews, and psychometric testing** [1706.05092]. The top five reconciled concepts were “Identify vulnerabilities and failures,” “Identify attacks against CIA triad and authentication,” “Devise a defense,” “Identify the security goals,” and “Identify potential targets and attackers,” with adversarial thinking identified as “a vital core of cybersecurity” [1706.05092].

“Experiences and Lessons Learned Creating and Validating Concept Inventories for Cybersecurity” extends that design logic and reports a 25-item CCI, five items per concept, approximately one hour in length, built for students after a first course in cybersecurity [2004.05248]. The paper explicitly frames validation as a chain of evidence and reports pilot testing with **142 students** from **6 schools**, **12 experts**, and approximately **7 cognitive interviews**, together with early classical test theory evidence that the CCI was “sufficiently reliable” but “too difficult as a whole” [2004.05248]. For CTF competency measurement, this work contributes a rigorous template: define scope, identify constructs, elicit misconceptions, build scenario-centered items, and validate iteratively.

“The CATS Hackathon: Creating and Refining Test Items for Cybersecurity Concept Inventories” makes the item-construction process more operational. Each test item has three parts—a **scenario**, a **stem**, and **five answer choices**—and the teams used misconception-based distractors, iterative review, and category coverage tracking [1901.09286]. The paper emphasizes that assessment items should be “easy for experts to answer but hard for students with poor or incomplete conceptual understanding” [1901.09286]. For a CTF Competency Index, this suggests that competence measurement should not begin from challenge categories alone; it should begin from a construct model, a target population, and observable novice–expert differences.

A plausible implication is that a rigorous CTF Competency Index inherits two principles from this lineage. First, it should separate **construct definition** from **scoring convenience**. Second, it should distinguish between **conceptual understanding**, **operational execution**, and **misconception-driven failure**, rather than treating all failed solves as equivalent.

## 3. Competency dimensions across CTF formats and roles

“CTF for education” provides the clearest direct account of how CTF format shapes the meaning of competence. It divides CTFs into four categories—**attack-based CTFs**, **defense-based CTFs**, **Jeopardy CTFs**, and **gamified / wargame CTFs**—and explicitly argues that these formats are not interchangeable [2601.17543]. Attack-based events emphasize workflows such as reconnaissance, scanning and enumeration, selecting an attack path, gaining access, escalating privileges, and stealing or submitting flags. Defense-based events emphasize securing services, maintaining functionality, detecting and responding to outside threats, and balancing security with business or service needs. Jeopardy events emphasize breadth of knowledge and discrete challenge-by-challenge progress. Gamified and wargame formats emphasize realism, exploration, time for information gathering, and contextualized multi-step attack reasoning [2601.17543].

| CTF format | Direct emphasis | CCI-relevant signal |
|---|---|---|
| Attack-based | reconnaissance, exploitation, privilege escalation | offensive discovery and attack efficiency |
| Defense-based | service hardening, response, availability | operational defense and service continuity |
| Jeopardy | category breadth, discrete tasks | foundational competence and measurable progress |
| Gamified / Wargame | realism, exploration, multi-step context | deep understanding and realistic operational skill |

This format-sensitive view is reinforced by “Capture the Flag for Team Construction in Cybersecurity,” which operationalizes CTF performance as a participant-by-role score matrix \(S\) of size \(p \times r\), where each entry \(S_{ij}\) is the \(i\)-th participant’s score in the \(j\)-th role category [2206.08971]. The role taxonomy comprises **Investigation**, **Design**, **Analysis**, **Implementation**, **Testing and Evaluation**, and **Coordination**. The paper does not define a “CTF Competency Index” explicitly, but it provides a concrete prototype: per-category CTF scores function as sub-index values, participant capacity is the sum of category scores, and team capacity is the sum of participant scores within a team [2206.08971]. Role assignment is then solved with the Hungarian algorithm after transforming scores into costs.

A plausible implication is that some CCI designs should be **role-aware** rather than merely challenge-aware. In such settings, the index is less a single number than a structured profile of strengths across offensive, defensive, analytical, implementation, and coordination dimensions.

## 4. Formalizations, signals, and scoring logic

The most explicit multidimensional scoring logic remains the six-factor agent CCI of CTFJudge [2508.05674]. In the reported experiments, the six factors are equally weighted, so the operational CCI is effectively the mean of six judge-assigned scores. The paper’s case studies show the intended dynamic range: the “slithery” case scores **1.0** on every factor, the “maze” case scores **0.0** on every factor, and the “for-1black0white” case receives **0.75** on all six criteria despite failing to retrieve the flag because the trajectory still demonstrates substantial competence [2508.05674]. This is the clearest current example of **partial-credit CTF competency measurement**.

Other work expands the set of measurable signals beyond final solves. “Battle Ground: Data Collection and Labeling of CTF Games to Understand Human Cyber Operators” introduces **keystroke accuracy**,
\[
A = \frac{K}{C},
\]
where \(K\) is the length of the final submitted command buffer and \(C\) is the total number of keystroke events counted until submission [2307.10877]. The same paper reconstructs commands from raw keystrokes, labels them with MITRE ATT&CK using Pathfinder, and reports that more experienced players issued more commands and had higher mean keystroke accuracy, while team score itself did not show an obvious relation to those behavioral measures [2307.10877]. A plausible implication is that process telemetry can reveal competence dimensions that raw scoreboard outcomes miss.

“A Human Study of Cognitive Biases in Web Application Security” provides a complementary caution against equating flag count with competence. In its Satisfaction of Search study, **6 participants**, or **35.3%** of the valid sample, were classified into the satisfied group; those exhibiting Satisfaction of Search found **25% fewer flags on average**, with a linear mixed-effects result of \(t=-2.413\), estimated reduction \(=2\) flags, and \(p=0.0291\) [2505.12018]. The paper’s central warning is that open-ended multi-target CTF score can be materially depressed by stopping behavior and bias, not only by lack of technical skill [2505.12018].

“Measuring and Augmenting Large Language Models for Solving Capture-the-Flag Challenges” isolates a different pair of sub-competencies: **technical knowledge** and **technical knowledge matching/application**. It defines two phases of CTF solving—**Understanding** and **Exploiting**—and builds **CTFKnow** with **3,992 questions**, comprising **1,996 single-choice** items and **1,996 open-ended** items [2506.17644]. The paper reports that large models have high recognition-level technical knowledge but much lower scenario-matched application accuracy. For example, GPT-4o reaches **87.83%** on single-choice questions, while GPT-4-Turbo reaches **51.90%** on the open-ended questions, and the paper’s central conclusion is that models “master the vast majority of technical knowledge encountered in most CTF contexts” yet “exhibit poor capability in matching technical knowledge to specific CTF scenarios” [2506.17644]. For a CCI, this supports separate subscores for **knowledge recognition** and **knowledge application**.

A further extension appears in “Towards Improving IDS Using CTF Events,” where the challenge score depends jointly on flag submission and IDS alert burden. The paper defines \(a\) as max points, \(b\) as min points, and \(s\) as steepness, then scores alert burden with
\[
f(x) = \max(b, a - (s * \ln(x)) * (a - b)).
\]
In the proof-of-concept event, a perfect stealthy solution corresponded to **500 points**, minimum points were **100**, and the text describes steepness as **0.2** [2501.11685]. This demonstrates a different CCI design principle: competence may be evaluated not only by whether a challenge is solved, but by **how** it is solved.

## 5. Platforms, archives, and benchmark substrates

Persistent platforms and executable benchmarks supply much of the infrastructure on which a CTF Competency Index can be built. “CTF Archive: Capture, Curate, Learn Forever” addresses the educational weakness of ephemeral CTFs by preserving challenges on **pwn.college** with a **VS Code interface**, **in-browser terminal**, optional **GUI desktop**, and **persistent storage** [2512.01233]. Its preservation workflow collects metadata such as event, year, category, and point value; reconstructs original environments; documents deviations in per-challenge `REHOST.md`; and deploys challenges as **containerized services** with uniform entry points [2512.01233]. The paper reports **650+ challenges** in narrative sections, **700 available** in Table 2, and **7,395 solves** across categories including Cryptography, Binary Exploitation, Reverse Engineering, Web Exploitation, Forensics, OSINT, Blockchain, Radio Frequency, Social Engineering, Steganography, and MISC [2512.01233]. A plausible implication is that a persistent archive with stable metadata and user-linked solve data is a strong substrate for longitudinal CCI modeling.

For AI-agent evaluation, “CTF-Dojo” provides a large executable runtime rather than a persistent archive. It reports **658** CTF-style challenges after decontamination, spanning **50 competitions** from **2011–2025**, with category counts of **228 Crypto**, **38 Forensics**, **163 Pwn**, **123 Rev**, **21 Web**, and **85 Misc** [2508.18370]. CTF-Forge generates `Dockerfile`, `docker-compose.yml`, and `challenge.json`, and the authors report **650** stable and reproducible environments, with the pipeline run **3 independent times** and **98% (650)** consistently passing validation [2508.18370]. This work supports an execution-grounded CCI in which competence is measured in live containers with verifiable feedback.

“CTFusion: A CTF-based Benchmark for LLM Agent Evaluation” pushes the same idea toward contamination-resistant **live** evaluation. It evaluates agents on **LIVE CTFs** through a control panel, isolated agent runners, an MCP server, a submission proxy, and CTFd integration [2605.11504]. Its core technical contribution is preserving per-agent independence under a single team account while forwarding only the first correct flag per challenge to the competition server [2605.11504]. The paper’s main empirical warning is that static benchmarks can inflate capability: average performance is **14.4%** on **NYU CTF Bench** but **6.3%** on **Live CTFs** [2605.11504]. For a CCI, this implies that **freshness** and **contamination resistance** are validity properties, not implementation details.

A plausible synthesis is that three benchmark substrates now coexist. Archives such as CTF Archive favor **persistent practice and longitudinal learner measurement**. Executable suites such as CTF-Dojo favor **reproducible training and testbed construction**. Live systems such as CTFusion favor **novelty and contamination resistance**. A mature CCI may need all three.

## 6. Validity, limitations, and emerging directions

The literature is unusually clear that CTF performance is not a transparent measure of underlying competence. The strongest human-side warning is cognitive bias: Satisfaction of Search can reduce observed output without reducing technical capacity [2505.12018]. The strongest agent-side warning is contamination and benchmark leakage: static reused tasks can overstate capability, especially when agents are given web access or benchmark packaging details permit shortcut retrieval [2605.11504]. These findings argue against interpreting a single solve-rate number as a complete or context-free CCI.

Challenge design itself also affects validity. “Towards Improving IDS Using CTF Events” shows that an evasion-oriented CTF can reveal real IDS weaknesses, including default configurations that miss an exploit and static detection rules that are bypassed through alternative vulnerabilities such as **CVE-2024-45195** rather than the intended **CVE-2024-38856** route [2501.11685]. At the same time, the same paper documents measurement threats from challenge opacity, infrastructure inconsistencies, logging gaps, and format unfamiliarity [2501.11685]. A plausible implication is that CCI results should always be interpreted together with challenge-design metadata: what the challenge was intended to elicit, what infrastructure actually exposed, and how much of performance reflects the target competency rather than incidental friction.

A further direction comes from “Beyond the Flag,” which proposes the **Ethical-Cognitive Apprenticeship in Cybersecurity (ECAC)** framework with five phases—**Foundational Modeling**, **Scaffolding the Arena**, **Coaching and Articulation**, **Ethical Dilemma Injections**, and **Reflective Exploration**—for integrating CTFs into K–12 education [2602.16921]. That paper does not define a numerical CCI, but it explicitly argues that CTF competence in educational settings includes not only technical skill but also higher-order thinking, collaboration, ethics, reflection, and autonomy/self-efficacy [2602.16921]. This suggests that future CCIs, especially for learners rather than autonomous agents, may become more **developmental**, **ethically integrated**, and **equity-aware**.

In current research, then, the CTF Competency Index is best understood not as a settled standard but as an emerging family of assessment models. Its most mature formalization is the six-factor trajectory metric of CTFJudge [2508.05674]. Its strongest methodological roots come from concept-inventory construction and misconception-based educational assessment [1706.05092]. Its most informative behavioral signals come from telemetry, ATT&CK labeling, keystroke reconstruction, and bias-sensitive process analysis [2307.10877]. Its most credible benchmark substrates are executable and increasingly live [2508.18370][2605.11504]. Across these lines of work, the common conclusion is consistent: a serious CCI must measure more than whether a flag was captured. It must also measure the competence revealed by **reasoning, action, adaptation, context, and the conditions under which success or failure occurred**.

Source: https://www.emergentmind.com/topics/ctf-competency-index-cci