Cognitive Penetration Testing

• Cognitive penetration testing is a reasoning-intensive approach that integrates memory, planning, and adaptive decision-making to interpret partial findings and revise strategies over long attack horizons.
• It employs AI techniques such as reinforcement learning and large language models to enable sequential, context-aware decision-making in dynamic cybersecurity environments.
• Recent methods emphasize persistent memory, validation, and error recovery while addressing safety, dual-use, and governance challenges in both offensive security and AI system testing.

Cognitive penetration testing denotes penetration testing treated as a reasoning-intensive process of interpreting partial findings, selecting plausible next actions, maintaining context across long attack horizons, and revising strategy as evidence accumulates. Recent literature does not use the term in a single standardized way: in offensive-security automation it usually refers to AI-assisted or agentic systems that support or automate sequential pentesting decisions, whereas in AI governance it can also denote pre-deployment testing of an AI system’s own reasoning under adversarial manipulation (Curtis et al., 13 Dec 2025, Aydin, 19 Aug 2025). In both usages, the focus shifts away from static exploit syntax or isolated cybersecurity question answering toward cognition: memory, planning, hypothesis revision, tool selection, validation, and adaptation.

1. Conceptual foundations

In the offensive-security literature, penetration testing is increasingly described as an iterative decision process rather than a linear script. CIPHER states the point directly: real pentesting is not merely recalling exploit syntax or enumerating tools, but interpreting partial findings, choosing a plausible next action, targeting the right service or weakness, and updating the plan as results accumulate (Pratama et al., 2024). The systematic review "The Role of AI in Modern Penetration Testing" reaches a similar conclusion in broader terms, describing AI-assisted pentesting as machine-supported decision-making in which systems can learn from experience, model attacker behavior, prioritize promising actions, and adapt strategies to unfolding conditions (Curtis et al., 13 Dec 2025).

This cognitive framing changes how the pentesting lifecycle is understood. The review organizes the field around a NIST 800-115-inspired four-phase model—Preparation / Reconnaissance, Discovery / Vulnerability Analysis, Exploitation, and Reporting / Remediation—and concludes that current AI support is concentrated in Discovery and Exploitation rather than end-to-end workflow reasoning (Curtis et al., 13 Dec 2025). A plausible implication is that “cognitive” pentesting, in the strongest sense, would require competence not only in exploit execution but in the transitions between phases: deciding what to enumerate, when to pivot, how to interpret failed attempts, and how to translate offensive findings into remediation.

The older RL literature framed the same problem in control-theoretic terms. "Deep hierarchical reinforcement agents for automated penetration testing" models autonomous pentesting as a Markov Decision Process $$\langle \mathcal{S}, \mathcal{A}, \mathcal{P}, \mathcal{R}, \gamma \rangle$$ and argues that large discrete action spaces make even simplified cyber environments difficult for conventional deep Q-learning (Tran et al., 2021). "Modeling Penetration Testing with Reinforcement Learning Using Capture-the-Flag Challenges" emphasizes a related point: the central challenge is not only choosing actions, but discovering the hidden structure of the target system itself (Zennaro et al., 2020). Together, these papers establish the technical intuition behind the term: penetration testing is cognitively demanding because it is a sequential, partially informed search problem with branching contingencies and sparse feedback.

2. Core cognitive functions

Recent systems decompose pentesting into a set of recurring cognitive functions. CIPHER is explicit that its target capability is “what should I do next?” reasoning: given ports, credentials, service banners, or failed attempts, the model should suggest the most likely next action and explain why (Pratama et al., 2024). Pentest-R1 formalizes the same pattern through a Thought-Command-Observation trajectory format, where each step contains a rationale, an executable action, and the environment’s response (Kong et al., 10 Aug 2025). This suggests a common operational unit for cognitive pentesting: state interpretation, action proposal, and feedback assimilation.

Memory is another central function. APT-Agent argues that many LLM pentesting failures arise because models cannot preserve enough operational context to avoid getting lost over multi-step attack chains, and it introduces a command-specific memory architecture that stores stage-specific logs of commands and their outcomes (Li et al., 24 May 2026). Red-MIRROR makes memory even more central through its Shared Recurrent Memory Mechanism, which preserves agent observations in a persistent shared structure and feeds summarized, role-specific state back into planning and execution (Khang et al., 28 Mar 2026). TermiAgent reaches a similar diagnosis from a different direction, treating long-context forgetting as a primary reason prior agents collapse on realistic environments and replacing flat transcript retention with Located Memory Activation over a Penetration Memory Tree (Mai et al., 11 Sep 2025).

Validation and error recovery are equally important. Red-MIRROR’s dual-phase reflection separates local payload checking from global strategic reassessment, so that failed exploit attempts can trigger either payload refinement or path-level replanning (Khang et al., 28 Mar 2026). APT-Agent’s hybrid rectification module addresses a narrower but operationally critical problem: hallucinated exploit names, malformed modules, and missing parameters in Metasploit-oriented workflows (Li et al., 24 May 2026). Pentest-R1 treats this as a training problem, using online RL in an interactive CTF environment so the model learns from command success, command failure, and terminal task completion rather than from text imitation alone (Kong et al., 10 Aug 2025).

A further cognitive function is abstraction across levels of control. Pen-Strategist splits the problem into strategy formation and step prediction: a strategy model derives the next high-level move from the current Pen Test Tree, previous step, and previous result, while a step model maps that strategy into a concrete next action and associated MCP servers (Ginige et al., 6 May 2026). This separation between strategic reasoning and executable action selection is recurrent across the literature and marks a shift away from monolithic chat-style assistants toward systems that approximate the layered decision structure of human pentesting.

3. Architectures and methods

The methodological literature on cognitive penetration testing is heterogeneous. A 2025 systematic literature review of 58 peer-reviewed studies reports that, among the 54 papers that could be assigned to a pentesting phase, 42 papers—approximately 77%—used reinforcement learning, while only four used LLMs (Curtis et al., 13 Dec 2025). This makes RL the dominant paradigm in the broader automation literature, especially for Discovery and Exploitation.

Within RL, several distinct approaches appear. HA-DRL addresses action-space explosion by decomposing a large discrete action space into smaller sub-action spaces selected by a hierarchy of DQN agents, using an algebraic action decomposition of the form $$a_{out}^{i+1} = a_{out}^{i} \times \beta^{i+1} + a^{i+1}$$ (Tran et al., 2021). Earlier CTF-oriented RL work instead emphasizes the trade-off between model-free learning and injected prior knowledge through lazy loading, state aggregation, and imitation learning (Zennaro et al., 2020). More recent LLM-centered work uses RL differently: Pentest-R1 combines offline RL from more than 500 real-world walkthroughs with online RL in InterCode-CTF, arguing that long-horizon error recovery is learned more effectively from interaction than from chain-of-thought prompting alone (Kong et al., 10 Aug 2025).

The LLM-based architectural strand is broader. CIPHER is a domain-specific 7B assistant built on Mistral-7B-Instruct-v0.3, trained with supervised fine-tuning and DPO on a mixture of general assistant data and offensive-security corpora, including over 300 Hack The Box write-ups, GTFOBins, LOLBAS, HackTricks, PayloadsAllTheThings, and open-source pentesting documentation (Pratama et al., 2024). APT-Agent organizes a fully automated loop around tactic selection, command generation, output translation, hybrid exploit rectification, and stage-aware memory, evaluating end-to-end exploitation on Metasploitable 2 (Li et al., 24 May 2026). Red-MIRROR uses a Planner, Collector, Exploiter, Summarizer/Analyzer, and RAG-backed knowledge layer tied together through SRMM and dual-phase reflection (Khang et al., 28 Mar 2026). TermiAgent uses a Reasoner, Assistant, Executor, Memory Module, and Arsenal Module, with the exploit arsenal built from 1,378 Dockerized exploits and 1,077 curated Metasploit exploits (Mai et al., 11 Sep 2025).

A strategy-centric design also appears. Pen-Strategist fine-tunes a Qwen-3-14B model with LoRA and GRPO on a 2,165-example reasoning dataset derived from 240 Hack The Box and VulnHub machines, then pairs it with a dual-head CNN classifier that predicts the next step class and MCP server set from the generated strategy and explanation (Ginige et al., 6 May 2026). This is one of the clearest examples of cognitive decomposition in the literature: state interpretation and strategy derivation are treated as distinct from action and tool grounding.

4. Benchmarks and empirical evaluation

As the field matured, evaluation shifted from toy tasks and exploit demonstrations toward workflow-sensitive and operationally realistic benchmarks.

Benchmark or protocol	Objective	Distinctive feature
FARR Flow (Pratama et al., 2024)	Next-step prediction	35 unseen HTB machines and 2124 formulated questions
AutoPenBench (Gioacchini et al., 2024)	End-to-end pentest tasks	33 tasks with command and stage milestones
Vulnerability-discovery protocol (Conde et al., 11 May 2026)	Validated finding-level scoring	108 expert-annotated vulnerabilities on three targets
TermiBench (Mai et al., 11 Sep 2025)	Shell acquisition	510 hosts, 25 services, and 30 CVEs

FARR Flow is designed around Findings, Action, Reasoning, and Results. Each evaluation prompt includes accumulated findings and prior results and asks for “the most likely action to do next,” making it a benchmark for local next-step cognition rather than full autonomous execution (Pratama et al., 2024). AutoPenBench instead evaluates generative agents on 33 tasks—22 in-vitro and 11 real-world—with unrestricted Bash execution, SSH, and milestone-based scoring across target discovery, infiltration, detection, exploitation, flag capture, and success (Gioacchini et al., 2024). The vulnerability-discovery protocol of "From Controlled to the Wild" shifts the target again: instead of task completion or trajectory similarity, it scores validated vulnerability discovery using semantic matching, maximum bipartite matching, precision, recall, $F1$, $F_{0.5}$, severity score, and CWE coverage (Conde et al., 11 May 2026). TermiBench departs furthest from CTF conventions by defining success as obtaining a shell rather than finding a flag, starting from subnet or IP only, and adding benign service noise to 510 host instances spanning 30 CVEs and 25 services (Mai et al., 11 Sep 2025).

The empirical picture is mixed but informative. On FARR Flow, CIPHER DPO 7B achieved the best total average score, 52.59, versus 52.07 for Llama 3 70B, 50.11 for GPT-3.5 Turbo, and 49.82 for Qwen1.5 72B; on the difficulty-stratified variant it reached 50.96 on insane machines, compared with 48.74 for Llama 3 70B (Pratama et al., 2024). On AutoPenBench, the fully autonomous agent in the benchmark paper achieved a 21% Success Rate across the 33 tasks, while the assisted agent achieved 64% (Gioacchini et al., 2024). Pentest-R1 later reported a 24.2% success rate on AutoPenBench and 15.0% on Cybench in unguided tasks, with ablations showing that offline and online RL stages are both necessary (Kong et al., 10 Aug 2025).

In web-centric autonomous exploitation, Red-MIRROR reported an 86.0 percent overall success rate on the XBOW benchmark, outperforming PentestAgent (50.0 percent), AutoPT (46.0 percent), and the VulnBot baseline (6.0 percent), and a 93.99 percent subtask completion rate (Khang et al., 28 Mar 2026). APT-Agent, evaluated on seven vulnerable services in Metasploitable 2, achieved an 84.29% end-to-end exploitation success rate, compared to 48.57% for Script Kiddie and 18.57% for PentestGPT under matched conditions (Li et al., 24 May 2026). By contrast, TermiBench is deliberately pessimistic about current capability: it reports that existing systems can hardly obtain system shells under realistic conditions, and in its experiments PentestGPT achieved 0/0 shell/root success across the 230-target real-world slice (Mai et al., 11 Sep 2025).

These results support a narrow but increasingly consistent conclusion: specialization, persistent memory, exploit grounding, and reflective validation improve performance materially, but realistic end-to-end autonomous penetration testing remains substantially harder than CTF-oriented evaluation suggests.

5. Human operators, attacker psychology, and cognitive bias

A persistent theme in the literature is that cognitive penetration testing is often better understood as augmentation than replacement. CIPHER is explicitly positioned as a helper “for ethical researcher” and as a reasoning scaffold for beginners rather than an autonomous exploit system, providing next-step guidance, tool suggestions, and explanations from accumulated findings (Pratama et al., 2024). The systematic review likewise concludes that AI is “not positioned to replace human pentesters, but to augment their capabilities,” especially because reconnaissance and reporting remain underdeveloped while most AI support is concentrated in midstream tactical decision-making (Curtis et al., 13 Dec 2025).

A second strand of research studies attacker cognition directly. PsybORG$^+$ models APT-style behavior under cognitive vulnerabilities such as loss aversion, confirmation bias, and sunk cost fallacy, using a Hidden Markov Model over cyber stage and bias state; under its simulator assumptions, Bayesian inference reached 0.965 accuracy for inferring bias state, and a decision-tree model reached 0.95 for loss aversion, 0.99 for confirmation bias, and 0.83 for sunk cost fallacy (Huang et al., 2024). This work does not build an operational pentesting assistant, but it extends cognitive penetration testing toward bias-sensitive adversary emulation and deception design.

"Risk Psychology & Cyber-Attack Tactics" moves in a related direction with human red-teamers rather than simulated attackers. Using multilevel mixed-effects Poisson regression over 1,964 technique uses from 33 cybersecurity professionals in a simulated enterprise environment, it finds that psychometric scales predict technique-specific ATT&CK usage while expertise level and treatment condition do not significantly predict technique patterns (Kim et al., 23 Oct 2025). Higher CRT, for example, is associated with more lateral movement and less reconnaissance, collection, credential access, initial access, and privilege escalation. A plausible implication is that “cognitive penetration testing” can also mean modeling how human cognitive differences shape tactic selection, not only building AI systems that execute those tactics.

Taken together, these works broaden the term’s scope. In one line, cognition is the target capability of AI pentesting systems: memory, planning, contextual reasoning, and explanation. In another, cognition is a property of the adversary being modeled: biases, framing effects, risk propensity, and reflective style. Both lines converge on the same claim that offensive behavior cannot be reduced to a static exploit library.

6. Limits, safety, governance, and future directions

The literature is equally clear about current limits. The systematic review finds only seven papers addressing Preparation and only five addressing Reporting, with PenHeal singled out as the only tool in that review explicitly addressing the final Reporting / Remediation stage (Curtis et al., 13 Dec 2025). CIPHER acknowledges that FARR evaluates local next-step prediction rather than a closed-loop branching simulation, because the benchmark appends the ground-truth result after each question regardless of model correctness (Pratama et al., 2024). The realistic vulnerability-discovery protocol of "From Controlled to the Wild" improves operational validity, but it still does not directly score internal planning quality, causal understanding, exploit-chain reasoning depth, calibration of uncertainty, or recovery from tool failures except insofar as they affect final findings (Conde et al., 11 May 2026). APT-Agent, despite strong results, is evaluated only on RECON, EXPLOIT, and EXFILTRATE in a single-host Metasploitable 2 setting and does not yet cover multi-host attack paths, lateral movement, or persistence (Li et al., 24 May 2026).

Safety and dual use are persistent concerns. CIPHER does not release its model weights and instead plans to release the FARR Flow augmentation and evaluation code (Pratama et al., 2024). APT-Agent explicitly calls for sandboxed environments, access controls, and human oversight (Li et al., 24 May 2026). Red-MIRROR proposes role-based access control, audit logging, RAG knowledge gating, non-destructive mode, and hard iteration limits (Khang et al., 28 Mar 2026). These safeguards reflect a common judgment that offensive-capable cognitive agents are useful for defense and evaluation, but risky to deploy without strong operational constraints.

A newer governance-oriented meaning of cognitive penetration testing appears in "CIA+TA Risk Assessment for AI Reasoning Vulnerabilities". There, CPT denotes pre-deployment testing of AI systems themselves against reasoning-level attacks such as authority hallucination, context poisoning, source interference, cognitive overflow, and attention hijacking, within an extended security model of Confidentiality, Integrity, Availability, Trust, and Autonomy (Aydin, 19 Aug 2025). Its empirical validation reports strong architecture dependence: identical defenses can produce effects ranging from 96% reduction to 135% amplification of vulnerabilities. This makes pre-deployment CPT a governance requirement in that framework, and suggests that the term may increasingly span both sides of the human–AI security interface: AI that performs penetration testing, and penetration testing applied to AI cognition.

The likely near-term trajectory is therefore plural rather than singular. One direction extends offensive agents through better memory, broader exploit coverage, multimodal interaction, and more realistic evaluation without leaking future state (Pratama et al., 2024, Kong et al., 10 Aug 2025). Another extends cognitive modeling to attacker profiling, deception, and psychology-informed defense (Huang et al., 2024, Kim et al., 23 Oct 2025). A third turns CPT into a release-gating practice for reasoning-capable AI systems under adversarial inputs (Aydin, 19 Aug 2025). Across these directions, the unifying idea remains stable: penetration testing becomes “cognitive” when success depends less on one-shot exploit recall and more on sustained interpretation, planning, validation, and adaptation under uncertainty.