Separate model capability from harness and system-design effects

Determine whether the observed capability gap between the Kimi K2.5 legacy human-in-the-loop PentestGPT system and the Claude Opus 4.8 autonomous PentestGPT system is caused by model capability, harness design, execution paradigm, autonomy, memory architecture, or their interaction.

Background

The comparison contrasts multiple variables simultaneously: Kimi K2.5 versus Claude Opus 4.8, human-mediated command execution versus autonomous execution, different PentestGPT frameworks, and different memory systems. Because no single factor is held constant while another changes, the reported progression is descriptive rather than causal.

Resolving this issue requires evaluating multiple models within one autonomous framework while keeping execution, framework, and memory conditions fixed. This would clarify whether the stronger results arise primarily from the frontier model, the autonomous harness, or the combination.

References

We cannot say whether the model or the harness accounts for the gap.

— Big Enough to Break Out: Tracking the Rising Capability of LLM Penetration-Testing Agents  (2609.10780 - Lovelace et al., 9 Sep 2026) in Section 8, Threats to Validity, paragraph “Confounded cross-setup comparison”; Conclusion

We compare with two baselines, Step-wise VLM and SeeAct-V, but have not tested the Jev-Mobile workflow with Jev replaced by a small VLM; that comparison would clarify whether the observed behavior depends on typed decision-model execution or on the delegated workflow more generally.

— Jev-Mobile: Jev as an Executor for Mobile GUI Agents  (2609.30186 - Zhang, 24 Sep 2026) in Section 7, Limitations

The deepest caveat is inherited from \S\ref{sec:incident2}: whether the harness causes the scores, versus the models alone, is unmeasured until a boundary-correct ablation is run.

— Kepler: Auditable World Models for ARC-AGI-3  (2610.00834 - Wu, 30 Sep 2026) in Section 6.2, “Incident 2: the control group rebuilt the harness,” and Section 8, “Limits,” paragraph “Scope of the release evidence”

A factorial comparison is therefore left to future work.

— Accurate but Not Humble: Evaluating Epistemic Humility in LLM Agents under Knowledge Conflict  (2610.12360 - Sun et al., 8 Oct 2026) in Limitations, subsection “Backbone and harness are coupled”

WebDevJudge~\citep{webdevjudge2025}, which benchmarks LLM and multimodal judges for web development, illustrates the importance of understanding these capabilities as it finds that its standalone GUI-agent workflow actually underperforms its strongest static judges due to errors arising during planning and execution.

— Interaction-as-a-Judge  (2610.17895 - Chi et al., 13 Oct 2026) in Related Work, paragraph “Interactive evaluation”

It remains unclear whether further gains should be sought primarily through better use of available information and tools, through additional inference (for example, in a multi-agent system), or through a combination of the two. Further analysis is needed to determine their relative contributions and whether they can be meaningfully separated.

— Improving Environment-Agnostic Coding Agents: The Role of Scaffolding  (2610.22289 - Rodionov, 15 Oct 2026) in Section 7.4, “Implications and future directions” (also reiterated in Section 8, “Conclusions”)