Determinants and Limits of LLM Security-Tool Orchestration: A Study with HexStrike-AI

Abstract: LLM agents driving security tool suites over the Model Context Protocol are increasingly common. Yet the factors that bound their capability remain poorly characterized: how much depends on the model versus the client that drives it, whether constraining the agent to the orchestrator's own tools helps, and where capability is limited by reasoning rather than by missing tools. Using HexStrikeAI, an open-source orchestrator that exposes 150+ tools, as a testbed, we follow a methodology that evaluates the system, diagnoses its failures, and applies targeted improvements. We run 86 picoCTF challenges across seven categories and three difficulty tiers, under three tool-access regimes and three model/client configurations (774 trials). We then apply corrections to existing tools, agent-behavior changes, and eleven new capability tools, and re-run the previously-unsuccessful trials. The diagnosis isolates the driving client as a first-order factor for a fixed model (a 2.1 * gap between two DeepSeek clients) and a monotonic difficulty gradient, with the largest gains in the mid tier. The overall solve rate rises from 55.4% to 72.0%, and every configuration improves significantly (paired McNemar p < 0.001, non-overlapping 95% confidence intervals). The residual failures are reasoning- or environment-bound rather than missing-tool. A 60-run stability sub-study finds single-run verdicts reproducible (17/20 unanimous). We discuss what the results imply for how such orchestrators should be evaluated, and we are explicit about the limits: the study uses a single benchmark, the fixes were tuned on the same challenges they were evaluated on, and the client effect is demonstrated for one model only, so its generality to other models remains a hypothesis.

• The paper establishes that the execution client is a key determinant of LLM performance, with a 2.1× variation in solve rates across configurations.
• The study employs a rigorous baseline–intervention methodology over 774 trials, demonstrating significant improvements (up to +21.7pp) after targeted capability and behavioral interventions.
• The findings highlight that behavioral corrections exceed pure tool augmentation in effectiveness, underscoring persistent reasoning and environmental challenges in complex tasks.

Determinants and Limits of LLM Security-Tool Orchestration: Analysis of HexStrike-AI

Architectural Overview and Evaluation Design

The study investigates the operational boundaries and determinants of LLM agents orchestrating complex security tool suites, using HexStrike-AI as a testbed. HexStrike-AI exposes a vast set of over 150 security utilities to LLMs via the Model Context Protocol (MCP), coupled with a Flask backend and FastMCP interface. The evaluation assesses agent performance across 86 picoCTF challenges, spanning seven categories and three difficulty levels, under three distinct tool-access regimes and three model/client configurations, resulting in 774 trials. The study utilizes a baseline–intervention methodology, systematically diagnosing failure modes and attributing recoveries to either tool augmentation or behavioral correction. The architecture incorporates logging, measurement for detailed verdict attribution, and robustness instrumentation.

Figure 1: Architecture of HexStrike-AI and study workflow, illustrating orchestration layers and the evaluation-and-intervention protocol.

Empirical Findings: Success Factors and Bottlenecks

The experimental matrix reveals that the execution client is a critical determinant of task success, independent of the model. With DeepSeek held constant across two clients (RooCode, 5ire), a 2.1$\times$ variation in solve rate (59.7% vs. 27.9%) emerges, affirming the harness effect hypothesized in recent agent studies and benchmarks (Yao et al., 27 May 2026, Zhang et al., 7 May 2026, Mayoral-Vilches et al., 27 May 2026). The model–client confounding is pronounced with Claude, where the client cannot be separated. Tool-access regime impacts are minor except for 5ire, whose strict regime leads to substantial improvements attributable to enforced tool constraints. Baseline solve rates decline sharply with challenge difficulty, with saturation in Easy tier (75.7%) and attrition in Hard (30.6%). Web and General categories are consistently weak, especially in higher tiers where HexStrike's baseline lacks specialized utilities.

Figure 2: Baseline solve rates across difficulty and category, highlighting pronounced attrition in Hard challenges and deficiency in Web/General.

Targeted Intervention: Augmenting Capability and Behavioral Fixes

The intervention phase introduces eleven new capability tools targeting previously identified challenge bottlenecks and rectifies behavioral default failures. Following the intervention, overall solve rates increase from 55.4% to 72.0%, with significant improvement across all configurations (Claude: +11.2pp, RooCode: +16.7pp, 5ire: +21.7pp). Importantly, a majority of the recoveries are attributed to behavioral and default fixes, rather than to pure tool augmentation. Redirecting agent reasoning toward existing tools (notably pwntools for binary exploits) and prompt upgrades yielded more recoveries (79 cases) than all the new tools combined (49 cases).

Figure 3: Configuration-wise solve rate improvement after intervention, with non-overlapping 95% Wilson confidence intervals.

Figure 4: Post-fix category-difficulty mapping, showing universal improvement but persistent capability floor in Hard challenges.

Failure Characterization and Statistical Robustness

Residual failures post-intervention are predominantly reasoning- or environment-bound, rather than tool-bound. Failure taxonomy includes insight deficits, interactive/real-time barriers, and multi-step stateful chains, with persistent unsolvables limited to complex cryptographic and multi-user interactive challenges. Statistical analysis confirms robustness of improvements (all $p<0.001$ and no interval overlap), and stability sub-study demonstrates outcome reproducibility: single-run verdicts are unanimous in 85% of trials (100% for RooCode, 70% for 5ire), establishing a low run-to-run variance baseline. Leakage from tool-access constraints is configuration-dependent, with weaker clients defaulting to native tool use despite prompt restrictions.

Practical and Theoretical Implications

The study substantiates that the orchestration harness—not merely the model—is a first-order driver of LLM agent performance in security automation contexts. This finding demands explicit harness disclosure and benchmarking in agent evaluations (Zhang et al., 7 May 2026). The results also suggest diminishing returns for tool catalog expansion in the absence of behavioral and reasoning support, reinforcing recent empirical observations from CTF agent evaluations (Deng et al., 19 Feb 2026). While intervention improved mid-tier challenge outcomes most, the persistent floor in complex cryptographic and stateful interactive tasks indicates reasoning and environmental barriers that are unlikely to yield to incremental tooling. Methodologically, the study highlights the necessity for reproducibility audits and explicit measurement of adherence to tool-access constraints.

Limitations and Prospective Directions

Several limitations constrain generalization: evaluation is based on a CTF-centric benchmark, intervention tuning and evaluation share the same challenge set, and the client–model hypothesis is demonstrated on DeepSeek only. Consequently, future research must target (a) harness benchmarking across diverse models and clients, (b) ablation studies isolating prompt and tooling effects, and (c) evaluations in real engagement settings decoupled from flag-centric constructs.

Conclusion

HexStrike-AI provides a rigorous platform for diagnosing the determinants and limits of LLM-driven security tool orchestration. The empirical evidence confirms the primacy of the client harness, the bounded benefit of pure tool augmentation, and the resilience of environmental and reasoning barriers. The study prompts a shift in agent benchmarking practices, emphasizing harness transparency and multi-level validation, with implications for practical deployments in offensive security automation and for the theoretical modeling of agent workflows.