MalSkillBench: A Runtime-Verified Benchmark of Malicious Agent Skills

Abstract: AI coding agents such as Claude Code and Gemini CLI increasingly extend themselves with third-party skills: markdown packages bundling natural-language instructions, executable scripts, and tool permissions. Because a skill is at once code and agent-facing instruction, it introduces a supply chain dependency whose risk is neither pure code nor pure prompt. Detection tools have never been measured against verified ground truth spanning this hybrid space, leaving their effectiveness unknown and wild-only evaluations biased. We present MalSkillBench, the first runtime-verified benchmark of malicious agent skills: 3,944 malicious skills labeled along a three-dimensional taxonomy of 108 cells. Of these, 3,214 come from a closed-loop Generate-Verify-Feedback pipeline admitting only samples whose malicious behavior fires inside a Docker sandbox under system-call monitoring and an LLM judge; we add 703 in-the-wild and 4,000 matched benign skills. Our measurements are consistent: code injection reaches 94.5% verification yield but prompt injection only 75.8%, the same fragility that later makes it hard to detect; the wild sample is narrow, dominated by one cryptocurrency-theft campaign (86.6% one behavior, 81% from two accounts) with a small but architecturally new tail attacking the agent control plane; the strongest skill-specific detector reaches 98.4% recall on code injection yet collapses on prompt-injection and agent-control attacks, and wild-only scoring swings the ranking by up to 66 recall points; supply-chain scanners and prompt-injection defenses each see only half of a skill, and no combination recovers the code-instruction relationship. Detecting malicious skills therefore requires reasoning jointly over task intent, code, and instructions. We release the dataset, pipeline, baselines, and results.

• The paper introduces a taxonomy-driven benchmark that runtime-verifies 3,944 malicious skills and 4,000 benign samples using a closed-loop generate–verify–feedback pipeline.
• Empirical results show code injection and mixed attacks achieve 94.5% and 91.9% realizability respectively, while prompt injection and steganographic attacks lag behind.
• Detector evaluations reveal that single-domain tools underperform, emphasizing the need for joint reasoning over code and instructions for robust threat detection.

MalSkillBench: Runtime-Verified Benchmarking of Malicious Agent Skills

Introduction and Motivation

MalSkillBench presents a systematic infrastructure for evaluating malicious agent skill detection, addressing the hybrid threat surface arising from composable agent extensions (“skills”) which blend executable code and natural language instructions in single distributable artifacts. Contemporary AI coding agents such as Claude Code and Gemini CLI have rapidly adopted skill ecosystems for extensibility, but extant detection methodologies are hampered by: (1) the absence of public, runtime-verified ground truth; (2) empirical attack surfaces biased toward particular vectors (notably, code injection/delivery); and (3) methodological fragmentation where tools are benchmarked in isolation with incomparable protocols.

MalSkillBench seeks to close these gaps via a taxonomy-driven, runtime-verified dataset of 3,944 labeled malicious skills and 4,000 benign pairs, spanning code injection (CI), prompt injection (PI), and mixed-vector hybrid attacks, covering 108 distinct taxonomy cells, with both wild-collected and systematically synthesized samples. A critical insight motivating this work is the structural migration of attack patterns from classic software supply chain (e.g., malicious PyPI/NPM packages) and indirect prompt-injection social engineering—demonstrated with cases exhibiting functional code and instruction migration into the skill context.

Figure 1: Attack-pattern migration. Case 1: near-identical reverse shells in a PyPI package and a malicious skill. Case 2: a social-engineering frame migrated from a phishing email to SKILL.md prerequisites.

Benchmark Construction: Methodology and Taxonomy

The benchmark architecture operationalizes a closed-loop generate–verify–feedback pipeline. Malicious skill synthesis operates across a three-dimensional taxonomy (attack vector, malicious behavior, insertion strategy), enabling comprehensive coverage. The knowledge base is mined from real-world CI (IntelliGraph) and PI (multiple jailbreak corpora) artifact collections, each LLM-labeled for precise mapping into the taxonomy. For each vector-behavior-strategy cell, an LLM-driven agent inserts a malicious payload into a benign template, yielding realistic and diverse camouflage.

Candidate skills are admitted into the benchmark only if runtime verification inside an instrumented agent sandbox confirms malicious behavior via system calls or high-confidence LLM semantic judgments, ensuring empirical ground truth rather than nominal label claims. This protocol prevents overfitting to annotation artifacts and reduces confirmation ambiguity.

Figure 2: Overview of the benchmark construction framework.

Empirical Analysis of Attack Realizability

A key outcome of the closed-loop generation is the quantified realizability of attacks across the taxonomy. MalSkillBench identifies marked disparity in realizability between vectors: CI and MIXED attacks are realized at 94.5% and 91.9% rates, respectively, while PI reaches only 75.8%. Steganographic prompt injection is particularly fragile (62.5%). The technical explanation is that CI and MIXED leave concrete, auditably observable runtime artifacts (network/file/process), whereas instruction-level PI relies on semantic shifts in agent behavior often not fully manifest during evaluation.

Figure 3: Realizability rate by insertion strategy for CI, PI, and MIXED.

PI reasoning attacks (agent-control plane, e.g., role hijack, goal hijacking, instruction override) are especially difficult to automate and validate, both due to dependency on nuanced agent LLM behavior and weak external side effects.

Figure 4: Runtime evidence of realized samples. (a) CDF of Layer-2 judge confidence for accepted PI skills, with the dashed line at $\theta = 0.7$. (b) Matched runtime evidence items per realized sample, by attack vector.

Empirical Analysis of Malicious Skills in the Wild

Analysis of 703 wild-collected skills reveals extreme empirical concentration: 86.3% of samples operate by fake-prerequisite dependency impersonation, and 86.6% yield Malware Delivery (B4). Authorship is highly centralized, with two accounts responsible for 81% of wild samples. Thus, the real-world attack corpus is not representative of the overall taxonomy—most attacks are supply chain pattern clones, providing misleadingly high apparent coverage for payload-centric detectors.

In contrast, a minority of wild samples exhibit agent-native exploits (1.7%), such as session lifecycle hooks, agent identity redefinition, and top-level instruction overrides. These attacks structurally differ from package malware and reflect vulnerabilities unique to agent-oriented reasoning and control mechanisms.

Figure 5: Wild sample overview (703 skills). (a) Delivery mechanism: fake-prerequisite dependency impersonation versus other. (b) Behavior distribution over the taxonomy. Bubble area $\propto\sqrt{\text{count}}$.

Evaluation of Skill-Specific Detectors

Twelve representative academic and industry tools were evaluated on the uniform benchmark. The best skill-specific configuration (Sentry Full) achieves 88.6% F1 and 98.4% recall, with 937 false positives among 4,000 benign samples. However, even this model’s coverage is non-uniform: recall collapses at instruction-only and agent-control attacks. All other detectors, including LLM-based and commercial static analyses, present a precision-recall tradeoff. High-recall configurations over-fire (excessive FPR), while high-precision static signature models (e.g., Snyk, VirusTotal) fundamentally miss attacks not using known code patterns or dependencies.

Figure 6: Detection recall by malicious behavior and insertion strategy.

Critically, wild-only ranking artificially inflates signature-based detector performance due to the narrow empirical composition discussed above. On wild-only data, VirusTotal is propelled to near-top ranking, a 66-point recall shift compared to the full benchmark.

Figure 7: Per-detector recall on the full benchmark versus the wild subset.

Transferability of Single-Domain Security Tools

A salient negative result is that neither supply chain scanners nor prompt-injection defenses, repurposed in isolation or naive combination, achieve reliable skill detection. Supply-chain tools trigger on code-borne artifacts, blind to malicious instructions, while PI detectors over-trigger on benign code-heavy SKILL.md, both yielding unacceptably high FPR or missing most attacks. OR-combinations inherit maximum FPR, AND-combinations collapse recall.

Figure 8: Pairwise combinations of transferred supply-chain scanners and prompt-injection defenses. OR flags a skill when either tool flags it, while AND flags only when both do.

Joint reasoning over instructions and executable code—binding intent and implementation—is necessary for robust skill detection. Summative evidence from single-modal tools cannot close this gap.

Implications and Future Directions

MalSkillBench’s results establish that maliciousness in AI agent skills is a cross-modal judgment: the semantic relationship between “advertised task” and the union of code and instructional authority shaping agent actions. This recasts the detection problem as joint intent–artifact–authority auditing, beyond rule matching or behavioral blacklisting.

From a practical standpoint, skill-specific detectors must evolve to integrate context-sensitive, cross-modal semantic analysis, validated with runtime-exercised, taxonomy-balanced ground truth. The current ecosystem of wild-only evaluation and single-domain heuristics provides misleading assurance.

The theoretical implication is the clear need for joint execution-intent reasoning frameworks—possibly leveraging causal LLMs trained over code-instruction-task triads, coupled with interpretable runtime monitoring aligned to agent plans.

Conclusion

MalSkillBench delivers the first runtime-verified, taxonomy-complete benchmark for malicious agent skill detection, disambiguating the effectiveness and limitations of state-of-the-art tools. The study demonstrates that the semantic alignment or misalignment between a skill's instructions, code, and advertised task is the fundamental axis of both realizability and detectability. The open-source dataset, evaluation harness, and systematic findings directly facilitate principled future research on this prominent threat vector for AI-driven software supply chains.

(2606.07131)