MalSkillBench: Verified Malicious Skills

• MalSkillBench is a runtime-verified benchmark of malicious AI coding agent skills, categorizing artifacts via a three-dimensional taxonomy of code, prompt, and mixed injection vectors.
• It uses a closed-loop Generate-Verify-Feedback pipeline to admit samples that demonstrate observable malicious behavior under Docker sandbox monitoring.
• The benchmark underpins detector evaluations, revealing high recall for code injections but significant detection gaps for prompt and agent-control behaviors.

Searching arXiv for the specified papers and topic to ground the article in current research. MalSkillBench is a runtime-verified benchmark of malicious agent skills for AI coding agents such as Claude Code and Gemini CLI. In this setting, a skill is simultaneously code, natural-language instruction, and a permission-bearing extension, so the relevant attack surface is neither pure code nor pure prompt. MalSkillBench was introduced as “the first runtime-verified benchmark of malicious agent skills,” with 3,944 malicious skills labeled across a three-dimensional taxonomy of 108 cells, 4,000 matched benign skills, and a verification pipeline that admits only samples whose malicious behavior fires inside a Docker sandbox under system-call monitoring and an LLM judge (Guo et al., 5 Jun 2026). The term “MalSkillBench” also appears in adjacent 2026 literature as a colloquial name for SkillSafetyBench and as a shorthand in one consolidated description of SkillVetBench, which makes nomenclature somewhat broader than the title of a single benchmark paper (Jin et al., 12 May 2026, Hossain et al., 30 May 2026).

1. Naming and scope

The canonical use of the name refers to the benchmark introduced in “MalSkillBench: A Runtime-Verified Benchmark of Malicious Agent Skills” (Guo et al., 5 Jun 2026). Its stated goal is to provide ground truth for the “full hybrid attack surface—code injection, prompt injection, and their combinations—so that detectors can be measured reliably.”

At the same time, nearby work uses closely related terminology for distinct evaluation targets. This is important because the benchmarks differ in unit of analysis, threat model, and metric design.

Benchmark term in the literature	Stated focus	Scale stated in the paper
MalSkillBench	“runtime-verified benchmark of malicious agent skills”	3,944 malicious skills and 4,000 benign skills
SkillSafetyBench	“runnable benchmark for evaluating such skill-mediated safety failures”; described in one summary as a “colloquial name for SkillSafetyBench”	155 adversarial cases across 47 tasks
SkillVetBench	“two-stage security vetting benchmark for open agentic skill ecosystems”; described in one summary as “referred to here as MalSkillBench”	78 confirmed-malicious skills and 22 benign skills

This naming overlap suggests that “MalSkillBench” has functioned both as a proper benchmark name and as an informal umbrella label for malicious-skill or skill-facing security evaluation. A plausible implication is that citations should be anchored to the exact arXiv identifier rather than to the name alone when comparing results across papers.

2. Corpus design and taxonomy

MalSkillBench’s dataset composition is explicit. The malicious corpus contains 3,944 skills in total: 3,214 generated via a closed-loop Generate-Verify-Feedback pipeline, 703 collected “in the wild” and manually verified, and 27 additional samples drawn from existing tool test suites. The benign side contains 4,000 high-download skills from ClawHub, used to measure false positives (Guo et al., 5 Jun 2026).

The benchmark organizes malicious skills with a three-dimensional taxonomy. The first dimension is Attack Vector, with three categories: CI (Code Injection), PI (Prompt Injection), and MIXED. The second dimension is Malicious Behavior, with 15 categories. The paper distinguishes B1–B9 as code-level behaviors—Data Exfiltration, Credential Theft, Remote Code Execution, Malware Delivery, Persistence, Reverse Shell, Ransomware, Resource Abuse, and Privilege Escalation—and B10–B15 as agent-control behaviors—Role Hijack, Safety Bypass, Instruction Override, System Prompt Leak, Goal Hijacking, and Content Manipulation. The third dimension is Insertion Strategy, which varies by vector: CI uses New Script File, Function Append, Function Inject, and Inline Code Block; PI uses Full Camouflage, Partial Injection, and Steganographic; MIXED uses Download + Execute, Config + Load, and Fetch + Run (Guo et al., 5 Jun 2026).

The resulting coverage matrix has 108 cells: 36 for CI, 45 for PI, and 27 for MIXED. This structure matters because it operationalizes the claim that malicious skills are hybrid artifacts. The benchmark therefore encodes not only what the attack does, but also where it is embedded and how it is staged.

3. Generate-Verify-Feedback pipeline

MalSkillBench’s synthetic portion is built by a closed-loop Generate-Verify-Feedback pipeline. In the Generation Agent, knowledge retrieval uses real malicious code snippets from IntelliGraph, covering 3,026 PyPI packages, for CI attacks, and real jailbreak payloads from prompt-injection corpora, covering 20,961 samples, for PI attacks. Template selection then samples a benign “facade” skill from a pool of 3,458 SkillsMP skills across 12 categories. Skill synthesis prompts an abliterated Qwen3.5-35B LLM with the target vector-behavior-strategy triple, retrieved examples, and the benign template, while constraining the output so that CI embeds executable payload in the correct script location, PI leaves scripts syntactically benign and hides instructions per strategy, and MIXED stages an intermediate artifact in markdown for the script to consume. The artifact output is parsed into a skill bundle containing SKILL.md, scripts/, assets/, and _expected.json, where _expected.json declares the expected observable behaviors $E$ (Guo et al., 5 Jun 2026).

The Verification Agent executes the synthesized skill in a Docker container with OpenCode agent, Python runtime, strace -f, and inotifywait, triggering the skill with a prompt matching its advertised benign purpose. It collects a system-call trace $$\tau = \{\tau_{\text{net}}, \tau_{\text{file}}, \tau_{\text{proc}}\}$$ and agent text output $o$. Verification then proceeds in two layers. Layer 1 is deterministic evidence matching: for CI and MIXED, the trace is matched against IOCs in $E$, such as connect() to a malicious domain or execve() of a payload; for PI, the output is matched against declared keyword IOCs and canonical refusals. Layer 2 is semantic LLM judgment: for CI and MIXED, GPT-5.4-mini judges each expected behavior as Observed, NotObserved, or Ambiguous with confidence $\text{conf}(e)\in[0,1]$; for PI, it judges the output into $$\{\text{triggered}, \text{detected}, \text{ineffective}\}$$ with confidence. The pass condition is stated as $$\text{pass} \iff \exists\,e\in E:\; m_{1}(e)\;\lor\;\text{conf}(e)\ge\theta$$ with threshold $\theta = 0.7$. Rejected samples produce feedback $$(\text{reason}, E_{\text{observed}}, \text{suggestion})$$ and are retried up to $r=3$ times (Guo et al., 5 Jun 2026).

This design is central to the benchmark’s claim of verified ground truth. The benchmark does not rely on unverified synthetic labels; rather, generated samples are admitted only when malicious behavior is observed in the sandbox.

4. Verification yield and empirical structure of the wild sample

MalSkillBench reports distinct runtime verification yields by vector. CI yield = 94.5%, PI yield = 75.8%, and MIXED yield = 91.9% (Guo et al., 5 Jun 2026). The paper explicitly notes that “code injection reaches 94.5% verification yield but prompt injection only 75.8%, the same fragility that later makes it hard to detect.” This suggests that prompt-layer attacks are not merely a detection problem; they are also less stable as executable, benchmarkable artifacts.

The wild sample contains 703 skills. Its delivery mechanism is heavily concentrated: 607/703 (86.3%) use fake-prerequisite dependency impersonation. Its behavior distribution is similarly narrow: 609/703 (86.6%) map to Malware Delivery (B4) via the fake dependency. Authorship is concentrated as well: two accounts produced 569/703 (81%) of the wild samples, largely a crypto-theft campaign targeting wallet keys and seed phrases (Guo et al., 5 Jun 2026).

The paper emphasizes a smaller but architecturally distinct tail of wild attacks. The novel agent-control tail (1.7% of wild) includes session lifecycle persistence by inserting hooks into .claude/settings.json, identity or goal rewriting by editing the agent’s system prompt to “serve Project Exodus,” and instruction hierarchy override through top-of-document “SYSTEM OVERRIDE” policies (Guo et al., 5 Jun 2026). In context, this matters because a wild-only sample dominated by one campaign can conceal behaviors that are strategically important but numerically rare.

5. Detector evaluation and the limits of partial defenses

MalSkillBench evaluates 12 skill-specific detector configurations on the full benchmark of 3,944 malicious and 4,000 benign skills. The strongest overall result is Sentry Skill Scanner (full, static+LLM) with Recall 98.4%, Precision 80.5%, and $$\tau = \{\tau_{\text{net}}, \tau_{\text{file}}, \tau_{\text{proc}}\}$$0, with 937 false positives. A more balanced point is AI-Infra-Guard (Tencent) at Recall 86.6%, Precision 84.6%, and $$\tau = \{\tau_{\text{net}}, \tau_{\text{file}}, \tau_{\text{proc}}\}$$1, with 620 false positives. Cisco Skill Scanner (LLM mode) achieves Recall 92.7% and Precision 71.4%, but with 1,465 false positives. At the conservative end, Snyk Agent Scan and VirusTotal produce low false positives—82 and 4—but low recall, 28.7% and 21.6%, respectively (Guo et al., 5 Jun 2026).

The paper identifies systematic detection gaps. “All drop sharply on PI-only and agent-control behaviors,” reaching 20–40% recall on B12–B15 (Guo et al., 5 Jun 2026). This is a substantive benchmark result, not merely a failure of any single tool. It indicates that the hardest cases are those where malicious intent resides in instructions, role manipulation, or cross-component coordination rather than in obvious payload code.

A further finding concerns evaluation methodology. On wild-only vs. full benchmark recall, VirusTotal jumps from 21.6% recall (full) to 87.9% recall (wild), a change of +66 points; Snyk Agent Scan increases by +49 points, while AI-Infra-Guard and static tools fall by 13–21 points (Guo et al., 5 Jun 2026). The benchmark therefore argues that wild-only scoring is misleading because it ranks tools according to a narrow wild distribution rather than according to the full attack space.

Transferred tools reinforce the same point. Supply-chain scanners such as OSSGadget and SAP reach recall ≈ 97–99% but flag ≈ 90% of benign skills; prompt-injection defenses such as DataSentinel and NeMo Guardrails reach recall ≈ 96–100% but flag ≈ 90–100% of benign skills. The paper’s conclusion is direct: “Supply-chain scanners and prompt-injection defenses each see only half of a skill, and no combination recovers the code-instruction relationship” (Guo et al., 5 Jun 2026).

6. Evasion, downstream use, and behavior-centric auditing

The MalSkillBench dataset is used directly in “Cloak and Detonate: Scanner Evasion and Dynamic Detection of Agent Skill Malware,” which studies adaptive bypasses against static and hybrid skill scanners (Ji et al., 2 Jul 2026). That paper treats MalSkillBench as a publicly archived corpus, begins from the 703 skill bundles collected from OpenClaw/ClawHub, and retains 622 skills whose payload was fully reproducible for its “Wild” evaluation, alongside 150 benign skills randomly sampled from popular marketplace bundles.

Its adversarial contribution, SkillCloak, has two strategies. Structural Obfuscation rewrites visible payload indicators into semantically equivalent forms through operators such as zero-width insertion, confusables, case-flip, synonym substitution, getattr/chr, var-splice, per-line split, per-file relocation, blind-spot directory moves, extension renaming, and size padding. Self-Extracting Skill (SFS) Packing hides the malicious skill from the install-time view, then decrypts and restores it during agent execution (Ji et al., 2 Jul 2026).

The evasion results are severe. Across eight scanners and 1,613 in-the-wild malicious skills, SFS Packing bypasses every scanner at over 90%, while Structural Obfuscation bypasses over 80% on most static scanners and reaches 96% on a hybrid scanner (Ji et al., 2 Jul 2026). Example scanner-specific numbers include Huifer, where raw flagged findings fall by 96.3% under Structural Obfuscation and by 99.8% under SFS Packing, and Vigile, with 84% reduction under Structural Obfuscation and 99.9% under SFS Packing (Ji et al., 2 Jul 2026).

Motivated by this, the paper proposes SkillDetonate, a behavior-centric runtime auditor. SkillDetonate combines on-demand closure lift, which observes instructions materialized during execution, with marker-based systemwide taint analysis, which tracks sensitive-data flows across the agent context, files, processes, and network operations. On the 622-skill MalSkillBench “Wild” subset, SkillDetonate detects 87.6% of original skills, 83.8% of obfuscated skills, and 90.5% of packed skills, for 87.3% average detection; on a separate SkillJect benchmark it averages 96.7% detection with 2% benign false-positive rate (Ji et al., 2 Jul 2026). This downstream use frames MalSkillBench not only as a detector benchmark but also as a pressure test for robustness against adaptive scanner evasion.

7. Relationship to adjacent benchmarks

MalSkillBench belongs to a broader 2026 family of skill-security benchmarks, but adjacent papers target different layers of the problem.

SkillSafetyBench evaluates “skill-mediated safety failures” rather than malicious-skill detection. It includes 155 adversarial cases across 47 tasks, 6 risk domains, and 30 safety categories, with reusable skills, task-relevant skill materials, local artifacts, runnable environments, and case-specific rule-based verifiers. It measures Attack Success Rate (ASR) and shows overall ASR ranging from ≈15.5% to ≈50.3%, with Context Trust ≈ 58.7% as the highest mean per-domain vulnerability and Supply Chain ≈ 20.1% as one of the lower ones (Jin et al., 12 May 2026). The conceptual distinction is that SkillSafetyBench studies how benign requests can be subverted by non-user “skill-facing” materials, whereas MalSkillBench studies malicious skills themselves as supply-chain artifacts.

SkillVetBench, presented in “Benchmarking Security Risk Detection and Verification in Open Agentic Skill Ecosystems,” frames the problem as two-stage vetting: semantic vetting followed by sandbox execution. It uses 78 confirmed-malicious skills from the live OpenClaw ClawHub repository and 22 benign skills as controls, scoring skills with a Skill Agentic Risk Score (SARS) over five dimensions—Instruction Fidelity Risk, Data Gravity, Action Irreversibility, Blast Radius, and Chain Amplification—then escalating suspicious skills to runtime verification in an isolated Docker-based OpenClaw sandbox (Hossain et al., 30 May 2026). Its experiments report that static baselines miss up to 89% of instruction-layer threats, while the benchmark’s semantic stage flags all 78 confirmed-malicious skills with zero false negatives and zero false positives on the 22 benign controls (Hossain et al., 30 May 2026).

Taken together, these benchmarks define complementary evaluation regimes. MalSkillBench is centered on runtime-verified malicious skill artifacts and detector benchmarking; SkillSafetyBench is centered on agent safety under skill-facing attack surfaces; SkillVetBench is centered on two-stage vetting in an open agentic skill ecosystem. A plausible implication is that meaningful security evaluation for agent skills now spans at least three layers: malicious-skill detection, skill-mediated safety failure, and post-vetting runtime confirmation.