---
title: 'MalSkillBench: Verified Malicious Skills'
url: https://www.emergentmind.com/topics/malskillbench
type: topic
---

# MalSkillBench: Verified Malicious Skills

Searching arXiv for the specified papers and topic to ground the article in current research.
MalSkillBench is a runtime-verified benchmark of malicious agent skills for AI coding agents such as Claude Code and Gemini CLI. In this setting, a skill is simultaneously code, natural-language instruction, and a permission-bearing extension, so the relevant attack surface is neither pure code nor pure prompt. MalSkillBench was introduced as “the first runtime-verified benchmark of malicious agent skills,” with 3,944 malicious skills labeled across a three-dimensional taxonomy of 108 cells, 4,000 matched benign skills, and a verification pipeline that admits only samples whose malicious behavior fires inside a Docker sandbox under system-call monitoring and an LLM judge [2606.07131]. The term “MalSkillBench” also appears in adjacent 2026 literature as a colloquial name for SkillSafetyBench and as a shorthand in one consolidated description of SkillVetBench, which makes nomenclature somewhat broader than the title of a single benchmark paper [2605.12015] [2606.00925].

## 1. Naming and scope

The canonical use of the name refers to the benchmark introduced in “MalSkillBench: A Runtime-Verified Benchmark of Malicious Agent Skills” [2606.07131]. Its stated goal is to provide ground truth for the “full hybrid attack surface—code injection, prompt injection, and their combinations—so that detectors can be measured reliably.”

At the same time, nearby work uses closely related terminology for distinct evaluation targets. This is important because the benchmarks differ in unit of analysis, threat model, and metric design.

| Benchmark term in the literature | Stated focus | Scale stated in the paper |
|---|---|---|
| MalSkillBench | “runtime-verified benchmark of malicious agent skills” | 3,944 malicious skills and 4,000 benign skills |
| SkillSafetyBench | “runnable benchmark for evaluating such skill-mediated safety failures”; described in one summary as a “colloquial name for SkillSafetyBench” | 155 adversarial cases across 47 tasks |
| SkillVetBench | “two-stage security vetting benchmark for open agentic skill ecosystems”; described in one summary as “referred to here as MalSkillBench” | 78 confirmed-malicious skills and 22 benign skills |

This naming overlap suggests that “MalSkillBench” has functioned both as a proper benchmark name and as an informal umbrella label for malicious-skill or skill-facing security evaluation. A plausible implication is that citations should be anchored to the exact arXiv identifier rather than to the name alone when comparing results across papers.

## 2. Corpus design and taxonomy

MalSkillBench’s dataset composition is explicit. The malicious corpus contains 3,944 skills in total: 3,214 generated via a closed-loop Generate-Verify-Feedback pipeline, 703 collected “in the wild” and manually verified, and 27 additional samples drawn from existing tool test suites. The benign side contains 4,000 high-download skills from ClawHub, used to measure false positives [2606.07131].

The benchmark organizes malicious skills with a three-dimensional taxonomy. The first dimension is **Attack Vector**, with three categories: **CI (Code Injection)**, **PI (Prompt Injection)**, and **MIXED**. The second dimension is **Malicious Behavior**, with 15 categories. The paper distinguishes **B1–B9** as code-level behaviors—Data Exfiltration, Credential Theft, Remote Code Execution, Malware Delivery, Persistence, Reverse Shell, Ransomware, Resource Abuse, and Privilege Escalation—and **B10–B15** as agent-control behaviors—Role Hijack, Safety Bypass, Instruction Override, System Prompt Leak, Goal Hijacking, and Content Manipulation. The third dimension is **Insertion Strategy**, which varies by vector: CI uses New Script File, Function Append, Function Inject, and Inline Code Block; PI uses Full Camouflage, Partial Injection, and Steganographic; MIXED uses Download + Execute, Config + Load, and Fetch + Run [2606.07131].

The resulting coverage matrix has 108 cells: 36 for CI, 45 for PI, and 27 for MIXED. This structure matters because it operationalizes the claim that malicious skills are hybrid artifacts. The benchmark therefore encodes not only *what* the attack does, but also *where* it is embedded and *how* it is staged.

## 3. Generate-Verify-Feedback pipeline

MalSkillBench’s synthetic portion is built by a closed-loop Generate-Verify-Feedback pipeline. In the **Generation Agent**, knowledge retrieval uses real malicious code snippets from IntelliGraph, covering 3,026 PyPI packages, for CI attacks, and real jailbreak payloads from prompt-injection corpora, covering 20,961 samples, for PI attacks. Template selection then samples a benign “facade” skill from a pool of 3,458 SkillsMP skills across 12 categories. Skill synthesis prompts an abliterated Qwen3.5-35B LLM with the target vector-behavior-strategy triple, retrieved examples, and the benign template, while constraining the output so that CI embeds executable payload in the correct script location, PI leaves scripts syntactically benign and hides instructions per strategy, and MIXED stages an intermediate artifact in markdown for the script to consume. The artifact output is parsed into a skill bundle containing `SKILL.md`, `scripts/`, `assets/`, and `_expected.json`, where `_expected.json` declares the expected observable behaviors \(E\) [2606.07131].

The **Verification Agent** executes the synthesized skill in a Docker container with OpenCode agent, Python runtime, `strace -f`, and `inotifywait`, triggering the skill with a prompt matching its advertised benign purpose. It collects a system-call trace \(\tau = \{\tau_{\text{net}}, \tau_{\text{file}}, \tau_{\text{proc}}\}\) and agent text output \(o\). Verification then proceeds in two layers. **Layer 1** is deterministic evidence matching: for CI and MIXED, the trace is matched against IOCs in \(E\), such as `connect()` to a malicious domain or `execve()` of a payload; for PI, the output is matched against declared keyword IOCs and canonical refusals. **Layer 2** is semantic LLM judgment: for CI and MIXED, GPT-5.4-mini judges each expected behavior as Observed, NotObserved, or Ambiguous with confidence \(\text{conf}(e)\in[0,1]\); for PI, it judges the output into \(\{\text{triggered}, \text{detected}, \text{ineffective}\}\) with confidence. The pass condition is stated as  
$$
\text{pass} \iff \exists\,e\in E:\; m_{1}(e)\;\lor\;\text{conf}(e)\ge\theta
$$  
with threshold \(\theta = 0.7\). Rejected samples produce feedback \((\text{reason}, E_{\text{observed}}, \text{suggestion})\) and are retried up to \(r=3\) times [2606.07131].

This design is central to the benchmark’s claim of verified ground truth. The benchmark does not rely on unverified synthetic labels; rather, generated samples are admitted only when malicious behavior is observed in the sandbox.

## 4. Verification yield and empirical structure of the wild sample

MalSkillBench reports distinct runtime verification yields by vector. **CI yield = 94.5%**, **PI yield = 75.8%**, and **MIXED yield = 91.9%** [2606.07131]. The paper explicitly notes that “code injection reaches 94.5% verification yield but prompt injection only 75.8%, the same fragility that later makes it hard to detect.” This suggests that prompt-layer attacks are not merely a detection problem; they are also less stable as executable, benchmarkable artifacts.

The wild sample contains **703 skills**. Its delivery mechanism is heavily concentrated: **607/703 (86.3%)** use fake-prerequisite dependency impersonation. Its behavior distribution is similarly narrow: **609/703 (86.6%)** map to Malware Delivery (B4) via the fake dependency. Authorship is concentrated as well: **two accounts produced 569/703 (81%)** of the wild samples, largely a crypto-theft campaign targeting wallet keys and seed phrases [2606.07131].

The paper emphasizes a smaller but architecturally distinct tail of wild attacks. The **novel agent-control tail (1.7% of wild)** includes session lifecycle persistence by inserting hooks into `.claude/settings.json`, identity or goal rewriting by editing the agent’s system prompt to “serve Project Exodus,” and instruction hierarchy override through top-of-document “SYSTEM OVERRIDE” policies [2606.07131]. In context, this matters because a wild-only sample dominated by one campaign can conceal behaviors that are strategically important but numerically rare.

## 5. Detector evaluation and the limits of partial defenses

MalSkillBench evaluates **12 skill-specific detector configurations** on the full benchmark of 3,944 malicious and 4,000 benign skills. The strongest overall result is **Sentry Skill Scanner (full, static+LLM)** with **Recall 98.4%**, **Precision 80.5%**, and **\(F_1 = 88.6\%\)**, with **937 false positives**. A more balanced point is **AI-Infra-Guard (Tencent)** at **Recall 86.6%**, **Precision 84.6%**, and **\(F_1 = 85.6\%\)**, with **620 false positives**. **Cisco Skill Scanner (LLM mode)** achieves **Recall 92.7%** and **Precision 71.4%**, but with **1,465 false positives**. At the conservative end, **Snyk Agent Scan** and **VirusTotal** produce low false positives—**82** and **4**—but low recall, **28.7%** and **21.6%**, respectively [2606.07131].

The paper identifies systematic detection gaps. “All drop sharply on PI-only and agent-control behaviors,” reaching **20–40% recall on B12–B15** [2606.07131]. This is a substantive benchmark result, not merely a failure of any single tool. It indicates that the hardest cases are those where malicious intent resides in instructions, role manipulation, or cross-component coordination rather than in obvious payload code.

A further finding concerns evaluation methodology. On **wild-only vs. full benchmark recall**, **VirusTotal jumps from 21.6% recall (full) to 87.9% recall (wild)**, a change of **+66 points**; **Snyk Agent Scan** increases by **+49 points**, while **AI-Infra-Guard** and static tools fall by **13–21 points** [2606.07131]. The benchmark therefore argues that wild-only scoring is misleading because it ranks tools according to a narrow wild distribution rather than according to the full attack space.

Transferred tools reinforce the same point. Supply-chain scanners such as OSSGadget and SAP reach **recall ≈ 97–99%** but flag **≈ 90% of benign skills**; prompt-injection defenses such as DataSentinel and NeMo Guardrails reach **recall ≈ 96–100%** but flag **≈ 90–100% of benign skills**. The paper’s conclusion is direct: “Supply-chain scanners and prompt-injection defenses each see only half of a skill, and no combination recovers the code-instruction relationship” [2606.07131].

## 6. Evasion, downstream use, and behavior-centric auditing

The MalSkillBench dataset is used directly in “Cloak and Detonate: Scanner Evasion and Dynamic Detection of Agent Skill Malware,” which studies adaptive bypasses against static and hybrid skill scanners [2607.02357]. That paper treats MalSkillBench as a publicly archived corpus, begins from the **703 skill bundles** collected from OpenClaw/ClawHub, and retains **622 skills whose payload was fully reproducible** for its “Wild” evaluation, alongside **150 benign skills** randomly sampled from popular marketplace bundles.

Its adversarial contribution, **SkillCloak**, has two strategies. **Structural Obfuscation** rewrites visible payload indicators into semantically equivalent forms through operators such as zero-width insertion, confusables, case-flip, synonym substitution, `getattr/chr`, var-splice, per-line split, per-file relocation, blind-spot directory moves, extension renaming, and size padding. **Self-Extracting Skill (SFS) Packing** hides the malicious skill from the install-time view, then decrypts and restores it during agent execution [2607.02357].

The evasion results are severe. Across eight scanners and **1,613 in-the-wild malicious skills**, **SFS Packing bypasses every scanner at over 90%**, while **Structural Obfuscation bypasses over 80% on most static scanners and reaches 96% on a hybrid scanner** [2607.02357]. Example scanner-specific numbers include **Huifer**, where raw flagged findings fall by **96.3%** under Structural Obfuscation and by **99.8%** under SFS Packing, and **Vigile**, with **84% reduction** under Structural Obfuscation and **99.9%** under SFS Packing [2607.02357].

Motivated by this, the paper proposes **SkillDetonate**, a behavior-centric runtime auditor. SkillDetonate combines **on-demand closure lift**, which observes instructions materialized during execution, with **marker-based systemwide taint analysis**, which tracks sensitive-data flows across the agent context, files, processes, and network operations. On the **622-skill MalSkillBench “Wild” subset**, SkillDetonate detects **87.6%** of original skills, **83.8%** of obfuscated skills, and **90.5%** of packed skills, for **87.3% average detection**; on a separate SkillJect benchmark it averages **96.7% detection** with **2% benign false-positive rate** [2607.02357]. This downstream use frames MalSkillBench not only as a detector benchmark but also as a pressure test for robustness against adaptive scanner evasion.

## 7. Relationship to adjacent benchmarks

MalSkillBench belongs to a broader 2026 family of skill-security benchmarks, but adjacent papers target different layers of the problem.

**SkillSafetyBench** evaluates “skill-mediated safety failures” rather than malicious-skill detection. It includes **155 adversarial cases across 47 tasks, 6 risk domains, and 30 safety categories**, with reusable skills, task-relevant skill materials, local artifacts, runnable environments, and case-specific rule-based verifiers. It measures **Attack Success Rate (ASR)** and shows overall ASR ranging from **≈15.5%** to **≈50.3%**, with **Context Trust ≈ 58.7%** as the highest mean per-domain vulnerability and **Supply Chain ≈ 20.1%** as one of the lower ones [2605.12015]. The conceptual distinction is that SkillSafetyBench studies how benign requests can be subverted by non-user “skill-facing” materials, whereas MalSkillBench studies malicious skills themselves as supply-chain artifacts.

**SkillVetBench**, presented in “Benchmarking Security Risk Detection and Verification in Open Agentic Skill Ecosystems,” frames the problem as two-stage vetting: semantic vetting followed by sandbox execution. It uses **78 confirmed-malicious skills** from the live OpenClaw ClawHub repository and **22 benign skills** as controls, scoring skills with a **Skill Agentic Risk Score (SARS)** over five dimensions—Instruction Fidelity Risk, Data Gravity, Action Irreversibility, Blast Radius, and Chain Amplification—then escalating suspicious skills to runtime verification in an isolated Docker-based OpenClaw sandbox [2606.00925]. Its experiments report that static baselines miss **up to 89%** of instruction-layer threats, while the benchmark’s semantic stage flags all **78 confirmed-malicious skills** with **zero false negatives** and **zero false positives** on the **22 benign controls** [2606.00925].

Taken together, these benchmarks define complementary evaluation regimes. MalSkillBench is centered on runtime-verified malicious skill artifacts and detector benchmarking; SkillSafetyBench is centered on agent safety under skill-facing attack surfaces; SkillVetBench is centered on two-stage vetting in an open agentic skill ecosystem. A plausible implication is that meaningful security evaluation for agent skills now spans at least three layers: malicious-skill detection, skill-mediated safety failure, and post-vetting runtime confirmation.

Source: https://www.emergentmind.com/topics/malskillbench