- The paper introduces MaliciousSkillBench, a conflict-filtered benchmark of 9,740 Skills—7,505 malicious and 2,235 benign—consolidated from 13 public sources with artifact-level provenance, evidence, deduplication, and reuse controls.
- Learned static detectors achieve 0.882–0.932 Random Macro-F1 but fall to 0.653–0.665 under Source-Disjoint evaluation, retaining 94.4–95.7% malicious recall while falsely flagging 62.0–64.4% of benign Skills.
- The benchmark shows existing scanners do not solve the trade-off: Cisco and SkillSpector keep benign false-positive rates near 1% but detect 2.5% or 0% of malicious Skills, while SkillFortify reaches 25.3% recall at 49.9% false positives.
Motivation and problem statement
Agent Skills extend LLM agents with installable packages that combine natural-language instructions, scripts, resources, and service configuration. Because an installed Skill functions as trusted procedural authority inside the agent's workflow, a malicious Skill constitutes a pre-installation supply-chain risk: prior work documents credential theft, behavior manipulation, triggered backdoors, and concealed unsafe side effects in such artifacts. The authors observe that existing Skill-security datasets are not interchangeable: they release different objects (complete artifacts, task pairs, runtime-verified cases, vulnerability records, scanner verdicts), rely on different evidence regimes, and overlap substantially through duplicated or scaffold-related content. Naively concatenating published rows therefore inflates apparent scale and can place the same content on both sides of a train/test split.
MaliciousSkillBench addresses this by consolidating 13 frozen public sources into a single artifact-level detection benchmark with explicit identity, reuse, evidence, and conflict accounting, and by evaluating both learned detectors and off-the-shelf scanners under protocols that separate random-split performance from structural and source-conditioned generalization.
Benchmark construction
The pipeline proceeds in four stages: source qualification, evidence-preserving canonicalization, identity/reuse control, and conflict-filtered finalization.
Source qualification. Of 13 frozen sources, 11 satisfy a conservative Core rule requiring a recoverable Skill artifact, a preserved source-native malicious claim with supporting evidence, inert static content, and traceability to a pinned revision. These contribute 8,414 raw malicious artifacts spanning wild, synthetic, injected/backdoored, and test-fixture provenance. Two sources (SRC002 MaliciousAgentSkillsBench and SRC004 SkillLeakBench) required documented recovery from author-provided historical research snapshots because their public releases contain metadata but not raw artifacts; four ambiguous SRC004 multi-Skill cases remain outside Core. Sources releasing only task pairs or silver scanner labels (e.g., Skill-Inject, ClawHub Security Signals) are retained as Auxiliary rather than promoted to ground truth — notably, even literal "malicious" scanner verdicts remain canonical intent uncertain under the mapping policy.
Canonicalization. Every record receives a stable canonical ID while retaining source ID, label, revision, artifact type, provenance, label strength, evidence level, hashes, and lineage metadata. Intent, provenance, confidence, and evidence are modeled as orthogonal axes; Core malicious ground truth contains only strong (8,237) or moderate (177) labels supported by constructed, human+runtime, static, or runtime evidence — never scanner-only supervision.
Identity and reuse control. Three notions of relatedness are kept distinct: exact SHA-256 identity, conservative normalized-text identity (formatting-only normalization that preserves substantive text), and operational structural families at a frozen similarity threshold of 0.68 over template-normalized text. Deduplication reduces 8,414 raw records to 7,562 exact-unique and 7,539 normalized-unique identities, partitioned into 4,588 structural families (3,219 singletons; largest family 146 identities). Cross-source reuse is substantial: for example, SRC009 and SRC013 share no exact or normalized hash yet share 26 structural families covering 42–63% of their normalized units. A blinded positive-only review of 72 sampled within-family pairs found all to be same-template at high confidence, though this validates only sampled inherited pairs and leaves clustering recall and threshold semantics unvalidated.
Benign pool and conflicts. From 48,217 benign candidates, only 2,251 raw Main-benign artifacts pass a gate requiring an inspectable artifact plus strong/moderate benign evidence; the remaining ~46k Auxiliary negatives are excluded from primary ground truth. Thirty-four normalized identities receive conflicting malicious/benign labels across sources; all are conservatively excluded rather than adjudicated. The frozen benchmark contains 9,740 Skills: 7,505 malicious and 2,235 benign, and every protocol starts from this same master table.
Threat landscape characterization
Using only direct or deterministic strong-semantic source-native mappings, 4,983 of 7,505 malicious identities (66.4%) receive multi-label annotations across 11 attack categories. Execution/Code Delivery dominates at 3,320 identities (66.6% of the mapped subset), followed by Instruction/Goal/Memory Manipulation (1,671) and Privilege/Tool/Authority Abuse (1,013). Conservative derived-impact mappings cover only 2,128 identities (28.4%), with 1,888 having both mappings.
The key finding is source concentration: SkillTrustBench alone contributes 2,755 of the 3,320 execution-coded identities and 1,258 of the 1,671 instruction-manipulation identities, while other categories draw on different sources entirely. No single existing dataset represents the observed threat landscape, which is itself presented as a benchmark finding motivating multi-source construction and source-aware evaluation. The authors correctly note these are descriptive co-occurrences, not causal transitions, and that aggregate distributions should not be read as population estimates beyond the collected sources.
Evaluation design
All evaluations use the frozen master table under four protocols: Random (label-stratified 70/10/20), Source-Balanced Random (a composition diagnostic), Malicious-Structural-Disjoint (each of the 4,588 families atomic across partitions), and Source-Disjoint (SRC009/SRC011/SRC012 held out entirely after removing eight cross-boundary identities). Learned baselines use only inert primary Skill instruction text — word TF-IDF with logistic regression or linear SVM, and char TF-IDF SVM — with source/provenance/family metadata excluded from features. Three off-the-shelf scanners (Cisco-local-behavioral, SkillFortify-offline, SkillSpector-static) run with pre-registered gates on the same static artifacts. Leakage audits confirm each protocol satisfies its declared disjointness contract, and non-zero overlaps on uncontrolled axes (e.g., 321 cross-partition structural families under Source-Disjoint) are explicitly distinguished from leakage.
Finding 1: random splits overstate robustness
Random Macro-F1 spans 0.882–0.932 across the three learned models. Structural-family disjointness costs little (a drop of only 0.016–0.038), whereas Source-Disjoint collapses Macro-F1 to 0.653–0.665. The strongest word TF-IDF SVM scores 0.932 / 0.916 / 0.665 on Random / Structural-Disjoint / Source-Disjoint. Controlling scaffold reuse therefore changes performance modestly relative to the gap created by holding out entire sources, indicating that source identity bundles collection venue, construction procedure, documentation style, and labeling policy into a dominant shift factor.
Finding 2: the main cross-source failure is benign over-flagging
On Source-Disjoint, learned detectors retain 94.4–95.7% malicious recall but incur 62.0–64.4% benign false-positive rates. The word-SVM flags 340 of 545 held-out benign Skills while missing only 37 of 839 malicious ones; SRC011 alone contributes 293 false positives. A format-only baseline (Random Macro-F1 0.486) confirms lexical content, not coarse formatting, drives performance. Class balancing lowers the word-SVM's Source-Disjoint FPR from 62.4% to 43.3% and raises Macro-F1 to 0.710, and audited scaffold sanitization leaves the gap intact (sanitized Source-Disjoint Macro-F1 0.643–0.662) — so neither class imbalance nor the seven audited construction-wrapper families explains the degradation, though the paper concedes these controls do not establish a unique causal mechanism given entangled source factors. An auxiliary frozen MiniLM embedding baseline shows the same regime (Source-Disjoint recall 0.895, FPR 0.749), suggesting sparse features are not solely responsible. Exploratory leave-one-source-out results show wide heterogeneity (Macro-F1 0.329–0.733 across held-out sources).
Finding 3: existing scanners do not resolve the trade-off
On the same Source-Disjoint test, Cisco and SkillSpector keep benign FPR at 1.1% and 0.55% but detect only 2.5% and 0% of malicious Skills; SkillFortify reaches 25.3% recall at 49.9% FPR. No evaluated detector simultaneously achieves high malicious recall and low benign FPR on held-out sources. SkillFortify also benefits from the related SRC012/SkillFortifyBench holdout: excluding SRC012 drops its Source-Disjoint Macro-F1 from 0.349 to 0.254 and worsens both recall and FPR — reported as source-overlap sensitivity, not memorization. Threat-stratified analysis shows SkillFortify's recall varies widely by category (81.5% on Credential Access down to 25.8% on Integrity/Output Manipulation), conditional on the mapped subset and confounded by source composition. Only three technical abstentions occur across 29,220 scanner jobs, none in Source-Disjoint.
Limitations and threats to validity
The paper is explicit about its boundaries. "Comprehensive" means traceable consolidation of 13 frozen public sources; coverage is bounded by them. Source-Disjoint measures source-conditioned stress with provenance, construction, labeling, and class composition coupled — it is not a causal estimate of unseen-source difficulty, and the held-out sources have markedly different label mixtures (SRC009 is 100% malicious, SRC011 4.4%). Attack mappings cover 66.4% of malicious identities and impact mappings 28.4%, leaving substantial unmapped portions. The Main-benign pool favors label confidence over ecological balance, so benign source coverage is narrower than malicious coverage — which directly contributes to the over-flagging failure mode being measured. All detection is static on primary Skill text, excluding package-level and runtime behavior; SkillSpector runs without its LLM mode; Cisco ran with a yara-x dependency deviation. Structural families carry no campaign or attack-ontology semantics, and the blind validation covers only sampled inherited pairs. Five credential-bearing malicious records are released only as sanitized representations that are not bit-for-bit identical to the frozen experimental inputs.
Conclusion
MaliciousSkillBench consolidates fragmented Skill-security resources into a conflict-clean, deduplicated 9,740-unit benchmark with explicit provenance, evidence, reuse, and threat accounting. Its characterization shows existing datasets cover complementary, source-specific slices of the attack surface, and its evaluation establishes a persistent two-sided trade-off: learned static detectors preserve high malicious recall by over-flagging unfamiliar benign Skills, while fixed scanners reduce false alarms only at sharply lower recall. The open question the paper leaves is whether any detector can achieve high malicious recall and low benign FPR jointly across held-out sources, and what additional benign diversity or representation would close the source-conditioned gap that survives both balancing and scaffold-sanitization controls.