SkillFortifyBench: Formal AI Supply Chain Security

• SkillFortifyBench is a benchmark for formal supply chain security of agentic AI skills, defining a rigorous five-phase lifecycle.
• It employs static capability reasoning, SAT-based dependency resolution, and trust score algebra to ensure soundness, confinement, and correct dependency management.
• The benchmark offers a balanced dataset of 540 skills with controlled malicious classes to reproducibly validate formal security properties.

SkillFortifyBench is a benchmark for formal supply chain security of agentic AI skills, introduced alongside the SkillFortify framework in "Formal Analysis and Supply Chain Security for Agentic AI Skills" (Bhardwaj, 27 Feb 2026). It is designed to support rigorous, reproducible evaluation of formal methods for agent skill ecosystems, rather than heuristic scanners alone. The benchmark targets the supply chain surfaces that arise across the five-phase skill lifecycle—Install, Load, Configure, Execute, and Persist—and operationalizes end-to-end evaluation of static capability reasoning, SAT-based dependency resolution with lockfile semantics, and trust score algebra. Its stated purpose is to enable guarantees such as soundness, confinement, and correct dependency resolution on a balanced, reproducible corpus of malicious and benign skills.

1. Motivation and threat model

SkillFortifyBench is motivated by the rapid expansion of agentic AI skill ecosystems such as OpenClaw and Anthropic Agent Skills, combined with the absence of formal supply chain defenses in those ecosystems (Bhardwaj, 27 Feb 2026). The threat context given for the benchmark is explicit: the ClawHavoc campaign infiltrated over 1,200 malicious skills into the OpenClaw marketplace during January–February 2026, MalTool catalogued 6,487 malicious tools that largely evade signature-based scanners such as VirusTotal, and large-scale scans found 26.1% of skills vulnerable. Twelve reactive defense tools appeared as fast-follow responses, but the benchmark is positioned against the limitation that all of them are heuristic; even Cisco’s documentation is quoted as stating that “no findings does not mean no risk.”

The benchmark formalizes a five-phase lifecycle with distinct attack surfaces. Install includes typosquatting and dependency confusion. Load includes prompt injection in metadata. Configure includes over-permissioning. Execute includes data exfiltration and privilege escalation. Persist includes leaky logs and state poisoning. These phases are captured by the DY-Skill attacker model, which adapts Dolev–Yao to the skill supply chain.

The admissible attacker operations are intercept, inject, modify, and drop messages or skills; forge skills; and compromise registries, under a perfect-sandbox assumption paralleling perfect cryptography. The maximality theorem states that for any symbolic attacker $A'$ on the supply chain, there exists a DY-Skill attacker and trace $\tau'$ that reproduces every observable effect, expressed as: for every observable $o$ in an $A'$ execution, there exists $o'$ in $\tau'$ with $\mathrm{effect}(o)=\mathrm{effect}(o')$. This is the formal basis for treating the benchmark as an evaluation environment for the maximal symbolic attacker within the stated assumptions.

2. Dataset scope, balance, and attack classes

SkillFortifyBench contains 540 skills, split evenly between 270 malicious and 270 benign skills (Bhardwaj, 27 Feb 2026). The corpus is also balanced across three formats: 180 Claude Code skills, 180 MCP servers, and 180 OpenClaw skills, with each format containing 90 malicious and 90 benign samples. The balance is exact by construction, and the paper states that this choice is intended to avoid bias in aggregate metrics and to focus evaluation on formal guarantees rather than class priors.

Format	Total	Split
Claude Code	180	90 malicious / 90 benign
MCP servers	180	90 malicious / 90 benign
OpenClaw	180	90 malicious / 90 benign

The malicious portion is organized into 13 attack or vulnerability categories. Each malicious skill embeds exactly one primary attack type, which yields clean attribution and perfect ground truth during evaluation.

Attack ID	Description
A1	Data exfiltration via HTTP
A2	Data exfiltration via DNS
A3	Credential theft from environment variables
A4	Arbitrary code execution
A5	File system tampering
A6	Privilege escalation (capability escalation)
A7	Steganographic data exfiltration
A8	Prompt injection payloads
A9	Reverse shells
A10	Cryptocurrency mining
A11	Typosquatting (Install-phase)
A12	Dependency confusion (Install-phase/dependency poisoning)
A13	Encoded/obfuscated payloads (e.g., base64+shell)

Per-attack-type counts are fixed for controlled per-class evaluation; examples given are A1 = 30, A2 = 18, and A13 = 26. The benchmark’s first version does not include dedicated malicious classes for TOCTOU races, lockfile tampering, or explicit trust manipulation artifacts. Instead, lockfile integrity and trust algebra behavior are evaluated through separate tasks on the same corpus.

The benign skills are described as realistic categories such as file utilities, data formatting, and simple API wrappers, with correct least-privilege declarations. This suggests that the benchmark is intended not merely to separate obviously malicious code from obviously benign code, but to test formal reasoning about declared capabilities, information flow, and dependency behavior under realistic packaging conventions.

3. Construction pipeline and canonical representation

The benchmark is primarily synthetic and curated rather than a scraped marketplace dump (Bhardwaj, 27 Feb 2026). Its pattern sources are ClawHavoc, MalTool, and CVE-2026-25253, with additional taxonomy and distribution validation against "Agent Skills in the Wild" and "Malicious Agent Skills in the Wild." The stated rationale for this design is perfect ground truth and reproducibility.

Generation is constructive and deterministic. The benchmark generator composes skill manifests and minimal implementations from parameterized templates, instantiating attack patterns with seeded randomness such as variable names and benign distractors. Parsers normalize all formats to a unified internal representation

$$s=(id, name, version, declared\ capabilities, code, dependencies).$$

Because labels are assigned during generation, annotation errors are eliminated and inter-annotator agreement is not applicable.

The formal maliciousness criterion is also explicitly specified. A skill is malicious if its concrete code or configuration contains at least one of the following: an unauthorized information-flow path from a sensitive source to an untrusted sink; an operation requiring capabilities strictly exceeding declared capabilities; or an install-time deception pattern such as typosquatting or dependency confusion. Benignness is defined as semantics confined to declared capabilities, with no sensitive-to-untrusted flows and no install-time deception.

The normalized schema includes name, version, format, manifest, code/config, and dependencies. The manifest records declared capabilities per resource—filesystem, network, environment, shell, skill_invoke, clipboard, browser, and database—mapped to levels {NONE, READ, WRITE, ADMIN}. The benchmark supports three canonical packaging forms: Claude Code skills encoded as .md with YAML frontmatter, MCP server JSON, and OpenClaw YAML manifests.

Lockfile semantics are part of the benchmark design. A deterministic skill-lock.json contains the version of the lockfile schema, timestamp, tool version, and a skills map from name to {version, format, sha256 hash, declared capabilities, dependencies, trust_score, analysis status}, plus an integrity field defined as the sha256 of the canonical skills JSON object. Canonicalization uses sorted keys, stable solver ordering, and canonical JSON formatting to ensure reproducibility and tamper detection.

4. Benchmark tasks and formal properties

SkillFortifyBench is structured to exercise multiple formal mechanisms rather than a single classifier (Bhardwaj, 27 Feb 2026). The first task family is static capability analysis and information flow. Abstract interpretation over a four-level capability lattice is used so that inferred capabilities $\mathrm{Cap}_I(s)$ form a sound over-approximation of concrete behavior, and the violation check enforces

$\mathrm{Cap}_I(s)\preceq \mathrm{Cap}_D(s).$

The stated soundness theorem is: if $\tau'$0, then every concrete access $\tau'$1 satisfies $\tau'$2. This is the core formal guarantee behind the benchmark’s use for capability overreach analysis.

The second task family is SAT-based Agent Dependency Graph resolution. Variables $\tau'$3 denote selecting skill $\tau'$4 at version $\tau'$5, and the benchmark evaluates satisfiability under five classes of constraints: at most one version, root requirements, dependencies, conflicts, and capability bounds. The theorem reported is that the conjunction of constraints $\tau'$6–$\tau'$7 is satisfiable if and only if a secure installation exists, and that any satisfying model yields both a secure installation and a deterministic lockfile. This makes dependency closure, version constraints, conflict handling, and policy enforcement part of a single formally justified resolution procedure.

The third task family is capability-based sandbox policy checking. The benchmark uses verification operations Permits, IsSubsetOf, and Violations, together with the static confinement theorem:

$\tau'$8

This is described as a design-time guarantee.

The fourth task family is trust score algebra. The intrinsic trust score is defined from provenance $\tau'$9, behavioral $o$0, community $o$1, and historical $o$2 signals, with non-negative weights summing to 1:

$o$3

Propagation is given by

$o$4

and trust decay by

$o$5

The benchmark evaluates the theorem of trust monotonicity: if any signal increases and weights are non-negative, then $o$6; the algebra is monotone and bounded.

Taken together, these tasks make the benchmark a testbed for formal supply chain security properties rather than only empirical malicious-sample detection. A plausible implication is that SkillFortifyBench is intended to validate proof obligations across the full skill lifecycle, from registry-facing installation decisions to execution-time capability constraints encoded statically.

5. Evaluation protocol and headline results

The evaluation protocol uses standard binary-classification metrics with explicit formulas (Bhardwaj, 27 Feb 2026):

$o$7

The paper reports 95% Wilson confidence intervals for Recall and FPR, and states that the F1 confidence interval is derived analytically from the observed $o$8 and the Wilson interval for Recall, corroborated by resampling.

The labeling policy for evaluation is operational rather than latent: a skill is flagged malicious if at least one finding of severity at least MEDIUM is produced by the formal analyzer, while benign skills have no such findings. Beyond static detection, the protocol also exercises SAT-based ADG resolution and lockfile generation, capability-based policy checks, and trust score algebra computation.

The headline results reported on 540 skills are: precision $o$9, recall $A'$0, F1 $A'$1 with 95% CI $A'$2, and false positive rate $A'$3 with 95% CI $A'$4. Static scanning of the full corpus takes 1.378 s total, or 2.55 ms per skill. SAT-based ADG resolution on synthetic graphs up to 1,000 skills is reported at 92.4 ms, approximately 0.09 ms per skill. Deterministic lockfile generation is described as near-instant for the benchmark corpus.

The paper also states that sound abstract interpretation plus information flow detected 8 additional steganographic exfiltration samples beyond pattern matching. The observed 0% false positive rate on the benign set is tied directly to the benchmark’s soundness-oriented design, and deterministic lockfiles plus upheld trust monotonicity are presented as formal properties observed in evaluation.

6. Baselines, limitations, and practical use

The benchmark is discussed against a set of baseline tools evaluated conceptually rather than through direct head-to-head benchmarking on the corpus: Snyk agent-scan, Cisco skill-scanner, ToolShield, and MCPShield (Bhardwaj, 27 Feb 2026). The reason given for the absence of a direct comparison is reproducibility and licensing. The failure modes emphasized are that heuristics may miss multi-step or obfuscated flows, may not provide formal absence guarantees, and may verify content hashes without semantically justifying dependency resolution. By contrast, the benchmark is designed so that lockfile integrity reflects the satisfying assignment of SAT resolution together with capability and trust constraints.

The practical distribution model is explicit. The repository is hosted at https://github.com/varun369/skillfortify, released under MIT, and installable with pip install skillfortify. The benchmark is organized into /bench/claude, /bench/mcp, and /bench/openclaw, each split into /malicious and /benign, with filenames reflecting attack IDs where applicable. Generation seeds and templates are included so that the corpus can be recreated byte-identically. The evaluation commands are:

• skillfortify scan <bench_root> for E1/E2/E7,
• skillfortify lock <bench_root> for E4/E6,
• skillfortify trust <skill> for E5,
• skillfortify sbom <bench_root> to output CycloneDX ASBOM.

Several limitations are stated directly. Install-time relational attacks such as typosquatting and dependency confusion require registry metadata, so static local analysis has limited power; the paper reports 50% detection for A11 and 0% for A12. TOCTOU and lockfile-tampering samples are not present as malicious artifacts in version 1, although lockfile integrity and determinism are validated as system properties. Dynamic behaviors such as payload fetch-after-install and natural-language disguise attacks are beyond static version 1. The skills each contain one primary attack for attribution, whereas real malware may combine techniques and use heavier obfuscation. The benchmark is balanced by design, so real-world prevalence may differ.

The ethical framing is also constrained: malicious samples are synthetic and minimal, not self-propagating, and intended solely for offline analysis; they should not be executed in a privileged environment. The repository provides warnings and runs analyzers in read-only modes by default.

Taken as a whole, SkillFortifyBench is a formally grounded benchmark for agentic AI skill supply chains. Its distinctive contribution is not merely a labeled corpus, but a corpus coupled to formal attacker modeling, abstract interpretation, confinement checks, SAT-based dependency resolution, deterministic lockfile semantics, and trust algebra. This suggests a shift in evaluation emphasis from heuristic detection alone toward end-to-end validation of formal security properties across the agent skill lifecycle.