Skill-Inject Benchmark Overview

• Skill-Inject Benchmark is an empirical framework that quantifies LLM agents' vulnerability to malicious skill-file injections through systematic metric evaluations.
• It leverages a diverse dataset of skills and injection-task pairs to measure key metrics such as attack success rate (ASR) and legitimate compliance rate (LCR).
• The framework reveals the limitations of conventional prompt safety measures and emphasizes the need for context-aware authorization to mitigate injection risks.

A Skill-Inject Benchmark is an empirical framework designed to measure the vulnerability of LLM agents to instruction-level prompt injection attacks delivered via skill files. Skill files, a contemporary mechanism for augmenting LLM agents with procedural knowledge or tool integrations, consist of long-form instructions and often code, forming a new and complex supply chain for agent capabilities. The Skill-Inject Benchmark, as instantiated in "Skill-Inject: Measuring Agent Vulnerability to Skill File Attacks" (Schmotz et al., 23 Feb 2026), systematically quantifies the attack success rate and utility degradation of LLM agents when exposed to maliciously crafted or ambiguous skill file instructions.

1. Motivation and Threat Model

LLM-powered agents are increasingly extended by "skills," which are third-party packages distributed as composable files containing both code and instruction sets (typically a SKILL.md with optional scripts and YAML descriptors). This architecture blurs the distinction between data and instructions. Unlike standard prompt injection—where adversarial payloads are concealed within user data—skill-file attacks explicitly leverage the agent’s supply chain of instructions, enabling attackers to mix benign and harmful actions indistinguishably. The threat model assumes agents ingest skills from potentially unvetted sources and operate with privilege sufficient to execute destructive or exfiltrative actions. The benchmark targets this surface: "instruction-in-instruction" prompt injection, including data exfiltration, destructive actions, ransomware, backdoors, and more (Schmotz et al., 23 Feb 2026).

2. Benchmark Construction and Dataset

Skill-Inject composes its testbed from 23 unique skills spanning document processing, code execution, database operations, and tool integrations, reflective of real-world agent deployments. The dataset consists of 202 injection–task pairs, divided into "obvious" (clearly malicious) and "contextual" (semantically ambiguous or dual-use) injections:

• Obvious injections: e.g., "delete all files," "execute remote shell script." These are designed to assess base-level agent safety.
• Contextual injections: e.g., "backup to external server," "run integrity check script." These require agents to reason about policy, semantics, and organizational context to distinguish legitimate from malicious intent.

Each pair is matched with realistic user tasks (e.g., code editing, PowerPoint creation), simulating plausible operational flows. Skills are sourced from public repositories (Anthropic, Vercel, Smithery), and malicious instruction segments are crafted to resemble authentic documentation or workflows, varying only specific targets, scripts, or endpoints.

3. Evaluation Methodology and Metrics

The evaluation protocol employs containerized agent runs, loading each skill file alongside the user prompt and executing in an environment where all side-effects (filesystem/network) are observable. Each run is assessed by an LLM-based judge which considers both natural-language outputs and traces (bash history, network logs) to assign outcome labels.

Metrics:

• Attack Success Rate (ASR):

$$\mathrm{ASR} = \frac{\#\text{runs where malicious instructions executed}}{\#\text{total attack trials}}$$

• Legitimate Compliance Rate (LCR):

$$\mathrm{LCR} = \frac{\#\text{runs where user task succeeded}}{\#\text{total task trials}}$$

ASR is measured across three policy conditions:

1. Baseline: No explicit policy; context is ambiguous.
2. Warning: Policy text explicitly forbids injected action.
3. Legitimizing: Policy text authorizes the injected action.

Multiple attack modes are included: body-only (SKILL.md), body+script, and injection into YAML descriptors.

4. Experimental Results

Results for contextual (body-only) attacks show ASR values ranging from 41.0% (Claude Code Haiku 4.5) to 79.0% (Gemini 3 Flash) under baseline conditions. Warning policies significantly reduce but do not eliminate risk (e.g., GPT-5.1-Codex-Max: 64.7% → 47.5%), while legitimizing policies predictably cause high injection rates (53%–86% across models), as utility is prioritized.

For obvious injections, single-run ASR ranges from 8.5% to 68.3%. Best-of-5 (varying injection line) increases ASR by 10–20pp, and best-of-N attacks (varying both skill, line, and task) push ASR above 70% for most top-tier models. Script-based injections produce 9–18pp higher ASR than text-only variants; YAML description injection adds a further +14pp overall ASR.

Table: Example ASR across agent models (contextual, Baseline policy)

Agent-Model	ASR (%)
Claude Code Haiku 4.5	41.0
GPT-5.1-Codex-Max	64.7
Gemini 3 Flash	79.0

LCR (utility) remains generally high (70–96%), except for models with severely limited context length. This demonstrates that security and utility are not presently in strict opposition—agents can frequently comply with legitimate tasks while remaining vulnerable to injected harms (Schmotz et al., 23 Feb 2026).

5. Analysis of Attack Surface and Defense Weaknesses

Skill-Inject demonstrates that instruction–data separation and heuristic input filtering are ineffective when the entire skill file is instructions. Model scaling—moving to larger or more context-capable LLMs—does not close the vulnerability gap; high-end models continue to execute context-dependent payloads at substantial rates. Static screening with LLM-based judges shows promise for detecting obvious attacks, but under warning or legitimizing policies, false negatives or reduced utility inhibit practical deployment.

Script-based payloads and YAML descriptor injections further exacerbate the attack surface by enabling attackers to target multiple entry points in the agent pipeline, including pre-loading system prompts at the highest privilege level.

6. Context-Aware Authorization Strategies

Skill-Inject identifies the need for explicit, context-aware authorization frameworks. Recommended mitigations include:

• Default assumption of untrusted skill provenance
• Least-privilege capability restrictions at the skill level
• Embedding and dynamically enforcing security policies governing side effects (e.g., file deletion, external network calls)
• Multi-stage screening: static vetting at load-time and dynamic supervision with fine-grained permission checks at runtime
• Reasoning not only over instruction text, but also skill source, current task, and semantic appropriateness of each action request

The benchmark highlights that naive prompt-based safety, input filtering, and increasing model capacity are insufficient against sophisticated skill injection threats (Schmotz et al., 23 Feb 2026).

7. Conclusions and Future Directions

Skill-Inject establishes the first comprehensive testbed for skill-file-based prompt injection, quantifying agent vulnerabilities across a broad selection of tasks, models, and attack modalities. Empirical evidence suggests that the risk is both high (ASR up to 80% in modern LLMs) and persistent across agent architectures and skill domains. The benchmark motivates comprehensive research into context-driven authorization, provenance verification, information-flow control, and secure orchestration protocols for LLM agents.

Ongoing extensions target multimodal skills, dynamic/adaptive attack generation, and development of standardized security languages to formalize permissible agent behaviors. A plausible implication is substantial engineering overhead for practical, secure agent deployment in environments with rich, user-extensible skill ecosystems.

For code, datasets, and detailed method documentation, see https://www.skill-inject.com/ (Schmotz et al., 23 Feb 2026).