SkillsBench Benchmark Overview

• SkillsBench Benchmark is a suite of containerized, requirement-driven tests designed to measure the marginal effect of structured agent skills in LLM workflows.
• It employs deterministic verifiers and paired experimental designs to assess curated and self-generated skills across varied domains, including software engineering and IoT.
• Empirical results highlight that focused, domain-aligned skill modules boost pass rates, emphasizing the importance of alignment, abstraction, and contextual compatibility.

SkillsBench Benchmark is a suite of requirement-driven, containerized benchmarks designed to rigorously assess the utility of agent skills—structured procedural knowledge packages that augment LLM agents at inference time. Unlike traditional model or agent benchmarks that measure baseline capabilities, SkillsBench-style evaluations explicitly quantify the marginal effect of structured skills across diverse, realistic tasks using deterministic verifiers, standardized agent configurations, and paired experimental designs. This family of benchmarks—including the canonical SkillsBench, SWE-Skills-Bench, and IoT-SkillsBench—represents the first systematic frameworks for evaluating, selecting, and optimizing skill artifacts in practical LLM agent workflows (Li et al., 13 Feb 2026, Han et al., 16 Mar 2026, Li et al., 20 Mar 2026).

1. Overview and Motivation

Agent skills are structured, portable procedural units (typically SKILL.md with stepwise instructions, example code, and heuristics) that can be loaded by LLM-based agents at inference time. Their adoption has been rapid, with open repositories hosting tens of thousands of skills across domains. However, prior to SkillsBench, there was no standardized methodology for measuring their actual, domain-specific return on investment relative to baseline agent capability. This lack of standardized benchmarking left researchers and practitioners without empirical guidance for skill authoring, curation, or selection. SkillsBench and its variants directly address this gap by treating skills as first-class experimental variables, benchmarking the effect size of curated or self-generated skills across paired conditions on authentic, end-to-end tasks (Li et al., 13 Feb 2026, Han et al., 16 Mar 2026).

2. Benchmark Design and Methodology

SkillsBench tests 84 containerized tasks spanning 11 domains (Software Engineering, Healthcare, Cybersecurity, Manufacturing, etc.) under three agent configurations: no skills, curated skills, and self-generated skills. Skills are curated from a large-scale quality audit (47,150+ skills; top quartile by completeness, specificity, leakage audit). Evaluation is deterministic: each agent-task trajectory is scored via automated, domain-specific verifiers within Dockerized, resource-constrained environments. The primary metric is pass rate, averaged over independent agent runs and tasks. Absolute improvement ($\Delta_{\mathrm{abs}}$) and normalized gain (Hake's gain, $g$) quantify skill impact. Self-generated skills (authored by the agent) are assessed separately to establish whether agents can bootstrap the procedural knowledge they benefit from consuming (Li et al., 13 Feb 2026).

SWE-Skills-Bench targets software engineering (SWE) specifically, pairing 49 agent skills across six SWE subdomains with 565 real-world development tasks extracted from authentic GitHub repositories. Each requirement is defined by a four-part document (background, requirement, file operations, and deterministic acceptance criteria). Evaluation is run in cloned Docker environments, using LLM-generated PyTest unit-tests for mechanical, binary scoring. Pass rates, delta improvement, token overhead, and cost efficiency are explicitly quantified for each skill (Han et al., 16 Mar 2026).

IoT-SkillsBench extends the paradigm to hardware-in-the-loop (HIL) embedded and IoT development, spanning 42 tasks over three representative platforms (AVR+Arduino, Xtensa+ESP-IDF, ARM+Zephyr) and 23 peripherals. Agents are tested under no skills, LLM-generated skills, and human-expert skills, with both compilation and hardware behavioral correctness as measured endpoints. Pass@1 and Pass@5 (proportion of tasks passed on first or any of five trials) are the primary metrics (Li et al., 20 Mar 2026).

3. Dataset Construction

Benchmark	# Tasks	Domains / Subdomains	Skill Types	Key Features
SkillsBench	84	11 (incl. SWE, Healthcare)	Curated, Self-gen	Human-audited, containerized, deterministic verifiers
SWE-Skills-Bench	565	6 SWE subdomains	49 public	Repo-pinned SWE, PyTest-based, pass/token metrics
IoT-SkillsBench	42	3 platforms, 23 peripherals	LLM, Human-expert	HIL, behavioral/compilation pass, feedback loops

Skills are subject to quality audits: procedural, portable, domain-applicable, and leakage checked. Tasks are stratified by domain, difficulty, and infrastructural realism (e.g., fixed commit-pinning, hardware pin-maps, CI-leakage checks). Acceptance criteria are always measurable—each maps directly to automated or hardware-based testing with no fuzzy matching (Li et al., 13 Feb 2026, Han et al., 16 Mar 2026, Li et al., 20 Mar 2026).

4. Evaluation Protocols and Metrics

Deterministic, paired evaluation is central. Each task is solved under multiple conditions: without skills, with curated skills, and with self-generated (agent-authored) skills. For each configuration:

• SkillsBench
  • Pass Rate ($p$): Fraction of tasks for which all verifier tests pass, averaged across multiple runs.
  • Absolute improvement ($\Delta_{\mathrm{abs}}$) and normalized gain ($g$):

  $$g = \frac{p_\mathrm{with} - p_\mathrm{no}}{100\% - p_\mathrm{no}} \times 100\%$$

• SWE-Skills-Bench

  • Per-skill pass rate, utility delta ($\Delta P$), token overhead ($\rho$), and cost efficiency ($\mathrm{CE}$):

  $$\Delta P(s) = \mathrm{Pass}^+(s) - \mathrm{Pass}^-(s)$$

  $g$0

  $g$1

• IoT-SkillsBench

  • Pass@1, Pass@5, and detailed breakdown by agent configuration and task difficulty.
  • Token cost per agent node (manager/coder).
  • Ground truth via HIL behavioral validation, with hardware oracles and human-in-the-loop for ambiguous cases (Li et al., 13 Feb 2026, Han et al., 16 Mar 2026, Li et al., 20 Mar 2026).

5. Experimental Results

Aggregate Uplift and Domain Variance:

In SkillsBench, curated skills yield a mean pass rate improvement of +16.2 percentage points, but effects vary widely: +4.5 pp in Software Engineering versus +51.9 pp in Healthcare. Self-generated skills show no reliable benefit (average –1.3 pp). Focused skills (2–3 concise modules, 500–2,000 tokens each) maximize gains; excessive or overly comprehensive skills degrade performance, and negative deltas are not rare (16/84 tasks).

SWE-Skills-Bench shows marginal mean uplift: average pass-rate increased only +1.2%. Of 49 skills tested, 39 yielded zero improvement, 7 provided meaningful gains (up to +30%), and 3 degraded performance (up to –10%). Token overhead spanned from –77.6% (token savings) to +450.8% (massive overhead) even in cases with no improved correctness. Only highly specialized skills (e.g., financial-risk calculation) manifested significant utility; broad or generic skills were redundant with base model knowledge or harmful due to contextual mismatch (Han et al., 16 Mar 2026).

IoT-SkillsBench found that no-skills agents solve only basic hardware tasks (100% Level 1, <40% Level 2, ~15% Level 3). LLM-generated skills offered inconsistent, often insufficient improvements, occasionally reinforcing incorrect assumptions. Human-expert skills, by contrast, resulted in near-perfect behavioral correctness (Pass@5 up to 100% Level 1/2, 98% Level 3) across all platforms, with moderate token overhead (~3k–4k tokens) (Li et al., 20 Mar 2026).

6. Factors Determining Skill Utility

Empirical outcomes demonstrate that skill injection is not a generic panacea. Three factors predominantly determine skill utility:

• Domain Fit: Only skills encoding missing, domain-specific procedural gaps (e.g., precise risk formulas) provide reliable benefit. Generic skills typically duplicate knowledge already present in pretrained weights.
• Level of Abstraction: Highly procedural, checklisted, or pattern-driven skills outperform rigid, monolithic templates with hardcoded parameters, which can harm performance by anchoring agents to irrelevant defaults.
• Contextual Compatibility: Skills need tight alignment with the task's actual stack—framework version, conventions, peripheral mappings. Version or abstraction mismatches introduce spurious or conflicting guidance, leading to hallucinations or misconfigurations (e.g., injecting Linkerd v1beta1 patterns into a v1beta3 codebase) (Han et al., 16 Mar 2026).

7. Implications and Best Practices

The collective benchmarks suggest clear guidelines for skill design and deployment:

• Curated human expertise consistently outperforms self-generated or excessively verbose skills.
• Limit skills to 2–3 focused modules; excessive skill count or length dilutes impact.
• Abstract, checklist-driven skills strike a safer balance than concrete code templates unless the latter encode critical procedural knowledge absent in foundation models.
• Skill selection should dynamically match task and repository metadata (language, domain, framework) to avoid injecting irrelevant or harmful skills.
• In production, actively monitor token overhead relative to correctness gain; disable or prune skills that inflate context windows without measurable benefit.
• Paired, deterministic evaluation (“with-skills” vs “no-skills”) is essential for measuring real augmentation, as absolute pass rates miss marginal effects.
• Skill injection should be reserved for high-leverage, high-risk tasks where model pretraining demonstrably fails (Han et al., 16 Mar 2026, Li et al., 13 Feb 2026).

8. Benchmark Infrastructure and Extensibility

SkillsBench and its domain-specific variants offer open-source, reproducible infrastructure supporting extension to new platforms, domains, and agent orchestration patterns. New tasks can be instantiated by providing natural language specifications, reference implementations, and verifiable acceptance criteria mapped to automated or hardware-based test harnesses. The methodology facilitates automated skill synthesis research, benchmark development for multi-agent tool use, and studies of skill-interactions or interference. Strict task and skill curation protocols (leakage audits, realistic scenarios) are emphasized to maintain ecological validity and standardization across studies (Li et al., 13 Feb 2026, Li et al., 20 Mar 2026).

---

SkillsBench benchmarks provide a rigorous foundation for empirical study of the real-world marginal utility of agent skills, establishing critical guidance for the design, selection, and evaluation of skill-augmented LLM agent systems. Their evidence underscores: skills are not a free lunch—their efficacy is domain-, abstraction-, and context-dependent, and must be paired with precise operational protocols to maximize tangible benefit.