SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks

Abstract: Agent Skills are structured packages of procedural knowledge that augment LLM agents at inference time. Despite rapid adoption, there is no standard way to measure whether they actually help. We present SkillsBench, a benchmark of 86 tasks across 11 domains paired with curated Skills and deterministic verifiers. Each task is evaluated under three conditions: no Skills, curated Skills, and self-generated Skills. We test 7 agent-model configurations over 7,308 trajectories. Curated Skills raise average pass rate by 16.2 percentage points(pp), but effects vary widely by domain (+4.5pp for Software Engineering to +51.9pp for Healthcare) and 16 of 84 tasks show negative deltas. Self-generated Skills provide no benefit on average, showing that models cannot reliably author the procedural knowledge they benefit from consuming. Focused Skills with 2--3 modules outperform comprehensive documentation, and smaller models with Skills can match larger models without them.

• The paper introduces a benchmark showing curated Skills boost LLM agent success by an average of +16.2 percentage points across 84 tasks.
• The paper finds that self-generated Skills yield negligible or negative performance, highlighting the necessity for human-curated procedural expertise.
• The paper demonstrates that focused modular Skills improve cost-performance trade-offs, enabling smaller models to outperform larger ones at reduced API costs.

SkillsBench: A Systematic Evaluation Framework for Agent Skills in LLM-Based Agents

Introduction

The paper "SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks" (2602.12670) presents a rigorous benchmarking methodology and empirical assessment for evaluating the impact of structured Agent Skills as augmentation for LLM-based agents. Agent Skills, defined as procedural, modular, and portable packages of workflow knowledge, have seen rapid ecosystem growth, yet prior to this work, there was no standardized, quantitative measure of their effect on agent performance. The SkillsBench benchmark introduces paired, deterministic evaluation of LLM-agents across 84 real-world tasks, 7 commercial agent-model configurations, and three augmentation conditions, yielding detailed insight into when and how Skills deliver value.

Figure 1: Agent architecture stack and resolution rates across 7 agent-model configurations on 84 tasks. Curated Skills (beige) improve performance by +16.2pp on average; self-generated Skills (amber) provide negligible or negative benefit.

Benchmark Design and Task Specification

SkillsBench is constructed to enable paired, controlled evaluation of Skills augmentation. Tasks span 11 professional domains such as software engineering, finance, healthcare, energy, and robotics (Figure 2), stratified by human completion time to cover a wide spectrum of task complexity and required expertise.

Figure 2: SkillsBench consists of tasks spanning 11 domains.

Each task is a containerized, self-contained module comprising a human-authored instruction, a reproducible environment, an oracle solution, and a deterministic programmatic verifier. Strict authoring and leakage-prevention policies ensure instructions are human-authored and Skills are truly procedural, not instance-specific. The construction pipeline aggregates Skills from open-source repositories, commercial partner contributions, and the Claude Code Skill ecosystem, with extensive automated and manual quality filtering.

Figure 3: SkillsBench pipeline overview, from Skill aggregation to multi-phase filtering and final evaluation across agents and conditions.

Evaluation Protocol and Experimental Setup

SkillsBench evaluates agents under three augmentation conditions for each task: (1) no Skills (baseline LLM+agent), (2) curated Skills (human-authored procedural packages), and (3) self-generated Skills (agent-authored Skills in the context of the given task). Commercial agent harnesses evaluated include Claude Code, Gemini CLI, and Codex CLI, each interfacing with their respective cutting-edge LLMs (Claude, Gemini, GPT).

The primary metric is deterministic pass rate, evaluated via binary programmatic verifiers. All agent-task interactions and augmentations are fully reproducible, and Skills are injected as system-level context, never referenced explicitly in instructions.

Main Findings

1. Skills Yield Substantial but Non-Uniform Benefit

Curated Skills increase average agent pass rate by +16.2 percentage points (pp) across 7,308 trajectories, with strong variance both across domains (from +4.5pp in Software Engineering to +51.9pp in Healthcare) and model-harness configurations. The most pronounced improvements occur in tasks and domains where baseline models lack procedural coverage (e.g., clinical harmonization, manufacturing workflows), while gains are minimal—or negative—where LLM priors are strong or where Skills introduce cognitive overhead.

2. LLMs Cannot Reliably Author Their Own Procedural Skills

Self-generated Skills, obtained by prompting agents to articulate procedural knowledge before solving a task, provide negligible or even negative benefit relative to the no-Skills baseline, with an average delta of –1.3pp. This result counters the hypothesis that LLMs' latent knowledge can compensate for curated domain expertise. Failure analysis reveals self-generated Skills are frequently coarse, incomplete, or miss procedural idiosyncrasies required by verifiers, especially in specialized domains.

3. Efficiency–Performance Tradeoff

Skills consistently shift the Pareto cost–performance frontier upwards for all evaluated agent-models. Specifically, smaller/cheaper models equipped with curated Skills can match or outperform larger/more expensive models lacking them (Figure 4). For instance, Gemini Flash (with Skills) outperforms Gemini Pro (no Skills) at 44% lower per-task API cost.

Figure 4: Skills shift the pass rate–cost Pareto frontier; Gemini 3 Flash with Skills offers dominant cost–performance trade-off.

4. Focused Modular Skills Outperform Exhaustive Documentation

Quantitative stratification reveals that Skills packages comprising 2–3 focused modules deliver the greatest benefit (+18.6pp), while providing 4+ Skills or exhaustive procedural documentation yields sharply diminished or even negative returns. Overlong Skills introduce context overload and may impair agent performance via injection of superfluous or conflicting strategies.

Task-Level and Failure Mode Analysis

A dense, paired matrix of task-by-model outcomes reveals a strongly clustered profile of task difficulty: some tasks are solved by all models with Skills, others remain unsolved even by top models, and a significant long-tail of tasks transition from failure to success only with curated Skills (Figures 11–13).

Figure 5: Task pass rates per model with curated Skills show stark transitions from universally solvable/easier tasks (top) to unsolved/hard cases (bottom).

Figure 6: Skills uplift by task: The predominance of blue indicates broad benefit, though red cells expose specific negative impacts for some task–model pairs.

Failure taxonomy analysis shows that Skills primarily reduce “quality below threshold” and “incomplete solution” failures. Marginal increase in timeouts is attributable to agents exploring more sophisticated solution trajectories enabled by Skills guidance. Critical negative delta tasks are often those that, due to strong model priors, are hampered by Skills introducing conflicting or distracting procedures.

Skills Ecosystem Characteristics

Meta-analysis of 47,150 public Skills artifacts demonstrates that most Skills are lightweight (median $\sim$1.5k tokens), documentation-focused (dominant markdown/text), and span diverse domains (Figures 6–10). The benchmark specifically selects high-quality Skills from the upper quartile of this ecosystem to control for confounds from low-quality or incomplete artifacts.

Figure 7: Token distribution of SKILL.md files—Skills are typically compact, indicating portability for inference-time augmentation.

Figure 8: Skills span documentation, git/version control, code quality, reflecting diverse developer needs with no dominant category.

Theoretical and Practical Implications

SkillsBench establishes that empirical evaluation of agent augmentation must consider not just whether context helps, but how much and under which tight conditions. The finding that modular, human-curated procedural Skills offer meaningful performance boosts—while naïve, self-generated Skills do not—has direct implications for practical agent system deployment and Skill ecosystem curation. These insights further clarify the limitations of context augmentation: there are diminishing returns to context size, and human domain expertise cannot currently be reliably synthesized automatically from LLMs themselves.

On the theoretical side, this work encourages future research to operationalize Skills-centric evaluation as standard practice, to further delineate which task types and agent architectures are most receptive to procedural intervention, and to invest in methods for automatic Skill synthesis with controllable quality guarantees.

Limitations and Future Directions

SkillsBench is currently focused on CLI-based, containerizable tasks; generalization to graphical, multi-modal, or multi-agent coordination environments remains an area for future work. While SkillsBench is immune to LLM-as-judge subjectivity by virtue of programmatic verification, possibilities for distributional shift and memorization remain. The negative or null results for self-generated Skills motivate deeper study into Skill composition, Skill quality-aware evaluation, and causally rigorous ablations (e.g., controlling for context length vs. procedural structure).

Conclusion

SkillsBench introduces a principled, large-scale framework for benchmarking the real-world efficacy of Agent Skills in LLM-based agents. Substantial, but non-uniform, gains from curated Skills and the ineffectiveness of self-generated Skills demonstrate that procedural augmentation is context-dependent and cannot simply rely on LLMs' in-context learning abilities. These findings motivate the use of paired, Skills-aware evaluation to guide continued development of agent augmentation ecosystems, selection mechanisms, and future procedural knowledge distillation methods.