STDAW: Realistic Agentic Skill Evaluation

• STDAW is a realistic evaluation framework that exposes agent skill fragility by assessing retrieval, adaptation, and harm under progressively challenging conditions.
• It leverages a corpus of 34,198 real-world skills and a progressive realism ladder, comparing curated skills, distractor interference, and full retrieval from noisy repositories.
• Results reveal that imperfect skill retrieval can degrade performance below baseline, underscoring the need for refined, task-specific skill adaptation strategies.

This paper is very likely the one introducing “HarmfulSkillBench”-like conditions, but it does not define a benchmark literally named HarmfulSkillBench. Instead, it introduces a realistic evaluation framework for agentic skill usage that exposes when skills become fragile, misleading, or effectively harmful for LLM agents.

What benchmark / evaluation setup it introduces

The paper’s main benchmark setup is a progressive evaluation on SkillsBench under increasingly realistic conditions, plus a transfer test on Terminal-Bench 2.0.

Core idea

Prior skill benchmarks were “too idealized”:

• hand-curated skills are directly provided,
• skills are tightly tailored to the task,
• no need to search large skill libraries.

This paper instead studies what happens when agents must:

1. select useful skills from many available ones,
2. retrieve skills from a large real-world library,
3. adapt skills that are only partially relevant.

Main skill collection

They build a large corpus of:

• 34,198 real-world skills
• sourced from public repositories / aggregation sites
• filtered to permissive licenses (MIT, Apache 2.0)
• deduplicated and cleaned

This is the “in the wild” retrieval setting.

Benchmark tasks

They evaluate on:

• SkillsBench: 84 tasks used (3 removed due to known issues)
• Terminal-Bench 2.0: 89 tasks

Progressive evaluation settings

From easiest to hardest:

1. Curated + forced load Curated skills are provided and the agent is told to load them all.
2. Curated Curated skills are available, but the agent must decide whether to use them.
3. Curated + distractors Curated skills are mixed with distractor skills retrieved from the 34k corpus.
4. Retrieved (w/ curated) Agent retrieves top-5 skills from the 34k corpus, but curated skills are still in the pool.
5. Retrieved (w/o curated) Agent retrieves top-5 skills from the 34k corpus and the curated task-specific skills are removed.
6. No skills Baseline.

This is the paper’s main realism ladder.

---

Does it relate to agentic skill usage in realistic settings?

Yes—this is exactly the paper’s focus.

The paper argues that real agentic skill use is not just “having skills in context”; it requires:

• searching for skills,
• choosing among noisy candidates,
• reading/adapting general-purpose documentation,
• deciding whether not to use a skill at all.

They emphasize that the practical challenges are:

• skill selection,
• skill retrieval,
• skill adaptation.

The 34k-skill corpus is central because it simulates the realistic situation where an agent must search a large and noisy skill repository, rather than being handed the perfect skill.

---

Retrieval setup and formulas

They build a search system with:

• BM25 keyword search
• dense semantic search using Qwen3-Embedding-4B
• hybrid search via Reciprocal Rank Fusion (RRF)

RRF formula

For hybrid search, they define:

$\text{RRF score} = \sum_s \frac{w_s}{k + r_s}$

where:

• $r_s$ = rank under search method $s$,
• $w_s$ = weight for that method,
• $k = 60$ = fusion constant.

For hybrid with content, they also incorporate both metadata and full skill content embeddings, using a weighted similarity combination.

Retrieval metric

They evaluate retrieval quality with:

$\text{Recall@}k$

defined as the fraction of ground-truth curated skills appearing in the top-$k$ retrieved results, averaged over tasks.

Retrieval results

Best retrieval method: agentic hybrid search with content

• Recall@3: 57.3–57.7%
• Recall@5: 65.5%
• Recall@10: 68.3%

Agentic hybrid search outperforms direct one-shot semantic search because the agent can iteratively refine queries and inspect candidates.

---

Are the settings harmful, fragile, or challenging for LLM agents?

Short answer

Yes, in the sense of fragility and challenge.

The paper shows that skills are not reliably beneficial under realistic conditions, and in the hardest settings they can become effectively harmful because they hurt performance relative to no skills.

Main findings on pass rates

The key result is that pass rates decrease as the setting becomes more realistic.

Claude Opus 4.6

• Curated + forced load: 55.4%
• Curated: 51.2%
• Curated + distractors: 43.5%
• Retrieved (w/ curated): 40.1%
• Retrieved (w/o curated): 38.4%
• No skills: 35.4%

Claude still stays slightly above baseline in the hardest setting, but only by 3.0 points.

Kimi K2.5

• Curated: 38.9%
• Retrieved (w/ curated): 33.5%
• Retrieved (w/o curated): 19.8%
• No skills: 21.8%

Here, the hardest retrieved setting is below no-skills baseline.

Qwen3.5-397B-A17B

• Curated: 31.6%
• Retrieved (w/ curated): 26.7%
• Retrieved (w/o curated): 19.7%
• No skills: 20.5%

Again, the hardest retrieved setting is worse than no skills.

Interpretation

This is the strongest evidence that the setup is “harmful” or at least skill-fragile:

• poorly matched retrieved skills can waste context,
• lead agents to follow misleading instructions,
• and reduce performance below baseline.

So although the paper does not call it “HarmfulSkillBench,” the hardest setting is very much aligned with that idea.

---

Skill refinement methods

The paper studies two refinement strategies to mitigate the fragility.

1) Query-specific refinement

This is the stronger method.

The agent:

• reads the task,
• inspects retrieved skills,
• attempts the task,
• self-evaluates what was useful or misleading,
• composes improved skills tailored to the task.

It can synthesize across multiple skills and merge useful pieces.

2) Query-agnostic refinement

This is offline skill improvement without knowing the target task.

Each skill is improved independently using:

• synthetic queries,
• A/B-style self-evaluation,
• the Anthropic skill-creator meta-skill.

This is cheaper at inference time but less adaptive.

---

Effects of refinement on pass rates

SkillsBench, retrieved (w/ curated)

Claude

• Retrieved: 40.1%
• • Query-specific: 48.2%
• • Query-agnostic: 42.0%

So query-specific refinement recovers much of the lost performance.

Qwen

• Retrieved: 26.7%
• • Query-specific: 30.8%
• • Query-agnostic: 26.2%

Again query-specific helps more.

Kimi

• Retrieved: 33.5%
• • Query-specific: 26.7%

This is a notable failure case: refinement can hurt if the model misjudges skill usefulness.

SkillsBench, retrieved (w/o curated)

Claude

• Retrieved: 38.4%
• • Query-specific: 37.9%
• • Query-agnostic: 37.4%

Kimi

• Retrieved: 19.8%
• • Query-specific: 23.1%

Qwen

• Retrieved: 19.7%
• • Query-specific: 21.5%
• • Query-agnostic: 24.6%

The gains are weaker here because the retrieved skills are less relevant overall.

Terminal-Bench 2.0

This benchmark has no curated skills; retrieval is from the general 34k collection.

Claude

• No skills: 57.7%
• Retrieved: 61.4%
• • Query-specific: 65.5%
• • Query-agnostic: 63.3%

This is an especially clear gain.

The paper reports similar improvements for Kimi and Qwen, though exact numbers differ by model.

---

Main message and implications for benchmark design

Main finding

Skills are not robustly helpful by default. Their utility is fragile and highly dependent on:

• skill quality,
• retrieval quality,
• model/harness behavior,
• whether the skill is actually relevant.

Why this matters

The paper argues that prior skill benchmarks overestimate skill usefulness because they:

• give curated, task-specific skills directly,
• avoid retrieval,
• avoid noisy pools,
• and ignore adaptation cost.

A realistic benchmark should instead test:

• retrieval from large corpora,
• distractor robustness,
• whether skills help more than no-skill baselines,
• and whether agents can ignore irrelevant skill artifacts.

Practical implication

In agent systems, skills should not be treated as automatically beneficial. You need:

• good retrieval,
• selective loading,
• and possibly refinement.

Otherwise, skills may be ignored or even actively harm performance.

---

Does this paper match the notion of HarmfulSkillBench?

Yes, conceptually.

Not by name, but by behavior.

It provides exactly the kind of evidence a “HarmfulSkillBench” would want:

• a realistic skill retrieval environment,
• noisy 34k-skill library,
• cases where retrieved skills degrade performance below no-skill baselines,
• and a diagnosis of when skills become harmful.

So the paper is best understood as:

a benchmark for fragile and sometimes harmful skill usage in LLM agents, especially under realistic retrieval and adaptation conditions.

If you want, I can also turn this into a compact table of settings / metrics / results / conclusion.