Not All Skills Help: Measuring and Repairing Agent Knowledge

Abstract: LLM agents can improve without weight updates by accumulating natural-language skills from experience, but current systems entrust every decision about which skills to keep and how to apply them to LLM judgment alone. We argue that this conflates two distinct roles: generating a skill from experience is a creative act that judgment handles well, while deciding whether that skill actually helps requires empirical evidence across many tasks. Measuring per-skill causal contributions via randomized masking, we find that skill libraries exhibit pervasive causal heterogeneity: individual skills routinely help on some task types while hurting on others, yet their opposing effects cancel in aggregate, making them invisible to global curation methods. We propose ASSAY, a framework that separates generation from curation: it computes a per-skill causal attribution on a small development set, restructures the library offline, and suppresses skills with negative predicted effect for each test task. Across seven base models spanning four providers and two benchmarks (AppWorld and tau-bench), ASSAY consistently improves over prior skill-curation approaches. On AppWorld's hardest split, DeepSeek-V3 achieves 69.3% task-goal completion (47.4% relative improvement), a new state of the art among all published methods including weight-tuned approaches. On tau-bench retail, GPT-4.1 improves by 8.7% relative, advancing past o4-mini, o1, and GPT-4.5 on the public leaderboard without any weight modification. Ablation traces the dominant gain to per-task masking, confirming that the bottleneck is matching skills to tasks at inference time, not removing bad skills globally. Code is available at https://github.com/aiming-lab/assay.

• The paper introduces randomized masking to estimate each skill’s task-specific causal effect, finding that over 90% of skills show substantial heterogeneity and that global averages can hide harmful regressions.
• Assay restructures, retires, merges, and selectively masks skills per task, improving AppWorld performance across seven models and raising DeepSeek-V3 from 47.0% with ReAct to 69.3%.
• Per-task masking delivers the largest ablation gain and can repair harmful skill libraries, but its effectiveness depends on development-task coverage, model-specific attribution, and safeguards for low-similarity test cases.

Motivation: the unchecked skill lifecycle

LLM agents can improve without weight updates by distilling successful trajectories into natural-language skills and injecting them into the context window. The paper's central observation is that every decision in this lifecycle—generation, retention, retrieval, application—is delegated to LLM judgment operating within individual tasks. Nothing verifies that a retained skill actually helps across the task distribution. The authors argue this conflates two distinct roles: generating a skill from a single experience is a creative act well suited to LLM judgment, whereas deciding whether a skill helps requires empirical evidence aggregated over many tasks.

The empirical motivation comes from tracing failures on AppWorld and $\tau$-bench. Skills essential on their source tasks become pure overhead elsewhere: multi-step purchase rules exhaust step budgets on simple tasks; exchange procedures derail straightforward cancellations. A small set of skills accounts for a disproportionate share of regressions, and the same skill can help one task type while hurting another.

Measuring causal heterogeneity

The paper measures per-skill causal effects via randomized masking, following the logic of Shapley-style attribution and randomized ablation (2606.15390). For each of $K$ trials, a random mask includes each skill independently with probability $f$; the causal score of skill $s_j$ on development task $d_i$ is the difference-in-means estimator of its average treatment effect, marginalised over co-occurring skills. Two row statistics drive all curation decisions: the global score $\bar{C}(j)$ (row mean) and heterogeneity $H(s_j)$ (row range).

The resulting attribution reveals what the authors call causal heterogeneity: on the GPT-5.1 attribution, over 90% of skills have per-task causal ranges exceeding the split threshold of 0.40—nearly every skill helps on some tasks and hurts on others. Critically, skills with high heterogeneity but near-zero global scores are invisible to any curation method that evaluates skills by aggregate score alone, because opposing effects cancel. Concrete examples include a Venmo contact-validation rule ($\bar{C} = -0.03$, ranging from $+0.50$ to $-0.67$) and an API response-checking rule ($K$0, ranging from $K$1 to $K$2); both would survive any global curation threshold.

The authors are candid about statistical limitations here: at $K$3 masks, per-cell permutation tests achieve only 38.5% power for detecting a $K$4 effect, so single-skill significance testing cannot support the claim. Instead they validate heterogeneity through six outcome-level tests—masking diversity (96% selective drops, mean Jaccard 0.30), attribution alignment between dropped and kept skills (gap 0.147, Mann-Whitney $K$5), sign-reversing cases, split separation, reverse-masking controls, and bootstrap sign stability (~71%/70% versus a 50% null). This is a reasonable evidentiary strategy, though it means the heterogeneity claim rests on convergent indirect evidence rather than per-cell inference.

The Assay framework

Assay operationalises the separation between generation and curation in three stages, all derived from a single attribution matrix computed once per base model.

Offline restructuring applies three operations ordered to prevent information loss. Split rewrites heterogeneous skills into two conditional variants with explicit trigger conditions, guided by the per-task causal score vector; each split must pass a development gate requiring no regression on any attribution task. Retire removes skills with negligible global scores. Merge deduplicates near-identical variants by embedding similarity. Splitting runs first because a heterogeneous skill has near-zero global score and would be incorrectly retired otherwise. Notably, splitting is the one point where LLM judgment re-enters curation—the authors acknowledge this reintroduces subjectivity, bounded only by candidate limits and the development gate.

Per-task masking frames inference-time skill selection as risk minimization. For a test task, the predicted causal effect of each skill is a kernel-weighted projection of the attribution row onto the $K$6 nearest development tasks, using temperature-scaled softmax attention weights. Skills with predicted effect below a threshold are suppressed, subject to two safeguards: protected templates (pagination, data validation, identity resolution) are exempt, and if too few skills survive, the full library is used. The removal-only asymmetry is deliberate—missing a critical skill causes catastrophic failure, while retaining a mildly harmful skill among many has diluted effect.

The masking rule rests on an independence approximation (skills contribute independently to success) and on nearest-neighbour transfer of causal evidence. Coverage analysis shows this assumption is strained: mean cosine similarity to the top-8 development tasks is only 0.469 on test_challenge, with 69.1% of tasks below 0.5 similarity. Gains persist even in the lowest-coverage quartile (+4.8 pp), aided by the fallback mechanism, which activates on 21.1% of challenge tasks.

Main results

Experiments span seven models across four providers, two benchmarks, and fixed hyperparameters throughout, with the attribution matrix recomputed per model since causal structure depends on the interpreting model.

On AppWorld, every model improves on both splits. Three findings stand out:

Model	Baseline	Best prior method	Assay
GPT-5.1	ReAct 52.5	ACE 49.9	66.4
DeepSeek-V3	ReAct 47.0	ACE 63.1	69.3
GPT-4.1	ReAct 50.4	CUGA 57.6	64.0

First, DeepSeek-V3 reaches 69.3% TGC on test_challenge—a claimed state of the art among all published methods including weight-tuned ones, a 47.4% relative improvement over bare ReAct. Second, and more diagnostic, the upstream ACE library degrades GPT-5.1 on test_challenge (52.5 → 49.9): a library designed to help made the agent strictly worse. Per-difficulty breakdown shows uncurated libraries help Level 1 but hurt Levels 2–3; per-task masking recovers the degraded levels, with the largest gain on Level 3 (43.1 → 71.3). Third, a reverse-masking control (suppressing positively-scored skills instead) degrades performance by 4.7 pp, confirming that attribution direction—not mere context reduction—drives the gain.

On $K$7-bench retail, GPT-4.1 improves 8.7% relative (68.0 → 73.9), advancing past o4-mini, o1, and GPT-4.5 on the public leaderboard without weight modification; GPT-5.4 gains 9.5% relative, reaching the top-5 range. However, two models show exactly zero gain: GPT-5.1 (62.6 → 62.6) and Sonnet 4.5 (73.0 → 73.0). The authors attribute this to high baseline competence saturating the benefit of prompt-time injection, supported by the observation that the strongest-prior model is most resistant to external skill injection under domain transfer from AppWorld-curated libraries. This is a genuine boundary condition for context engineering rather than a uniform failure mode.

A sequential ablation isolates component contributions on GPT-5.1 / test_normal: templates add +6.0 pp, offline restructuring +2.0 pp, and per-task masking +7.5 pp—the largest single increment. This confirms the paper's central thesis: the bottleneck is matching skills to tasks at inference time, not removing bad skills globally.

Limitations and open questions

Several limitations bear directly on the strength of the claims. The state-of-the-art comparison on AppWorld relies partly on a leaderboard submission (Qwen3-14B, weight-tuned) with no accompanying publication. The attribution matrix is estimated on only 15 development tasks and extrapolated via nearest-neighbour weighting, with substantially degraded coverage on the harder split; per-cell power remains low at the chosen mask budget, though the authors note 30–50 masks would raise power to 80–98%. The split operation reintroduces LLM subjectivity, bounded but not eliminated by the development gate; a fully measurement-driven splitting procedure remains open. The framework assumes a fixed library, so extension to online settings where skills are continuously added would require incremental attribution updates—an explicitly unresolved question. Finally, the null results on $K$8-bench suggest diminishing returns of prompt-time skill injection as base competence strengthens, raising the question of where the crossover point lies for future frontier models.

Conclusion

This paper provides the first systematic characterisation of causal heterogeneity in agent skill libraries and a practical remedy. Its core contribution is methodological reframing: skill effects are predominantly conditional rather than uniformly good or bad, so curation must operate per task rather than globally. The randomized-masking attribution is cheap (180 rollouts per model), model-specific, and layered on top of any existing skill-generation pipeline without weight updates. The strong aggregate gains, the demonstrated case where an uncurated library actively harms performance, and the ablation pinpointing per-task matching as the dominant factor together make a persuasive case that measurement, not judgment, should govern skill application.