SkillCraft: Compositional Skill Benchmarking

• SkillCraft is a formal framework for modeling and evaluating compositional skill acquisition and reuse in structured, long-horizon tasks.
• It employs hierarchical grammars and protocol-driven documentation to enable persistent skill libraries and systematic cross-task benchmarking.
• Empirical results show that SkillCraft enhances transfer, efficiency, and generalization in LLM-agent and RL settings, reducing token usage by up to 80%.

SkillCraft encompasses a family of formalisms and benchmarks designed to elicit, document, and measure compositional skill acquisition and reuse in both artificial and human agents. Across disparate research traditions—LLM-based software agents, computational craft documentation, and unsupervised skill induction in RL—SkillCraft denotes systems for constructing, abstracting, and applying skills in long-horizon, structured task domains. Its central concerns are protocol and grammar-driven workflow abstraction, persistent skill libraries, and rigorous empirical assessment of generalization, efficiency, and transfer.

1. Formal Definitions and Core Concepts

SkillCraft benchmarks and frameworks adopt a formal, hierarchical approach to skill and workflow modeling. In LLM-agent tool-use contexts (Chen et al., 28 Feb 2026), a task $T$ is a finite sequence of atomic tool calls:

$T = (t_1, t_2, \ldots, t_n),$

where each $t_i$ invokes an atomic API endpoint. Tool composition is formalized as a function $C$ mapping an ordered subsequence of tool calls to a single composite operator $t^*$:

$C: \{t_{i_1},\ldots,t_{i_k}\} \rightarrow t^*,$

with $t^*$ encapsulating the original chain as a reusable black-box. A skill is a parameterized code wrapper—or script—$S:\Theta \rightarrow \mathrm{Result}$, encapsulating the composition and exposing it via a uniform call interface.

In computational craft documentation (Batra et al., 12 Jun 2025), a skill or workflow $S$ is generated via an elementary grammar $G=(N,\Sigma,P,S)$ over domains of actions, states, and structural operators, with production rules capturing compositional, branching, and improvisational patterns. In unsupervised skill discovery (Harvey et al., 30 Jan 2026), a skill denotes a discrete sub-segment of agent trajectory, and a hierarchy of skills is derived via grammar induction (e.g., Sequitur).

2. Scenario Design, Toolsets, and Grammar Representation

SkillCraft scenarios span multiple domains:

• In LLM-agent benchmarks (Chen et al., 28 Feb 2026), the suite comprises 21 task families across six domains (Entertainment, Reference, Education, Developer, Science, Food). Each provides 4–7 RESTful atomic tools, augmented with standard I/O primitives. Task instances are parametrized along quantitative (number of entities $T = (t_1, t_2, \ldots, t_n),$0 per task) and structural (chain length $T = (t_1, t_2, \ldots, t_n),$1 per entity) axes to control the compositional complexity, yielding 126 distinct tasks at varying difficulty.
• The elementary craft grammar (Batra et al., 12 Jun 2025) models workflows via nonterminals (actions $T = (t_1, t_2, \ldots, t_n),$2, state-things $T = (t_1, t_2, \ldots, t_n),$3, reflective loops $T = (t_1, t_2, \ldots, t_n),$4, annotations, segments, branches, revision-loops). Terminals encode primitive actions and material/tool tokens. Production rules compose basic $T = (t_1, t_2, \ldots, t_n),$5–$T = (t_1, t_2, \ldots, t_n),$6–$T = (t_1, t_2, \ldots, t_n),$7 triples into segments, allow controlled branching (alternatives), and revision cycles (undo/redo), with explicit annotation and granularity controls.
• In HiSD for RL (Harvey et al., 30 Jan 2026), the input is unlabelled trajectories in high-dimensional observation spaces. Atomic skills correspond to segmented subsequences, and mid-level composite skills are induced hierarchically via CFG rule extraction. Skilful behavior is thus represented as sequences and trees over a learned grammar.

3. Protocols for Skill Acquisition, Composition, and Reuse

Skill acquisition in SkillCraft is tightly specified:

• Agent Protocol (LLM context): The test-time protocol (Chen et al., 28 Feb 2026) iterates over task instances. For each, it attempts to match and invoke a previously saved skill from the agent’s persistent library. Failing that, it decomposes the task into atomic tool-calls, scripts the sequence as a new candidate skill, and—upon verification—saves it for future reuse. Formally, cache lookup seeks

$T = (t_1, t_2, \ldots, t_n),$8

with execution preferring existing $T = (t_1, t_2, \ldots, t_n),$9; unsuccessful cases revert to atomic planning.

• Grammar-Driven Craft Documentation: Video-to-grammar pipelines (Batra et al., 12 Jun 2025) ingest narrated expert demonstrations, automatically tokenize detected actions/states, and segment workflows into connectable grammar units using multimodal LLMs. Human participants edit, annotate, and branch/fork workflows via interactive graph interfaces for shareable, community-scale skill repositories.
• Unsupervised Skill Discovery (RL): HiSD (Harvey et al., 30 Jan 2026) optimizes a batchwise Sinkhorn/ASOT objective to segment trajectories into primitive skills, then applies a modified Sequitur algorithm to induce hierarchical grammars over observed sequences, enabling unsupervised extraction and abstraction of repeating sub-tasks.

4. Evaluation Metrics and Empirical Assessment

SkillCraft domains deploy robust, quantitative metrics:

• LLM-Agent Benchmarks (Chen et al., 28 Feb 2026):
  • Instance-level Success Rate: Fraction of tasks achieving $t_i$0 correctness.
  • Skill-enabled Success Rate: Identical, but with Skill use permitted.
  • Efficiency Gains:

  $t_i$1

  computed for tokens, tool calls, or dialog turns. - Correlation: Pearson’s $t_i$2 quantifies the link between skill-execution rate ($t_i$3) and task success indicator ($t_i$4).

• Craft Grammar Annotation (Batra et al., 12 Jun 2025):

  • Coverage: Share of expert-identified actions parsed: 85%.
  • Annotation Accuracy: Post-edit grammar fidelity: 92%.
  • User Satisfaction: Likert mean: 4.1 (creators), 4.0 (consumers).
• HiSD Segmentation and Hierarchy (Harvey et al., 30 Jan 2026):
  • Mean-over-Frames (MoF): Proportion of frame-level labels matching ground truth.
  • Segment-level F1, mean IoU: Alignment quality.
  • Hierarchy Metrics: Unique tree count (consistency), average depth, branching factor.

Empirical findings substantiate the centrality of reusable skill abstraction. LLM agents achieve up to 80% token reduction (e.g., 1.23M→0.26M, $t_i$5) and improved success rates with skill use; skill execution rate strongly correlates with success ($t_i$6). In RL, HiSD hierarchies materially accelerate sparse-reward learning (e.g., optimal reward in $t_i$7k steps; PPO baselines fail).

5. Failure Modes, Limitations, and Qualitative Insights

Empirical analyses reveal several salient limitations:

• Flat Planning Inefficiency: Repeated serialization of atomic operations induces overhead, context saturation, and error propagation (“forgotten plan” phenomena (Chen et al., 28 Feb 2026)).
• Depth vs. Validation in Hierarchies: Deep recursive skills (hierarchical composition) often underperform shallow, validated flat skills due to compounded debugging overhead and error amplification.
• Tacit Knowledge Gaps: In craft, embodied or haptic details—e.g., tension “in the fingers”—are poorly captured even with granular grammars, leading practitioners to supplement with in-person demonstration (Batra et al., 12 Jun 2025).
• Generalization: Skills learned on “easy” instances transfer with negligible loss ($t_i$8 execution rate) to harder variants and across models. However, poorly designed skills degrade efficiency (negative $t_i$9).

6. Applications, Transfer, and Impact

SkillCraft constructs enable:

• Persistent, Community Knowledge Repositories: The grammar-driven approach supports versioned workflow sharing—analogous to “GitHub for making”—with branching, annotation, and refinement capabilities for both explicit and improvisational practice (Batra et al., 12 Jun 2025).
• Accelerated Agent Learning: Grammar-induced hierarchies in RL (HiSD) yield subroutines matching or exceeding ground-truth hierarchies in consistency, depth, and downstream utility (Harvey et al., 30 Jan 2026).
• Cross-Model and Cross-Task Skill Transfer: Skill abstractions authored by one agent or model demonstrably generalize in execution and efficiency to other agents, supporting model-agnostic library growth (Chen et al., 28 Feb 2026).
• Systematic Benchmarking: Explicit protocol and lightweight MCP (“Minimal Control Primitives”) APIs—save_skill, list_skills, get_skill, execute_skill—provide rigorous platforms for cross-method comparison in tool-use and workflow synthesis.

7. Prospects and Future Directions

SkillCraft research foregrounds several avenues:

• Beyond Episodic Evaluation: Persistent, cross-task, and cross-domain skill accumulation is required for robust agent benchmarking.
• Meta-Learning and Library Optimization: Future protocols may require agents to develop meta-strategies for skill curation, growth, and pruning.
• Multi-Model Integration: Benchmarks extending to multi-model skill exchange, robustness in hierarchical debugging, and shared libraries represent open challenges.
• Human–AI Collaborative Authoring: Grammar-based, human-in-the-loop systems enable a living, evolving repository that spans explicit procedures and tacit improvisational knowledge.
• Minimalist Interfaces: Results suggest that simple, composable skill-APIs are sufficient to uncover core agent competence in abstraction and reuse, without complex bespoke architectures.

SkillCraft thus defines a rigorous paradigm for the study, benchmarking, and operationalization of skill abstraction, composition, and knowledge sharing across both artificial and human domains, with empirical protocols and formal grammars as foundational elements for advancing collectively reusable expertise (Chen et al., 28 Feb 2026, Batra et al., 12 Jun 2025, Harvey et al., 30 Jan 2026).