RESOURCE2SKILL: Distilling Executable Agent Skills from Human-Created Multimodal Resources

Abstract: Skills are a useful abstraction for software agents, turning human and agent experience into reusable procedural knowledge. Yet existing skill libraries are mostly hand-written, text-centric, or derived from agent traces, leaving tutorial videos and other multimodal human resources largely underused. We present RESOURCE2SKILL, a framework that distills multimodal resources, including tutorial videos, repositories, articles, and reference artifacts, into executable skills for software agents. RESOURCE2SKILL organizes these skills as a hierarchical multimodal Skill Wiki, where each entry combines structured text, code, visual examples, metadata, and provenance. This design preserves complementary signals from different resources: videos capture temporal operations and visual effects, code captures executable tool patterns, and articles or artifacts provide conceptual and stylistic grounding. At inference time, agents retrieve and compose relevant skills from the wiki; when coverage is insufficient, the same construction operator can acquire new skills online. Across seven practical authoring domains, RESOURCE2SKILL improves average overall score by +11.9 percentage points over no-skill agents and outperforms strong harness baselines in 26 of 28 main-aggregate model-domain cells. Ablations confirm the value of multimodal skill format, hierarchical organization, source diversity, selection strategy, and online acquisition.

• The paper introduces a unified pipeline that distills multimodal resources into executable skills organized within a hierarchical wiki.
• It demonstrates significant performance gains with an average +11.9 point improvement over no-skill agents across diverse software domains.
• The framework supports scalable online acquisition and interpretable procedural memory, enhancing both real-time adaptation and agent robustness.

Distilling Multimodal, Executable Skills for Software Agents: An In-Depth Analysis of Resource2Skill

Introduction and Motivation

The development of capable, generalist software agents necessitates more than access to factual knowledge. Procedural know-how—reusable, executable, and context-sensitive routines—enables agents to tackle diverse tasks in practical environments. Traditional skill libraries for agents are predominantly text-based, hand-authored, or derived from interaction traces, leaving a considerable portion of human expertise, especially those encapsulated in multimodal tutorial content (videos, code repositories, articles, artifacts), underexploited. "RESOURCE2SKILL: Distilling Executable Agent Skills from Human-Created Multimodal Resources" (2606.29538) introduces Resource2Skill, a framework which systematically mines, distills, and organizes such multimodal resources into a hierarchical, executable Skill Wiki shared by offline and online acquisition pipelines.

Figure 1: Resource2Skill distills multimodal resources into a hierarchical Skill Wiki across seven creative software domains.

Resource2Skill Framework Overview

Resource2Skill is structured around a unified resource-to-skill pipeline comprising resource distillation, hierarchical skill organization, skill selection, execution, and on-demand online acquisition.

1. Resource Distillation: A construction operator $f_\theta$ processes multimodal domain-specific resources (videos, repositories, articles, artifacts), extracting candidate skills represented as structured tuples containing text, code, visual examples, metadata, and provenance. Rigorous deterministic post-processing ensures completeness, provenance, deduplication, modality consistency, and executability.
2. Hierarchical Multimodal Skill Wiki: Extracted skills are organized into a domain-specific taxonomy, facilitating both semantic grouping and efficient retrieval during inference.
3. Skill Selection and Execution: Given a user brief, the agent navigates the wiki using a two-stage MetaBrowse protocol: lexical scoring (BM25) narrows candidates using both content and hierarchical paths, followed by an LM-based selector which composes a minimal subset for enactment. Code from skill entries, when present, is executed directly via backend-specific APIs, while text and visual cues are adapted when code is absent.
4. Online Skill Acquisition: When coverage is insufficient, an online operator invokes the same resource-to-skill construction pipeline, mining new skills to fill capability gaps without conflating offline and online skill pools.

   Figure 2: Resource2Skill pipeline. A construction operator distills resources into the hierarchical Skill Wiki; selection and composition are integrated into agent execution, with online acquisition invoked for out-of-distribution requirements.

Empirical Evaluation

Benchmark and Setup

Resource2Skill is evaluated across seven heterogeneous creative software domains: slide design, spreadsheet authoring, web authoring, 3D scene creation (Blender), real-time 3D (UE5), CAD, and music production (Reaper). Each domain involves a pool of 80 screened benchmarks, and comparative evaluations include the full Resource2Skill pipeline (with skills), its no-skill ablation, and two off-the-shelf agentic system baselines (ClaudeCode-H, Codex-H).

Automatic judging (GPT-5.4 vision or GPT-4o audio) is combined with domain-specific rubrics. All systems are evaluated under matched agent backends and benchmark splits, enforcing strong statistical rigor by paired brief selection.

Main Results

With skill access, Resource2Skill yields a consistent and strong improvement across all domains and model backends:

• Average overall score improves by $+11.9$ points over no-skill agents.
• Resource2Skill outperforms both agentic-harness baselines in 26 out of 28 main-aggregate cells.
• Gains are largest in domains dense with procedural conventions (Excel, Blender, Web, and notably UE5), with per-domain improvements frequently exceeding $+14$ to $+40$ points in the most challenging settings.
• All cell-wise improvements are statistically significant ($p < 10^{-3}$), with most at $p < 10^{-8}$, and consistent human study validation confirms automatic judge preference.

Library Scaling and Online Acquisition

Figure 3: (a, left) Skill-pool scaling: Increasing the library size leads to monotonic performance gains which saturate near 200 skills. (b, right) Wiki organization: The full hierarchical, multimodal wiki interface outperforms both no skills and flat pure-text access designs across all domains.

• Performance scales monotonically with the number of skills up to saturation near 200-400 entries per domain. Major gains are observed with the first 200 skills, after which marginal improvements diminish.
• Online acquisition is effective only as a gap-filling mechanism for capabilities absent in the offline pool, yielding a $+21.6$-point lift on stress-test splits targeting such gaps, but only negligible gain ($+0.7$) on standard benchmarks.

Ablation Studies

Skill Representation and Wiki Structure:

Hierarchical organization and multimodal representation (combining text, code, visuals) are critical for performance. Pure text skills as a flat list yield some gains ($+2.5$—$8.2$ points lower than the full system), but the integrated, hierarchical wiki with code and visuals consistently dominates.

Source Modality:

Video is a non-substitutable source for procedural signal; its removal leads to an average $+11.9$0 points loss. Adding code, articles, or artifacts on top of video provides smaller but consistent incremental coverage and robustness.

Skill Representation:

Text-only skills reach $+11.9$1, but enabling both visual and code content lifts mean performance to $+11.9$2, confirming the value of a true multimodal bundle.

Selection Strategy:

The hierarchy-then-LM selection policy outperforms all retrieval-only or random candidates, especially for tasks requiring nuanced judgment and code grounding.

Implications and Prospects

The results demonstrate that distilling procedural knowledge directly from diverse, human-generated multimodal resources enables a new class of software agents with substantially improved procedural competence and generalization. This establishes a paradigm wherein agent procedural memory is incrementally constructed and maintained through scalable, semi-automatic pipelines sensitive to domain structure and resource diversity.

Practical Implications:

• Such skill wikis can be incrementally deployed and extended, making agent capabilities transparent, debuggable, and maintainable—a major advantage over latent-weight LMs or hardwired prompts.
• The maturity of the underlying framework suggests that future agentic systems can integrate online, real-time acquisition pipelines to adapt to evolving software and user needs without manual intervention.
• The explicit, multimodal nature of the skills supports inspection, reuse, and collaborative authoring in agentic toolchains.

Theoretical Implications and Future Directions:

• The demonstrated statistical effectiveness of multimodal procedural distillation raises new questions in representation learning for procedural tasks, and points to the value of cross-modal transfer and alignment.
• The clean separation of resource processing, skill abstraction, and hierarchical retrieval offers a modular foundation for future research in autonomous agent self-improvement, open-world generalization, and compositional problem solving.
• Challenges remain in scaling such systems beyond domains possessing rich programmatic interfaces and widely available procedural content, and in fully automating robust parameter grounding and dynamic skill composition.

Conclusion

Resource2Skill (2606.29538) defines an operational pipeline for transforming human multimodal procedural knowledge into structured, executable agent skills, yielding significant performance gains and outscoring strong agentic baselines. The findings underscore the utility of a blended, hierarchical, and diversely-sourced Skill Wiki, and open the path for new research in lifelong agent skill acquisition, robust procedural modeling, and extensible agentic memory architectures.