SkillForge: Self-Evolving Agent Skills

• SkillForge is a set of methodologies that explicitly represent and optimize agent skills for iterative improvement across various technical domains.
• In enterprise cloud support, the framework leverages historical data and failure analysis to continuously refine skills with measurable boosts in consistency.
• As a benchmark and tool-forging mechanism, SkillForge enables safe, traceable execution in Minecraft tasks and MetaForge's dynamic agent environments.

SkillForge most prominently denotes a self-evolving framework for constructing, evaluating, and refining domain-specific agent skills for enterprise cloud technical support, introduced in "SkillForge: Forging Domain-Specific, Self-Evolving Agent Skills in Cloud Technical Support" (Liu et al., 9 Apr 2026). In adjacent arXiv usage, the same term also denotes a curated Minecraft benchmark for instruction-following agents in "GROOT: Learning to Follow Instructions by Watching Gameplay Videos" (Cai et al., 2023), is the benchmark targeted by Preference Goal Tuning in "Optimizing Latent Goal by Learning from Trajectory Preference" (Zhao et al., 2024), and appears as an online tool-forging mechanism inside MetaForge (Wei et al., 1 Jun 2026). Across these usages, the common theme is the operationalization of skills as explicit artifacts that can be evaluated, adapted, and, in some settings, continuously improved.

1. Scope and nomenclature

The term “SkillForge” is not monosemous in the recent literature. It refers to distinct technical objects with different problem settings, evaluation protocols, and artifact types.

Usage of “SkillForge”	Paper	Technical referent
Enterprise support framework	(Liu et al., 9 Apr 2026)	Self-evolving framework for cloud technical support skills
Minecraft benchmark	(Cai et al., 2023)	Human-in-the-loop benchmark of 30 Minecraft tasks
Benchmark target for latent-goal tuning	(Zhao et al., 2024)	Task suite used to evaluate PGT on 17 representative tasks
Mechanism inside MetaForge	(Wei et al., 1 Jun 2026)	Online tool-forging and recycling capability

This terminological overlap matters because the enterprise framework and the Minecraft benchmark solve different problems. The former focuses on domain-grounded skill artifacts for production support agents; the latter evaluates open-world instruction-following control in Minecraft. A plausible implication is that the shared label reflects a broader research shift toward treating skills as explicit units of optimization rather than as implicit behavior emerging only from end-to-end prompting or policy pretraining.

2. Enterprise SkillForge as a skill artifact framework

In the enterprise setting, SkillForge addresses deployment of LLM-powered agents for cloud technical support, where the central requirement is high-quality, domain-specific skills that encode procedural workflows, tool usage, and organizational knowledge (Liu et al., 9 Apr 2026). The framework was introduced because generic skill creators often fail for three reasons: they lack domain grounding, they provide no mechanisms to trace execution failures back to specific parts of the skill, and they stagnate after deployment because there is no structured way to turn operational evidence into targeted refinements.

A skill is represented as a self-contained, file-based package that avoids arbitrary code execution for enterprise safety and traceability. The package consists of SKILL.md, which stores instructions and workflow logic; references/, which stores domain knowledge files and tool schemas; and references/tools.json, which stores schemas for verified, pre-defined system tools. For expository convenience, the paper denotes a skill as $S=(I,T,R)$, where $I=\texttt{SKILL.md}$, $T=\texttt{tools.json}$, and $R=\texttt{references/}$. The agent treats $S$ as a meta-tool: it dynamically loads $I$ and $R$ into context, adjusts tool permissions using $T$, executes the task, and emits a customer-facing response. The skill is injected into runtime context via a dual-message mechanism carrying hidden API instructions rather than by altering the global system prompt.

All agent-side components interact with these files through a sandboxed Virtual File System. The VFS supports read, write, rename, copy, and standard utilities such as grep, head, tail, and find, with full logging and versioning. This constrained environment reduces production risk by eliminating arbitrary script execution, narrows the action space for failure attribution, and preserves traceability of every change to the skill artifact.

The initial skill is produced by a Domain-Contextualized Skill Creator using an enterprise-internal LLM, Qwen3-Max. Its synthesis process combines Workflow Mining from historical support tickets, Tool Mining from operation logs and internal registries, Knowledge Extraction from internal KB and official product documentation, and Skill Synthesis into a templated layout. The resulting SKILL.md organizes Background Knowledge, Scenario Triage, Per-Scenario Handling, FAQ for long-tail cases, and a Reference Index. Constraints require evidence traceability, non-linear branching, symptom-to-cause-to-resolution links, and explicit escalation paths.

3. Creation, diagnosis, and iterative self-evolution

The enterprise framework is organized as a five-phase pipeline: initialization, execution and monitoring, failure analysis, aggregation and diagnosis, and optimization (Liu et al., 9 Apr 2026). Initialization produces Skill_v0 from enterprise data. During execution and monitoring, a business agent handles tasks with Skill_vn, and a held-out LLM-judge evaluates the customer-facing reply for consistency with historical expert responses. Bad cases then enter the self-evolution loop.

Failure analysis is explicitly four-dimensional. The Failure Analyzer processes each inconsistent or partially consistent case along Knowledge, Tool, Clarification, and Style. Knowledge errors include missing, incorrect, contradictory, outdated, misapplied, or not surfaced information. Tool errors include missed call, wrong tool selection, wrong parameters, repeated calls, misinterpretation, or underutilization. Clarification errors include over-asking, under-asking, and wrong focus. Style errors include robotic, verbose, cold tone, or other expression issues. The analyzer emits per-dimension severities and issue types, then derives failure_categories, overall_severity, overall_verdict, and primary_category through deterministic aggregation logic.

Diagnosis converts failure patterns into localized skill defects. A ReAct-based Skill Diagnostician reads SKILL.md and references/ through the VFS, parses aggregated evidence, attributes failure categories to concrete defects in specific locations, and emits both a Diagnostic Report and a machine-parsable Optimization Plan. The paper gives canonical mappings such as knowledge:missing to absent scenario knowledge in SKILL.md, tool:missed_call to missing guidance on when to call a tool, tool:wrong_params to insufficient parameter documentation, clarification:over_asking to overly broad information-gathering rules, and style:verbose to lack of concise guidelines.

Optimization is deliberately conservative. The Skill Optimizer applies three guardrails: minimal modification, do no harm, and evidence-based editing. It merges similar recommendations, determines insertion points respecting SKILL.md structure, applies edits by priority, adds examples drawn from expert references, deduplicates, runs safety checks, and commits Skill_vn+1. Knowledge augmentation uses enterprise search tools to fill missing knowledge, placing core decision rules in SKILL.md and detailed background in references/.

This creates a closed creation–evaluation–refinement loop: creation, deployment, failure analysis, diagnosis, and optimization. In experiments the authors executed three rounds, denoted v1, v2, and v3. The framework is also compatible with human-in-the-loop operations because historical tickets and online rejects can serve as the feedback signal for later evolution.

4. Evaluation methodology and empirical findings in cloud technical support

The evaluation covers five production scenarios from a major cloud provider: S1 Account, S2 Domain, S3 DNS, S4 OSS, and S5 ECS, over 1,883 tickets and 3,737 tasks (Liu et al., 9 Apr 2026). A task is defined as a single-turn dialogue within a ticket that advances resolution. Data are split into a development set for evolution and a held-out evaluation set consisting of tickets never seen during evolution.

Skill quality is measured by an LLM-judge through Consistency Rate, with three verdicts: Consistent, Partially Consistent, and Inconsistent. The reported metrics are

$$\text{Strict CR}=\frac{C}{N}, \qquad \text{Lenient CR}=\frac{C+P}{N},$$

where $N$ is the number of evaluated tasks, $I=\texttt{SKILL.md}$0 the count judged consistent, and $I=\texttt{SKILL.md}$1 the count judged partially consistent. The judge was validated against human annotations with more than 90% agreement on a sample.

The baseline comparison distinguishes S_generic, S_domain, and S_manual. S_generic uses Claude Code with Claude Sonnet 4.5 plus tool schemas mined from operation logs but no access to domain docs or ticket content. S_domain is the Domain-Contextualized Skill Creator of SkillForge. S_manual is a human expert-authored skill encoding best practices. Across all five scenarios, S_domain outperforms S_generic on historical-ticket consistency. On average, S_domain improves Strict CR by +4.3 percentage points and Lenient CR by +3.6 points over S_generic, with the largest Strict CR gain in S5 at +7.2pp. Because S_generic already includes mined tool schemas, these gains isolate the contribution of domain workflows and knowledge grounding.

Self-evolution then yields monotonic gains from all starting points. On held-out evaluation sets, three rounds improve both Strict and Lenient CR for S_manual, S_domain, and S_generic. Starting from S_manual, gains reach +10.99 Strict and +12.21 Lenient CR by v3; starting from S_domain, +9.23 and +8.00; starting from S_generic, +11.60 and +4.90. The paper emphasizes that improvements accumulate steadily and do not plateau within three iterations. The v3 evolved skill also surpasses a mature legacy baseline composed of decision-tree workflows plus expert prompts by +13.76pp Strict CR on the same held-out set.

The OSS case study in Appendix F shows the mechanism of diagnosis-to-rewrite at finer granularity. Among 108 bad cases, Style appears in 108 cases, Clarification in 97, Knowledge in 76, and Tool in 59. Representative failures include missing knowledge about signed URL access to private buckets, incorrect knowledge on mirror back-to-origin configuration, and missing tooling for billing or resource package queries. The resulting Optimization Plan prioritizes filling critical knowledge gaps, refining clarification strategy and style through a “minimum information set” and concise-first guidance, and improving tool invocation logic by adding missing tools, specifying trigger conditions, and adding result-interpretation guidance.

Category-wise analysis shows steady improvements in Tool and Style, declining Clarification errors, and Knowledge failures that drop early and then plateau. The paper attributes that plateau to a natural boundary imposed by coverage of the underlying KB and available tools. This suggests that the self-evolution loop is especially effective for file-localized workflow and response-policy defects, while some residual knowledge limitations remain infrastructure-bound.

5. Minecraft SkillForge as an instruction-following benchmark

In the Minecraft literature, SkillForge is a curated, human-in-the-loop benchmark designed to evaluate diverse open-world skills under instruction following (Cai et al., 2023). It contains 30 tasks grouped into six categories: collect, explore, craft, tool, survive, and build. Tasks are goal-driven but open-ended in execution, and rubrics specify graded completion levels, allowing partial credit and preference-based human judgments.

The benchmark runs in Minecraft 1.16.5 via MineRL and MCP-Reborn, with first-person RGB observations at $I=\texttt{SKILL.md}$2 resized to $I=\texttt{SKILL.md}$3, a 70-degree field of view, night vision for dark scenes, and VPT’s hierarchical discrete action space. Evaluation combines a human-rated Elo tournament and programmatic checks for a subset of tasks. For each task, each agent produces 10 videos of 600 frames, human raters compare video pairs using task-specific rubrics, and 1,500 total pairwise comparisons yield converged Elo. The benchmark also reports programmatic success criteria on nine representative tasks such as enchant sword, smelt food, use bow, sleep, lead animals, build snow golem, build obsidian, and dig three down and fill one up. GROOT, which conditions on reference gameplay videos rather than text, closes the human-machine gap and exhibits an approximately 70% winning rate over the best generalist agent baseline, corresponding to an Elo advantage of about 150 points.

This benchmark later becomes the evaluation target for Preference Goal Tuning, which studies prompt brittleness in instruction-following foundation policies (Zhao et al., 2024). PGT operates on two policies, GROOT and STEVE-1, and tunes only the latent goal representation $I=\texttt{SKILL.md}$4 while freezing the policy backbone. The benchmark in that work comprises more than 30 tasks across the same six categories, but the experiments evaluate 17 representative tasks because trivially easy and near-zero-success tasks are excluded. PGT collects trajectories, labels them as positive and negative via human preference or reward-based ranking, and optimizes a trajectory-level DPO-style objective in latent-goal space. In the reported configuration, the system uses 500 rollouts per task, 150 positive and 150 negative samples, Adam with learning rate $I=\texttt{SKILL.md}$5, $I=\texttt{SKILL.md}$6, full-batch gradient descent, and up to three iterations of recollection and tuning.

Empirically, PGT reports average relative improvements of +72.0% and +81.6% in in-distribution settings over 17 tasks for GROOT and STEVE-1 respectively, with out-of-distribution improvements of +73.8% for GROOT and +36.9% for STEVE-1. It also surpasses full fine-tuning by 13.4% on average in out-of-distribution environments and consistently beats the best human-selected prompt. In this usage, SkillForge functions not as a self-evolving skill artifact system but as a demanding benchmark for studying latent-goal alignment, robustness to world variation, and continual-learning efficiency.

6. SkillForge inside MetaForge: on-demand tool forging and recycling

Within MetaForge, “SkillForge” denotes the online creation and reuse of executable tools in a multimodal agent framework (Wei et al., 1 Jun 2026). MetaForge factorizes behavior into Decide, Retrieve, Adapt, and Forge, with validated forged tools recycled back into a Dynamic Tool Pool, yielding a closed judge–retrieve–adapt–forge–recycle loop. A unified orchestration policy chooses among answering directly, invoking an existing tool, or forging a new one on demand.

The tool pool contains Executable Tools, Model Tools, and Forged Tools, all sharing a unified interface $I=\texttt{SKILL.md}$7. A forged tool is represented as a triplet $I=\texttt{SKILL.md}$8, where $I=\texttt{SKILL.md}$9 is Markdown containing name, function, scenarios, and calling conventions; $T=\texttt{tools.json}$0 is a JSON schema containing parameters, types, and constraints; and $T=\texttt{tools.json}$1 is a Python script set with parameterized I/O executed by the runtime. Validation uses

$T=\texttt{tools.json}$2

and only validated skills are registered back into the pool. Capacity-constrained management then performs semantic deduplication and usage-based retention.

The policy is trained with Group Relative Policy Optimization over multi-turn trajectories. The composite reward

$T=\texttt{tools.json}$3

assigns credit to answer correctness, efficiency through a latency-normalized invocation-cost penalty, forged-skill creation and reuse, and syntactic parsability. This makes Decide, Retrieve, Adapt, and Forge jointly optimized rather than hard-wired as separate modules.

MetaForge is trained on 9,500 samples across 19 datasets in six categories and evaluated on 12 benchmarks with 24,005 test samples. With Qwen3-VL-8B-Instruct as the base policy, Qwen3-VL-235B-A22B-Instruct as the forging model, and GPT-5 as judge, it reports ALL AVG 88.14 with IID tools and 88.98 with OOD tools, surpassing 16 baselines. Efficiency metrics include Average Interaction Turns of approximately 2.28 for IID and 2.30 for OOD, end-to-end latency of 10.99s and 4.97s, and generated tokens of 280.87 and 336.65. Tool-call success reaches 96.59% under IID tools and 98.24% under OOD tools, while Forged Skill Registration Success is 64.7% and 91.7% respectively. In this context, SkillForge is neither a benchmark nor a support-specific refinement loop; it is a mechanism for on-demand synthesis, validation, registration, and later reuse of executable skills.

7. Limitations, reproducibility, and broader significance

Across the three main usages, the term is associated with explicit skill representations, iterative improvement, and evaluation under distributional variation, but the limitations differ by setting. In enterprise support, the main constraints are knowledge coverage, possible misdiagnosis or overfitting during refinement, domain shift to emerging issues, privacy and security requirements, and the need for human oversight when major knowledge or tool additions occur (Liu et al., 9 Apr 2026). The paper does not release code or data because enterprise processing remains within the provider’s privacy boundary, but it documents the VFS environment, LLM-judge prompt and schema, failure-analysis schema, aggregation logic, Diagnostician and Optimizer workflows, and Skill Creator template structure.

For PGT on the Minecraft benchmark, the limiting factors are environment interaction cost, sparse positives on very hard tasks, sensitivity to data diversity and label noise, and dependence on whether the initial goal encoder can express the target concept (Zhao et al., 2024). That paper documents the environment, OOD protocols, hyperparameters, PEFT baselines, and continual-learning baselines, but does not explicitly claim a public code release. For MetaForge, current forged skills are executable programs suited to programmatic operations, evaluation focuses on image–text rather than audio or video, and future deployments may need to model monetary cost and context budget in addition to latency (Wei et al., 1 Jun 2026).

A recurrent misconception is to treat SkillForge as a single standardized system. The literature instead uses the name for three linked but distinct ideas: a production-oriented self-evolving skill framework, a benchmark for open-world instruction following, and a tool-forging mechanism in multimodal agents. What unifies them is an artifact-centric view of capability: skills are represented explicitly, assessed with structured criteria, and updated either by deployment feedback, preference optimization, or online synthesis. This suggests a broader research trajectory in which agent competence is managed through inspectable, versioned, and reusable skill objects rather than only through monolithic prompt engineering or full-parameter retraining.