Skill1: Unified Skill Learning in RL

• Skill1 is a unified reinforcement learning framework that evolves skill-augmented agents by maintaining a persistent library of parameterized skills.
• It integrates end-to-end modules—query generation, candidate re-ranking, skill-conditioned execution, and distillation—to jointly optimize task success under a single reward objective.
• Empirical evaluations on ALFWorld and WebShop demonstrate that Skill1 outperforms traditional modular and decoupled approaches through effective reward decomposition and skill library management.

Skill1 is a unified reinforcement learning (RL) framework for evolving skill-augmented agents that maintain, reuse, and expand a persistent library of parameterized skills. It operationalizes the co-evolution of three coupled agent capabilities—skill selection, skill utilization, and skill distillation—under a single shared task-outcome objective. By integrating library querying, skill-conditioned policy execution, and experience-driven skill distillation within a single trainable policy, Skill1 achieves robust performance gains over prior modular or reward-decoupled approaches, as demonstrated on benchmarks such as ALFWorld and WebShop. The framework employs decomposed credit assignment from sparse task reward, resulting in stable, interpretable skill libraries and state-of-the-art success rates compared to alternative baselines (Shi et al., 7 May 2026).

1. Formal Model and Skill Library

Skill1 considers the skill-augmented agent problem as a partially observable Markov decision process (POMDP) where each agent state comprises a task instruction $x$, environment state $e$, and a persistent skill library $\mathcal{B} = \{s_1, s_2, ...\}$. A skill $s \in \mathcal{B}$ contains both (a) a natural-language strategy $s.\mathrm{strat}$ (describing how to act) and (b) a short natural-language scenario description $s.\mathrm{desc}$ (characterizing the skill’s applicability). The policy $\pi_\theta$ operates over this joint space, producing both environment actions and library-management subroutines.

The lifecycle of each skill comprises three stages:

1. Selection: Retrieve and select a task-relevant skill $z$ from $\mathcal{B}$.
2. Utilization: Condition the agent’s policy execution on $z.\mathrm{strat}$ to solve the current task.
3. Distillation: Summarize a successful trajectory into a new candidate skill and admit it to $e$0 (possibly evicting a low-utility skill when the library exceeds a size threshold).

Prior approaches often treat these phases as independent modules or optimize them under isolated heuristics, resulting in conflicting gradients and suboptimal coordination. Skill1 instead trains all phases toward maximization of expected task-outcome reward:

$e$1

where $e$2 denotes terminal task success (Shi et al., 7 May 2026).

2. Policy Architecture and Sequential Workflow

The Skill1 agent’s policy $e$3 is implemented via a sequential, end-to-end architecture composed of four core modules:

• A. Query Generation: Given a task instruction $e$4, the policy produces a one-sentence query $e$5. A frozen encoder $e$6 computes embeddings for both $e$7 and skill scenario descriptions, retrieving the top-$e$8 most similar skills $e$9 by embedding similarity.
• B. Candidate Re-ranking: Conditioned on $\mathcal{B} = \{s_1, s_2, ...\}$0, a re-ranking module outputs a permutation $\mathcal{B} = \{s_1, s_2, ...\}$1 over $\mathcal{B} = \{s_1, s_2, ...\}$2 candidates, sampling $\mathcal{B} = \{s_1, s_2, ...\}$3. The highest-ranked skill, $\mathcal{B} = \{s_1, s_2, ...\}$4, is selected for task conditioning.
• C. Skill-conditioned Utilization: For up to $\mathcal{B} = \{s_1, s_2, ...\}$5 environment steps, actions are drawn as $\mathcal{B} = \{s_1, s_2, ...\}$6, building an interaction trajectory $\mathcal{B} = \{s_1, s_2, ...\}$7.
• D. Skill Distillation: Upon a successful trajectory ($\mathcal{B} = \{s_1, s_2, ...\}$8), new skill tokens are generated:
  • $\mathcal{B} = \{s_1, s_2, ...\}$9,
  • $s \in \mathcal{B}$0,
  • and $s \in \mathcal{B}$1 is admitted to $s \in \mathcal{B}$2 (Shi et al., 7 May 2026).

A simplified pseudocode is:

$\pi_\theta$7

Each module’s output feeds into the next, and the entire pipeline is jointly differentiable and optimized toward a unified task reward.

3. Unified Reinforcement Learning Credit Assignment

A central innovation is the decomposition of the single, sparse task-outcome reward signal $s \in \mathcal{B}$3 into two signals:

• Low-frequency trend $s \in \mathcal{B}$4: A smoothed baseline (LPF of $s \in \mathcal{B}$5), which credits the skill-selection step—encouraging the retrieval of globally high-utility skills.
• High-frequency variation $s \in \mathcal{B}$6: Used to credit distillation, rewarding creation of skills that outperform the library’s current best.

Credit assignment is structured as follows:

• Utilization reward $s \in \mathcal{B}$7 trains query and environment-action modules.
• Selection (re-ranking) reward $s \in \mathcal{B}$8 is NDCG between sampled permutation and library utility ranking.
• Distillation reward $s \in \mathcal{B}$9, where $s.\mathrm{strat}$0 is the utility of the best candidate in $s.\mathrm{strat}$1, incentivizes novel, valuable skill creation.

The collective objective is:

$s.\mathrm{strat}$2

with $s.\mathrm{strat}$3. Policy gradients are estimated via GRPO for utilization/distillation and REINFORCE for re-ranking. Per-skill utilities $s.\mathrm{strat}$4 are exponentially averaged from rollout successes (Shi et al., 7 May 2026).

4. Empirical Evaluation and Benchmark Comparison

Skill1 was evaluated on two benchmarks:

• ALFWorld: Text-based household planning tasks requiring multi-step reasoning.
• WebShop: Simulated e-commerce task environment.

Skill1 was compared to:

• Training-free baselines: Zero-Shot, ReAct, Reflexion, ExpeL, Mem0.
• Parameter-only RL: PPO, RLOO, GRPO, GiGPO.
• Skill-based RL: EvolveR, Mem0+GRPO, SkillRL, RetroAgent.

Performance metrics included task success rates and purchase scores (WebShop). Results indicate that Skill1 achieved:

• ALFWorld: $s.\mathrm{strat}$5 average success (vs. $s.\mathrm{strat}$6 for best skill-RL baseline RetroAgent; +2.6 pts).
• WebShop: $s.\mathrm{strat}$7 success (vs. $s.\mathrm{strat}$8 baseline) and a $s.\mathrm{strat}$9 point improvement over pure RL (GiGPO: $s.\mathrm{desc}$0 on ALFWorld).

These results demonstrate the advantage of Skill1’s explicit skill-reuse and unified credit assignment over both modular and ablated variants (Shi et al., 7 May 2026).

5. Co-Evolution Dynamics and Ablation Findings

Training analysis tracked three indicators:

• Selection precision (mean utility of retrieved skills): rapidly approaches $s.\mathrm{desc}$1 within $s.\mathrm{desc}$2 steps.
• Utilization success rate and distillation positive rate (rollouts where $s.\mathrm{desc}$3): both accelerate around step $s.\mathrm{desc}$4–$s.\mathrm{desc}$5, stabilizing near $s.\mathrm{desc}$6.

This dynamics demonstrates sequential mutual reinforcement: high-quality selection leads to easier downstream solving and richer novel-skill distillation.

Ablation studies reveal:

• Without the skill library: $s.\mathrm{desc}$7 avg success (–16.6 pts).
• Without distillation: $s.\mathrm{desc}$8 (–5.1 pts).
• Without selection: $s.\mathrm{desc}$9 (–5.7 pts).
• Without re-ranking ($\pi_\theta$0): $\pi_\theta$1 (–3.5 pts).
• Without distill reward ($\pi_\theta$2): $\pi_\theta$3 (–2.6 pts).
• With both $\pi_\theta$4, $\pi_\theta$5: $\pi_\theta$6 (–7.3 pts).

Every component and credit signal contributes additively and complementarily; omitting either selection trend or distillation variation diminishes both library quality and task performance (Shi et al., 7 May 2026).

6. Interpretation, Scope, and Limitations

Skill1 demonstrates that a single-objective, reward-decomposed RL framework can achieve unified co-evolution of skill selection, utilization, and library expansion—without task-independent heuristics or separate reward signals. The design ensures skills are compact, non-redundant, and dynamically adapted to the agent’s experience distribution.

A plausible implication is that such unified frameworks may more generally overcome modular bottlenecks in skill-based RL. However, the results in (Shi et al., 7 May 2026) are focused on text-environment tasks, with both the skill representations and credit assignment relying on sparse outcome rewards. Scalability to rich, continuous, or high-stakes environments remains an open topic.

Skill1’s performance gains, robust training dynamics, and interpretable modularity establish it as a state-of-the-art solution for persistent skill library evolution in agentic RL settings (Shi et al., 7 May 2026).