Skill Reward Model (Skill-RM) Overview

• Skill-RM is a unifying framework that attaches explicit reward signals to reusable skills, spanning robotics, language agents, and evaluation scenarios.
• It employs diverse formulations—from value functions and intrinsic rewards to utility estimators and multi-branch schedulers—to guide skill selection and optimization.
• The model enables efficient planning and decision-making by integrating counterfactual evaluation, structured orchestration, and dynamic reward scheduling.

Skill Reward Model (Skill-RM) denotes a family of formulations in which reusable skills are assigned explicit reward, value, or utility signals and then selected, composed, validated, pruned, or evolved through optimization. In the supplied literature, the “skill” may be a task-agnostic robot controller with a value function, a latent macro-action, a textual procedure stored in a language-agent skill bank, or a reusable Reward-Evaluation Skill that an LLM judge executes as an agent. Accordingly, a Skill-RM may appear as an additive plan-value model over robot skills, a context-dependent marginal-utility estimator for candidate language skills, an intrinsic reward family induced by a skill discriminator, a dynamic reward scheduler over subskill channels, or a structured evaluation procedure that orchestrates rubrics, references, checklists, and verifiers (Xue et al., 2024, Zhang et al., 7 Jun 2026, Adeniji et al., 2022, Huang et al., 5 May 2025, Chen et al., 2 Jun 2026).

1. Conceptual scope

The literature does not use Skill-RM in a single narrow sense. In robotic planning, it refers to per-skill rewards or value functions that are composed into a plan objective. In language agents, it refers to skill-level utility signals that determine whether a textual procedure should be stored, retrieved, revised, or pruned. In open-ended evaluation, it refers to a reusable reward-evaluation procedure that turns heterogeneous evidence into a judgment. In unsupervised RL, it refers to intrinsic rewards that differentiate skills by their state occupancy or transition semantics. Taken together, these works indicate that Skill-RM is best understood as a reward-bearing interface between a skill abstraction and a downstream decision process (Xue et al., 2024, Shi et al., 7 May 2026, Chen et al., 2 Jun 2026, Xiao et al., 17 Jun 2025).

Setting	Skill form	Skill-RM signal
Robotic planning	Independently trained, task-agnostic controller plus value function	$$\sum_{k=1}^K V^{\pi_k}(\bm{x}_{k_0}) + \Psi(\overline{\bm{x}_{K_T}})$$
Language agents	Textual procedures, natural-language strategies, or Markdown skill files	$u(x,s)$, $U_s$, or $\Delta_k$
Unsupervised / model-based RL	Discrete or latent skill $z$	$q_\phi(\tau,z)$, $r^{\rm sd3}_z(s)$, or $R_\phi(h,z)$
Reward judging	Reward-Evaluation Skill $$\mathcal{S}_{\text{RM}}=(\mathcal{M}_{\text{RM}},\mathcal{U}_{\text{RM}})$$	Structured judgment $z=(\mathcal{E},d)$ and readout $u(x,s)$0

The breadth of the concept is visible in the underlying skill representations. LSP models each primitive as an MDP $u(x,s)$1 with a policy $u(x,s)$2 and value function $u(x,s)$3 (Xue et al., 2024). SAPO stores skills as JSON objects with title, principle, and when_to_apply, whereas Skill1 stores each skill as a two-part natural-language object with strat and desc, and SKILL0 stores skills as Markdown files grouped by task and category (Zhang et al., 7 Jun 2026, Shi et al., 7 May 2026, Lu et al., 2 Apr 2026). Skill-RM for reward evaluation defines a reward skill as $u(x,s)$4, while SD3 and SkiMo operate over discrete or latent skill variables $u(x,s)$5 in skill-conditioned policies (Chen et al., 2 Jun 2026, Xiao et al., 17 Jun 2025, Shi et al., 2022).

2. Formal reward constructions

In robot skill planning, the most explicit plan-level Skill-RM appears in Logic-Skill Programming. LSP defines the total value of a plan as

$u(x,s)$6

with each initial skill state $u(x,s)$7 depending on the chosen subgoal through $u(x,s)$8. This coupling is central: the value function is not only a local execution score but a function over potential subgoal instantiations. The per-skill reward is skill-centric, of the form $u(x,s)$9, encoding tracking error, effort, and, for pushing, minimal face switching. The paper explicitly tests whether value functions preserve the ordering of cumulative reward across states, and reports TT value-function agreement of approximately $U_s$0–$U_s$1 depending on skill, compared with SAC/PPO at approximately $U_s$2–$U_s$3; in the detailed summary this appears as $U_s$4–$U_s$5 versus $U_s$6–$U_s$7 (Xue et al., 2024).

SkiMo adopts a related but model-based construction at the macro-action level. It learns a skill dynamics model

$U_s$8

and, during downstream RL, extends the model with a skill-level cumulative reward predictor

$U_s$9

where

$\Delta_k$0

The Skill-RM here is the mapping from current latent state and skill latent to the cumulative reward of executing that skill over its fixed horizon. Planning is then performed directly in skill space rather than over primitive actions (Shi et al., 2022).

IRM makes the reward-model interpretation explicit in a different way. It treats the skill discriminator learned during unsupervised skill discovery as an intrinsic reward family

$\Delta_k$1

and selects the downstream skill by minimizing EPIC distance to the extrinsic task reward:

$\Delta_k$2

The key property is that skill selection can be done without environment interaction, because the Skill-RM is the discriminator-derived reward function itself rather than empirical rollout return (Adeniji et al., 2022).

SD3 gives a density-based intrinsic formulation. Its inter-skill reward is

$\Delta_k$3

and it adds an intra-skill exploration term

$\Delta_k$4

The combined intrinsic reward is

$\Delta_k$5

This formulation makes the Skill-RM a joint density-and-novelty estimator over $\Delta_k$6, with the KL term interpreted as a latent pseudo-count bonus (Xiao et al., 17 Jun 2025).

AHRS broadens the notion further by decomposing a single robotic skill into sub-reward channels. It uses a multi-branch value network, one branch per reward component, and a weighted policy gradient in which each branch contributes $\Delta_k$7. With the auxiliary branch, the final advantage is

$\Delta_k$8

The weights are computed from LLM-generated rules over per-branch returns and variances, and the method reports an average $\Delta_k$9 performance improvement across multiple high-degree-of-freedom robotic tasks. In this formulation, the Skill-RM is not a single scalar reward head but a time-varying reward composition mechanism over subskill channels (Huang et al., 5 May 2025).

3. Language-agent skill utilities

In language agents, Skill-RM commonly appears as a utility estimator over reusable textual procedures. SAPO defines skill-level marginal utility through matched rollout groups. For a query $z$0, base rollouts are sampled under the currently retrieved skills $z$1, and skill-augmented rollouts are sampled under $z$2. The intra-prompt marginal utility is

$z$3

and for grouped induction units the cross-prompt score is

$z$4

Promotion to the long-term bank occurs only if

$z$5

The same marginal-utility signal trains the policy as a skill generator and becomes a retrieval-time scoring proxy through a learned skill-generation likelihood (Zhang et al., 7 Jun 2026).

Skill1 pursues unification through a single task-outcome signal. It defines skill utility as an exponential moving average

$z$6

uses this to create a ranking reward

$z$7

and defines a distillation reward

$z$8

The combined objective is

$z$9

with $q_\phi(\tau,z)$0 in experiments. The framework reports $q_\phi(\tau,z)$1 average success on ALFWorld and shows that removing either the ranking or distillation credit degrades performance (Shi et al., 7 May 2026).

SKILL0 uses an implicit Skill-RM in the form of policy-dependent helpfulness. Every $q_\phi(\tau,z)$2 steps it evaluates each skill file with and without the skill on a validation sub-task set and defines

$q_\phi(\tau,z)$3

The current active skills are filtered by $q_\phi(\tau,z)$4, ranked by $q_\phi(\tau,z)$5, and limited by a linearly decaying budget

$q_\phi(\tau,z)$6

At the final stage, $q_\phi(\tau,z)$7 and inference uses no skills. The method reports improvements of $q_\phi(\tau,z)$8 for ALFWorld and $q_\phi(\tau,z)$9 for Search-QA while maintaining fewer than $r^{\rm sd3}_z(s)$0k tokens per step (Lu et al., 2 Apr 2026).

Skill-R1 turns skill revision itself into a recurrent RL problem. For generation $r^{\rm sd3}_z(s)$1, it defines intra-generation advantage

$r^{\rm sd3}_z(s)$2

inter-generation advantage

$r^{\rm sd3}_z(s)$3

and combined advantage

$r^{\rm sd3}_z(s)$4

This yields a bi-level GRPO objective over skills across generations. On GAIA, overall accuracy rises from $r^{\rm sd3}_z(s)$5 for Vanilla GRPO to $r^{\rm sd3}_z(s)$6 for Skill-R1 (GRPO), and on WebWalker from $r^{\rm sd3}_z(s)$7 to $r^{\rm sd3}_z(s)$8 (Vishe et al., 10 May 2026).

4. Reward modeling as reusable evaluation skill

The most literal use of the term appears in reward judging for LLM post-training. Skill-RM defines a Reward-Evaluation Skill as

$r^{\rm sd3}_z(s)$9

where $R_\phi(h,z)$0 is a procedural specification and $R_\phi(h,z)$1 is a resource bank of rubrics, references, checklists, verifiers, and aggregation rules. Each resource is represented as

$R_\phi(h,z)$2

and the agentic judge produces a structured judgment

$R_\phi(h,z)$3

from which the reward or selection decision is deterministically read out as $R_\phi(h,z)$4. On the Qwen3.5-27B backbone, the average score across RewardBench2, RM-Bench, and JudgeBench rises from $R_\phi(h,z)$5 for the baseline judge to $R_\phi(h,z)$6 for Skill-RM, and to $R_\phi(h,z)$7 when sample-specific resources are added. A notable ablation shows that merely appending the same resources to the prompt reduces the average to $R_\phi(h,z)$8, indicating that the gain comes from structured orchestration rather than raw context volume (Chen et al., 2 Jun 2026).

Eval-Skill reaches a related endpoint through offline skill evolution rather than manual skill-bank design. It synthesizes one domain-level evaluation skill from only $R_\phi(h,z)$9 cases per domain in two progressive stages, workflow generation followed by principle generation, with exploration and selection interleaved across both stages. The final skill is directly injected into judge context, so reward guidance becomes context evolution rather than parameter training or per-query rubric generation. On RewardBench 2, the method improves vanilla judging by $$\mathcal{S}_{\text{RM}}=(\mathcal{M}_{\text{RM}},\mathcal{U}_{\text{RM}})$$0 for Qwen3-8B and $$\mathcal{S}_{\text{RM}}=(\mathcal{M}_{\text{RM}},\mathcal{U}_{\text{RM}})$$1 for DeepSeek-V4-Flash, and the paper reports that online rubric methods can underperform vanilla judging on the same benchmark (Yue et al., 5 Jun 2026).

These two lines also clarify a recurring distinction. A rubric is typically a flat criterion list or an online-generated query-specific artifact, whereas a skill can encode workflow, conditional branching, calibration rules, and output contracts. This suggests that Skill-RM in evaluation is less a scalar predictor than a reusable decision procedure whose final numeric output is a readout of structured evidence (Chen et al., 2 Jun 2026, Yue et al., 5 Jun 2026).

5. Representations and optimization mechanisms

Different Skill-RM instantiations rely on markedly different representations, but each is designed to make skill-level evaluation tractable. LSP uses Tensor Train factorization and TTPI to approximate value functions over the whole skill state space, with storage $$\mathcal{S}_{\text{RM}}=(\mathcal{M}_{\text{RM}},\mathcal{U}_{\text{RM}})$$2 instead of $$\mathcal{S}_{\text{RM}}=(\mathcal{M}_{\text{RM}},\mathcal{U}_{\text{RM}})$$3. Planning alternates between symbolic MCTS with UCB1 and continuous subgoal optimization with CEM-MD, where the scalar objective returned to symbolic search is exactly

$$\mathcal{S}_{\text{RM}}=(\mathcal{M}_{\text{RM}},\mathcal{U}_{\text{RM}})$$4

The role of the Skill-RM is therefore algorithmic as well as semantic: it is the low-level evaluator that biases the symbolic search tree toward high-value skeletons (Xue et al., 2024).

SkiMo jointly learns a skill repertoire, a latent skill dynamics model, and skill-level reward/value predictors, then plans in skill space by MPC with CEM over skill sequences. The skill dynamics

$$\mathcal{S}_{\text{RM}}=(\mathcal{M}_{\text{RM}},\mathcal{U}_{\text{RM}})$$5

and the cumulative skill reward predictor $$\mathcal{S}_{\text{RM}}=(\mathcal{M}_{\text{RM}},\mathcal{U}_{\text{RM}})$$6 reduce the effective planning horizon by a factor $$\mathcal{S}_{\text{RM}}=(\mathcal{M}_{\text{RM}},\mathcal{U}_{\text{RM}})$$7 and allow downstream learning of sparse-reward long-horizon tasks in imagination (Shi et al., 2022).

AHRS uses a multi-branch value network in which each branch corresponds to a distinct reward component. Rule generation is done offline by an LLM from task description, environment description, reward code, and reference rules; rule selection is done online every $$\mathcal{S}_{\text{RM}}=(\mathcal{M}_{\text{RM}},\mathcal{U}_{\text{RM}})$$8 epochs using per-branch current means, historical returns over $$\mathcal{S}_{\text{RM}}=(\mathcal{M}_{\text{RM}},\mathcal{U}_{\text{RM}})$$9 checkpoints, and variances. The final weight is

$z=(\mathcal{E},d)$0

and the weighted advantages are combined in the PPO update. The Skill-RM is thus a scheduler over a decomposed value landscape rather than a monolithic reward function (Huang et al., 5 May 2025).

ROSKA represents rewards directly as TorchScript-compatible Python functions generated and edited by GPT-4o. Reward candidates are evolved in dynamic populations,

$z=(\mathcal{E},d)$1

and policy initialization for each candidate reward is chosen by fusing the current best policy with a random initialization,

$z=(\mathcal{E},d)$2

with $z=(\mathcal{E},d)$3 selected by Short-Cut Bayesian Optimization. The paper reports an average normalized improvement of $z=(\mathcal{E},d)$4 across various high-dimensional robotic skill learning tasks, framing the Skill-RM as dynamic, policy-aware reward code rather than a fixed universal objective (Huang et al., 2024).

A common misconception is that a Skill-RM must be a learned scalar head. The cited implementations instead use value-function sums, discriminator-induced intrinsic rewards, utility tables, matched rollout estimators, executable reward code, and agentic evaluation skills. This suggests that the unifying property is not architecture but the existence of a reusable skill-level reward interface coupled to search, planning, storage, or post-training.

6. Design principles, limitations, and open problems

Several design principles recur. First, a Skill-RM must usually evaluate counterfactual or hypothetical skill instantiations rather than only on-policy trajectories. LSP therefore requires value functions that are accurate across the whole state/goal domain, not merely near training rollouts (Xue et al., 2024). SAPO requires pre-storage validation because skill utility is context-dependent and later rollout feedback conflates multiple retrieved skills; matched rollout groups are introduced precisely to isolate marginal contribution (Zhang et al., 7 Jun 2026). SKILL0 likewise measures skill helpfulness relative to the current policy, then withdraws skill context only when the policy no longer benefits from it (Lu et al., 2 Apr 2026).

Second, many systems rely on explicit structure to control non-stationarity. Skill1 smooths skill quality with an EMA utility table and then derives distinct rewards for ranking and distillation from the same task-outcome signal (Shi et al., 7 May 2026). Skill-R1 separates within-generation from across-generation credit, allowing a skill revision to be rewarded even when individual rollouts remain noisy (Vishe et al., 10 May 2026). AHRS and ROSKA both make reward shaping policy-dependent: one through branch reweighting, the other through reward–policy co-evolution (Huang et al., 5 May 2025, Huang et al., 2024).

Third, the literature repeatedly identifies scaling limits. LSP lists scaling CEM-MD, dynamic TT composition, uncertainty modeling, and integration with LLM task planners as open problems (Xue et al., 2024). SAPO notes noisy reward gaps, instability early in training, and changing policy as sources of non-stationary utility (Zhang et al., 7 Jun 2026). SKILL0 highlights dependence on environment reward, coarse-grained skill credit assignment at the level of entire files, offline relevance grouping, and scalability to large skill ecosystems (Lu et al., 2 Apr 2026). SD3 points to the discrete skill assumption, density-estimation fidelity, computational cost from evaluating all skills in the denominator, and sensitivity to $z=(\mathcal{E},d)$5 and $z=(\mathcal{E},d)$6 (Xiao et al., 17 Jun 2025). Skill-RM for evaluation remains constrained by manually and LLM-assisted curated reward skills, inference overhead from multi-step judging, and limited validation beyond current text-only settings; Eval-Skill further reports that mixed-domain settings, weaker instruction-following backbones, and hard domains such as healthcare remain challenging (Chen et al., 2 Jun 2026, Yue et al., 5 Jun 2026).

Across these works, Skill-RM is best characterized as a unifying abstraction for attaching reward semantics to skills at the level where decisions are actually made. In robotics, this level may be the subgoal-conditioned primitive or latent macro-action; in language agents, it may be the stored textual procedure or skill revision; in reward judging, it may be the evaluation protocol itself. This suggests that future Skill-RM research will depend less on converging to a single canonical model class than on improving reusable interfaces between skill abstractions, heterogeneous evidence, and downstream optimization objectives.