Skill-RM: Unifying Heterogeneous Evaluation Criteria via Agent Skill

Abstract: Reward models (RMs) provide critical feedback signals for LLM post-training, notably in reinforced fine-tuning (RFT) and reinforcement learning (RL) pipelines. However, current reward evaluation relies on heterogeneous criteria such as rule-based verifiers, ground-truth references, procedural checklists, and complex rubrics, where a unified mechanism to integrate all types of evidence remains unexplored. To this end, we propose Skill Reward Model (Skill-RM), a unified framework that reformulates reward modeling as the execution of a reusable Reward-Evaluation Skill. By treating reward computation as a structured agentic task, Skill-RM provides a consistent interface to orchestrate heterogeneous resources, dynamically selecting and aggregating evidence tailored to the specific requirements of each input. This approach enables the reward model to move beyond static evaluation, ensuring consistency and transparency across diverse tasks. Extensive experiments on reward benchmarks and downstream applications, including best-of-N selection and reinforcement learning, demonstrate that Skill-RM consistently outperforms traditional judge baselines. Our findings suggest that Skill-RM not only provides a unified solution for reward modeling but also achieves superior performance through the strategic and dynamic orchestration of evidence. The code is at https://github.com/Qwen-Applications/Skill-RM.

• The paper demonstrates that reward modeling can be reformulated as an agent skill by executing structured protocols over heterogeneous evidence.
• It introduces a modular design that dynamically selects task-specific resources such as rubrics, checklists, and verifiers, ensuring criterion-level traceability.
• Experimental results show that Skill-RM outperforms traditional methods, achieving up to 97.8 accuracy on best-of-N selection and improved LLM alignment.

Skill-RM: Agentic Unification of Heterogeneous Reward Modeling

Introduction and Motivation

LLM alignment fundamentally relies on reward models (RMs) to operationalize complex behavioral desiderata during RL or reinforced fine-tuning. Traditional reward modeling typically outputs scalar scores or pairwise preferences via parameterized scorers, but emerging demand for high-fidelity reward assignment—in factuality verification, code execution, instruction following, and agentic trajectories—requires dynamic integration of diverse evidence types: rubrics, tool-based verifiers, reference answers, checklists, aggregation rules, and more.

Existing RM pipelines either statically encode such signals as undifferentiated contexts, bundle them into flattened prompts, or restrict evaluation to isolated mechanisms. This procedural opacity impedes interpretability, flexibility, and adaptation to new evidence types. The presented work introduces the Skill Reward Model (Skill-RM) (2606.03980), which reconceptualizes reward modeling as the execution of an agent skill—an explicit, filesystem-based procedural artifact that manages heterogeneous resources, orchestrates resource selection and usage in response to input requirements, and exposes a modular computation trace for criterion-level auditing.

Framework: Skill Mediation for Reward Modeling

Skill-RM treats the reward model as a reusable agent skill comprising a (i) procedural specification that dictates the protocol for evaluation, (ii) a structured resource bank comprising rubrics, references, checklists, verifiers, and aggregation rules, and (iii) a deterministic readout function to synthesize the structured evidence trace into scalar or selection signals.

This architecture implements reward computation as an agentic trace over the evaluation instance: the judge model incrementally loads and invokes resources as necessitated by the evaluation protocol, collects criterion-level evidence, and records an auditable computation trace. This design guarantees that the final reward is an explicit function of selected resources and synthesized observations.

Figure 1: Overview of Skill-RM. Evaluation logic is encoded as an agent-executable procedure with explicit invocation of rubrics, checklists, verifiers, and aggregation rules, progressing from resource selection to trace generation to deterministic reward readout.

Compared with black-box approaches or heuristic prompt composition, this modular abstraction enables:

• Adaptive resource invocation: Task-specific resource selection, enabling the judge to dynamically load only relevant evidence modalities.
• Criterion-level traceability: Structured evidence separated by criterion, with explicit decision fields for audit and diagnosis.
• Unification of heterogeneous paradigms: Scalar, pairwise, rubric-conditioned, tool-augmented, and resource-calibrated reward assignments are all expressed as procedural skill executions.

Experimental Results

Reward Benchmarking

Extensive evaluation is conducted on RewardBench2, RM-Bench, JudgeBench, and downstream RL tasks (IFEval, IFBench, AdvancedIF), all with controlled backbone ablations.

Skill-RM (Qwen3.5-27B) achieves an average reward benchmark score of 86.2, outperforming Qwen3.5-27B LLM-as-a-Judge (83.9) and all other generative, scalar, and rubric-based competitors. The gains are robust across major benchmarks and parameterizations.

When protocol-exposed, sample-specific resources are mounted via the skill, the Skill-RM average rises further to 89.1. Appending the same resources directly as context—without skill mediation—results in substantially weaker or even negative yield (81.0), highlighting that structured resource organization, not mere exposure, drives the improvement.

Best-of-N Selection

In fixed-pool best-of-10 reranking on JETTS families, Skill-RM achieves accuracy up to 97.8, matching the oracle upper bound on GSM8K and delivering clear gains over baseline judges on IFEval and HumanEval+. Smaller but positive deltas are observed for more challenging code-oriented pools.

Figure 2: Best-of-10 response selection accuracy across JETTS benchmarks confirms that Skill-RM consistently surpasses direct LLM judging and narrows the gap with the oracle upper bound.

Instruction-Following Policy Optimization

For RL reward provision (using GRPO on VerInstruct), Skill-RM delivers the highest average accuracy (45.9) across downstream IF benchmarks, marginally exceeding prior bests (VerIF: 44.7, Tulu3: 45.1). The effect persists in anchored-pairwise ablations: skill-mediated rewards outperform both context-augmented and tool-only approaches, demonstrating that compositional, protocol-driven reward computation best supports generalization in policy optimization.

Theoretical and Practical Implications

Theoretical Integration

By casting reward modeling as a skill—a dynamic, auditable agentic protocol—Skill-RM aligns with advances in agent architecture (Xu et al., 12 Feb 2026, Jiang et al., 24 Feb 2026) and leverages the lifecycle management, applicability conditions, and modular compositionality of agent skills. It bridges the gap between static RM architectures and the fine-grained, multimodal evaluative demands of real LLM applications.

Practical Advantages

• Traceability: Skill-RM judgments are procedural computations, not opaque scores; criterion-level evidence, resource provenance, and aggregation decisions are retained for later audit, diagnosis, and improvement.
• Adaptivity: New judgment resources can be registered as skill-accessible entries without loss of protocol discipline; e.g., integrating a newly discovered verifier or checklist for a novel task.
• Generalization: Superior performance in both generic (reward benchmarks) and highly contextualized (instruction following, tool-centered verification) regimes.

Limitations and Directions for Future Work

Skill-RM presently targets text-based reward modeling and leverages manually curated skills. Open theoretical challenges include extension to multimodal settings, compositional aggregation over subjective preferences, and automatic skill construction or continual updating. Additionally, agentic execution incurs inference-time overhead compared to scalar RMs, motivating research in resource caching, execution policy optimization, and adaptive trace truncation.

Conclusion

Skill-RM (2606.03980) redefines reward modeling as a structured agent skill: a compositional, interpretable, and highly adaptive mechanism for unifying heterogeneous evaluation criteria. Extensive evaluation demonstrates that explicit procedural mediation of resources—the operational core of agent skills—yields robust, scalable improvements in reward model accuracy and auditability. This formulation provides a scalable pathway to high-fidelity, trustworthy LLM alignment and sets the stage for agentic reward modeling beyond current text-centric benchmarks.