General Reward Model: Methods & Applications

• General Reward Models are versatile systems mapping context and outputs to structured reward signals, unifying scalar and structured evaluations.
• They integrate generative architectures, supervised fine-tuning, and reinforcement learning to enhance interpretability and performance across modalities.
• GRMs enable practical applications in LLM alignment, speech quality assessment, and robotic process manipulation while addressing reward hacking and calibration challenges.

A General Reward Model (GRM) is a versatile, often generative, parametric function designed to encode and evaluate outcomes, trajectories, or system outputs—according to learned or inferred principles—across diverse decision-making, alignment, or evaluation tasks. Unlike traditional scalar reward models that output single scores for isolated samples, a GRM leverages the representation and reasoning abilities of large neural models (e.g., LLMs or multimodal LLMs) to assess, explain, and rank outputs, sometimes over sequences and with structured, interpretable outputs such as natural-language critiques or stepwise progress. Modern GRMs unify both supervised and reinforcement learning objectives, often scale across modalities, and can be constructed from generative architectures, specialized fusions of domain expertise, or even derived directly from standard next-token predictors via inverse RL. Their development addresses the need for improved generalization, calibration, and interpretability in reward modeling for large-scale language, speech, vision, agentic, and robotic systems.

1. Formal Definitions and Core Architectures

A GRM is most generally formulated as a (learned) mapping

$(x, y) \mapsto \text{reward signal}$

where $x$ is a context (prompt, state, or task specification), $y$ an output (action, trajectory, answer), and the reward signal may be a scalar, vector, structured sequence, or probability distribution.

• In LLM alignment, GRMs are often implemented as conditional generative models $p_\theta(r, y | x, \{A_i\})$ producing both a natural-language chain-of-thought (CoT) and preference label over answer set $\{A_i\}$ (Chen et al., 20 Jun 2025, Wang et al., 17 Jun 2025).
• In speech quality assessment, GRMs ingest pairs of raw audio, encode each via a neural frontend, and output both a multi-aspect critique and numeric quality scores (Cao et al., 1 Oct 2025, Zhang et al., 11 Nov 2025).
• In robotic manipulation, GRMs process multi-view temporally paired images plus textual task specification, returning stepwise or global progress estimates and dense reward differentials suitable for policy shaping (Tan et al., 29 Dec 2025).

Approaches include:

• Autoregressive generative reward modeling (LLM-style reasoning and token-level scoring)
• Hybrid architectures with scalar reward heads behind shared representations
• Potential-based and history-dependent transformations (for non-Markovian shaping)
• Model merging (domain integration via parameter interpolation)
• Direct extraction from next-token LLMs by inversion of the soft Bellman operator (Li et al., 29 Jun 2025)

2. Training Paradigms and Objectives

GRMs are typically trained using a mixture of supervised, unsupervised, and RL-based objectives:

• Supervised Fine-Tuning (SFT):
  • Next-token cross-entropy on gold rationales, scalar preference labels, or paired scores.
  • Commonly used on curated datasets of labeled preferences (Cao et al., 1 Oct 2025, Wang et al., 17 Jun 2025, Zhang et al., 11 Nov 2025).
• Reinforcement Learning Objectives (e.g., PPO, GRPO, DAPO):
  • Reward obtained by comparing the model’s ranking or score outputs with human or synthetic preferences (Chen et al., 20 Jun 2025, Cao et al., 1 Oct 2025).
  • Mixed with difficulty-aware shaping (e.g., MOS-difference in speech) for fine-grained sensitivity.
• Self-training and semi-supervised objectives:
  • GRAM-R² employs self-training on unlabeled data by generating pseudo-labels and rationales, using a separate preference-proving model for synthetic rationales (Wang et al., 2 Sep 2025).
  • Label smoothing and hidden-state regularization improve generalization and robustness (Wang et al., 17 Jun 2025, Yang et al., 14 Jun 2024).
• RL with verifiable rewards in closed-loop agentic systems:
  • GRMs are co-evolved with policy models as learned verifiers, periodically calibrated using small real-data injections to prevent drift and reward hacking (Sun et al., 16 Oct 2025).
• Domain adaptation via model merging:
  • Combine a generic reward model and a domain-specialized SFT model with parameter interpolation to create GRMs that balance general preference and domain expertise (Lin et al., 1 Jul 2024).

3. Application Domains and Modal Adaptations

GRMs have been deployed and evaluated across a broad spectrum of domains:

Domain/Task	Model Input/Output	Notable GRM Mechanism	Key Results
LLM alignment (text)	(prompt, answer)	LLM: generative CoT, logits, tokens	Pretrained + label smoothing yields OOD gains (Wang et al., 17 Jun 2025)
Speech quality	(audio₁, audio₂, [task])	Audio encoder + critique + scores	MOS-aware reward narrows fine-grained gap (Cao et al., 1 Oct 2025)
Robotic process manipulation	(task, images pre/post)	Multi-view ViT fusion + hop progress	Policy-invariant shaping avoids reward trap (Tan et al., 29 Dec 2025)
Multi-modal alignment	(vision, text)	MLLM generative critiques + scores	RL-trained GRM outperforms score-RM by +18% (Zhou et al., 24 May 2025)
Agentic self-learning/QA	(retrieval, policy output)	Shared transformer, generative verifier	Co-evolution avoids reward hacking (Sun et al., 16 Oct 2025)
Semi-supervised learning	(feat, pseudo-label)	Generator, cross-attn rewarder	1.5–4.0 pp error reduction, ×1.5–3.7 speedup (Li et al., 2023)

This breadth highlights the flexibility of GRMs, which unify scalar, structured, and interpretability-aware outputs for reward modeling across disparate input types and environments.

4. Interpretability, Generalization, and Calibration

A unique feature of GRMs is their potential for interpretable, structured outputs:

• Natural-language critiques: Explicit reasoning steps or critiques are often generated, improving transparency and user trust (Chen et al., 20 Jun 2025, Cao et al., 1 Oct 2025, Zhang et al., 11 Nov 2025).
• Multi-dimensional scoring: Decomposition of complex reward axes (e.g., helpfulness, personalization, naturalness), reducing reward hacking and enabling more precise alignment objectives (Zhu et al., 21 Oct 2025).
• Sequential/stepwise scoring: For robotics and agentic domains, step-wise or trajectory-level progress scoring enhances reward granularity and enables intermediate feedback (Tan et al., 29 Dec 2025, Xia et al., 25 Feb 2025).
• Generalization and regularization: Shared hidden-state regularization, label smoothing, domain merging, and self-training pipelines demonstrably improve generalization to out-of-distribution (OOD) prompts and disfavored tasks (Wang et al., 17 Jun 2025, Wang et al., 2 Sep 2025, Lin et al., 1 Jul 2024).

Calibration and scaling:

• MOS-aware rewards and difficulty shaping: Adaptive reward functions that emphasize challenging distinctions yield improved fine-grained discrimination, closing the gap with scalar models in hard cases (Cao et al., 1 Oct 2025).
• Inference-time scaling: Parallel sampling and voting over GRM outputs, optionally meta-filtered, achieves further gains and allows compute-performance scaling beyond that achievable via model size alone (Liu et al., 3 Apr 2025).
• Meta reward models: Use small RMs to filter GRM-generated samples for higher-quality aggregation during inference (Liu et al., 3 Apr 2025).

5. Theoretical Foundations and Policy Invariance

Several works formalize the properties of GRMs:

• Potential-based and generalized reward matching: GRM is a plug-and-play, history-sensitive transformation that converts arbitrary intrinsic motivation signals into potential-shaped rewards, preserving optimality for Markov and future-agnostic IM signals (Forbes et al., 16 Oct 2024). The matching function $m_{t,t'}$ allows for a continuum of shaping reward timescales.
• Endogenous reward extraction from LLMs: For any next-token–trained LLM, the latent "generalist reward model" can be recovered by inverting the soft Bellman operator, with provable equivalence to offline inverse RL objectives and bounded policy error improvements under RL fine-tuning (Li et al., 29 Jun 2025). This removes the need for explicit reward-model training and supports self-improving alignment pipelines.
• Policy-invariant reward shaping in robotics: Dense process rewards from GRMs can be incorporated into RL agents using policy-invariant shaping terms, avoiding semantic reward traps and ensuring the preservation of optimal task policies (Tan et al., 29 Dec 2025).

6. Limitations and Open Challenges

Despite their flexibility and strong empirical performance, GRMs face open issues:

• Annotation cost and critique quality: Collecting large-scale, high-fidelity rationales and multi-dimensional human feedback remains expensive (Zhu et al., 21 Oct 2025).
• Computational cost: Generative and in-context reasoning require more inference compute than scalar models. Inference-time scaling (parallel sampling, voting) incurs additional overhead, especially in groupwise evaluation (Liu et al., 3 Apr 2025, Zhou et al., 24 May 2025).
• Reward hacking and drift: Autonomous agents or policies trained solely against poorly-calibrated or static GRMs are vulnerable to gaming and reward drift. Continuous co-training, curriculum adaptation, and periodic calibration with real, verified data are effective but not foolproof solutions (Sun et al., 16 Oct 2025, Wang et al., 2 Sep 2025).
• Domain specificity and scalability: Model merging and domain fusion improve performance but require careful balancing to avoid degrading general preference scoring (Lin et al., 1 Jul 2024).
• Theoretical unification: The connection among generative reward modeling, classical pairwise ranking, and RL reward shaping is now clearer, but extensions to richer open-ended tasks, multi-objective rewards, and more robust safety validation remain ongoing research (Wang et al., 17 Jun 2025, Wang et al., 2 Sep 2025, Zhou et al., 24 May 2025).

7. Outlook and Future Directions

Research on General Reward Models is converging on increasingly data- and compute-efficient, interpretable, and domain-adaptive reward modeling paradigms. The blueprint established by leading works includes:

• Structured, interpretable, and multi-dimensional outputs for richer downstream learning and evaluation signals,
• Data-efficient pre-training and self-training with minimal labeled preference data via large-scale use of unlabeled or synthetic data,
• Pointwise and groupwise voting strategies for scalable, robust judgment aggregation at inference time,
• Shared architectures and co-evolution with policies to prevent reward model drift and support continual agent improvement,
• Plug-and-play policy-invariant shaping to preserve optimal decision boundaries,
• Training-free reward extraction from pre-trained LLMs as a theoretically grounded solution that bypasses data bottlenecks (Li et al., 29 Jun 2025),

with ongoing developments targeting improved efficiency, safety, robustness to reward hacking, open-endedness, and seamless multi-modal and multi-domain integration.

---

References:

(Cao et al., 1 Oct 2025, Chen et al., 20 Jun 2025, Zhang et al., 11 Nov 2025, Zhu et al., 21 Oct 2025, Xia et al., 25 Feb 2025, Yang et al., 14 Jun 2024, Liu et al., 3 Apr 2025, Wang et al., 2 Sep 2025, Li et al., 2023, Forbes et al., 16 Oct 2024, Wang et al., 17 Jun 2025, Tan et al., 29 Dec 2025, Zhou et al., 24 May 2025, Sun et al., 16 Oct 2025, Lin et al., 1 Jul 2024, Li et al., 29 Jun 2025).