Explicit Reward Models (EX-RMs)

• Explicit Reward Models are explicit formulations that compute scalar rewards directly from inputs using transparent, parameterized functions.
• They decompose tasks into modular components, supporting applications like RLHF, hierarchical reward structuring, and multi-objective optimization.
• EX-RMs improve generalization and reduce reward hacking by leveraging semantic representations rather than relying solely on token-level probabilities.

An explicit reward model (EX-RM) is a class of reward model in which the mapping from input (often a prompt–response pair or a trajectory in reinforcement learning) to a scalar (or vector) reward is made directly and transparently available, typically via a dedicated parameterized function or an explicit modular structure. In contrast to implicit reward models—where the reward emerges from log-probabilities or the structure of the base model itself—EX-RMs feature direct, auditable computation of rewards, facilitating interpretability and controllability. Explicit reward modeling has become central across RL from human feedback (RLHF), structured reinforcement learning, and LLM alignment research due to its advantages in interpretability, generalization, and robustness.

1. Formal Definitions and Core Mechanisms

Explicit reward models utilize an explicit parameterization to compute rewards. A standard form attaches a linear output head to a LLM or agent's encoder, such that for a prompt–response pair $(x, y)$ and hidden representation $h_{x, y}$,

$r(x, y) = \langle w, h_{x, y} \rangle,$

where $w$ is a learnable weight vector. EX-RMs can also take automata-based forms in reinforcement learning, where a finite-state machine known as a reward machine defines the reward through structured transitions, such as $\mathcal{R} = (U, u_0, \delta_u, \delta_r)$ with $U$ a set of abstract states and $\delta_r: U \times 2^P \to \mathbb{R}$ assigning rewards based on high-level events (Icarte et al., 2021).

Recent EX-RMs, especially in the context of RLHF and LLM alignment, include multi-objective regression heads, modular attribute decomposition, and chain-of-thought reasoning modules, all designed to provide clearly interpretable, decomposable reward signals (Wang et al., 18 Jun 2024, Guo et al., 20 May 2025).

2. Structured, Decomposable, and Modular EX-RMs

One foundational instance of EX-RMs in reinforcement learning is the reward machine (RM) formalism (Icarte et al., 2021). Here, task rewards are decomposed into subproblems via automata: each RM state corresponds to a subtask or phase, and transitions encode high-level event-driven logic. This approach allows for hierarchical extensions (hierarchical reward machines, HRMs), wherein reward machines call sub-machines, supporting efficient representation and learning of multi-level objectives (Furelos-Blanco et al., 2022).

Model Type	Structure	Example Domains
Flat Reward Machine	Single automaton with transitions for the whole task	Single-phase RL tasks
Hierarchical Reward Machine	RM with callable sub-RMs, supporting task decomposition	Long-horizon RL, RLHF
Multi-objective EX-RM	Outputs vector of ratings per attribute	LLM alignment, safety

Explicit modularity allows the integration of interpretable dimensions. For instance, multi-objective EX-RMs output a rating vector (e.g., honesty, verbosity, safety), which is then weighted by a gating network to produce a scalar score, facilitating transparency and contextual adaptation (Wang et al., 18 Jun 2024): $R = g_\phi(f_\theta(x))^\top r',$ with $g_\phi$ a context-specific gating function and $r'$ the adjusted vector of attribute ratings.

3. Learning and Optimization Techniques

Learning explicit reward models typically involves maximizing the likelihood (or minimizing a suitable loss) of observed preference data under a parameterized reward function. For pairwise preference data, the standard loss is: $$r_\text{EX-RM} = \arg\max_\phi \; - \mathbb{E}_{(x, y_w, y_l) \sim \mathcal{D}}[ \log \sigma(r_\phi(x, y_w) - r_\phi(x, y_l)) ],$$ where $\sigma(z) = 1 / (1 + \exp(-z))$ and $y_w$/$y_l$ are preferred/rejected responses.

For RMs as automata, learning involves discrete optimization: traces are collected, mapped to abstract observations, and the RM structure is fit via mixed-integer linear programming, constraint programming, or local search to optimize prediction error criteria (Icarte et al., 2021). This approach decomposes a non-Markovian or partially observable RL problem into a Markovian product space over observations and RM states.

Explicit models based on multi-dimensional feedback can also be learned using regression losses: $$\min_{\theta, w} \mathbb{E}_{x, y, r} \| w^\top f_\theta(x \oplus y) - r \|_2^2,$$ where $r$ encodes ratings for distinct objectives (Wang et al., 18 Jun 2024).

For explicitly-causal models, targeted augmentations enforce sensitivity to causal attributes and invariance to spurious ones, resulting in composite losses that penalize both incorrect preferences and incorrect ties (Srivastava et al., 19 Jun 2025).

4. Generalization, Robustness, and Comparative Properties

Empirical findings show that EX-RMs generalize better than implicit reward models (IM-RMs) and implicit DPO-based models (DPORMs), especially under out-of-distribution (OOD) shifts (Lin et al., 5 Sep 2024, Razin et al., 10 Jul 2025). The core reason is that EX-RMs operate primarily on hidden semantic representations, fostering robustness to superficial token-level changes (such as paraphrasing or formatting). In contrast, IM-RMs and DPORMs—whose rewards are functions of token-level log probabilities—often key off surface features, resulting in significant generalization gaps under token-level distribution shifts.

Theoretical results confirm that EX-RMs trained via margin-based updates in hidden space maintain correct ranking under paraphrase, whereas IM-RMs' update coefficients depend on token identity, failing to preserve intended rewards under non-semantic perturbations (Razin et al., 10 Jul 2025). Similarly, multi-objective and modular EX-RMs reveal, via ablation and benchmarking, enhanced interpretability and reduced vulnerability to reward hacking, verbosity bias, and spurious signals (Wang et al., 18 Jun 2024, Srivastava et al., 19 Jun 2025).

5. Applications, Extensions, and Benchmarking

EX-RMs underpin a wide range of practical alignment and RL systems:

• Policy Alignment in RLHF: Serving as reward functions for best-of-N sampling, beam search, and direct policy optimization in language, vision, code, and agent domains (Xia et al., 25 Feb 2025, Wang et al., 6 May 2025).
• Multi-modal and Chain-of-Thought Rewards: UnifiedReward-Think demonstrates that long-step CoT explicit reward models can generalize across vision tasks (image/video captioning, understanding, and generative evaluation), providing both process-level and outcome-level interpretability (Wang et al., 6 May 2025).
• Principle-Following Models: RewardAnything extends EX-RMs to follow dynamically supplied natural language instructions, allowing flexible, context-dependent reward assessment without retraining (Yu et al., 4 Jun 2025).
• Causal Rubrics and Robustness: Methods like Crome enforce causal reasoning in EX-RMs, mitigating reward hacking and focusing on genuine drivers of response quality via targeted data augmentations (Srivastava et al., 19 Jun 2025).
• Energy-Based Models: EBRM refines existing (potentially miscalibrated) EX-RMs via post-hoc energy-based training, capturing reward uncertainty to improve safety-critical alignment (Lochab et al., 17 Apr 2025).
• Evaluation: Comprehensive benchmarks (e.g., RewardBench, RABench, RM-Bench), as well as metrics for reward overoptimization ($\gamma$), facilitate rigorous assessment of EX-RM alignment quality and robustness (Kim et al., 19 May 2025, Yu et al., 4 Jun 2025).

6. Limitations and Open Research Problems

EX-RMs, despite their strengths, exhibit limitations rooted in:

• Dependency on Representational Quality: EX-RMs' generalization depends on the hidden space of the underlying model; if the representations do not encode sufficient semantic or causal information, robustness may suffer (Razin et al., 10 Jul 2025).
• Annotation and Feedback Challenges: Fine-grained, multi-dimensional, or principle-driven EX-RMs require carefully annotated or scripted feedback, which is resource-intensive to collect. Frameworks that handle ordinal feedback (with “tied” or “graded” responses) have shown improved generalization, yet demand annotation protocols that ensure unbiased, informative labels (Liu et al., 19 Nov 2024).
• Reward Hacking and Spurious Correlations: Even explicit models are susceptible to reward hacking if trained on datasets with systematic artifacts (e.g., verbosity, formatting), making robust augmentation and evaluation methodologies essential (Srivastava et al., 19 Jun 2025, Kim et al., 19 May 2025).
• Scalability and Computation: Richer EX-RMs (e.g., with hierarchical automata or chain-of-thought modules) offer increased expressiveness at the cost of increased memory or computational overhead (Furelos-Blanco et al., 2022, Wang et al., 6 May 2025).

7. Forward Directions

The field is progressing towards even greater transparency, flexibility, and causal coherence in EX-RMs:

• Causal-Aware and Robustness-Centric Training: Integrating targeted augmentations and synthesizing counterfactuals to ensure reward models focus on true quality drivers (Srivastava et al., 19 Jun 2025).
• Dynamic, Editable Principles: Enabling principle-following reward models that interpret arbitrary natural language criteria, allowing on-the-fly adaptation to application- and deployment-specific needs (Yu et al., 4 Jun 2025).
• Process-Level Rewards and Reasoning: Leveraging chain-of-thought generation, modular subgoal decomposition, and explicit attribute-based evaluation to align not only the outcome but the reasoning pathway of agents and LLMs (Wang et al., 6 May 2025, Guo et al., 20 May 2025).
• Probabilistic and Energy-Based Methods: Explicitly modeling uncertainty over reward assignments to increase alignment robustness, especially in safety-sensitive applications (Lochab et al., 17 Apr 2025).
• Refined Evaluation: Adopting robust multi-faceted benchmarks and overoptimization-aware performance metrics to better assess alignment fidelity and the link between reward models and downstream policy quality (Kim et al., 19 May 2025).

Future research is likely to focus on causal alignment frameworks, scalable and principled data generation, improved human-in-the-loop protocols, and the integration of explicit reward models with adaptive, multi-modal learning agents.