Explicit Reward Models (EX-RMs)

• Explicit Reward Models are explicit formulations that compute scalar rewards directly from inputs using transparent, parameterized functions.
• They decompose tasks into modular components, supporting applications like RLHF, hierarchical reward structuring, and multi-objective optimization.
• EX-RMs improve generalization and reduce reward hacking by leveraging semantic representations rather than relying solely on token-level probabilities.

An explicit reward model (EX-RM) is a class of reward model in which the mapping from input (often a prompt–response pair or a trajectory in reinforcement learning) to a scalar (or vector) reward is made directly and transparently available, typically via a dedicated parameterized function or an explicit modular structure. In contrast to implicit reward models—where the reward emerges from log-probabilities or the structure of the base model itself—EX-RMs feature direct, auditable computation of rewards, facilitating interpretability and controllability. Explicit reward modeling has become central across RL from human feedback (RLHF), structured reinforcement learning, and LLM alignment research due to its advantages in interpretability, generalization, and robustness.

1. Formal Definitions and Core Mechanisms

Explicit reward models utilize an explicit parameterization to compute rewards. A standard form attaches a linear output head to a LLM or agent's encoder, such that for a prompt–response pair $(x, y)$ and hidden representation $h_{x, y}$,

$r(x, y) = \langle w, h_{x, y} \rangle,$

where $w$ is a learnable weight vector. EX-RMs can also take automata-based forms in reinforcement learning, where a finite-state machine known as a reward machine defines the reward through structured transitions, such as $\mathcal{R} = (U, u_0, \delta_u, \delta_r)$ with $U$ a set of abstract states and $\delta_r: U \times 2^P \to \mathbb{R}$ assigning rewards based on high-level events (2112.09477).

Recent EX-RMs, especially in the context of RLHF and LLM alignment, include multi-objective regression heads, modular attribute decomposition, and chain-of-thought reasoning modules, all designed to provide clearly interpretable, decomposable reward signals (2406.12845, 2505.14674).

2. Structured, Decomposable, and Modular EX-RMs

One foundational instance of EX-RMs in reinforcement learning is the reward machine (RM) formalism (2112.09477). Here, task rewards are decomposed into subproblems via automata: each RM state corresponds to a subtask or phase, and transitions encode high-level event-driven logic. This approach allows for hierarchical extensions (hierarchical reward machines, HRMs), wherein reward machines call sub-machines, supporting efficient representation and learning of multi-level objectives (2205.15752).

Model Type	Structure	Example Domains
Flat Reward Machine	Single automaton with transitions for the whole task	Single-phase RL tasks
Hierarchical Reward Machine	RM with callable sub-RMs, supporting task decomposition	Long-horizon RL, RLHF
Multi-objective EX-RM	Outputs vector of ratings per attribute	LLM alignment, safety

Explicit modularity allows the integration of interpretable dimensions. For instance, multi-objective EX-RMs output a rating vector (e.g., honesty, verbosity, safety), which is then weighted by a gating network to produce a scalar score, facilitating transparency and contextual adaptation (2406.12845): $R = g_\phi(f_\theta(x))^\top r',$ with $g_\phi$ a context-specific gating function and $r'$ the adjusted vector of attribute ratings.

3. Learning and Optimization Techniques

Learning explicit reward models typically involves maximizing the likelihood (or minimizing a suitable loss) of observed preference data under a parameterized reward function. For pairwise preference data, the standard loss is: $$r_\text{EX-RM} = \arg\max_\phi \; - \mathbb{E}_{(x, y_w, y_l) \sim \mathcal{D}}[ \log \sigma(r_\phi(x, y_w) - r_\phi(x, y_l)) ],$$ where $\sigma(z) = 1 / (1 + \exp(-z))$ and $y_w$/$y_l$ are preferred/rejected responses.

For RMs as automata, learning involves discrete optimization: traces are collected, mapped to abstract observations, and the RM structure is fit via mixed-integer linear programming, constraint programming, or local search to optimize prediction error criteria (2112.09477). This approach decomposes a non-Markovian or partially observable RL problem into a Markovian product space over observations and RM states.

Explicit models based on multi-dimensional feedback can also be learned using regression losses: $$\min_{\theta, w} \mathbb{E}_{x, y, r} \| w^\top f_\theta(x \oplus y) - r \|_2^2,$$ where $r$ encodes ratings for distinct objectives (2406.12845).

For explicitly-causal models, targeted augmentations enforce sensitivity to causal attributes and invariance to spurious ones, resulting in composite losses that penalize both incorrect preferences and incorrect ties (2506.16507).

4. Generalization, Robustness, and Comparative Properties

Empirical findings show that EX-RMs generalize better than implicit reward models (IM-RMs) and implicit DPO-based models (DPORMs), especially under out-of-distribution (OOD) shifts (2409.03650, 2507.07981). The core reason is that EX-RMs operate primarily on hidden semantic representations, fostering robustness to superficial token-level changes (such as paraphrasing or formatting). In contrast, IM-RMs and DPORMs—whose rewards are functions of token-level log probabilities—often key off surface features, resulting in significant generalization gaps under token-level distribution shifts.

Theoretical results confirm that EX-RMs trained via margin-based updates in hidden space maintain correct ranking under paraphrase, whereas IM-RMs' update coefficients depend on token identity, failing to preserve intended rewards under non-semantic perturbations (2507.07981). Similarly, multi-objective and modular EX-RMs reveal, via ablation and benchmarking, enhanced interpretability and reduced vulnerability to reward hacking, verbosity bias, and spurious signals (2406.12845, 2506.16507).

5. Applications, Extensions, and Benchmarking

EX-RMs underpin a wide range of practical alignment and RL systems:

• Policy Alignment in RLHF: Serving as reward functions for best-of-N sampling, beam search, and direct policy optimization in language, vision, code, and agent domains (2502.18407, 2505.03318).
• Multi-modal and Chain-of-Thought Rewards: UnifiedReward-Think demonstrates that long-step CoT explicit reward models can generalize across vision tasks (image/video captioning, understanding, and generative evaluation), providing both process-level and outcome-level interpretability (2505.03318).
• Principle-Following Models: RewardAnything extends EX-RMs to follow dynamically supplied natural language instructions, allowing flexible, context-dependent reward assessment without retraining (2506.03637).
• Causal Rubrics and Robustness: Methods like Crome enforce causal reasoning in EX-RMs, mitigating reward hacking and focusing on genuine drivers of response quality via targeted data augmentations (2506.16507).
• Energy-Based Models: EBRM refines existing (potentially miscalibrated) EX-RMs via post-hoc energy-based training, capturing reward uncertainty to improve safety-critical alignment (2504.13134).
• Evaluation: Comprehensive benchmarks (e.g., RewardBench, RABench, RM-Bench), as well as metrics for reward overoptimization ($\gamma$), facilitate rigorous assessment of EX-RM alignment quality and robustness (2505.12763, 2506.03637).

6. Limitations and Open Research Problems

EX-RMs, despite their strengths, exhibit limitations rooted in:

• Dependency on Representational Quality: EX-RMs' generalization depends on the hidden space of the underlying model; if the representations do not encode sufficient semantic or causal information, robustness may suffer (2507.07981).
• Annotation and Feedback Challenges: Fine-grained, multi-dimensional, or principle-driven EX-RMs require carefully annotated or scripted feedback, which is resource-intensive to collect. Frameworks that handle ordinal feedback (with “tied” or “graded” responses) have shown improved generalization, yet demand annotation protocols that ensure unbiased, informative labels (2411.12843).
• Reward Hacking and Spurious Correlations: Even explicit models are susceptible to reward hacking if trained on datasets with systematic artifacts (e.g., verbosity, formatting), making robust augmentation and evaluation methodologies essential (2506.16507, 2505.12763).
• Scalability and Computation: Richer EX-RMs (e.g., with hierarchical automata or chain-of-thought modules) offer increased expressiveness at the cost of increased memory or computational overhead (2205.15752, 2505.03318).

7. Forward Directions

The field is progressing towards even greater transparency, flexibility, and causal coherence in EX-RMs:

• Causal-Aware and Robustness-Centric Training: Integrating targeted augmentations and synthesizing counterfactuals to ensure reward models focus on true quality drivers (2506.16507).
• Dynamic, Editable Principles: Enabling principle-following reward models that interpret arbitrary natural language criteria, allowing on-the-fly adaptation to application- and deployment-specific needs (2506.03637).
• Process-Level Rewards and Reasoning: Leveraging chain-of-thought generation, modular subgoal decomposition, and explicit attribute-based evaluation to align not only the outcome but the reasoning pathway of agents and LLMs (2505.03318, 2505.14674).
• Probabilistic and Energy-Based Methods: Explicitly modeling uncertainty over reward assignments to increase alignment robustness, especially in safety-sensitive applications (2504.13134).
• Refined Evaluation: Adopting robust multi-faceted benchmarks and overoptimization-aware performance metrics to better assess alignment fidelity and the link between reward models and downstream policy quality (2505.12763).

Future research is likely to focus on causal alignment frameworks, scalable and principled data generation, improved human-in-the-loop protocols, and the integration of explicit reward models with adaptive, multi-modal learning agents.