RewardUQ: Uncertainty-Aware Reward Modeling

Updated 4 July 2026

RewardUQ is a unified framework for uncertainty-aware reward models that provides both scalar rewards and reliability estimates.
It enhances traditional reward modeling by incorporating uncertainty metrics to improve calibration and guide active learning.
The approach applies formal uncertainty quantification in step-wise verification to mitigate reward hacking and overoptimistic predictions.

RewardUQ denotes a line of work on uncertainty-aware reward modeling in which a reward model or verifier is required to produce not only a scalar preference or correctness signal, but also an estimate of how reliable that signal is. In contemporary alignment and verification settings, this idea appears in at least two closely related forms. First, RewardUQ is the name of a unified framework for uncertainty-aware reward models that evaluates reward models jointly on accuracy and calibration (Yang et al., 27 Feb 2026). Second, the term is used as a setting for uncertainty-aware step-wise verification with generative reward models, where uncertainty estimates are attached to intermediate reasoning-step judgments rather than only to final outcomes (Ye et al., 16 Feb 2025). Across these formulations, the common premise is that reward models, process reward models, and preference models are imperfect learned proxies for human judgment, and that overconfident errors can drive reward hacking, overoptimization, or unreliable verification (Yang et al., 27 Feb 2026).

1. Conceptual scope and motivation

RewardUQ arises from the observation that standard reward models in RLHF and related post-training pipelines typically output a point estimate even though they are trained on finite, noisy, and often limited preference data. The 2026 RewardUQ framework states this explicitly: most approaches rely on pointwise reward estimates that overlook the epistemic uncertainty in reward models arising from limited human feedback (Yang et al., 27 Feb 2026). The same concern appears in uncertainty-aware step-wise verification, where process reward models are described as noisy learned proxies for human judgment that are vulnerable to reward hacking and costly step-level annotation bottlenecks (Ye et al., 16 Feb 2025).

A central implication is that reward quality cannot be reduced to ranking accuracy alone. The RewardUQ framework therefore evaluates models along accuracy and calibration, while the step-wise verification formulation treats uncertainty as a control signal for selective acceptance or rejection of intermediate judgments (Yang et al., 27 Feb 2026, Ye et al., 16 Feb 2025). This suggests that RewardUQ is not merely a diagnostic perspective. In several papers, uncertainty is used operationally: for active learning, for rejecting low-confidence verifier outputs, or for reducing the influence of unreliable rewards during policy optimization (Yang et al., 27 Feb 2026, Pan et al., 18 Jun 2026).

Related work reinforces this motivation from several directions. “Uncertainty-aware Reward Model: Teaching Reward Models to Know What is Unknown” argues that deterministic reward models cannot express when they are unreliable, and distinguishes aleatoric uncertainty from epistemic uncertainty in reward prediction (Lou et al., 2024). “Reward Auditor” shifts the evaluation question from pointwise preference accuracy to suitability, defined as conditional reliability under real-world perturbations, and audits whether perturbations cause statistically significant degradation in reward-model confidence distributions (Zang et al., 30 Nov 2025). In a different domain, “Learning Human Objectives by Evaluating Hypothetical Behavior” uses ensemble disagreement as an approximation to reward-model uncertainty in order to synthesize informative queries and correct reward hacking before deployment (Reddy et al., 2019).

2. Formal foundations of uncertainty-aware reward modeling

The formal core of RewardUQ is the Bradley–Terry view of pairwise preference prediction. In the unified RewardUQ framework, given a prompt $x$ and two candidate completions $y$ and $y'$ , the preference probability is modeled as

$p(y\succ y' \mid x,y,y') = \sigma(r(x,y)-r(x,y')).$

A standard reward model is trained on preference triplets $(x,\chosen{y},\rejected{y})$ using binary cross-entropy: $\Lcal_{\text{base}(\theta;\Dcal_{\text{train})=\frac{1}{n}\sum_{(x,\chosen{y},\rejected{y})\in\Dcal_{\text{train} -\log \sigma\!\big(\rmodel(x,\chosen{y})-\rmodel(x,\rejected{y})\big).$ RewardUQ extends this by requiring the model to output both a reward and an uncertainty estimate: $\ub{\rmodel}(x,y)=\rmodel(x,y)+\beta\,\umodel(x,y), \qquad \lb{\rmodel}(x,y)=\rmodel(x,y)-\beta\,\umodel(x,y),$ which induces lower and upper preference-probability bounds under Bradley–Terry (Yang et al., 27 Feb 2026).

This formalization makes uncertainty interval overlap operational. RewardUQ defines predictions as confident or unconfident according to whether the confidence intervals of the preferred and rejected completions overlap, and then partitions evaluation outcomes into four rates: CT rate, UT rate, CF rate, and UF rate (Yang et al., 27 Feb 2026). The framework also introduces a ranking score

$\mathrm{RS}_\alpha = \frac{\mathrm{CT\ rate}{\mathrm{win\ rate}+\alpha(1-\mathrm{win\ rate})} - \frac{\mathrm{CF\ rate}{(1-\mathrm{win\ rate})+\alpha\,\mathrm{win\ rate},$

with $\alpha\in[0,1]$ , to favor models that are confidently correct and penalize models that are confidently wrong (Yang et al., 27 Feb 2026).

Calibration is treated as a first-class criterion. RewardUQ uses ECE for point probabilities and ELCE/EUCE, summarized as EBCE after symmetrization, for uncertainty bounds (Yang et al., 27 Feb 2026). This emphasis on bound calibration differentiates RewardUQ from reward modeling work that reports only accuracy or win rate. A closely related direction, UARM, replaces scalar rewards by quantile predictions and uses quantile-based conformal prediction to obtain calibrated intervals with marginal coverage

$\mathbb{P}[R \in \mathcal{I}(X)] \ge 1-\alpha,$

then interprets interval width as instance-wise reward uncertainty (Pan et al., 18 Jun 2026). This suggests a broader formal pattern within RewardUQ: the reward signal is elevated from a scalar score to a structured object containing both a central estimate and a reliability envelope.

3. RewardUQ for step-wise verification and process supervision

In mathematical reasoning, RewardUQ appears as uncertainty-aware step-wise verification with a generative process reward model. The setup considers a generator LLM $y$ 0 that produces a reasoning trace $y$ 1 for a question $y$ 2, while a verifier assigns step-wise correctness labels $y$ 3, where $y$ 4 means the $y$ 5-th step is correct and $y$ 6 means it contains an error (Ye et al., 16 Feb 2025). For a generative PRM, the step reward is

$y$ 7

and the overall trace score is

$y$ 8

The paper identifies this as the RewardUQ setting: uncertainty estimates are used to make the verifier more reliable when judging intermediate reasoning steps (Ye et al., 16 Feb 2025).

The main proposed uncertainty estimator is CoT Entropy, which prompts the judge-LM to generate a critique or rationale $y$ 9 before the final binary decision $y'$ 0. Predictive entropy for the step label is

$y'$ 1

and CoT Entropy approximates this by marginalizing over sampled rationales: $y'$ 2 In practice, the computation samples several CoT rationales and decisions at high temperature, clusters outputs by the binary decision $y'$ 3, normalizes the token probabilities for each sampled output using the label-token distribution, sums the probabilities for all samples leading to the same decision, and computes entropy over the final two decision masses (Ye et al., 16 Feb 2025). The paper emphasizes that this is not just majority voting.

The empirical study uses PRM800K, selecting 150 questions from the test split and yielding 1,152 labeled steps, of which only 11.2% are labeled as errors. The judge model is Qwen2-Math-72B-Instruct. Baselines include Random, Naive Entropy, P(True), SEU with MiniLM and NV-Embed embeddings, and CoT Entropy (Discrete). Evaluation uses AUROC, AUPRC, AU-F1C, and Rejection-F1 (Ye et al., 16 Feb 2025).

CoT Entropy is reported as the strongest method overall, with

AUROC: $y'$ 4
AUPRC: $y'$ 5
AU-F1C: $y'$ 6

It also achieves the best Rejection-F1 across thresholds, and the paper notes that naive entropy performs worse than random, while the discrete CoT Entropy variant underperforms the probability-aware version (Ye et al., 16 Feb 2025). The authors further decompose uncertainty into predictive, epistemic, and aleatoric parts using mutual information, finding that predictive uncertainty is best overall for identifying verifier mistakes, while epistemic uncertainty is nearly as good (Ye et al., 16 Feb 2025). This suggests that many verifier errors arise from the judge model’s own lack of knowledge rather than label noise.

4. The unified RewardUQ evaluation framework

The paper “RewardUQ: A Unified Framework for Uncertainty-Aware Reward Models” systematizes comparison across several uncertainty-aware reward-model families. It evaluates ENS-MLP, ENS-LoRA, MCD-DPO, and BAY-LIN under a common protocol and reports that no single uncertainty method dominates in all settings (Yang et al., 27 Feb 2026). The most consistent empirical finding is that model size and initialization have the most meaningful impact on performance, with task-aligned base models—especially Skywork-Reward-V2 Qwen3 variants—substantially benefiting fixed-backbone methods such as BAY-LIN and ENS-MLP (Yang et al., 27 Feb 2026).

The framework selects models by first requiring calibration thresholds, specifically $y'$ 7 and $y'$ 8, and then ranking by $y'$ 9 (Yang et al., 27 Feb 2026). Hyperparameter selection is performed on UltraFeedback, while final evaluation is done on RewardBench, with additional robustness experiments on the Skywork preference dataset and the Tulu 3 8B preference mixture (Yang et al., 27 Feb 2026). Training uses a single-node setup with four NVIDIA GH200 GPUs, one epoch, and method-specific parameter grids (Yang et al., 27 Feb 2026).

The paper’s interpretation is notable. Larger models often improve win rate, but their ranking score can flatten or even worsen because they become more overconfident, and confident false predictions are penalized by the new metric (Yang et al., 27 Feb 2026). Calibration results are described as more uniform than accuracy results, with representative settings generally showing ECE below $p(y\succ y' \mid x,y,y') = \sigma(r(x,y)-r(x,y')).$ 0 and EBCE below $p(y\succ y' \mid x,y,y') = \sigma(r(x,y)-r(x,y')).$ 1 (Yang et al., 27 Feb 2026). The authors explicitly argue that uncertainty estimates should be judged by downstream-relevant trade-offs: for active learning, models should identify samples that are confidently informative, while for safe alignment the most dangerous cases are confidently wrong predictions (Yang et al., 27 Feb 2026).

This framework is intentionally intrinsic rather than end-to-end. It does not run full PPO or DPO loops, but isolates the quality of uncertainty estimation from confounders in downstream optimization (Yang et al., 27 Feb 2026). A plausible implication is that RewardUQ is meant to function as a measurement substrate for uncertainty-aware reward modeling, rather than as a single uncertainty algorithm.

Several neighboring papers extend the RewardUQ agenda by turning uncertainty from an evaluation quantity into a control variable in optimization. UARM does this most explicitly. It trains a quantile reward model with pinball loss, calibrates intervals by conformal prediction, and defines interval width as the uncertainty measure. That uncertainty is then converted into a sample-specific reliability weight in Group Relative Policy Optimization through a heteroscedastic advantage

$p(y\succ y' \mid x,y,y') = \sigma(r(x,y)-r(x,y')).$ 2

The paper states that UARM significantly improves reward model calibration, reduces reward hacking, and enhances downstream alignment quality compared to standard GRPO and uncertainty-agnostic baselines (Pan et al., 18 Jun 2026).

URM and URME pursue a different decomposition. URM uses a probabilistic value head to model aleatoric uncertainty through attribute-wise normal distributions, while URME estimates epistemic uncertainty through ensemble disagreement (Lou et al., 2024). The paper reports that URM achieves 92.9 overall on RewardBench with a Llama3.1-8B backbone, surpassing several larger models, and that uncertainty-based filtering improves reward evaluation accuracy because lower uncertainty corresponds to higher evaluation accuracy (Lou et al., 2024).

Reward Auditor broadens the notion of reliability beyond clean benchmark accuracy. It defines suitability as conditional reliability under perturbation and audits 26 reward models under ten perturbation scenarios, including EF, PH, IU, IW, CN, ST, LE, SP, LC, and SLC (Zang et al., 30 Nov 2025). Its main finding is that perturbations often cause systematic degradation in the distribution of preference perception confidence, with 80.7% of RMs showing idiosyncratic vulnerability patterns in the chat subset and ST and LC emerging as especially severe vulnerabilities (Zang et al., 30 Nov 2025). This reframes RewardUQ from confidence estimation on isolated samples to inferential auditing of robustness under distribution shift.

Other adjacent works are related in a looser sense. “Reward Hacking Mitigation using Verifiable Composite Rewards” introduces a composite reward for medical multiple-choice QA that penalizes premature answer revelation and structural non-compliance, reporting lower hacking rates while preserving or improving accuracy, especially for Qwen2.5-3B SFT (CoT) + RM (Tarek et al., 19 Sep 2025). “UR $p(y\succ y' \mid x,y,y') = \sigma(r(x,y)-r(x,y')).$ 3: Unify RAG and Reasoning through Reinforcement Learning” uses a difficulty-aware curriculum and staged verifiable rewards to teach a model when retrieval should be invoked, but the paper explicitly does not model reward uncertainty or calibrated reward estimates (Li et al., 8 Aug 2025). “OmniQuality-R” introduces STD filtering and entropy gating to suppress unstable updates in multimodal reward-model post-training; this is variance-aware and RewardUQ-adjacent, but not a formal uncertainty-quantification framework (Lu et al., 12 Oct 2025).

6. Limitations, controversies, and research directions

The RewardUQ literature is explicit about unresolved issues. The step-wise verification paper notes that prompt design for the judge-LM could be improved, that the study is limited to one main dataset/model setting, and that reward hacking is not yet directly mitigated, although uncertainty estimates may help detect and prevent it during RL training or inference-time search (Ye et al., 16 Feb 2025). The unified RewardUQ framework likewise acknowledges that its evaluation is intrinsic rather than end-to-end and that its ranking score reflects a particular accuracy–confidence trade-off that may not capture every application’s utility (Yang et al., 27 Feb 2026).

A recurring controversy concerns what uncertainty should mean in reward modeling. Some methods emphasize epistemic uncertainty from model ignorance, as in ensemble disagreement, Bayesian linear heads, or conformal interval width (Yang et al., 27 Feb 2026, Reddy et al., 2019, Pan et al., 18 Jun 2026). Others explicitly model aleatoric uncertainty, treating human preference as intrinsically stochastic or attribute-distributed (Lou et al., 2024). In step-wise verification, predictive entropy and epistemic uncertainty are both informative, but predictive uncertainty performs best overall for mistake identification (Ye et al., 16 Feb 2025). This suggests that there is no single universally accepted decomposition or estimator.

Another unresolved issue is whether RewardUQ should be evaluated primarily by intrinsic metrics such as ECE, EBCE, or RS $p(y\succ y' \mid x,y,y') = \sigma(r(x,y)-r(x,y')).$ 4, or by downstream effects on policy optimization, active learning efficiency, and reward-hacking resistance. The literature contains both positions. RewardUQ as a framework is explicitly intrinsic (Yang et al., 27 Feb 2026), whereas UARM and the step-wise verification setting use uncertainty to control optimization or selective acceptance (Pan et al., 18 Jun 2026, Ye et al., 16 Feb 2025). Reward Auditor argues that even benchmark accuracy can obscure real vulnerabilities, and therefore shifts attention to perturbation-sensitive inferential auditing (Zang et al., 30 Nov 2025).

The broader direction is clear. RewardUQ is moving from an auxiliary evaluation layer toward a design principle for alignment systems in which reward signals are expected to be calibrated, selective, and uncertainty-aware. This suggests that future work will continue to connect reward uncertainty to active data acquisition, verifier abstention, safer RLHF, and robustness auditing under real-world perturbations (Yang et al., 27 Feb 2026, Zang et al., 30 Nov 2025, Pan et al., 18 Jun 2026).