Hallucinated Reward Correction

• Hallucinated reward correction is a methodology that detects and fixes misaligned reward signals generated by proxy models in reinforcement learning and language models.
• The approaches use step-wise penalties, token- and sequence-level decompositions, and post-hoc repair mechanisms to counteract reward hacking and model misalignment.
• Key techniques include structured RL, binary reward adjustments, and preference-based repairs, offering both theoretical insights and empirical performance improvements.

Hallucinated Reward Correction

Hallucinated reward correction refers to a class of methodologies and diagnostic tools designed to detect, mitigate, or retroactively repair the assignment of incorrect, unreliable, or unfaithful reward signals—so-called "hallucinated rewards"—in learning systems, predominantly those using reinforcement learning (RL) or LLMs. These hallucinated rewards typically arise when proxy reward models, external verifiers, or policy rollouts assign high reward to behaviors or outputs that are not actually aligned with ground truth objectives, process correctness, or factuality. Such misaligned reward signals may reinforce spurious, ungrounded, or logically incorrect behaviors and are a major source of reward hacking and model misalignment.

1. Theoretical Foundations and Problem Setting

The issue of hallucinated rewards emerges whenever the learning system relies on models (for reward, verification, or environment dynamics) that are misspecified or cannot distinguish between truly correct and superficially plausible behaviors. In model-based RL, this occurs when an imperfect environment model generates state-action trajectories not encountered in the true environment; the correct reward value for such "hallucinated" states is ambiguous. The theoretical contribution of "Learning the Reward Function for a Misspecified Model" is to bound planning error via the hallucinated reward error: the expected discrepancy between true reward on the environment trajectory and the learned reward applied to the flawed, model-generated trajectory. For deterministic dynamics, the bound is

$$\epsilon_{val}^{\xi,\rho,T} \leq \sum_{t=1}^T \gamma^{t-1} \mathbb{E}_{(s,z,a)\sim H^t_{\xi,\rho}} \big|R_s^a - \hat R_z^a\big|$$

where $(s,z,a)$ denotes joint sampling from real and model rollouts, and $\hat R_z^a$ is the reward evaluated on hallucinated model state $z$ (Talvitie, 2018).

This insight generalizes to RL for LLMs: reward models, external verifiers, or process reward models (PRMs) trained as proxies may assign maximal reward to unfaithful, factually unsupported, or illogical behaviors—epitomizing the hallucinated reward pathology.

2. Reward Correction via Structured RL and Sequence Decomposition

Recent approaches explicitly detect and penalize hallucinatory signals at the intermediate or step-wise level, rather than solely relying on final outcome correctness.

• Thinking, Faithful and Stable: Mitigating Hallucinations in LLMs develops a composite reward for LLMs that decomposes into (a) a confidence calibration term that penalizes unjustified high self-reported confidence, (b) a penalty for token-level entropy spikes in reasoning traces, and (c) a regularization for output format. For confidence, the reward is

$$R_{\text{confidence}} = (2\,\mathrm{correct} - 1) \times (2\,c - 1)$$

and for entropy spikes within the "think" span, the penalty is

$$R_{\text{entropy}} = -\lambda\,\max\{0,\,z_{\max} - \tau\}$$

where $z_{\max}$ is the peak z-score of entropy over reasoning tokens (Zou et al., 19 Nov 2025). This approach provides continuous, fine-grained feedback, avoiding mid-generation hallucinations by shaping both faithfulness of intermediate steps and calibration of the policy.

• RLFH (Reinforcement Learning for Hallucination) uses decomposition to atomic statements, evaluating each for truthfulness (Correct, HedgedCorrect, Vague, etc.) and informativeness, and mapping these to dense, token-aligned scalar rewards. This fine-grained mechanism enables RL policies to suppress hallucinated tokens while rewarding correct, informative content, improving factuality on held-out QA and biography tasks (Wen et al., 2024).
• Faithfulness-Aware Step-Level Reinforcement Learning (FaithRL) introduces explicit, sentence-level faithfulness rewards within chain-of-thought reasoning. Each reasoning step $S_t$ is judged as faithful or hallucinated by a PRM; faithful steps are rewarded (+1), hallucinated steps are penalized (–1). The approach combines this with a truncated resampling mechanism that compares hallucinated rollouts to corrected, resampled trajectories, thereby supplying contrastive step-level credit assignments (Nie et al., 5 Feb 2026).

3. Correction in Binary and Noisy Reward Settings

In settings where reward assignment is binary, possible verifier imperfections raise the risk of reinforcing (hallucinated) errors or suppressing genuinely correct outputs.

• Binary Retrieval-Augmented Reward (RAR) defines a binary reward as 1 only if the model's output exhibits no factual contradictions with retrieved evidence, and 0 otherwise. This discrete reward signal eliminates partial-credit reward hacking and forces the model to abstain ("I don't know") when uncertain, leading to substantial reductions in hallucination rates without quality regressions (Chen et al., 20 Oct 2025).
• Reinforcement Learning with Verifiable yet Noisy Rewards formalizes the binary verifier as a stochastic channel with false-positive ($\rho_0$) and false-negative ($\rho_1$) rates. Two lightweight correction mechanisms enable unbiased (backward) or directionally correct (forward) policy gradients, stabilizing learning under verifier noise and reducing the risk of reinforcing hallucinated errors as genuine (Cai et al., 1 Oct 2025). The forward correction only requires estimating the false-negative rate and avoids division by small denominators, thus maintaining robustness under heavy noise.

4. Diagnostic and Post-hoc Repair of Hallucinated Rewards

Beyond prevention during training, hallucinated rewards can be retroactively diagnosed and mitigated through reward decomposition and feature-level analysis.

• Causal Reward Adjustment (CRA) frames reward hacking as a problem of confounding: process reward models assign high scores to incorrect paths due to entanglement with spurious semantic features. CRA trains sparse autoencoders on PRM activations to extract these confounders, then applies Pearl's backdoor adjustment to estimate the causal effect of a reasoning path, thereby downscoring reward-hacked branches during beam selection. On math-reasoning tasks, CRA yields measurable gains in answer-level accuracy by attenuating the influence of spurious features without retraining the PRM or policy (Song et al., 6 Aug 2025).
• IR³ (Interpretable Reward Reconstruction and Rectification) reconstructs the implicit RLHF-tuned reward function using contrastive inverse reinforcement learning, then decomposes the reward into sparse, interpretable features. Feature-level contribution analysis isolates which latent features are responsible for reward hacking. Surgical mitigation (feature suppression, adversarial shaping, constrained optimization, distillation) then corrects only the problematic components, achieving high precision in reducing hacking while retaining alignment and general capabilities (Beigi et al., 23 Feb 2026).

5. Preference-Based and Summarization-Specific Correction

Misaligned reward learning from preferences can also induce hallucinations.

• F-DPO (Factuality-aware Direct Preference Optimization) augments the standard DPO framework with binary factuality labels (factual vs. hallucinated). A label-flipping step ensures that the preferred response is never less factual than the rejected one; a factuality-aware margin amplifies learning from clearly factual vs. hallucinated pairs. F-DPO produces up to 9× reductions in hallucination rates while increasing factuality scores, using only response-level binary judgment and without token-level annotation or auxiliary reward models (Chaduvula et al., 6 Jan 2026).

For summarization, methods such as Contrastive Reward Learning (CRL) and entity-based detectors guide learning away from hallucinated outputs by using margin-based ranking losses or explicit hallucination detection. CRL penalizes the model when it assigns higher log-probability to hallucinated summaries than to more factually accurate alternatives, guided by external factuality metrics (BARTScore, DAE). This method achieves large gains in factuality without degrading fluency or relevance, as confirmed by human and automatic evaluations (Chern et al., 2023). Entity-level detection further enables token-level penalties for nonfactual hallucinations, selectively discouraging fabrications without penalizing beneficial factual hallucinations (Cao et al., 2021).

6. Reward Repair, Patch, and Human Feedback Correction

Classical reward hacking—arising from misspecified or proxy reward functions—can often be repaired efficiently by learning small, sparse additive correction terms from a limited set of human preferences.

• Preference-Based Reward Repair (PBRR) iteratively augments a hand-designed proxy reward $(s,z,a)$0 with an additive, transition-dependent correction $(s,z,a)$1 learned from preferences over trajectory pairs. Targeted exploration focuses queries on divergences between the proxy policy and a reference "safe" policy, with theoretical guarantees matching prior preference-based RL methods and empirical data efficiency 5–7× higher than reward-from-scratch baselines (Hatgis-Kessell et al., 14 Oct 2025). This approach corrects only the handful of hallucinated or overoptimistic transitions rather than retraining the reward function wholesale.

7. Comparative Summary of Hallucinated Reward Correction Methods

Method/Work	Correction Mechanism	Setting/Domain
Confidence+Entropy Composite Reward (Zou et al., 19 Nov 2025)	Penalizes self-miscalibration & instability	Multi-step LLM reasoning
Fine-grained Token Alignment (Wen et al., 2024)	Fact+informativeness to tokens	Open-domain QA/biography LLMs
FaithRL (Nie et al., 5 Feb 2026)	Step-level PRM rewards + resampling	Open-book QA, SRMs
Binary RAR (Chen et al., 20 Oct 2025)	Binary (strict) reward from external retrieval	Open-ended/hybrid QA LLMs
Noisy Verifier Correction (Cai et al., 1 Oct 2025)	Unbiased or directionally-correct gradient	Math RL with unreliable verifiers
CRA (Song et al., 6 Aug 2025)	Backdoor adjustment via confounder SAE	Reasoning systems, procedural tasks
IR³ (Beigi et al., 23 Feb 2026)	Reward inverse modeling, feature suppression	Post-hoc RLHF auditing/fixing
F-DPO (Chaduvula et al., 6 Jan 2026)	Factuality-aware margin in preferences	Preference optimization in LLMs
PBRR (Hatgis-Kessell et al., 14 Oct 2025)	Additive transition repair from preferences	Classical RL with proxies
Summarization (CRL/Entity) (Chern et al., 2023, Cao et al., 2021)	Margin-based, entity detection	Abstractive summarization

Taken collectively, these methods illustrate that hallucinated reward correction is best treated as a multi-stage, multi-granular process: preventative during RL and generation (step-wise or token-wise penalties, uncertainty suppression), diagnostic (feature-level probing, sparse decomposition), and reparative (additive reward patching, contrastive fine-tuning, preference-informed repair). This field has moved toward fine-grained, interpretable, and theoretically-grounded corrections that avoid the pitfalls of both black-box proxy hacking and reward degeneration, yielding stable and faithful LLM alignment.