Train for Truth, Keep the Skills: Binary Retrieval-Augmented Reward Mitigates Hallucinations (2510.17733v1)

Abstract: LLMs often generate factually incorrect information unsupported by their training data, a phenomenon known as extrinsic hallucination. Existing mitigation approaches often degrade performance on open-ended generation and downstream tasks, limiting their practical utility. We propose an online reinforcement learning method using a novel binary retrieval-augmented reward (RAR) to address this tradeoff. Unlike continuous reward schemes, our approach assigns a reward of one only when the model's output is entirely factually correct, and zero otherwise. We evaluate our method on Qwen3 reasoning models across diverse tasks. For open-ended generation, binary RAR achieves a 39.3% reduction in hallucination rates, substantially outperforming both supervised training and continuous-reward RL baselines. In short-form question answering, the model learns calibrated abstention, strategically outputting "I don't know" when faced with insufficient parametric knowledge. This yields 44.4% and 21.7% fewer incorrect answers on PopQA and GPQA, respectively. Crucially, these factuality gains come without performance degradation on instruction following, math, or code, whereas continuous-reward RL, despite improving factuality, induces quality regressions.

• The paper introduces Binary RAR as an RL post-training method that assigns a binary, retrieval-verified reward to mitigate hallucinations in large language models.
• It leverages Group Relative Policy Optimization and BM25-based document retrieval to balance factual accuracy and model utility, achieving significant reductions in hallucination rates.
• Experimental results demonstrate substantial improvements in both long-form and short-form tasks, maintaining utility scores while enhancing uncertainty calibration.

Binary Retrieval-Augmented Reward for Hallucination Mitigation in LLMs

Introduction

The paper introduces Binary Retrieval-Augmented Reward (Binary RAR), an online RL post-training method for LLMs that directly targets extrinsic hallucination—generation of factually incorrect information not supported by training data. Unlike prior approaches that use continuous factuality scores or offline preference optimization, Binary RAR assigns a binary reward: a response receives a reward of one only if it is entirely factually correct according to retrieved evidence, and zero otherwise. This discrete reward structure is designed to resist reward hacking and maintain general model capabilities, addressing the persistent trade-off between hallucination reduction and utility preservation.

Figure 1: Overview of Binary RAR. Left: RL with Binary RAR assigns a binary reward based on retrieval-verified factual correctness. Right: Binary RAR achieves the best hallucination–utility tradeoff among all post-training baselines.

Methodology

RL Objective and Algorithm

Binary RAR is implemented via online RL, specifically using Group Relative Policy Optimization (GRPO), which is computationally efficient and stable for LM post-training. The RL objective is:

$$\max_{\pi_\theta} \mathbb{E}_{x \sim \mathcal{D}, y \sim \pi_\theta(\cdot|x)} \left[ r(x, y) - \beta \mathbb{D}_{\mathrm{KL}}(\pi_\theta(\cdot|x) \| \pi_{\mathrm{ref}}(\cdot|x)) \right]$$

where $r(x, y)$ is the binary reward, and $\beta$ controls KL regularization to prevent excessive drift from the base model.

Binary Retrieval-Augmented Reward Pipeline

• Retrieval: For each prompt-response pair $(x, y)$, top-$k$ relevant documents are retrieved from a pre-cached set using BM25, ensuring efficient evidence collection.
• Verification: A verifier LM (Qwen3-32B) checks for contradictions between the response and retrieved documents. If any contradiction is found, $r(x, y) = 0$; otherwise, $r(x, y) = 1$.
• Efficiency: The pipeline avoids claim decomposition, enabling single-pass verification and 2–4× throughput improvement over continuous reward methods like VeriScore.

Dataset Curation

Prompts are selected from WildChat, filtered for verifiable factual content using GPT-4.1. For each prompt, up to 10 relevant web pages are pre-cached, and instances with insufficient evidence are discarded. This ensures high-quality, diverse, and verifiable training data.

Experimental Results

Hallucination Reduction

Binary RAR achieves substantial reductions in hallucination rates across both long-form and short-form tasks:

• Qwen3-8B: Long-form hallucination rate drops from 61.9% to 37.5%; short-form from 60.6% to 27.6%.
• Qwen3-4B: Long-form from 66.2% to 37.7%; short-form from 68.7% to 41.9%.

These improvements surpass SFT, DPO, and RL with continuous rewards (e.g., VeriScore), which either provide modest gains or induce utility degradation.

Figure 2: Hallucination–utility tradeoff for ablations on KL coefficients (left) and reward designs (right).

Utility Preservation

Binary RAR maintains general capabilities across instruction following, reasoning, and coding benchmarks. For Qwen3-8B, the average utility score remains stable (62.2 vs. 61.6 for the base model), while RL with VeriScore shows clear degradation (59.6). Notably, AlpacaEval and ArenaHard scores are preserved, indicating robustness against reward hacking and overfitting to stylistic proxies.

Informativeness and Abstention

Binary RAR selectively filters uncertain content without sacrificing correct claims. On Biography, the number of correct claims remains nearly constant, while total claims decrease, indicating improved precision rather than loss of detail.

Figure 3: Informativeness in long-form generation. Left: Binary RAR cuts total claims but keeps correct claims nearly the same. Right: Binary RAR gives shorter answers with similar win rates.

In short-form QA, models learn calibrated abstention, replacing incorrect answers with "I don't know" when uncertain, while accuracy for attempted answers increases. This behavior reflects improved uncertainty calibration rather than knowledge loss.

Figure 4: Qwen3-8B before and after RL fine-tuning with Binary RAR. The fine-tuned model corrects factual errors and keeps relevant details.

Analysis

Reward Design and KL Regularization

Ablation studies show that low KL coefficients lead to degenerate solutions (e.g., overly short responses), while moderate KL preserves informativeness. Alternative reward designs (binary VeriScore, conflict-only VeriScore, rating-based RAR) are more sensitive to output style and verifier noise, leading to degraded utility or reward hacking. The effectiveness of Binary RAR arises from whole-response evaluation and strict binary correctness.

Qualitative Comparison

Binary RAR penalizes factual errors most reliably, while continuous rewards (VeriScore, LM Judge) are vulnerable to stylistic exploitation and may incentivize irrelevant or superficial content.

Figure 5: Comparing Binary RAR, VeriScore, and LM Judge on LM responses. Binary RAR penalizes factual errors most reliably.

Figure 6: Model outputs trained with Binary RAR provide detailed, structured guidance; VeriScore-trained models produce higher-level text.

Implications and Future Directions

Binary RAR demonstrates that simple, discrete reward signals can robustly mitigate hallucinations in LLMs without compromising general capabilities. The method is scalable, efficient, and resistant to reward hacking, making it suitable for continual post-training of large models. The binary reward structure encourages strategic abstention and improved uncertainty calibration, which are critical for deployment in high-stakes domains.

Future work may explore:

• Extending binary RAR to multi-modal or tool-augmented LMs.
• Integrating more sophisticated retrieval and verification mechanisms.
• Investigating the impact of binary RAR on model interpretability and calibration.
• Scaling to larger models and more diverse domains, including scientific and medical applications.

Conclusion

Binary Retrieval-Augmented Reward offers a principled and practical approach to hallucination mitigation in LLMs. By leveraging retrieval-verified binary rewards in online RL, the method achieves strong factuality improvements while preserving model utility and informativeness. The results suggest that discrete, contradiction-focused reward signals are preferable to continuous proxies for post-training factuality enhancement, providing a robust foundation for safer and more reliable LLM deployment.