Papers
Topics
Authors
Recent
Search
2000 character limit reached

To Answer or to Abstain: Mitigating Search-Agent Hallucinations via Abstention-Aware Reinforcement Learning

Published 12 Jul 2026 in cs.LG, cs.AI, and cs.CL | (2607.10738v1)

Abstract: Recent advances in equipping LLMs with search tools and outcome-reward reinforcement learning (RL) have achieved new state-of-the-art results on open-domain QA tasks. However, we argue that current training paradigms harbor a critical vulnerability: they predominantly reward correct answers but fail to penalize fabricated ones when retrieval fails, thereby implicitly exacerbating hallucinations. To address this, we propose Abstention-Aware Reinforcement Learning (AWA-RL), which dynamically shapes the abstention reward utilizing the model's query-specific prior capabilities and continuous on-policy training observations. We also introduce a novel metric, RA-F1, to measure the capability-reliability trade-off. Compared to non-abstaining baselines, AWA-RL boosts absolute precision by up to 10.3% and overall RA-F1 by 2.9%, with only marginal sacrifice in raw accuracy. These results confirm that AWA-RL successfully yields highly capable and reliable search agents. The code, data, and model weights are publicly available at https://github.com/zfj1998/AWA-RL.

Citations (2)

Summary

  • The paper introduces AWA-RL, a dynamic reward shaping mechanism that calibrates abstention decisions based on query-specific capability estimates.
  • It demonstrates up to 10.3% precision improvement and enhanced RA-F1 stability on multi-hop QA benchmarks using Group Relative Policy Optimization.
  • The approach decouples capability acquisition from abstention tuning, providing a scalable method for reliable, agentic open-domain question answering.

Abstention-Aware Reinforcement Learning for Search Agents: Mitigating Hallucinations with Dynamic Capability Calibration

Motivation and Problem Formulation

Current reinforcement learning (RL) paradigms for agentic LLMs in open-domain question answering (QA) optimize for outcome correctness but lack explicit incentives for abstention. This leads to systematic over-answering, where agents produce unsupported answers instead of safely abstaining when retrieval fails or the query lies beyond their capability. As retrieval-augmented generation evolves into agentic search, the dynamic nature of tool-use and iterative evidence gathering exacerbates the challenge: capability boundaries are not static but vary based on the trajectory, and prior abstention methods were largely designed for single-turn settings, only weakly generalizing to multi-turn search agents. Conventional approachesโ€”post-hoc pipeline training, static refusal rewards, or data manipulationsโ€”trigger either severe degradation of search/reasoning ability or catastrophic reward hacking, failing to establish reliable abstention behavior and decreasing factual grounding.

Methodological Contributions

AWA-RL (Abstention-aWAre Reinforcement Learning) introduces a dynamic reward shaping mechanism anchored to query-specific prior capability estimates, addressing the capability-reliability trade-off at its root. For each training query, the agentโ€™s baseline correctness probability is estimated from an abstention-free checkpoint, clipped to avoid premature refusal on inherently hard queries. Abstention reward is dynamically calibrated via a โ€œcourageโ€ factor (ฮณ\gamma), mapping the agentโ€™s confidence to reward, ensuring that the expected reward for answering exceeds refusal when capability is high. To counter lazy refusal and stabilize policy learning, batch-level empirical refusal rates are continuously monitored and penalized in real time, correcting deviations from theoretical boundaries with a piecewise penalty function. This formulation provides high responsivenessโ€”abstention behavior is sensitive yet stable with respect to the courage hyperparameter, and system utility (as measured by Reliability-Aware F1, or RA-F1) is maximized without decoupling capability acquisition from abstention calibration.

Experimental Framework

Evaluations are conducted on multi-hop QA benchmarks (HotpotQA, 2WikiMultiHopQA, MuSiQue), using Qwen2.5-7B-Instruct as the base agent, with an initial SFT phase bootstrapping search proficiency from teacher-generated trajectories. RL is implemented using Group Relative Policy Optimization (GRPO), supported by dense citation rewards adjudicated by external LLM judges, yielding composite advantage signals that decouple fine-grained search optimization from abstention calibration. Metrics target: Accuracy (Nc/Ntotal, proxy for recall), Precision (Nc/(Nc+Nw)), and RA-F1 (harmonic mean), measuring both agentic helpfulness and reliability in choosing to answer or refuse.

Empirical Results and Numerical Analysis

AWA-RL demonstrates statistically significant improvements in precision (up to +10.3%) and overall RA-F1 (+2.9%), with only marginal decreases in raw accuracy compared to non-abstaining paradigms. Conventional SFT or static-reward RL fail to internalize reliable abstention boundaries, yielding either unpredictable โ€œlazy refusalโ€ collapse or entrenched overconfidence. Even minimal static rewards cause reward hacking (e.g., >99% refusal), breaking the zero-sum trade-off. In contrast, AWA-RL converges rapidly and smoothly to stable refusal rates, with the optimal ฮณ\gamma yielding plateaued training curves in <30 steps. The dynamic reward mechanism consistently outperforms injected-data and static-penalty baselines across all benchmarks. Empirical tuning of ฮณ\gamma is straightforward, with abstention/answering calibration highly responsive and predictable.

Practical Implications, Limitations, and Theoretical Considerations

AWA-RL provides a scalable pathway for deploying reliable, self-aware agentic LLMs, aligning fine-grained capability estimates with abstention boundaries in a unified RL framework. The method is model-agnostic and theoretically extensible to other agentic RL domains (tool-use, mathematical reasoning, code synthesis), but current empirical validation is limited to Qwen2.5-7B and search-based QA. Computational overhead from prior estimation is modest but present; optimizing capability proxies or developing adaptive, end-to-end calibration mechanisms remains a pertinent direction.

The external judge and retrieval environment introduce downstream biases, potentially reinforcing disparities in information access for underrepresented domains or non-mainstream queries. The methodโ€™s explicit linking of abstention to prior capability implies that marginalized topicsโ€”where model performance is weakerโ€”may incur systematically higher refusal rates, mitigating hallucination but potentially exacerbating inequities.

Future Directions

AWA-RL raises key questions for agentic RL design: How can adaptive abstention calibration be implemented end-to-end? How can capability proxies be refined to minimize computational cost? Scaling to larger foundation models and non-QA agentic applications is required to confirm universal generality. Rigorous bias auditing and equitable capability expansion are critical for ethical deployment. Mechanisms for dynamic calibration beyond the strictly empirical courage parameterโ€”potentially integrating uncertainty modeling or meta-learningโ€”could further automate and refine abstention boundaries.

Conclusion

AWA-RL establishes a robust, lightweight, and effective paradigm for mitigating hallucinations in search-based agentic LLMs. By anchoring abstention rewards to query-specific prior capabilities and dynamically penalizing empirical deviation, this framework optimizes the capability-reliability trade-off and enables precise, scalable reliability calibration throughout RL training. The approach decouples capability acquisition from abstention tuning under a unified objective, facilitating trustworthy deployment in open-domain QA and broader agentic settings (2607.10738).

Paper to Video (Beta)

No one has generated a video about this paper yet.

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Tweets

Sign up for free to view the 1 tweet with 7 likes about this paper.