Mitigating Factual Hallucination in Large Reasoning Models via Mixed-Mode Advantage Regularization

Abstract: Large reasoning models (LRMs) improve LLM capabilities by generating explicit thinking traces before final answers. In factuality-oriented question answering (QA), such thinking often improves overall performance by helping the model recover relevant knowledge and refine its answers. However, we find that this benefit is not uniform at the instance level: explicit thinking can also overturn correct non-thinking answers and lead to factual drift. We refer to this failure mode as \emph{thinking-induced hallucination}. To explain this phenomenon, we formulate explicit thinking in factuality QA as a thinking residual over the model's direct-answer tendency, which can either recover missing knowledge or introduce unsupported associations. Based on this formulation, we propose MARGO, \underline{\textit{M}}ixed-Mode \underline{\textit{A}}dvantage \underline{\textit{R}}egularization for \underline{\textit{G}}rounded \underline{\textit{O}}ptimization, a reinforcement learning framework that uses non-thinking rollouts as same-model references in advantage estimation. By constructing mixed-mode rollout groups with both thinking and non-thinking trajectories, MARGO evaluates whether explicit thinking adds factual value beyond direct answering, thereby suppressing hallucination-prone thinking while preserving beneficial thinking behaviors. Experiments across multiple factuality-oriented QA benchmarks demonstrate that MARGO improves factual reliability over strong baselines, while evaluations on mathematical benchmarks show that it preserves general reasoning ability.

• The paper introduces MARGO, a novel RL framework that uses mixed-mode advantage regularization to reduce hallucinations in explicit reasoning traces.
• It leverages paired thinking and non-thinking trajectories to quantify and adjust reasoning residuals, ensuring improved factual reliability.
• Empirical results across multiple QA benchmarks demonstrate robust gains in accuracy while preserving overall general reasoning performance.

Mixed-Mode Advantage Regularization for Factual Hallucination Mitigation in LRMs

Motivation: Thinking-Induced Hallucination in Factuality QA

Large reasoning models (LRMs), when prompted to generate explicit reasoning traces (i.e., "thinking"), can both improve answer correctness and introduce factual hallucinations. In factuality-oriented question answering (QA), explicit thinking often refines responses by aggregating relevant knowledge, but the paper identifies a key failure mode whereby thinking overwrites correct non-thinking answers with incorrect ones—termed "thinking-induced hallucination". Empirical analysis across model scales using Qwen3 demonstrates that while explicit thinking corrects 11.74% of originally incorrect answers, it also induces hallucinations in 7.50% of originally correct non-thinking answers. This dual effect motivates a residual view of reasoning: explicit thinking acts as a residual atop the model's direct-answer tendency, potentially recovering missing knowledge or introducing unsupported associations.

Figure 1: Explicit thinking in LRMs acts as a residual over direct-answer tendency, and mixed-mode advantage regularization compares thinking rollouts to same-model direct responses for hallucination mitigation.

Method: Mixed-Mode Advantage Regularization for Grounded Optimization (MARGO)

To operationalize hallucination mitigation, the paper introduces MARGO—a reinforcement learning framework incorporating mixed-mode advantage regularization. Unlike conventional GRPO, which evaluates only thinking rollouts, MARGO samples both thinking and non-thinking trajectories for each factual question, using non-thinking rollouts as same-model references in advantage estimation.

For a question $x$, expected factual rewards for each mode ($\mu_{\mathrm{T}}(x)$ and $\mu_{\mathrm{N}}(x)$) are computed, with reasoning residual $$\Delta_{\mathrm{res}}(x) = \mu_{\mathrm{T}}(x) - \mu_{\mathrm{N}}(x)$$ quantifying the value of thinking. Mixed-mode advantage for a thinking rollout $y^{\mathrm{T}}$ decomposes as:

$$A^{\mathrm{T}}_\mathrm{mix}(x, y^{\mathrm{T}}) \propto [R(x, y^{\mathrm{T}}) - \mu_{\mathrm{T}}(x)] + (1-\alpha)\Delta_{\mathrm{res}}(x)$$

where the first term measures within-thinking relative quality and the second term adds residual-value adjustment. Hallucination-prone thinking trajectories are penalized when $\Delta_{\mathrm{res}}(x)<0$. Rewarding is conducted via semantic consistency evaluation by a parametric judge model (Qwen3-32B), reducing reliance on brittle string matching.

Empirical Results

Experiments span Qwen3-4B and Qwen3-8B across six factual QA benchmarks. MARGO consistently surpasses fixed-mode inference, adaptive mode-selection, SFT, and RL variants. On Qwen3-8B, MARGO achieves 29.57% average accuracy across benchmarks, a +1.98% gain over fixed thinking mode, and similarly robust improvements for Qwen3-4B. Notably, adaptive mode-selection baselines fail to consistently outperform fixed modes, underscoring that the utility of explicit thinking is trajectory- and model-specific, not reliably predictable at the question level.

Transition analysis reveals that MARGO systematically reduces the ratio $r_{1,0}$—cases where thinking overwrites correct direct answers with hallucinations—while increasing $r_{1,1}$, cases where correct answers persist under thinking. RL in a fixed mode has marginal effect on these ratios, confirming that regularization via mixed-mode advantage estimation is the primary driver for hallucination suppression.

Figure 2: MARGO reduces harmful correctness-to-incorrectness transitions ($r_{1,0}$) and increases correct-to-correct transitions ($\mu_{\mathrm{T}}(x)$0) relative to RL baselines.

Qualitative case studies further illustrate MARGO's effect: in prompts where fixed thinking drifts from direct factuality (“Christmas Island is administered by which country?”), MARGO preserves the correct answer by penalizing hallucination-inducing reasoning traces.

Figure 3: MARGO recovers correct answers lost to unsupported reasoning in fixed thinking mode.

Preservation of General Reasoning Ability

Despite its focus on factual hallucination mitigation, MARGO maintains or even slightly improves general reasoning performance on mathematical benchmarks (AMC23, AIME24, AIME25). Mean@16 accuracy increases up to +1.5% over strong baselines, demonstrating that hallucination suppression does not come at the expense of reasoning capacity.

Data Selection and Training Analysis

MARGO's efficacy relies on training over examples with pronounced factuality gaps between thinking and non-thinking modes. Ablation studies show that randomly sampled training examples yield limited improvement; targeted selection of mode-dependent factuality gaps leads to consistent gains, validating the importance of reference informativeness in mixed-mode regularization.

Practical and Theoretical Implications

MARGO’s residual-based optimization represents a principled direction for mode-aware fine-tuning of LRMs, mitigating explicit reasoning-induced hallucination while retaining beneficial reasoning integration. This trajectory-level regularization provides a scalable methodology for QA tasks where factuality is paramount and explicit reasoning carries intrinsic risk of factual contamination. The approach complements adaptive mode-selection and step-wise RL protocols, offering a grounded mechanism for robust factual reliability.

Practically, MARGO can be deployed in LRMs for information-seeking and educational applications, reducing hallucination rates without resorting to external verification. Theoretically, it demonstrates the utility of intra-model reference-based advantage estimation for controlling suboptimal reasoning drift in generative systems. Future work could extend mixed-mode advantage regularization to open-ended generation, retrieval-augmented models, and multimodal settings, exploring further the interplay between explicit reasoning and direct factual tendencies.

Conclusion

The paper characterizes, explains, and addresses thinking-induced hallucination in LRMs for factual QA. Through the introduction of MARGO, mixed-mode advantage regularization uses non-thinking rollouts as references to regularize explicit reasoning traces. Extensive experiments validate improved factual reliability, robust general reasoning retention, and mode-aware optimization. The insights indicate that optimizing LRMs requires nuanced balancing of explicit thinking and direct-answer behavior, informed by trajectory-level value estimation.