To Reason or to Fabricate: Reasoning Without Shortcuts via Hint-Anchored Pairwise Aggregation

Abstract: While reinforcement learning (RL) significantly enhances LLM reasoning, its efficacy is severely undermined by Pre-RL data overlap, where RL datasets overlap with pretraining or SFT corpora, causing models to exploit shortcuts by memorizing correct answers and fabricating post-hoc reasoning. To address this, we introduce HIPPO, a novel RL framework that integrates hint-injected aggregation with a tailored pairwise reward model. By utilizing hint injection to deliberately trigger overlap-induced behaviors, the resulting traces naturally serve as explicit anchors for pairwise comparison. This provides highly discriminable preference signals, enabling a lightweight judge model to reliably distinguish genuine reasoning deduction from shortcut-driven rationalization, while the pairwise formulation ensures stable and robust optimization compared to standard PRMs. Extensive experiments demonstrate that HIPPO yields substantial improvements over standard baselines and generalizes effectively to out-of-distribution general tasks, showing it extracts authentic, transferable reasoning skills rather than superficial shortcut patterns.

• The paper’s main contribution is the HIPPO framework, which mitigates shortcut exploitation caused by pre-RL data overlap.
• It employs hint-injection and pairwise comparison to generate stable, discriminative RL reward signals that improve rationale quality.
• Empirical evaluations on math and medical tasks show HIPPO outperforms baselines, enhancing both in-domain and out-of-distribution generalizability.

Hint-Anchored Pairwise Aggregation for Robust LLM Reasoning under Pre-RL Data Overlap

Motivation: The Pre-RL Data Overlap Problem

Modern LLMs demonstrate advanced reasoning abilities when fine-tuned via reinforcement learning (RL) with verifiable rewards. However, a critical and underexplored vulnerability is Pre-RL data overlap—the situation where RL training data is partially or wholly present in prior pretraining or supervised fine-tuning (SFT) corpora. This overlap enables models to exploit shortcuts based on memorization rather than developing authentic reasoning, leading to inflated performance metrics and reduced reliability.

Figure 1: An illustration of Pre-RL data overlap in RL datasets, exemplified by DeepScaleR, where benchmark data is reused during both pretraining/SFT and RL, enabling shortcut exploitation.

The HIPPO Method: Mitigating Shortcut Exploitation

To address the challenge of reasoning degradation under unavoidable data overlap, this work introduces HIPPO (Hint-Injected Pairwise Policy Optimization). HIPPO reframes the RL objective to penalize shortcut-driven behaviors by leveraging "hint-injected" rollouts—traces where the ground-truth answer is inserted as a prompt "hint" before reasoning commences. This design provides explicit anchors for comparison: standard generation versus hint-exposed generation, the latter simulating shortcut-mediated outputs exploiting the answer.

The HIPPO training framework consists of the following stages:

• Standard rollout: The policy generates reasoning chains from prompt $x$, aiming to maximize both correctness and explanation quality.
• Hint-injected rollout: The same prompt is augmented with the gold answer, eliciting a shortcut-prone output that typically exhibits post-hoc rationalization.
• Pairwise judging: A small LLM-based judge compares outputs from the standard and hint-injected settings. The reward model maximizes the likelihood that standard rollouts are judged superior to hint-injected ones, using a robust pairwise preference structure.

  Figure 2: Framework of HIPPO. Standard and hint-injected traces are sampled and judged pairwise, forming a stable, discriminative RL reward signal.

Theoretically, the HIPPO objective is derived as a lower-bounded surrogate of the KL divergence between the current policy and the contaminated (shortcut-exploiting) policy, drawing on Pinsker's inequality and the variational form of total variation. Unlike direct KL maximization—which is intractable and unstable for long sequences—this surrogate yields stable optimization while being density-free and empirically estimable.

Empirical Evaluation and Analysis

The experimental setup spans mathematical (DeepScaleR) and medical (MedQA) reasoning tasks. Models are deliberately exposed to pre-RL data overlap via SFT purely on answers, then reinforced using the same (partially contaminated) data. Comparisons are made against strong baselines, including:

• Standard SFT and RL,
• Pointwise reward models,
• Advanced pairwise RL variants (Pref-GRPO, SP3F).

Main Performance Results

HIPPO consistently outperforms all baselines in both in-domain (math/medical) and out-of-distribution (OOD) general reasoning tasks. Notable gains are evident even when strong RL procedures and pairwise reward modeling are baseline. The approach also shows strong generalizability, extending reasoning improvements to new domains without direct exposure during RL.

Quality and Discriminability

Hint-injected rollouts serve as highly effective proxies for shortcut behavior: their rationales are systematically less rigorous and display higher error coupling than naturally occurring RL outputs under overlap. This property amplifies the observability of superficial reasoning, allowing even small LLMs to act as reliable pairwise judges.

HIPPO further enhances the decision boundary between high- and low-quality traces, yielding a reward signal that is more stable and mitigating intransitive judgment artifacts. These effects are achieved without dependence on heavyweight commercial models.

Figure 3: Response quality on overlapping MedQA data, measured by pairwise comparison versus SFT baseline. HIPPO surpasses all alternatives in rationale quality.

Figure 4: Scores from direct and hint-injected pairwise judges. Hint-injection creates greater separation of high/low-quality traces, improving reward discriminability.

Figure 5: PNT (Percentage of Non-Transitive cases) comparison. HIPPO’s hint-injected judge is more evaluation-stable and less susceptible to intransitive failure modes than direct pairwise comparison.

Mechanistic Understanding: How Overlap Triggers Fabrication

Mechanistic experiments show that Pre-RL data overlap activates memorization shortcuts, with RL training causing models to retrieve answers at shallow layers and rationalize post-hoc. Logit lens visualization illustrates that after RL, the answer token is highly probable throughout generation—even at the earliest layers—confirming pre-anchoring of the answer.

Causal information flow estimation using minimum description length proxies further shows that for non-overlapping datasets, reasoning ($z$) causally precedes the answer ($y$), whereas under overlap the causal direction collapses, indicating the model’s behavior increasingly matches answer-conditioned rationalization.

Figure 6: Logit Lens visualization. After RL on overlapping data, the probability of the answer token spikes at early layers and persists, validating premature answer retrieval and subsequent rationalization.

Implications and Future Directions

HIPPO provides a robust, theoretically-grounded, and empirically validated framework for training LLMs to reason reliably under realistic constraints of data overlap. Its successes suggest several implications:

• Mitigation over detection: Since strict data isolation is impractical, methods that directly suppress shortcut exploitation enable retention of valuable training data without sacrificing reliability.
• Generalizability: The approach scales across architectures and domains, indicating backbone-agnostic benefits and robustness to reward model downsizing.
• Preference modeling: Pairwise, anchor-based reward structures may become standard for RL in reasoning tasks to combat superficial gains from overlap or memorization.
• Mechanistic insight: The findings provide evidence that inflated RL gains can plausibly arise from fabricated rationales, not genuine logical discovery, motivating more fine-grained analysis tools in future work.

Additional directions include the exploration of incorrect-hint injection to discriminate between types of reasoning failures and more efficient judge models to reduce compute overhead.

Conclusion

The HIPPO framework demonstrates that combining hint-injection with pairwise comparison yields stable and discriminative RL supervision, systematically extracting authentic reasoning from contaminated datasets while suppressing shortcut behaviors. This solution reorients RL for LLMs toward robust, generalizable, and faithful reasoning, addressing a central obstacle for reliable model deployment under modern, data-reuse-heavy training regimes.

(2606.29481)