Honesty over Accuracy: Trustworthy Language Models through Reinforced Hesitation (2511.11500v1)

Abstract: Modern LLMs fail a fundamental requirement of trustworthy intelligence: knowing when not to answer. Despite achieving impressive accuracy on benchmarks, these models produce confident hallucinations, even when wrong answers carry catastrophic consequences. Our evaluations on GSM8K, MedQA and GPQA show frontier models almost never abstain despite explicit warnings of severe penalties, suggesting that prompts cannot override training that rewards any answer over no answer. As a remedy, we propose Reinforced Hesitation (RH): a modification to Reinforcement Learning from Verifiable Rewards (RLVR) to use ternary rewards (+1 correct, 0 abstention, -$λ$ error) instead of binary. Controlled experiments on logic puzzles reveal that varying $λ$ produces distinct models along a Pareto frontier, where each training penalty yields the optimal model for its corresponding risk regime: low penalties produce aggressive answerers, high penalties conservative abstainers. We then introduce two inference strategies that exploit trained abstention as a coordination signal: cascading routes queries through models with decreasing risk tolerance, while self-cascading re-queries the same model on abstention. Both outperform majority voting with lower computational cost. These results establish abstention as a first-class training objective that transforms ``I don't know'' from failure into a coordination signal, enabling models to earn trust through calibrated honesty about their limits.

• The paper introduces Reinforced Hesitation, a novel reward structure that incorporates abstention to optimize risk calibration in LLMs.
• It demonstrates that increasing the risk penalty (λ) produces a Pareto-optimal family of models, with conditional accuracy surpassing 99% on answered instances.
• The study proposes cascading and self-cascading strategies that leverage calibrated abstention signals to enhance query routing and overall decision reliability.

Trustworthy LLMs via Reinforced Hesitation

Motivation: The Failure of Modern LLMs to Abstain

Modern LLMs deployed in domains with asymmetric and high-cost errors—such as medical, financial, legal, and infrastructure decision-making—are evaluated and trained almost exclusively for accuracy under paradigms such as RLHF and RLVR. However, these paradigms do not include explicit incentives for calibrated abstention. Empirical evaluations across GSM8K, MedQA, and GPQA demonstrate that state-of-the-art RLVR- or RLHF-trained models nearly never abstain, even when prompts are modified to warn of severe penalties for incorrect answers or to provide abstention instructions. Abstention rates remain below 1% while error rates exceed 10%, regardless of how strong the penalty for mistakes is during inference, demonstrating instructional prompting cannot compensate for training-time misalignment.

Reinforced Hesitation: A Training-Time Solution

The core contribution is Reinforced Hesitation (RH): a minimal but principled alteration of RLVR post-training in which the binary reward signal is replaced with a ternary structure: +1 for correct, 0 for abstention, and $-\lambda$ for incorrect answers. Here, $\lambda \ge 0$ is a domain-kernelized risk aversion parameter. Under this reward structure, a rational agent abstains when its posterior confidence falls below the threshold $\frac{\lambda}{1+\lambda}$, introducing domain-specific, interpretable risk calibration directly into the optimization objective.

Empirical studies on logic puzzles (Knights & Knaves) using Qwen3-1.7B demonstrate RH with increasing $\lambda$ produces a family of models tracing a Pareto-optimal frontier: low $\lambda$ yields aggressive answerers, high $\lambda$ creates conservative, error-minimizing specialists, and intermediate $\lambda$ produces models that selectively trade coverage for safety. Each model becomes optimal for different operational contexts, and no single model can uniformly dominate others across varying costs of errors.

Figure 1: RH induces a Pareto frontier; each model trained for different $\lambda$ achieves distinct coverage-risk tradeoffs, and cascading these models enables efficient triage by confidence regime.

Behavioral Dynamics and Selectivity

Training with RH leads to sharp behavioral differentiation depending on $\lambda$. With $\lambda=0$, the model maintains high answering coverage but a persistent 15% error rate. Moderate $\lambda$ leads to selective abstention concentrated on hard problems: for $\lambda=1$, abstention is 10% for easy and 60% for hard instances, with overall errors dropping below 2%. For $\lambda \geq 10$, the model abstains on nearly all hard instances and only answers on easy ones, with errors approaching zero.

Notably, conditional accuracy on answered instances increases dramatically with penalty, surpassing 99% for high-$\lambda$ models. Thus, abstention signals are highly selective: the model abstains on cases it would otherwise get wrong. This selectivity is empirically validated and visualized below.

Figure 2: As $\lambda$ increases, conditional accuracy on attempted instances jumps from 84% to >99%, with error rates collapsing and abstention precisely concentrating where models are likely to fail.

Exploiting Abstention: Cascading and Self-Cascading

RH-trained models provide a unique coordination signal: abstentions precisely identify instances outside the model's calibrated competence region. This behavioral structure can be operationalized through two advanced inference strategies:

• Cascading: Queries are routed first to highly risk-averse models (large $\lambda$); if abstaining, queries are forwarded down a chain of less conservative specialists. This procedure achieves 88% overall accuracy (compared to 84% for the baseline), with only 2.2 average queries per sample. The reduction in error rates, IDK rate collapse, and maintained coverage indicate efficiency unattainable by naive ensembling.

  Figure 3: Cascading achieves superior coverage, low error rates, and Pareto-dominant performance over majority vote baselines by exploiting the selective specialization induced by RH.

• Self-Cascading: The same model is re-queried multiple times for abstentions, leveraging stochasticity to search for valid reasoning paths. This yields substantial accuracy gains; e.g., for $\lambda=1$, correct rate increases from 77.5% to 92.5% with a re-query budget of 64. IDK rates decrease monotonically with budget, while error rates remain bounded.

  Figure 4: Self-cascading with re-query budget improves both accuracy and coverage by converting abstentions to answers, while constraining error growth.

Majority voting, in contrast, provides negligible gains in accuracy and fails to recover abstained cases, demonstrating the unique utility of RH-induced abstention as a coordination mechanism.

Figure 5: Majority voting delivers limited improvement in correct and IDK rates, highlighting its inefficacy relative to cascading and self-cascading.

Theoretical and Practical Implications

RH formalizes abstention as a first-class objective for trustworthy LLMs, directly operationalizing epistemic humility. Its reward structure introduces an explicit accuracy-coverage-risk trade-off at the training level, enabling fine-grained specialization. The Pareto-optimal family of models precludes one-size-fits-all solutions; deployment must select models tuned to contextual error costs rather than optimizing for leaderboard accuracy alone.

Furthermore, RH enables new system architectures wherein abstention provides actionable routing signals for adaptive computation—whether to other LLMs, systems, or human experts—mirroring real-world epistemic delegation.

Limitations and Future Directions

Current experiments focus on synthetic logic tasks with sparse ground truth and a single model scale (1.7B parameters); generalization to subjective tasks, larger models, and real-world deployment pipelines requires further paper. Automated or data-driven selection of $\lambda$ remains an open research direction. Extensions to allow soft or continuous abstention/confidence, as opposed to hard “I don't know” boundaries, would improve calibration in broader contexts. Finally, broad adoption calls for new evaluation benchmarks that explicitly encode asymmetric error costs and reward calibrated uncertainty alongside classical accuracy.

Conclusion

Reinforced Hesitation (RH) redefines post-training objectives for modern LLMs by embedding calibrated abstention directly into reward structure. Empirical results establish that prompt-based abstention, even with explicit instructions and penalties, fails due to intrinsic gradient-driven biases. In contrast, RH yields a Pareto frontier of models, optimal under different risk regimes, with abstention acting as a predictive, precise, and actionable coordination signal. These advances push the field to reconsider evaluation and deployment paradigms: in high-stakes contexts, epistemic humility achieved via RH-trained abstention is more valuable for trust than headline accuracy alone. The implications extend to future AI training pipelines emphasizing calibration, adaptive trust, and safe real-world deployment.

---

Citation: "Honesty over Accuracy: Trustworthy LLMs through Reinforced Hesitation" (2511.11500)