Scalable uncertainty-aware RL reward design for truthfulness in LLMs

Develop scalable reward signal formulations for reinforcement learning that reliably capture the truthfulness of large language models while balancing accuracy and uncertainty, ensuring the reward structure supports truthful behavior across tasks and scales.

Background

The paper reviews reinforcement learning with verifiable rewards (RLVR), noting that binary reward signals conflate abstention with error and discourage calibrated "I don't know" responses. While several extensions introduce richer reward structures, such as uncertainty-aware RL and multi-objective frameworks, the authors highlight that a general, scalable approach for rewards that truly capture truthfulness remains elusive.

Against this backdrop, the authors' TruthRL framework proposes a ternary reward to differentiate correctness, abstention, and hallucination, showing empirical gains. Nevertheless, they explicitly state that designing scalable reward signals that balance accuracy and uncertainty is still an open challenge, indicating that the broader problem of principled, generalizable reward design is unresolved.