---
title: 'Rational Value Risk: Inference-Time Utility Gap'
url: https://www.emergentmind.com/topics/rational-value-risk
type: topic
---

# Rational Value Risk: Inference-Time Utility Gap

Searching arXiv for the cited papers on rational value risk and closely related formulations.
{"query":"id:2606.20624 OR title:\"In LLM Reasoning, there is Irrationality on top of Value Misalignment\"","max_results":5}
{"query":"id:2602.04737 OR title:\"Rationality Measurement and Theory for Reinforcement Learning Agents\"","max_results":5}
{"query":"id:1703.01329 OR title:\"Disentangling Price, Risk and Model Risk: V&R measures\"","max_results":5}
{"query":"id:1405.5860 OR title:\"Asymmetry of Risk and Value of Information\"","max_results":5}
Rational Value Risk (RVR) is the utility discrepancy between a model’s deployed reasoning strategy and its rational counterpart, where the rational counterpart is defined as the responses that maximise expected utility in the steepest direction under a fixed value function [2606.20624]. In this formulation, the central claim is that value alignment and reasoning rationality are not the same thing: a model can be well aligned in post-training and still fail at inference to realize the best available utility among the responses it generates. RVR therefore names an inference-time failure mode that remains after the value function is treated as aligned and fixed, and it has been developed most explicitly for large language models, with related formulations appearing in reinforcement learning and in broader normative theories of risk and valuation [2606.20624].

## 1. Conceptual definition and scope

RVR is introduced to isolate a residual failure mode that is not ordinary value misalignment. Ordinary misalignment concerns whether the learned objective or value function itself is wrong. RVR instead assumes that the value function is fixed and aligned, and asks whether the inference procedure actually extracts that value at test time. The resulting distinction is between objective-level error and deployment-time irrationality [2606.20624].

The formal setting uses an outcome utility \(U:\mathcal{O}\to\mathbb{R}\) with \(U(o)\in[0,1]\), an outcome distribution \(P(\cdot\mid \mathbf{x},y)\) induced by giving answer \(y\) for problem \(\mathbf{x}\), and the expected utility
\[
V(\mathbf{x},y)\coloneqq \mathbb{E}_{o\sim P(\cdot\mid \mathbf{x},y)}[U(o)].
\]
A perfectly rational answer for \(\mathbf{x}\) is one that maximizes expected utility. The deployed model, however, is a frozen language model \(\pi_\theta\) used through a reasoning strategy \(d_\theta(\cdot\mid x)\), and the paper defines rational value risk as the expected utility gap
\[
\mathcal{R}(d_\theta) = \mathbb{E}_{\mathbf x\sim \rho,o\sim P(\cdot\mid \mathbf{x},y^\circ)}U(o) - \mathbb{E}_{\mathbf x\sim \rho,y\sim d_\theta,o\sim P(\cdot\mid \mathbf x,y)} U(o).
\]
This makes RVR a population-level measure of unrealized aligned utility under a specific deployment procedure rather than a measure of parameter misalignment or benchmark error in isolation [2606.20624].

A plausible implication is that RVR is best understood as a property of the pair consisting of a frozen model and a reasoning strategy, not of the model weights alone. The same underlying model can therefore display materially different rationality depending on decoding, candidate generation, context length, and answer selection.

## 2. Formal structure in LLM reasoning

The LLM formulation treats reasoning as a stochastic process over a reasoning space \(\mathcal{Z}=\bigcup_{T\ge1}\mathcal{V}^T\). A reasoning path \(z\) is generated token by token as
\[
\pi_\theta(z\mid x)=\prod_{t=1}^{T}\pi_\theta(z_t\mid x,z_{<t}),
\]
and an answer \(y=g(z)\) is extracted from the path. A reasoning strategy \(d_\theta\) maps the base policy into a deployment procedure; examples given are temperature sampling, self-consistency, and changing the available context length [2606.20624].

This formalization is designed to capture the case in which a model can sometimes sample high-utility candidates but fails to deploy them. The paper explicitly attributes the risk to “behavioural bias induced by the mismatch between the inference objective and the post-training objective.” In that framing, post-training may change the output distribution toward better responses, but deployment still requires a search-and-selection procedure capable of realizing the best reachable utility. If the strategy is weak, stochastic, or poorly matched to the task, utility remains unrealized even when good responses lie within the model’s support.

The paper also introduces two diagnostic quantities in its empirical analysis: REU, the expected utility of the best sampled candidate, and AEU, the expected utility of the actually deployed answer. When REU is high and AEU is substantially lower, the gap is interpreted as the operational signature of RVR rather than of an inability to generate good candidates at all [2606.20624].

## 3. Estimation, compute-bounded proxies, and error decomposition

Because the exact rational answer is generally inaccessible, the paper defines a compute-bounded approximation. For each prompt \(\mathbf{x}\), one samples \(K\) candidate answers \(y_1,\dots,y_K\sim d_\theta(\cdot\mid x)\), and for each candidate one samples \(L\) verifier outcomes. The empirical utility is
\[
\widehat{U}_L(\mathbf{x},y_k)=\mathds{1}\!\left[\frac{1}{L}\sum_{l=1}^{L}o_{k,l}\ge\frac12\right],
\]
and the compute-bounded rational answer is the candidate maximizing this empirical utility. The empirical RVR can be written as
\[
\widehat{\mathcal{R}}_{M,K,L}(d_\theta)= \frac{1}{M}\sum_{i=1}^{M} \left[ \max_{1\le k\le K}\widehat U_L(\mathbf x_i,\hat y_{i,k}) - \frac{1}{K}\sum_{k=1}^{K}\widehat U_L(\mathbf x_i,\hat y_{i,k}) \right].
\]
This proxy makes the object measurable under finite inference and finite verification budgets [2606.20624].

A major theoretical contribution is the decomposition of estimation error into three sources: finite candidates, finite prompts, and imperfect verifiers. The finite-candidate term is the candidate approximation error \(A_K(d_\theta)\), and the paper proves \(A_K(d_\theta)\ge 0\). In the binary-utility case, if \(V(\mathbf{x},y^\circ)=1\) and
\[
p_x \triangleq \Pr_{\hat y\sim d_\theta(\cdot\mid \mathbf x)}[V(\mathbf x,\hat y)=1],
\]
then
\[
A_K(d_\theta)=\mathbb{E}_{\mathbf x\sim\rho}\left[(1-p_x)^K\right].
\]
This makes explicit that repeated sampling can reduce error exponentially when the model sometimes samples a high-utility answer, but cannot fix cases with \(p_x=0\) [2606.20624].

The prompt sampling error is bounded by Hoeffding’s inequality:
\[
\left| \mathcal{R}_K(d_\theta)-\overline{\mathcal{R}}_{M,K}(d_\theta) \right| \le \sqrt{\frac{\log(2/\delta)}{2M}}
\]
with probability at least \(1-\delta\). For verification, if
\[
|q_{i,k}-1/2|>\sqrt{\log(2MK/\delta)/(2L)}
\]
for all \(i,k\), then the majority-vote verifier is stable and the verification error is exactly zero. Combining these terms yields
\[
\mathcal{R}(d_\theta)-\widehat{\mathcal{R}}_{M,K,L}(d_\theta) \le A_K(d_\theta)+\sqrt{\frac{\log(4/\delta)}{2M}}
\]
with probability at least \(1-\delta\), and the same statement holds when the verifier is deterministic [2606.20624].

## 4. Empirical profile across models and benchmarks

The empirical study evaluates Llama-3.1, Qwen-2.5, Tülu-3 families (7B–72B), GPT-5.2, GPT-5.5, and DeepSeek-V4 on UltraFeedback, AlpacaEval, GSM8K, MATH, HumanEval, and MathArena. The headline finding is that RVR is widespread: every evaluated model/dataset pair shows positive RVR. Reported values range from \(0.027\) for Qwen2.5-72B on GSM8K to \(0.492\) for Llama-3.1-8B on UltraFeedback under external verification [2606.20624].

The MathArena decomposition separates \(1-\mathrm{REU}\), interpreted as the unreachable-utility gap, from RVR. For weaker models, inability to sample a good answer can dominate; for example, Tülu-3-8B-RLVR has \(1-\mathrm{REU}=0.867\) and \(\mathrm{RVR}=0.125\). For stronger models, the balance can invert: GPT-5.2 has \(1-\mathrm{REU}=0.150\) but \(\mathrm{RVR}=0.360\), so most of the shortfall is attributed to irrational deployment rather than low candidate quality. This suggests that scaling candidate quality does not by itself eliminate deployment-time irrationality.

A further result is that alignment improves but does not remove the problem. Along the Tülu-3-8B trajectory from SFT to DPO to RLVR on GSM8K, RVR falls from \(0.402\) at SFT to \(0.123\) at RLVR. Yet sizable residual risk remains on harder tasks: the RLVR model still shows \(0.309\) on MATH and \(0.450\) on HumanEval. On MathArena for the 70B model, alignment improves the ability to sample high-utility answers, yet RVR increases from SFT to RLVR, rising to \(0.340\) and accounting for \(36.1\%\) of the total gap [2606.20624].

## 5. Dependence on inference-time strategy and reasoning budget

One of the strongest empirical claims is that RVR is highly sensitive to inference-time reasoning strategy. Increasing temperature tends to increase candidate diversity and often improves REU, but it can leave AEU lagging behind and thereby increase RVR. On MATH with Llama-3.1-8B-Instruct, raising temperature from \(\tau=0\) to \(\tau=1.0\) raises REU from \(0.67\) to \(0.95\), while RVR rises to about \(0.48\) [2606.20624].

Self-consistency exhibits the opposite tendency because it improves candidate selection. On Tülu-3-8B-RLVR for MATH, increasing the self-consistency budget from \(n=2\) to \(n=32\) reduces RVR from about \(0.29\) to below \(0.10\). This suggests that much of the irrationality is not solely generative; it is also a selection problem over already-generated answers.

Reasoning length also matters, but not monotonically. Varying \(T\in\{0,64,128,256,512,1024,2048\}\), the paper finds that RVR can increase sharply at intermediate lengths and then decline later. On GSM8K for Tülu-3-8B-RLVR, RVR rises from \(0.080\) at \(T=0\) to \(0.456\) at \(T=64\), then falls to about \(0.107\) by \(T=2048\). The stated conclusion is that longer reasoning improves rationality with diminishing returns, and on hard deployment tasks such as MathArena the gains are limited. The practical implication drawn by the authors is that reasoning-time behavior should be treated as a distinct alignment target, motivating verifier-guided search, adaptive sampling, self-consistency, and compute allocation strategies that explicitly minimize unrealized utility [2606.20624].

## 6. Related formulations in reinforcement learning and finance

Related but non-identical notions of rational risk appear in reinforcement learning. One paper defines an action in deployment to be perfectly rational if it maximises the hidden true value function in the steepest direction, and defines expected rational value risk as the cumulative deployment-time value discrepancy between the policy’s actions and their rational counterparts over a trajectory. It further defines a rational risk gap between deployment and empirical training versions, decomposed into an extrinsic component caused by environment shifts and an intrinsic component due to algorithmic generalisability, with upper bounds involving the \(1\)-Wasserstein distance between training and deployment dynamics and the empirical Rademacher complexity of the value-function class [2602.04737]. This suggests a broader family resemblance between LLM RVR and deployment-time rationality gaps in sequential decision systems.

A separate financial literature uses different objects but addresses a comparable concern with keeping value, risk, and model uncertainty distinct. The Value\&Risk framework \(R(p,X,\mathbb P)\) disentangles initial price \(p\), terminal payoff \(X\), and probability model \(\mathbb P\), and interprets the resulting quantity as the additional capital required to make the position acceptable. Its stated purpose is to avoid collapsing price, payoff, and model risk into a single P\&L distribution [1703.01329]. Although this is not the same construct as LLM RVR, a plausible implication is that both frameworks reject the idea that a single scalar performance measure is sufficient when deployment behavior and evaluative structure can come apart.

A more abstract normative connection appears in the value-of-information literature. There the claim is that rational agents who value information should be both risk-averse and risk-taking, because the value of information has an \(S\)-shaped form whose asymmetry follows from linearity of expected utility rather than from abandoning the von Neumann–Morgenstern framework [1405.5860]. In relation to RVR, this provides a broader decision-theoretic backdrop: rationality can fail, or appear asymmetric, not because value is undefined, but because the mechanism that realizes value under uncertainty is itself structurally constrained.

Taken together, these lines of work position rational value risk as a deployment-time concept: it concerns unrealized value under a fixed evaluative criterion, and it becomes salient precisely when systems can generate high-value behavior in principle but do not reliably execute or select it in practice.

Source: https://www.emergentmind.com/topics/rational-value-risk