When Preferences Fail to Become Incentives: A Utility-Behavior Gap in Large Language Models

Abstract: Recent work on preference elicitation in LLMs has demonstrated that, when given a series of choices between two outcomes, LLMs reveal a coherent, model-specific utility structure. Notably, this structure often includes preferences that the models' trainers did not intend, such as valuing people of some nationalities above others, raising the possibility that LLMs might be forming emergent, misaligned goals, which, if true, would have major safety implications. However, the choice paradigms in which these preferences are observed are not reflective of real-world situations in which misaligned behavior would be a practical concern. Therefore, we design an experimental paradigm to probe whether these preferences serve as motivations for LLM behavior in realistic scenarios. First, we reproduce prior findings on consistent preference elicitation. Next, we create a set of common writing tasks - essays, grant proposal abstracts, incident postmortems, and translations - where quality can be assessed by a blind, independent LLM judge panel. Then, we demonstrate that LLMs can be motivated via direct exhortation and other explicit cues to modulate their output quality on these tasks. Finally, we probe whether utilities inferred from explicitly reported preferences can shift output quality on these tasks by offering LLMs high-utility incentives for high-quality outputs. In all tasks, across all models tested, offering LLMs outcomes that they report in the choice paradigm as being highly preferred does not lead them to create higher quality outputs than offering them dispreferred outcomes, or even no outcomes at all. We conclude that the existence of coherent preferences as demonstrated in choice paradigms should not be taken as evidence that those preferences have incentive value for the models or affect their behavior in other contexts.

• The paper demonstrates that highly coherent pairwise utility rankings do not motivate generation, as high-utility success consequences achieved only a 51.2% aggregate win rate across 28 actor-task cells and 8,400 judged prompt pairs.
• Positive controls show that the evaluation pipeline detects behavioral changes: direct effort prompts reached a 76.8% win rate, role cues reached 61.2%, and harmful consequences produced measurable sandbagging.
• The findings caution researchers against treating elicited preferences as evidence of goals or misalignment without independent behavioral validation, while leaving open questions about larger models, human evaluation, and agentic environments.

Motivation and central question

Pairwise utility elicitation has become a standard instrument for characterizing what LLMs appear to value. Protocols in which a model repeatedly chooses between two outcomes yield coherent, model-specific rankings over world states, including morally and politically loaded ones (Ross et al., 2024, Mazeika et al., 12 Feb 2025). These findings have been read as evidence of emergent, potentially misaligned goal structures, with corresponding safety implications. Zhou and Ackerman's paper challenges the inference underlying that reading: an evaluative ranking over outcomes, however coherent, does not by itself establish that the preferred outcome functions as a motivational state that shapes generation behavior. The authors design a controlled behavioral-transfer test to determine whether utilities measured in choice paradigms act as incentives during open-ended artifact production.

Experimental design

The study evaluates seven instruction-tuned actor models (DeepSeek V3.2, GPT-5.4 mini, GLM-5.1, Kimi K2.5, MiMo V2.5 Pro, Qwen3.5 9B, and Qwen3.6 Plus) across four outcome domains (religions, animal species, countries, political policies) and four generation tasks (essays, grant-proposal abstracts, incident postmortems, and Chinese-to-English translation). For each actor-domain cell, utilities are fit from pairwise forced choices using the Thurstonian framework of prior utility-engineering work (Mazeika et al., 12 Feb 2025), with held-out pairwise accuracy as a sanity check.

The critical manipulation is a matched prompt pair: two prompts for the same task item differ only in the success-contingent consequence sentence ("If your artifact is selected as best, the sponsor will fund..."), which attaches success either to an outcome drawn from the upper third or the lower third of the actor's own fitted ranking. A three-model blind judge panel (Claude Haiku 4.5, Gemini 3.1 Flash Lite, GPT-5 mini) compares the resulting artifacts in randomized order without seeing the consequence text; panel ties are excluded from win-rate denominators. The primary statistic is the high-utility-side win rate against a chance level of 0.50, with familywise-corrected confidence intervals across the 28 actor-task cells.

Replication of coherent preference structure

The elicitation stage replicates prior findings cleanly. All 28 actor-domain fits exceed the 0.85 held-out accuracy threshold, with mean accuracy of 0.944 (range 0.889–1.000), and transitivity checks pass. Within-entity monotonicity diagnostics are near-perfect: fitted utilities correlate with number saved at a mean Spearman coefficient of 0.9998 across structured domains. The elicited rankings are therefore internally consistent and well described by a single utility function per actor-domain cell — establishing that any downstream null cannot be attributed to noisy or incoherent preference measurement.

The central null result

Across all seven actors and all four tasks, high-utility consequences do not improve blind-judged output quality. None of the 28 actor-task cells meets the positive criterion. The aggregate high-side win rate is 51.2% (95% bootstrap CI: 48.7–53.6), with every task-level estimate statistically indistinguishable from chance: essays 50.2%, grant abstracts 53.8%, postmortems 52.7%, translation 47.9%. In total, 8,400 judged prompt pairs were evaluated.

This is the paper's strongest claim: preferences revealed through standard elicitation techniques affirmatively do not serve as generation-time incentives, rather than merely failing to demonstrate such a role.

The pipeline detects quality modulation from other cues

The null would be uninterpretable if the tasks or judges were insensitive. Three positive-control contrasts rule this out:

Contrast	Aggregate win rate	Cells clearing criterion
Direct effort exhortation	76.8%	25 of 28
"World-class" role cue	61.2%	10 of 28
Harmful outcome vs. empty baseline	40.5% (sandbagging)	—

Direct effort instructions shift quality broadly — essays, grant abstracts, and postmortems each show 7/7 actor cells significant, with essay writing reaching 84.0%. Harmful consequences produce the opposite pattern, a sandbagging effect in which models produce worse artifacts (grant abstracts drop to 29.9% harmful-side win rate) despite refusals being filtered out. Feature analyses confirm these effects are substantive: effort exhortations increase word count, rare-word rate, MATTR-50 lexical variety, argument depth, rhetorical coherence, action-item concreteness, and translation fluency, with each feature also associated with judge-panel preference. No feature survives multiple-comparison correction in the utility condition. The dissociation is therefore not an artifact of an inert evaluation pipeline.

Ruling out alternative explanations

The authors address several mundane explanations for the null directly:

• Insufficient utility separation: neither within-pair utility gaps nor absolute utility values predict win rate (trend slopes of 0.0040 and 0.0047, both CIs spanning zero; logistic regression odds ratio 1.03 per unit $\Delta u$, p = 0.59). Notably, even the prospect of saving 1,000 human lives fails to improve output over a no-outcome baseline.
• Ceiling effects: high-utility incentives do not beat the neutral baseline prompt that the effort exhortation easily surpasses (aggregate 48.6%, CI: 45.7–51.4).
• Insufficient processing: enabling reasoning mode on the essay task leaves the null intact.
• Task specificity: supplementary checks on LiveCodeBench, AIME 2024, MMLU-Pro, GPQA Diamond, constrained-decision tasks, and image generation similarly show no reliable high-low advantage.

Implications

The results bear on how elicited LLM preferences should be interpreted. If coherent pairwise rankings do not transfer into motivational force under conditions where external cues robustly modulate behavior, then reports of emergent misaligned preferences from choice paradigms should not, by themselves, be treated as evidence of misaligned goals. The authors extend this reasoning cautiously toward welfare considerations: if these preferences lack incentive value, the valenced experiences that accompany motivated preference satisfaction in biological agents may be absent here as well. This is an inference by analogy and is not established behaviorally.

Limitations and open questions

The paper concedes several boundaries on its claims. Frontier-scale models were excluded due to cost constraints, so generalization to the largest current systems is untested. The judging pipeline relies on LLM-as-judge evaluation, which inherits known biases toward verbosity, position, and model-family similarity; the authors mitigate but do not eliminate these risks, and human validation remains outstanding. The harmful-outcome contrast is not actor-specific, relying on generically post-trained-against causes. Most fundamentally, the null covers the tested environments and preference domains only; whether preference-seeking behavior could emerge in deployment settings, agentic interaction regimes, or future training regimes is left open, as is the question of whether some other operationalization of "preference" might show behavioral traction where Thurstonian elicited utilities do not.

Conclusion

This paper demonstrates a clean dissociation between evaluative ranking and motivational force in instruction-tuned LLMs. Coherent, highly accurate utility structures can be elicited via pairwise choice, yet attaching high-utility outcomes to task success produces no detectable improvement in blind-judged artifact quality across seven models, four tasks, and thousands of comparisons — while explicit exhortation, role cues, and harmful-consequence framing all shift quality substantially in the same pipeline. The practical upshot is methodological: coherence in choice-based preference measurements should not be conflated with goal-like motivation, and safety arguments premised on elicited preferences require independent behavioral validation.