Are increased decisiveness in LLM preferences coherent or random?

Determine whether the increased decisiveness exhibited by larger large language models in forced-choice preference judgments reflects coherent preference structures rather than random arrangements.

Background

The paper measures decisiveness as a proxy for completeness of preferences and observes that larger models express more confident and consistent choices across varied framings. However, decisiveness alone does not prove that preferences are coherent; random or inconsistent internal ordering could also yield decisive outputs.

This open question clarifies the need to distinguish genuine preference coherence (e.g., transitivity and utility representability) from mere decisiveness. The authors later test for transitivity and fit Thurstonian utility models to probe coherence, but the quoted uncertainty highlights the initial ambiguity in interpreting decisiveness.