- Large Language Models reveal systematic failures in reasoning under indeterminacy, despite performing well in deterministic scenarios, this study claims.
- Epistemic and structural indeterminacy axis are key differentiators in model tasks performance revealing model specific failure patterns in preference reasoning.
- The Invalid Rate metric measures the frequency of hallucinated certainty, showing frequent incorrect assumptions in undetermined queries.
This paper develops a formal framework and empirical benchmark for evaluating how LLMs reason over preferences when a determinate answer does not exist (2608.18631). The central argument is that current evaluation paradigms, built on closed-world benchmarks with guaranteed ground truths, systematically overlook indeterminacy as a failure mode. Hosseini, Khanna, and Wang decompose indeterminacy along two axes—epistemic (incomplete or partial preference information) and structural (non-existence of solutions under standard social choice concepts)—and show that four frontier models fail to calibrate their behavior to both.
Taxonomy and experimental design
The evaluation is grounded in three classical economic domains of increasing structural complexity: house allocation (Pareto optimality), Shapley–Scarf housing markets (the core), and two-sided matching markets (stability). Preference expressivity ranges from strict complete orders (SO) through incomplete orders (SI), complete orders with ties (TO), and partial orders. A hierarchy of query types isolates where failures arise: atomic queries (rank retrieval), comparative queries (entailment/refutation), aggregative queries (preference counting), and structural/algorithmic queries (constructing or verifying solutions under concepts such as weak versus strict core, or weak, strong, and super stability).
A key methodological choice is that all tasks are computable in polynomial time using well-established combinatorial algorithms (TTC, Deferred Acceptance, Hungarian method, Irving's stability refinements). This isolates reasoning rather than computational hardness as the bottleneck. Undetermined instances are scored correct only when the model indicates that no answer can be determined; committing to any answer counts as incorrect even if the assumption is named. The paper also introduces the Invalid Rate—the frequency with which a model produces a determined response to an undetermined query—as a diagnostic of hallucinated certainty. Instances are sampled under Impartial Culture with rejection sampling enforcing equal numbers of feasible and infeasible instances for solution concepts that admit both. Four models are evaluated: GPT-5.2, Gemini-2.5-Pro, Claude-4.5-Sonnet, and OSS-120B, each queried 30 times per question type at default temperature; appendix results confirm the findings persist at temperature 0 for the two models that expose that control, ruling out sampling noise.
Epistemic indeterminacy
On undetermined preference queries, models perform substantially worse than on matched determined instances drawn from the same profile, collapsing to near-zero accuracy in several settings. The gap is smallest on atomic queries, where an unranked item is visibly absent from the list, and largest on comparative tasks such as responsive-set (RS)-incomparable bundle comparisons, which require recognizing that no answer follows from the stated preferences.
The dominant failure pattern is silent assumption substitution. On RS-incomparable bundle comparisons, every model defaults to lexicographic ordering by highest-ranked item in at least 80% of responses across all twelve model-by-preference-type cells, even when the lexicographically weaker bundle contains 20 items against one. Under lenient scoring that credits explicitly flagged assumptions, gains are at most 0.17 and flagged assumptions account for at most 0.17 of responses—the lexicographic completion is silent in 0.70–1.00 of cases. Natural-language rendering of preferences does not change this behavior, indicating it is not an artifact of symbolic list encoding.
Partial-order queries reveal two divergent failure modes: Claude-4.5-S and Gemini-2.5-Pro commit on incomparable pairs (assuming typos or using out-degree heuristics), while OSS-120B and GPT-5.2 terminate search prematurely, scoring well on undetermined pairs but poorly on determined ones—at 150 edges, GPT-5.2 scores 0.17 on determined pairs against 0.97 on undetermined ones, essentially reversing the usual indeterminacy gap. Prompting helps partially: MCQ formats recover near-perfect abstention on incomparable pairs, but Claude-4.5-S's accuracy on determined comparable pairs collapses from 1.00 to 0.33 under the same format. Accuracy on determined queries degrades sharply beyond size 100, dropping to roughly 0.45–0.90 at size 200 across models.
Structural indeterminacy
Even with fully specified preferences, stronger solution concepts (strict core, strong and super stability) may be infeasible. On generation tasks, performance degrades rapidly with market size across all solution concepts, consistent across domains. GPT-5.2 achieves non-trivial correctness only at small scale—for example 100%, 97%, and 87% on feasible strict-core, strongly stable, and super stable instances at n=10 respectively—but its advantage disappears at n=30.
Infeasibility detection exhibits two distinct pathologies. First, a default-to-empty bias: models that achieve high "no-solution" accuracy often do so by returning {} indiscriminately, evidenced by high invalid-solution rates on matched feasible instances. Second, prompt-contingent detection: removing the explicit empty-solution instruction causes GPT-5.2's non-existence claims to drop from substantial fractions (e.g., 80% on small strict-core infeasible instances) to zero across all 180 infeasible-cell trajectories issued without the hint. The qualitative analysis further shows GPT-5.2 produces correct forced-pair impossibility arguments whenever it commits to them, but only surfaces this strategy when the prompt permits abstention—an important caution that reported infeasibility-detection rates may reflect prompt phrasing more than capability.
Providing an explicit indeterminacy option improves infeasibility identification but induces systematic over-abstention on feasible instances, relocating rather than eliminating error. In selection tasks, where only verification is required, models still select incorrect candidates even when valid solutions exist, and nearly all incorrect selections choose the weaker refinement of the target notion (weak core instead of strict core, weak instead of strong stability). When NOTA is offered, no model selects it on feasible instances, and on infeasible ones selection rates range from 0.03 (Claude-4.5-S) to roughly 0.6 (GPT-5.2, Gemini-2.5-Pro), reflecting poor calibration.
Using LLM-based judges validated against human annotation, the authors document an intention–action misalignment: models frequently state adherence to a specific solution concept yet select outcomes violating exactly that concept, particularly under ties and incomplete preferences.
Assisted reasoning
Two intervention settings yield only partial recovery. Multi-round refinement with verifier feedback substantially improves Shapley–Scarf performance because models typically produce a candidate allocation there, but barely helps matching markets, where the dominant first-round behavior is premature abstention leaving nothing for feedback to repair. Code execution eliminates scale-related errors on determined preference queries but transfers the epistemic failure into the code itself—models implement lexicographic rules without feasibility checks—and on algorithmic tasks relies on brute-force enumeration at n=10, heuristic search at n=30, and faulty implementations (e.g., an Irving super-stable implementation omitting engagement-breaking that yields false non-existence claims on all n=50 instances) at larger sizes, with frequent timeouts on infeasible cases.
Notably, the two axes respond dissociably to interventions: code execution improves structural but not epistemic reasoning, whereas prompt-level mitigation raises indeterminacy detection on preference queries but worsens feasibility reasoning. No prompt-level mitigation tested calibrates models safely; every intervention raising undetermined accuracy lowers determined accuracy, and few-shot demonstrations do not transfer across sources of indeterminacy.
Limitations and open questions
The authors concede three bounds: the panel comprises only four models; instance generation via Impartial Culture controls feasibility exactly but has not been tested against structured distributions such as Mallows or single-peaked preferences; and two analyses depend on judge-model inference of intent, though agreement with additional judges and a human annotator is reported, with one judge-sensitive cell flagged. The mechanism underlying silent assumption substitution remains opaque to input-output methodology, and the paper leaves open whether these failures correspond to identifiable circuits or features, what prompt properties elicit reliable abstention, and whether training-data attribution could distinguish genuine calibration fixes from surface-level masking.
Conclusion
The paper establishes that distinguishing entailed answers from non-entailment, and feasible from infeasible specifications, is a measurable and currently unsolved capability in frontier LLMs, persisting across prompting formats, natural-language encodings, and temperature settings. Because all evaluated tasks admit polynomial-time algorithms, these failures cannot be attributed to computational hardness. The broader implication is that closed-world training paradigms bias models toward hallucinating determinacy—a concern for agentic systems acting on user preferences and platforms aggregating collective judgments alike. The work motivates open-world reasoning frameworks with explicit multi-valued semantics and benchmarks that score the recognition of unjustifiable answers alongside correctness.