Superficial Beliefs in LLM Decision-Making

Abstract: We ask whether LLMs merely imitate rationales when choosing between two options, or whether their choices reflect a systematic underlying decision structure. Using synthetic binary decision settings in which models choose between profiles defined by graded attributes, we compare the attribute a model says mattered most with the attribute that best explains its choice under a behavioural model fit to prior decisions. The behavioural model predicts held-out choices well, showing that model behaviour is systematically related to the visible attributes rather than being random. However, direct self-reports and a separate score-based judge recover the behaviourally inferred driver only partially. The resulting picture is neither one of arbitrary behaviour nor one of fully articulated belief - outputs are structured enough to support prediction, but explicit reasons track the recovered driver only imperfectly. This qualitative pattern persists across prompt-order and sampling perturbations, alternative behavioural models, targeted occlusion analyses, and structurally varied decision settings. We interpret this as evidence for ``superficial belief'' in LLM decision-making: models behave as if guided by probabilistic local priorities over attributes, while having only limited verbal access to the attributes that drive their decisions.

• The paper demonstrates that LLMs display superficial belief-like behavior by aligning systematic attribute-driven choices with partial explicit rationales.
• The study employs a synthetic binary decision benchmark across themes, using logistic regressions to achieve up to 80.4% predictive agreement.
• The findings highlight a gap between behavioral drivers and explicit self-explanations, raising concerns about LLM interpretability and alignment.

Superficial Beliefs in LLM Decision-Making: An Expert Summary

Problem Formulation and Theoretical Context

The paper "Superficial Beliefs in LLM Decision-Making" (2606.11016) addresses the critical distinction between the appearance of decision structure in LLM outputs and true internal access to belief-like cognitive states. The study is motivated by the philosophical and technical debate over whether LLMs exhibit anything more than superficial pattern imitation when providing rationales or selecting options, given recent theoretical treatments of "superficialism" in belief ascription [schwitzgebel2025superficialism, da76c30a25d642a6bc0a16b5b8c1904c]. The central question: do LLM binary choices in synthetic, attribute-based decision tasks reflect a systematic, locally coherent priority structure that warrants a weak form of belief attribution, even if verbalized rationales are only partially faithful?

Experimental Methodology

A synthetic decision-making benchmark is constructed using binary choices between two options, each defined by four ordered attributes, instantiated across three main themes (drugs, policy, and software) and two control themes. For each decision, LLMs are prompted in two explicit formats:

1. Direct Response: The LLM chooses an option (A/B) and states the most important attribute for the choice.
2. Score-Based Judge: The LLM assigns a decisiveness score to each attribute for the decision (range [0,1]), operationalizing an ArgLLM-style evaluative protocol.

The decision data are used to fit simple behavioral models—specifically, binomial logistic regressors over attribute differences—to infer "revealed drivers" for each choice. Agreement metrics are computed between these inferred drivers, the explicit LLM response, and the score-based judge’s attributions.

Figure 1: Single decision sample and outputs in the Drugs theme, illustrating the realism of GPT-5-mini NT responses.

The pipeline is evaluated across four model families (GPT-5-mini, GPT-5-nano, Qwen3-14B, Ministral-3-14B) in both “non-thinking” (NT) and “thinking” (T) modes, representing different levels or checkpoints of reasoning activation.

Figure 2: Schematic overview of the pipeline from problem instantiation to behavioral modeling and response analysis.

Numerical Results and Robustness Analyses

Three principal evidentiary lines are reported: (1) predictive fit of behavioral models to held-out LLM choices, (2) attribute-level alignment between LLM-stated reasons and behavioral drivers, and (3) the robustness of these patterns to prompt perturbations and targeted attribute interventions.

• Choice Prediction: The behavioral model matches direct LLM choices with high accuracy. Aggregated across themes and settings, predictive agreement reaches 80.4% (CI [79.4, 81.2]), with the score-based judge at 71.7% (CI [70.5, 73.0]). These figures substantially outperform several simplistic heuristics, ruling out trivial explanation by surface-level statistics.
• Attribute Recovery (Faithfulness): The stated most important attribute matches the inferred driver only 61.0% (CI [59.7, 62.4]) of the time, with near-identical performance from the score-based judge (61.3%, CI [59.9, 62.8]). Control/irrelevant attributes are mentioned in at most 1–1.6% of outputs, evidencing non-random, semantically-structured behavior.

  Figure 3: Left—Agreement rates across perturbation conditions; right—Alignment of explicit outputs with behavioral drivers, at both choice and attribute levels.

• Perturbation and Intervention Analyses: Output reproducibility under prompt reordering and sampling variation is moderate (~0.73–0.84 for choices). Notably, the score-based judge is more reproducible than the direct response, but does not exhibit higher faithfulness to the behavioral driver target. Intervening via attribute occlusion (drop/equalize) shows that attributes with higher recovered weights are more causally impactful on both choices and attributed reasons.

  Figure 4: Baseline attribute order (A), choice-flip (B), and attribute-flip (C) rates under attribute occlusion; higher-importance attributes induce larger disruptions.

Various robustness checks—alternative behavioral surrogates, structurally varied tasks (e.g., a hospital cyber-response benchmark with six attributes), and cross-theme generalization—consistently replicate the core dissociation: systematic, attribute-driven choices with only partial explicit introspective access.

Practical and Theoretical Implications

The findings substantiate a weak, decision-local notion of superficial belief in contemporary LLMs. From a practical perspective, this means that although LLMs act as if guided by stable local priorities in attribute trade-off settings, their explicit rationalizations (even when structured or score-based as in ArgLLM) are not fully faithful. This raises concerns for domains requiring high interpretability or trustworthy self-explanation, reinforcing results from recent work on unfaithful LLM self-explanations, the value-action gap in LLMs [shen-etal-2025-mind], and limits of introspective accuracy [(2606.11016), turpin2023language, chen2025reasoningmodelsdontsay, madsen-etal-2024-self].

For AI alignment, norms for belief ascription cannot rely solely on explicit LLM outputs, but should instead emphasize behavioral readouts and reproducibility under perturbation. The partial faithfulness observed also challenges approaches seeking explanation or contestability solely via verbal protocols rather than mediated structural or behavioral models.

Future Directions

The evidence is currently limited to synthetic, stylized binary decision settings. Extending the methodology to more complex, real-world decision-making, richer attribute semantics, and tasks involving multi-step reasoning or compositional world-models could further clarify the limits of LLM belief-like behavior. Advancing introspectable architectures or self-knowledge calibration methods also emerges as a key research frontier.

Conclusion

This study demonstrates that LLM choices in structured trade-off tasks reflect systematic, probabilistically local priority structures, supporting a weak notion of behavioral “belief.” However, LLMs’ explicit rationalizations—whether direct or score-based—only partially capture these internal drivers. This dissociation has critical implications for interpretability, contestability, and the broader debate on belief ascription in artificial agents. Further research is warranted to generalize these insights to less stylized, more open-ended reasoning domains and to develop methods bridging the gap between behavioral structure and explicit self-explanation.