Cross-model generality of the prior-substitution curve

Determine whether the prior-substitution curve has the same shape across other language-model families and model scales, including whether its slope is consistent across models.

Background

The empirical analysis is based primarily on a single eight-billion-parameter model, Llama-3.1-8B-Instruct. Although a parallel experiment with Qwen2.5-1.5B-Instruct was conducted, its near-constant recommendations and use of an earlier prompt make it unsuitable as a directly comparable replication.

The unresolved issue is whether the observed increase in identity-driven variation as financial disclosure decreases is specific to Llama-3.1-8B-Instruct or reflects a broader property of LLMs. In particular, the authors distinguish the level of the curve, which may vary across models, from its slope, whose cross-model behavior remains unresolved.

References

Whether the curve has the same shape in other families and at other scales is open. The finance-specific audits of \citet{foltyn2026worth} and \citet{agliata2026generative}, which span many models but hold the context fixed, suggest that demographic sensitivity varies considerably across models, which is a reason to expect the level of the curve to differ and an open question for its slope.

— Thin Evidence, Thick Priors: How Language Models Substitute Identity for Missing Financial Facts  (2610.07798 - Khetan et al., 6 Oct 2026) in Section Threats to Validity, paragraph “One model” (Section 7)