Closed-Source Model Generalization

Determine whether the modes-first convergence, tail variability, scale discordance, and prompt-sensitivity dynamics observed for open-weight large language models also hold in closed-source systems.

Background

The study evaluates nine open-weight LLMs from the Llama, Qwen, and Mistral families, using elicited probability distributions over discrete preference choices in travel, restaurant, and product domains. Its principal findings—including rapid stabilization of high-probability modes, persistent tail variability, and substantial discordance across model scales and families—are therefore established only for open-weight systems.

The authors note that closed-source APIs apply provider-side post-processing, which can confound direct distributional measurement. Consequently, whether the reported distributional dynamics generalize to closed-source models remains unresolved and is identified as an open question.