Origin of LLM probability judgment incoherence in autoregression

Establish whether the mechanism by which GPT-4, GPT-3.5-turbo, LLaMA-2-70b, and LLaMA-2-7b generate probability judgments originates from the implementation of the autoregressive training objective used in these models.

Background

The paper shows that LLMs (GPT-4, GPT-3.5-turbo, LLaMA-2-70b, LLaMA-2-7b) produce probability judgments that are often incoherent when evaluated via probabilistic identities, exhibiting human-like deviations from zero. It also finds an inverted-U-shaped mean-variance relationship in repeated judgments, paralleling human data.

To explain these patterns, the authors compare two models of human probability judgment—Probability Theory plus Noise (PT+N) and the Bayesian Sampler—and argue that LLM responses align more closely with the Bayesian Sampler model. They outline a theoretical link between the autoregressive objective in LLMs and implicit Bayesian inference using de Finetti’s theorem, suggesting that the mechanism for forming probability judgments could be driven by autoregression.

The conjecture focuses on the causal origin of the observed incoherence patterns and posits that they arise from the autoregressive training objective. Resolving this would clarify whether the training objective itself induces the statistical structures observed in LLM probability judgments.