---
title: 'BayesBench: Benchmarking LLM Bayesian Behavior'
url: https://www.emergentmind.com/topics/bayesbench
type: topic
---

# BayesBench: Benchmarking LLM Bayesian Behavior

BayesBench is the name used by two recent benchmark proposals for evaluating whether large language models exhibit Bayesian behavior under uncertainty. One BayesBench is a psychophysics-inspired benchmark for multimodal magnitude estimation and cue combination, designed to test whether models handle noisy text and image evidence in Bayes-consistent ways [2512.02719]. The other is a benchmark suite for multi-turn evidence accumulation, designed to compare LLM belief trajectories against Bayesian posterior updates in sequential inference settings [2606.30850]. In both cases, the object of evaluation is not only final-answer accuracy but the structure of belief revision itself: how a model integrates evidence, adapts to noise and context, and uses latent inference for downstream prediction.

## 1. Scope and conceptual orientation

BayesBench, in current usage, refers not to a single canonical benchmark but to two closely related LLM-evaluation programs centered on Bayesian rationality. The psychophysics-oriented benchmark asks whether models perform optimal cue combination and show Bayes-consistent behavioral shifts in controlled magnitude-estimation tasks over text and image. The multi-turn benchmark asks whether models update beliefs about hidden quantities in a way that matches a rational Bayesian reasoner when evidence arrives sequentially in conversation [2512.02719; 2606.30850].

The shared premise is that uncertainty handling should be evaluated as a process property rather than a terminal property. In the psychophysics setting, this means measuring whether modality weights shift with reliability, whether context alters prior reliance, and whether instruction-based priors induce the expected directional changes. In the sequential setting, it means eliciting a model’s belief after each turn and comparing the entire belief trajectory to a Bayesian posterior or posterior predictive trajectory. This suggests a common BayesBench agenda: controlled tests of latent inference, evidence accumulation, and uncertainty-sensitive prediction under repeated interaction.

Both benchmark lines also reject the sufficiency of raw accuracy as a standalone metric. The psychophysics paper explicitly argues that capability and strategy can dissociate, since a model can achieve near-perfect text accuracy yet fail to integrate visual cues efficiently. The multi-turn paper similarly shows that gains in latent inference do not reliably carry over to downstream prediction, exposing a gap between inferring latent structure and using it to rationally update beliefs about a target outcome [2512.02719; 2606.30850].

## 2. Psychophysics BayesBench

The psychophysics BayesBench introduces four magnitude-estimation tasks: marker location estimation, line ratio estimation, maze distance estimation, and duration estimation. The first three tasks are multimodal, with text-only, image-only, and text+image conditions; the duration task is text-based and uses transcript extracts from the AMI Meeting Corpus. For text-only presentation, line-ratio and marker-location are represented using ASCII, while maze distance is presented as a concise text description [2512.02719].

The benchmark organizes each estimation task into three sessions—short, medium, and long—using stimulus values drawn from different but overlapping ranges under a range-uniform prior. A rolling context of prior trials, including previous stimuli and the model’s own previous responses, is supplied to emulate memory in stateless API settings. The benchmark further includes ablations over steering, noise, and context. Steering manipulations include a verbal prompt instructing the model to behave like a Bayesian observer and a numerical range prompt that provides a prior-like range of previous observations. Noise manipulations apply Gaussian blur to the image modality, either at a constant level or in a gradual sequence. Context manipulations vary the context window and reverse stimulus order [2512.02719].

The benchmark’s behavioral layer uses both linear and Bayesian observer models. Its static Bayesian observer is defined by
\[
\mu_t = \frac{\tau_x}{\tau_x+\tau_p} x_t + \frac{\tau_p}{\tau_x+\tau_p} \mu_p,\quad y_t \sim \mathcal{N}(\mu_t, \sigma^2_{\text{dec}})
\]
with prior weight
\[
w_{\text{prior}}=\frac{\tau_p}{\tau_p+\tau_x}.
\]
It also includes a sequential Bayesian observer in Kalman-filter form, with prediction
\[
\mu_{t|t-1}=\mu_{t-1|t-1},\quad P_{t|t-1}=P_{t-1|t-1}+q
\]
and update
\[
K_t=\frac{P_{t|t-1}}{P_{t|t-1}+r},\quad \mu_{t|t}=\mu_{t|t-1}+K_t(x_t-\mu_{t|t-1}),\quad P_{t|t}=(1-K_t)P_{t|t-1}.
\]
Cue combination is benchmarked against equal weighting, fitted linear weighting, and inverse-variance Bayes-optimal fusion, with
\[
w_{\text{text}}=\frac{1/\sigma^2_{\text{text}}}{1/\sigma^2_{\text{text}}+1/\sigma^2_{\text{image}}}.
\]
An oracle Bayes cue-combination baseline is also defined through generalized least squares after calibrating each modality estimate to ground truth [2512.02719].

The benchmark reports three principal metrics. Accuracy is measured by normalized RMSE,
\[
\mathrm{NRMSE}=\mathrm{RMSE}_{\text{LLM}}/\mathrm{RMSE}_{\text{baseline}},
\]
where the baseline is a constant predictor that outputs the mean of the stimulus range. Cue-combination efficiency is measured by relative response efficiency,
\[
\mathrm{RRE}(m_{\text{ref}})=\mathrm{NRMSE}_{\text{ref}}/\mathrm{NRMSE}_{\text{LLM}}.
\]
Its main methodological novelty is the Bayesian Consistency Score, designed to detect Bayes-consistent directional shifts even when accuracy saturates. For each ablation,
\[
\Delta w_{\text{prior}}=w_{\text{prior}}^{(\text{ablation})}-w_{\text{prior}}^{(\text{base})},
\]
and
\[
s_a=\begin{cases}
+1 & \text{if }\Delta w_{\text{prior}}\ge 0,\\
-1 & \text{if }\Delta w_{\text{prior}}<0,
\end{cases}
\qquad \text{with } s_a=0 \text{ if } w_{\text{prior}}^{(\text{ablation})}>0.9.
\]
The score is then
\[
\mathrm{BCS}=\sum_a s_a,
\]
with range
\[
\mathrm{BCS}_{\min}=-15,\qquad \mathrm{BCS}_{\max}=15.
\]
The overall BayesBench score combines accuracy, efficiency, and consistency:
\[
S_{\text{BayesBench}}=\tfrac{1}{3}(A+E+C).
\]
The benchmark evaluates nine models—Claude 3.7 Sonnet, GPT-5 Mini, GPT-4o, Llama-4 Maverick, Qwen 2.5 VL 32B, Mistral 24B, Gemini 2.5 Flash Lite, Phi 4 Multimodal, and Gemma 3 4B—and also includes human judgments for calibration on the main tasks and selected ablations [2512.02719].

## 3. Multi-turn evidence-accumulation BayesBench

The multi-turn BayesBench formalizes conversational inference as sequential Bayesian updating over hidden quantities. It introduces three task families: Bayesian estimation, Bayesian prediction, and latent-framed Bayesian prediction. These are instantiated in four environments: coin flip, recommender system, social judgment, and medical triage [2606.30850].

Its common generative template defines a hidden quantity \(H\), a fixed environment specification \(\eta\), sequential observations \(X_t\), and, when applicable, a prediction target \(Y\):
\[
\begin{aligned}
H &\sim \pi_\eta, \\
Y \mid H = h &\sim g_\eta(\cdot \mid h), \\
X_t \mid H = h,\, X_{1:t-1} = x_{1:t-1} &\sim \ell_\eta(\cdot \mid h,\, x_{1:t-1}).
\end{aligned}
\]
The joint density factorizes as
\[
p_\eta(h,y,x_{1:T})=\pi_\eta(h)\, g_\eta(y\mid h)\, \prod_{t=1}^T \ell_\eta(x_t\mid h,x_{1:t-1}).
\]
The benchmark compares elicited LLM beliefs \(p_{\mathrm{LLM}}(\cdot\mid x_{1:t})\) against the corresponding Bayesian posterior or posterior predictive \(p_\eta(\cdot\mid x_{1:t})\), thereby making the entire belief trajectory the object of evaluation rather than only the final turn [2606.30850].

In the Bayesian-estimation family, the coin-flip environment uses a Beta-Bernoulli model:
\[
\Theta \sim \mathrm{Beta}(1,1), \qquad X_t\mid \Theta=\theta \sim \mathrm{Bernoulli}(\theta).
\]
With \(n_h=\sum_{i\le t}x_i\), the Bayesian posterior is
\[
p_\eta(\theta\mid x_{1:t})=\mathrm{Beta}(1+n_h,1+t-n_h),
\]
with posterior mean
\[
\mathbb{E}_{p_\eta}[\theta\mid x_{1:t}]=\frac{1+n_h}{2+t}.
\]
The model is probed by asking it to predict the next flip, so the probability assigned to heads is interpreted as the current estimate of the posterior mean bias. This yields a closed-form normative reference at every turn [2606.30850].

In the Bayesian-prediction family, the recommender-system environment introduces a latent categorical user type \(Z\in\{1,\ldots,K\}\), with \(K=4\), and a held-out rating target \(Y\). Observations are sequential ratings on fixed items:
\[
Z \sim \mathrm{Categorical}(\pi), \qquad X_t\mid Z=k \sim \mathrm{Categorical}(\phi_{m_t,k}), \qquad Y\mid Z=k \sim \mathrm{Categorical}(\phi_{m^\star,k}).
\]
The exact Bayesian reference for type inference is
\[
p_\eta(z = k \mid x_{1:t}) = \frac{\pi_k \prod_{i=1}^{t} \phi_{m_i,k}(r_i)} {\sum_{\ell=1}^{K} \pi_{\ell} \prod_{i=1}^{t} \phi_{m_i,\ell}(r_i)},
\]
and the posterior predictive over held-out ratings is
\[
p_\eta(y = r \mid x_{1:t},\, m^\star) = \sum_{k=1}^{K} p_\eta(z = k \mid x_{1:t})\,\phi_{m^\star,k}(r).
\]
This environment directly separates latent inference quality from downstream predictive use [2606.30850].

In the latent-framed Bayesian-prediction family, the hidden quantity is \(H=(Z_s,Z_f)\), where \(Z_s\) is a latent state and \(Z_f\) is a framing or persona variable. The state generates a clean signal \(\tilde X_t\) and the target \(Y\), while the framing variable transforms \(\tilde X_t\) into the observed \(X_t\). The key invariance is
\[
Y \perp Z_f.
\]
The social-judgment environment uses latent scenario and communication-style variables to predict Reddit verdicts \(\{\text{YTA},\text{NTA}\}\); the medical-triage environment uses latent clinical condition and communication style to predict urgency labels \(\{\text{Emergency},\text{Urgent},\text{Observation},\text{Routine}\}\). In both environments the benchmark distinguishes passive observation of evidence from active engagement with a user simulator, allowing belief trajectories to be compared under interaction and under passive receipt of the same underlying evidence [2606.30850].

The benchmark evaluates seven open-weight LLMs from 3B to 70B—LLaMA 3 3B, 8B, and 70B, and Qwen 2.5 3B, 7B, 14B, and 32B. Beliefs are elicited with multiple-choice probes, and a position-bias correction is applied by cyclic rotation of answer options. Performance is quantified by turn-wise total variation distance,
\[
\mathrm{TVD}_t(\cdot)=\frac{1}{2}\left\|p_{\mathrm{LLM}}(\cdot\mid x_{1:t})-p_\eta(\cdot\mid x_{1:t})\right\|_1,
\]
and mean absolute error,
\[
\mathrm{MAE}_t(\cdot)=\left|\mathbb{E}_{p_{\mathrm{LLM}}}[\cdot\mid x_{1:t}]-\mathbb{E}_{p_\eta}[\cdot\mid x_{1:t}]\right|.
\]
In environments without a closed-form posterior reference, the benchmark substitutes the ground-truth value in place of \(\mathbb{E}_{p_\eta}[\cdot]\) [2606.30850].

## 4. Empirical findings

The psychophysics BayesBench reports that strong models often adapt in Bayes-consistent ways, but accuracy does not guarantee robustness. The paper’s main dissociation example is GPT-5 Mini: in the maze-distance task its text-only NRMSE is nearly perfect at \(0.01\), while image-only NRMSE is \(0.2\), and yet the multimodal response still reflects image influence rather than near-zero image weight under Bayes-optimal fusion. By contrast, Llama-4 Maverick attains the highest BayesBench score, achieves the best multimodal NRMSE across all tasks, and outperforms the Bayesian reliability-weighted unbiased linear combination baseline. The paper interprets this as evidence that the model may be exploiting nonlinear cue integration beyond the benchmark’s linear Bayesian baseline. More generally, image conditions show higher Bayesian factor evidence than text, multimodal gains are not automatic, and smaller models such as Gemma 3 4B and Phi 4 Multimodal can exhibit relatively decent BCS despite lower accuracy [2512.02719].

The multi-turn BayesBench reports a different but related pattern. In the coin-flip environment, scaling improves separation between different true biases, but larger models often over-update and become too extreme relative to the Bayesian posterior, while smaller models under-update and remain too close to the middle. In the recommender system, type-posterior TVD improves with scale—LLaMA 3B at \(61.4 \pm 0.8\), Qwen 32B at \(47.3 \pm 3.8\), and LLaMA 70B at \(45.4 \pm 3.7\)—but rating-predictive TVD does not improve correspondingly. In the social-judgment environment, active engagement shifts beliefs toward NTA relative to passive observation, producing lower error on NTA cases and higher error on YTA cases. In medical triage, larger models infer communication profiles more accurately, sometimes reaching about \(0.80\) probability on the true profile by mid-conversation, but urgency prediction remains poorly calibrated in the middle categories, with Urgent often pushed upward into Emergency and Observation pushed toward either Urgent or Routine [2606.30850].

Taken together, these empirical results place two recurrent failure modes at the center of BayesBench. The first is a capability–strategy dissociation: a model may be highly accurate in one modality or on final outputs while still exhibiting poor reliability weighting, poor calibration, or non-Bayesian shifts under perturbation. The second is a latent-inference–prediction dissociation: a model may recover hidden structure, such as user type or patient communication style, but fail to translate that latent inference into normatively correct downstream predictions. Both benchmark lines therefore support a process-oriented interpretation of Bayesian behavior: what matters is not only whether a model can infer or predict, but whether the transformation from evidence to belief to action preserves Bayesian structure [2512.02719; 2606.30850].

## 5. Methodological significance and relation to other benchmark traditions

BayesBench belongs to a broader literature on Bayesian benchmarking, but its object of evaluation is distinctive. URSABench evaluates approximate Bayesian inference methods in deep neural network classification along the axes of uncertainty, robustness, scalability, and accuracy, with tasks such as in-domain prediction, misclassification detection, OOD detection, and decision-making under asymmetric cost [2007.04466]. posteriordb provides a database of posterior targets, models, datasets, and reference posterior draws for testing and developing Bayesian inference algorithms across a wide range of posterior geometries [2407.04967]. “Benchmarking for Bayesian Reinforcement Learning” defines performance criteria over large sets of MDPs drawn from prior and test distributions and explicitly incorporates offline and online computation time [1509.04064].

BayesBench differs from those resources by centering the evaluation target on behavioral Bayesianity in LLMs. The psychophysics BayesBench is closer to a controlled cognitive benchmark: it manipulates noise, context, and prompt-induced priors to test whether cue weighting and prior reliance shift in the Bayes-consistent direction. The multi-turn BayesBench is closer to a sequential inference benchmark: it treats conversation as a stream of evidence and evaluates whether elicited beliefs track posterior updates over time. This suggests that BayesBench extends Bayesian benchmarking from algorithms over formal posterior targets to agent-like systems whose outputs must be interpreted as evolving beliefs under partial information [2512.02719; 2606.30850].

Methodologically, the two BayesBench proposals also share design choices that align with stronger benchmark practice. Both construct controlled environments in which the latent structure is known by design. Both distinguish intermediate inference quality from downstream performance. Both introduce evaluation procedures that remain informative when accuracy saturates or when final-turn correctness alone obscures systematic deviations. The psychophysics benchmark does this via BCS, which scores directional adaptation under ablation. The multi-turn benchmark does it via turn-wise TVD and MAE, together with conditional-latent interventions that test whether explicit access to inferred hidden structure improves prediction [2512.02719; 2606.30850].

A plausible implication is that BayesBench fills a gap left by conventional LLM leaderboards. Standard multimodal or reasoning benchmarks often emphasize correctness under fixed prompts, but BayesBench-style evaluations target epistemic dynamics: uncertainty tracking, evidence integration, latent-state recovery, and invariance to style or delivery format. In that sense, BayesBench occupies an intermediate position between cognitive psychophysics, Bayesian ideal-observer analysis, and benchmark engineering for modern foundation models.

## 6. Limitations and open directions

The psychophysics BayesBench is explicit about several limitations: bounded task ranges, limited perturbation scope, API dependence, synthetic task construction, and a behavioral rather than mechanistic interpretation of results. Its use of controlled synthetic tasks is deliberate, but this also limits immediate transfer to naturalistic multimodal settings. The paper accordingly presents BayesBench as a complement to standard multimodal benchmarks rather than a replacement for them [2512.02719].

The multi-turn BayesBench likewise has important scope limits. Exact Bayesian references are only available in the coin-flip and recommender environments. In social judgment and medical triage, the benchmark relies on invariance tests, conditioning interventions, and ground-truth targets rather than exact posterior comparison. Belief measurement depends on elicitation format, the latent structures are deliberately simplified, and active-engagement experiments depend on user simulators with fixed style instructions rather than unconstrained human interaction [2606.30850].

Across both benchmark lines, the central open problem is the relation between elicited beliefs and internal computation. BayesBench evaluates whether behavior is Bayes-consistent, not how any such behavior is implemented. This leaves open whether observed improvements with scale reflect more faithful posterior computation, stronger pattern matching to benchmark structure, or post-training effects that sharpen confidence without improving normative calibration. The repeated appearance of over-updating, extreme-category attraction, and style-sensitive interaction suggests that post-training and conversational framing remain central obstacles to robust Bayesian behavior [2512.02719; 2606.30850].

A broader implication is that future BayesBench work will likely need richer latent structures, stronger naturalistic environments, and tighter links between belief elicitation and internal state. The psychophysics line points toward more realistic multimodal uncertainty tests; the sequential line points toward persistent user models, memory, and structured latent-state tracking in conversation. Both lines indicate that evaluating LLMs as Bayesian reasoners requires process-level benchmarks with explicit uncertainty semantics, not only task-level correctness.

Source: https://www.emergentmind.com/topics/bayesbench