BayesBench: Benchmarking LLM Bayesian Behavior

• BayesBench is a dual benchmark suite that evaluates LLMs’ Bayesian behavior using controlled multimodal and sequential inference tasks.
• It tests evidence accumulation, cue combination, and belief revision through psychophysics-inspired and multi-turn conversational environments.
• Findings highlight a notable dissociation between final-answer accuracy and the proper integration of uncertainty and latent inference.

BayesBench is the name used by two recent benchmark proposals for evaluating whether LLMs exhibit Bayesian behavior under uncertainty. One BayesBench is a psychophysics-inspired benchmark for multimodal magnitude estimation and cue combination, designed to test whether models handle noisy text and image evidence in Bayes-consistent ways (Ma et al., 2 Dec 2025). The other is a benchmark suite for multi-turn evidence accumulation, designed to compare LLM belief trajectories against Bayesian posterior updates in sequential inference settings (Samanta et al., 29 Jun 2026). In both cases, the object of evaluation is not only final-answer accuracy but the structure of belief revision itself: how a model integrates evidence, adapts to noise and context, and uses latent inference for downstream prediction.

1. Scope and conceptual orientation

BayesBench, in current usage, refers not to a single canonical benchmark but to two closely related LLM-evaluation programs centered on Bayesian rationality. The psychophysics-oriented benchmark asks whether models perform optimal cue combination and show Bayes-consistent behavioral shifts in controlled magnitude-estimation tasks over text and image. The multi-turn benchmark asks whether models update beliefs about hidden quantities in a way that matches a rational Bayesian reasoner when evidence arrives sequentially in conversation (Ma et al., 2 Dec 2025, Samanta et al., 29 Jun 2026).

The shared premise is that uncertainty handling should be evaluated as a process property rather than a terminal property. In the psychophysics setting, this means measuring whether modality weights shift with reliability, whether context alters prior reliance, and whether instruction-based priors induce the expected directional changes. In the sequential setting, it means eliciting a model’s belief after each turn and comparing the entire belief trajectory to a Bayesian posterior or posterior predictive trajectory. This suggests a common BayesBench agenda: controlled tests of latent inference, evidence accumulation, and uncertainty-sensitive prediction under repeated interaction.

Both benchmark lines also reject the sufficiency of raw accuracy as a standalone metric. The psychophysics paper explicitly argues that capability and strategy can dissociate, since a model can achieve near-perfect text accuracy yet fail to integrate visual cues efficiently. The multi-turn paper similarly shows that gains in latent inference do not reliably carry over to downstream prediction, exposing a gap between inferring latent structure and using it to rationally update beliefs about a target outcome (Ma et al., 2 Dec 2025, Samanta et al., 29 Jun 2026).

2. Psychophysics BayesBench

The psychophysics BayesBench introduces four magnitude-estimation tasks: marker location estimation, line ratio estimation, maze distance estimation, and duration estimation. The first three tasks are multimodal, with text-only, image-only, and text+image conditions; the duration task is text-based and uses transcript extracts from the AMI Meeting Corpus. For text-only presentation, line-ratio and marker-location are represented using ASCII, while maze distance is presented as a concise text description (Ma et al., 2 Dec 2025).

The benchmark organizes each estimation task into three sessions—short, medium, and long—using stimulus values drawn from different but overlapping ranges under a range-uniform prior. A rolling context of prior trials, including previous stimuli and the model’s own previous responses, is supplied to emulate memory in stateless API settings. The benchmark further includes ablations over steering, noise, and context. Steering manipulations include a verbal prompt instructing the model to behave like a Bayesian observer and a numerical range prompt that provides a prior-like range of previous observations. Noise manipulations apply Gaussian blur to the image modality, either at a constant level or in a gradual sequence. Context manipulations vary the context window and reverse stimulus order (Ma et al., 2 Dec 2025).

The benchmark’s behavioral layer uses both linear and Bayesian observer models. Its static Bayesian observer is defined by

$$\mu_t = \frac{\tau_x}{\tau_x+\tau_p} x_t + \frac{\tau_p}{\tau_x+\tau_p} \mu_p,\quad y_t \sim \mathcal{N}(\mu_t, \sigma^2_{\text{dec}})$$

with prior weight

$w_{\text{prior}}=\frac{\tau_p}{\tau_p+\tau_x}.$

It also includes a sequential Bayesian observer in Kalman-filter form, with prediction

$$\mu_{t|t-1}=\mu_{t-1|t-1},\quad P_{t|t-1}=P_{t-1|t-1}+q$$

and update

$$K_t=\frac{P_{t|t-1}}{P_{t|t-1}+r},\quad \mu_{t|t}=\mu_{t|t-1}+K_t(x_t-\mu_{t|t-1}),\quad P_{t|t}=(1-K_t)P_{t|t-1}.$$

Cue combination is benchmarked against equal weighting, fitted linear weighting, and inverse-variance Bayes-optimal fusion, with

$$w_{\text{text}}=\frac{1/\sigma^2_{\text{text}}}{1/\sigma^2_{\text{text}}+1/\sigma^2_{\text{image}}}.$$

An oracle Bayes cue-combination baseline is also defined through generalized least squares after calibrating each modality estimate to ground truth (Ma et al., 2 Dec 2025).

The benchmark reports three principal metrics. Accuracy is measured by normalized RMSE,

$$\mathrm{NRMSE}=\mathrm{RMSE}_{\text{LLM}}/\mathrm{RMSE}_{\text{baseline}},$$

where the baseline is a constant predictor that outputs the mean of the stimulus range. Cue-combination efficiency is measured by relative response efficiency,

$$\mathrm{RRE}(m_{\text{ref}})=\mathrm{NRMSE}_{\text{ref}}/\mathrm{NRMSE}_{\text{LLM}}.$$

Its main methodological novelty is the Bayesian Consistency Score, designed to detect Bayes-consistent directional shifts even when accuracy saturates. For each ablation,

$$\Delta w_{\text{prior}}=w_{\text{prior}}^{(\text{ablation})}-w_{\text{prior}}^{(\text{base})},$$

and

$$s_a=\begin{cases} +1 & \text{if }\Delta w_{\text{prior}}\ge 0,\ -1 & \text{if }\Delta w_{\text{prior}}<0, \end{cases} \qquad \text{with } s_a=0 \text{ if } w_{\text{prior}}^{(\text{ablation})}>0.9.$$

The score is then

$\mathrm{BCS}=\sum_a s_a,$

with range

$w_{\text{prior}}=\frac{\tau_p}{\tau_p+\tau_x}.$0

The overall BayesBench score combines accuracy, efficiency, and consistency: $w_{\text{prior}}=\frac{\tau_p}{\tau_p+\tau_x}.$1 The benchmark evaluates nine models—Claude 3.7 Sonnet, GPT-5 Mini, GPT-4o, Llama-4 Maverick, Qwen 2.5 VL 32B, Mistral 24B, Gemini 2.5 Flash Lite, Phi 4 Multimodal, and Gemma 3 4B—and also includes human judgments for calibration on the main tasks and selected ablations (Ma et al., 2 Dec 2025).

3. Multi-turn evidence-accumulation BayesBench

The multi-turn BayesBench formalizes conversational inference as sequential Bayesian updating over hidden quantities. It introduces three task families: Bayesian estimation, Bayesian prediction, and latent-framed Bayesian prediction. These are instantiated in four environments: coin flip, recommender system, social judgment, and medical triage (Samanta et al., 29 Jun 2026).

Its common generative template defines a hidden quantity $w_{\text{prior}}=\frac{\tau_p}{\tau_p+\tau_x}.$2, a fixed environment specification $w_{\text{prior}}=\frac{\tau_p}{\tau_p+\tau_x}.$3, sequential observations $w_{\text{prior}}=\frac{\tau_p}{\tau_p+\tau_x}.$4, and, when applicable, a prediction target $w_{\text{prior}}=\frac{\tau_p}{\tau_p+\tau_x}.$5: $w_{\text{prior}}=\frac{\tau_p}{\tau_p+\tau_x}.$6 The joint density factorizes as

$w_{\text{prior}}=\frac{\tau_p}{\tau_p+\tau_x}.$7

The benchmark compares elicited LLM beliefs $w_{\text{prior}}=\frac{\tau_p}{\tau_p+\tau_x}.$8 against the corresponding Bayesian posterior or posterior predictive $w_{\text{prior}}=\frac{\tau_p}{\tau_p+\tau_x}.$9, thereby making the entire belief trajectory the object of evaluation rather than only the final turn (Samanta et al., 29 Jun 2026).

In the Bayesian-estimation family, the coin-flip environment uses a Beta-Bernoulli model: $$\mu_{t|t-1}=\mu_{t-1|t-1},\quad P_{t|t-1}=P_{t-1|t-1}+q$$0 With $$\mu_{t|t-1}=\mu_{t-1|t-1},\quad P_{t|t-1}=P_{t-1|t-1}+q$$1, the Bayesian posterior is

$$\mu_{t|t-1}=\mu_{t-1|t-1},\quad P_{t|t-1}=P_{t-1|t-1}+q$$2

with posterior mean

$$\mu_{t|t-1}=\mu_{t-1|t-1},\quad P_{t|t-1}=P_{t-1|t-1}+q$$3

The model is probed by asking it to predict the next flip, so the probability assigned to heads is interpreted as the current estimate of the posterior mean bias. This yields a closed-form normative reference at every turn (Samanta et al., 29 Jun 2026).

In the Bayesian-prediction family, the recommender-system environment introduces a latent categorical user type $$\mu_{t|t-1}=\mu_{t-1|t-1},\quad P_{t|t-1}=P_{t-1|t-1}+q$$4, with $$\mu_{t|t-1}=\mu_{t-1|t-1},\quad P_{t|t-1}=P_{t-1|t-1}+q$$5, and a held-out rating target $$\mu_{t|t-1}=\mu_{t-1|t-1},\quad P_{t|t-1}=P_{t-1|t-1}+q$$6. Observations are sequential ratings on fixed items: $$\mu_{t|t-1}=\mu_{t-1|t-1},\quad P_{t|t-1}=P_{t-1|t-1}+q$$7 The exact Bayesian reference for type inference is

$$\mu_{t|t-1}=\mu_{t-1|t-1},\quad P_{t|t-1}=P_{t-1|t-1}+q$$8

and the posterior predictive over held-out ratings is

$$\mu_{t|t-1}=\mu_{t-1|t-1},\quad P_{t|t-1}=P_{t-1|t-1}+q$$9

This environment directly separates latent inference quality from downstream predictive use (Samanta et al., 29 Jun 2026).

In the latent-framed Bayesian-prediction family, the hidden quantity is $$K_t=\frac{P_{t|t-1}}{P_{t|t-1}+r},\quad \mu_{t|t}=\mu_{t|t-1}+K_t(x_t-\mu_{t|t-1}),\quad P_{t|t}=(1-K_t)P_{t|t-1}.$$0, where $$K_t=\frac{P_{t|t-1}}{P_{t|t-1}+r},\quad \mu_{t|t}=\mu_{t|t-1}+K_t(x_t-\mu_{t|t-1}),\quad P_{t|t}=(1-K_t)P_{t|t-1}.$$1 is a latent state and $$K_t=\frac{P_{t|t-1}}{P_{t|t-1}+r},\quad \mu_{t|t}=\mu_{t|t-1}+K_t(x_t-\mu_{t|t-1}),\quad P_{t|t}=(1-K_t)P_{t|t-1}.$$2 is a framing or persona variable. The state generates a clean signal $$K_t=\frac{P_{t|t-1}}{P_{t|t-1}+r},\quad \mu_{t|t}=\mu_{t|t-1}+K_t(x_t-\mu_{t|t-1}),\quad P_{t|t}=(1-K_t)P_{t|t-1}.$$3 and the target $$K_t=\frac{P_{t|t-1}}{P_{t|t-1}+r},\quad \mu_{t|t}=\mu_{t|t-1}+K_t(x_t-\mu_{t|t-1}),\quad P_{t|t}=(1-K_t)P_{t|t-1}.$$4, while the framing variable transforms $$K_t=\frac{P_{t|t-1}}{P_{t|t-1}+r},\quad \mu_{t|t}=\mu_{t|t-1}+K_t(x_t-\mu_{t|t-1}),\quad P_{t|t}=(1-K_t)P_{t|t-1}.$$5 into the observed $$K_t=\frac{P_{t|t-1}}{P_{t|t-1}+r},\quad \mu_{t|t}=\mu_{t|t-1}+K_t(x_t-\mu_{t|t-1}),\quad P_{t|t}=(1-K_t)P_{t|t-1}.$$6. The key invariance is

$$K_t=\frac{P_{t|t-1}}{P_{t|t-1}+r},\quad \mu_{t|t}=\mu_{t|t-1}+K_t(x_t-\mu_{t|t-1}),\quad P_{t|t}=(1-K_t)P_{t|t-1}.$$7

The social-judgment environment uses latent scenario and communication-style variables to predict Reddit verdicts $$K_t=\frac{P_{t|t-1}}{P_{t|t-1}+r},\quad \mu_{t|t}=\mu_{t|t-1}+K_t(x_t-\mu_{t|t-1}),\quad P_{t|t}=(1-K_t)P_{t|t-1}.$$8; the medical-triage environment uses latent clinical condition and communication style to predict urgency labels $$K_t=\frac{P_{t|t-1}}{P_{t|t-1}+r},\quad \mu_{t|t}=\mu_{t|t-1}+K_t(x_t-\mu_{t|t-1}),\quad P_{t|t}=(1-K_t)P_{t|t-1}.$$9. In both environments the benchmark distinguishes passive observation of evidence from active engagement with a user simulator, allowing belief trajectories to be compared under interaction and under passive receipt of the same underlying evidence (Samanta et al., 29 Jun 2026).

The benchmark evaluates seven open-weight LLMs from 3B to 70B—LLaMA 3 3B, 8B, and 70B, and Qwen 2.5 3B, 7B, 14B, and 32B. Beliefs are elicited with multiple-choice probes, and a position-bias correction is applied by cyclic rotation of answer options. Performance is quantified by turn-wise total variation distance,

$$w_{\text{text}}=\frac{1/\sigma^2_{\text{text}}}{1/\sigma^2_{\text{text}}+1/\sigma^2_{\text{image}}}.$$0

and mean absolute error,

$$w_{\text{text}}=\frac{1/\sigma^2_{\text{text}}}{1/\sigma^2_{\text{text}}+1/\sigma^2_{\text{image}}}.$$1

In environments without a closed-form posterior reference, the benchmark substitutes the ground-truth value in place of $$w_{\text{text}}=\frac{1/\sigma^2_{\text{text}}}{1/\sigma^2_{\text{text}}+1/\sigma^2_{\text{image}}}.$$2 (Samanta et al., 29 Jun 2026).

4. Empirical findings

The psychophysics BayesBench reports that strong models often adapt in Bayes-consistent ways, but accuracy does not guarantee robustness. The paper’s main dissociation example is GPT-5 Mini: in the maze-distance task its text-only NRMSE is nearly perfect at $$w_{\text{text}}=\frac{1/\sigma^2_{\text{text}}}{1/\sigma^2_{\text{text}}+1/\sigma^2_{\text{image}}}.$$3, while image-only NRMSE is $$w_{\text{text}}=\frac{1/\sigma^2_{\text{text}}}{1/\sigma^2_{\text{text}}+1/\sigma^2_{\text{image}}}.$$4, and yet the multimodal response still reflects image influence rather than near-zero image weight under Bayes-optimal fusion. By contrast, Llama-4 Maverick attains the highest BayesBench score, achieves the best multimodal NRMSE across all tasks, and outperforms the Bayesian reliability-weighted unbiased linear combination baseline. The paper interprets this as evidence that the model may be exploiting nonlinear cue integration beyond the benchmark’s linear Bayesian baseline. More generally, image conditions show higher Bayesian factor evidence than text, multimodal gains are not automatic, and smaller models such as Gemma 3 4B and Phi 4 Multimodal can exhibit relatively decent BCS despite lower accuracy (Ma et al., 2 Dec 2025).

The multi-turn BayesBench reports a different but related pattern. In the coin-flip environment, scaling improves separation between different true biases, but larger models often over-update and become too extreme relative to the Bayesian posterior, while smaller models under-update and remain too close to the middle. In the recommender system, type-posterior TVD improves with scale—LLaMA 3B at $$w_{\text{text}}=\frac{1/\sigma^2_{\text{text}}}{1/\sigma^2_{\text{text}}+1/\sigma^2_{\text{image}}}.$$5, Qwen 32B at $$w_{\text{text}}=\frac{1/\sigma^2_{\text{text}}}{1/\sigma^2_{\text{text}}+1/\sigma^2_{\text{image}}}.$$6, and LLaMA 70B at $$w_{\text{text}}=\frac{1/\sigma^2_{\text{text}}}{1/\sigma^2_{\text{text}}+1/\sigma^2_{\text{image}}}.$$7—but rating-predictive TVD does not improve correspondingly. In the social-judgment environment, active engagement shifts beliefs toward NTA relative to passive observation, producing lower error on NTA cases and higher error on YTA cases. In medical triage, larger models infer communication profiles more accurately, sometimes reaching about $$w_{\text{text}}=\frac{1/\sigma^2_{\text{text}}}{1/\sigma^2_{\text{text}}+1/\sigma^2_{\text{image}}}.$$8 probability on the true profile by mid-conversation, but urgency prediction remains poorly calibrated in the middle categories, with Urgent often pushed upward into Emergency and Observation pushed toward either Urgent or Routine (Samanta et al., 29 Jun 2026).

Taken together, these empirical results place two recurrent failure modes at the center of BayesBench. The first is a capability–strategy dissociation: a model may be highly accurate in one modality or on final outputs while still exhibiting poor reliability weighting, poor calibration, or non-Bayesian shifts under perturbation. The second is a latent-inference–prediction dissociation: a model may recover hidden structure, such as user type or patient communication style, but fail to translate that latent inference into normatively correct downstream predictions. Both benchmark lines therefore support a process-oriented interpretation of Bayesian behavior: what matters is not only whether a model can infer or predict, but whether the transformation from evidence to belief to action preserves Bayesian structure (Ma et al., 2 Dec 2025, Samanta et al., 29 Jun 2026).

5. Methodological significance and relation to other benchmark traditions

BayesBench belongs to a broader literature on Bayesian benchmarking, but its object of evaluation is distinctive. URSABench evaluates approximate Bayesian inference methods in deep neural network classification along the axes of uncertainty, robustness, scalability, and accuracy, with tasks such as in-domain prediction, misclassification detection, OOD detection, and decision-making under asymmetric cost (Vadera et al., 2020). posteriordb provides a database of posterior targets, models, datasets, and reference posterior draws for testing and developing Bayesian inference algorithms across a wide range of posterior geometries (Magnusson et al., 2024). “Benchmarking for Bayesian Reinforcement Learning” defines performance criteria over large sets of MDPs drawn from prior and test distributions and explicitly incorporates offline and online computation time (Castronovo et al., 2015).

BayesBench differs from those resources by centering the evaluation target on behavioral Bayesianity in LLMs. The psychophysics BayesBench is closer to a controlled cognitive benchmark: it manipulates noise, context, and prompt-induced priors to test whether cue weighting and prior reliance shift in the Bayes-consistent direction. The multi-turn BayesBench is closer to a sequential inference benchmark: it treats conversation as a stream of evidence and evaluates whether elicited beliefs track posterior updates over time. This suggests that BayesBench extends Bayesian benchmarking from algorithms over formal posterior targets to agent-like systems whose outputs must be interpreted as evolving beliefs under partial information (Ma et al., 2 Dec 2025, Samanta et al., 29 Jun 2026).

Methodologically, the two BayesBench proposals also share design choices that align with stronger benchmark practice. Both construct controlled environments in which the latent structure is known by design. Both distinguish intermediate inference quality from downstream performance. Both introduce evaluation procedures that remain informative when accuracy saturates or when final-turn correctness alone obscures systematic deviations. The psychophysics benchmark does this via BCS, which scores directional adaptation under ablation. The multi-turn benchmark does it via turn-wise TVD and MAE, together with conditional-latent interventions that test whether explicit access to inferred hidden structure improves prediction (Ma et al., 2 Dec 2025, Samanta et al., 29 Jun 2026).

A plausible implication is that BayesBench fills a gap left by conventional LLM leaderboards. Standard multimodal or reasoning benchmarks often emphasize correctness under fixed prompts, but BayesBench-style evaluations target epistemic dynamics: uncertainty tracking, evidence integration, latent-state recovery, and invariance to style or delivery format. In that sense, BayesBench occupies an intermediate position between cognitive psychophysics, Bayesian ideal-observer analysis, and benchmark engineering for modern foundation models.

6. Limitations and open directions

The psychophysics BayesBench is explicit about several limitations: bounded task ranges, limited perturbation scope, API dependence, synthetic task construction, and a behavioral rather than mechanistic interpretation of results. Its use of controlled synthetic tasks is deliberate, but this also limits immediate transfer to naturalistic multimodal settings. The paper accordingly presents BayesBench as a complement to standard multimodal benchmarks rather than a replacement for them (Ma et al., 2 Dec 2025).

The multi-turn BayesBench likewise has important scope limits. Exact Bayesian references are only available in the coin-flip and recommender environments. In social judgment and medical triage, the benchmark relies on invariance tests, conditioning interventions, and ground-truth targets rather than exact posterior comparison. Belief measurement depends on elicitation format, the latent structures are deliberately simplified, and active-engagement experiments depend on user simulators with fixed style instructions rather than unconstrained human interaction (Samanta et al., 29 Jun 2026).

Across both benchmark lines, the central open problem is the relation between elicited beliefs and internal computation. BayesBench evaluates whether behavior is Bayes-consistent, not how any such behavior is implemented. This leaves open whether observed improvements with scale reflect more faithful posterior computation, stronger pattern matching to benchmark structure, or post-training effects that sharpen confidence without improving normative calibration. The repeated appearance of over-updating, extreme-category attraction, and style-sensitive interaction suggests that post-training and conversational framing remain central obstacles to robust Bayesian behavior (Ma et al., 2 Dec 2025, Samanta et al., 29 Jun 2026).

A broader implication is that future BayesBench work will likely need richer latent structures, stronger naturalistic environments, and tighter links between belief elicitation and internal state. The psychophysics line points toward more realistic multimodal uncertainty tests; the sequential line points toward persistent user models, memory, and structured latent-state tracking in conversation. Both lines indicate that evaluating LLMs as Bayesian reasoners requires process-level benchmarks with explicit uncertainty semantics, not only task-level correctness.