--- title: Bayesian Elicitation with LLMs url: https://www.emergentmind.com/topics/bayesian-elicitation-with-llms type: topic --- # Bayesian Elicitation with LLMs Bayesian elicitation with large language models (LLMs) denotes a growing set of methodologies that harness probabilistic reasoning, structured uncertainty, and active information-gathering algorithms within LLM workflows. These approaches leverage LLMs both as sources and surrogates of expert knowledge, integrating Bayesian inference principles into tasks such as preference modeling, causal discovery, prior extraction, uncertainty quantification, and optimal information acquisition. This article surveys the dominant paradigms, mathematical frameworks, and empirical results in state-of-the-art Bayesian elicitation with LLMs, as surfaced across major recent research on arXiv. ## 1. Foundations: Bayesian Elicitation and LLMs Bayesian elicitation transforms latent expert knowledge, preferences, or priors into explicit, probabilistic objects, typically by querying human experts. LLMs substitute or augment humans both by encoding implicit world-knowledge acquired through pretraining and by generating, scoring, and adapting evidence in response to structured queries. A general Bayesian elicitation problem involves an unknown parameter $\theta$ (e.g., prevalence rate, utility function, graph structure), a prior $p(\theta)$ (possibly to be elicited), and a data-generation or feedback process $p(d\mid\theta)$. LLMs can be placed in several roles: - **Direct Bayesian estimator:** Given a prompt, LLMs generate point or interval estimates with an associated uncertainty claim, which serves as a "Bayesian expert opinion" [2604.01896]. - **Probabilistic knowledge extractor:** LLMs are queried for conditional, marginal, or structural relationships (e.g., Bayesian network CPTs) [2505.15918]. - **Uncertainty-aware surrogate:** LLMs are framed as posterior approximators or surrogates for Bayesian inference or active learning [2605.04764, 2508.21184]. - **Prior sampler:** Iterated in-context learning is used to sample from the model's implicit Bayesian prior [2406.01860]. LLM-based Bayesian elicitation strictly requires evaluation and calibration, since LLMs may exhibit systematic overconfidence, miscalibration, or domain-dependent biases [2604.01896]. ## 2. Mathematical and Algorithmic Frameworks ### 2.1 Probabilistic Modeling and Inference The core of Bayesian elicitation is updating beliefs over unknowns using Bayes' rule as new evidence (real or LLM-generated) accumulates. - **Expert knowledge extraction:** For Bayesian network parameterization, LLMs are prompted per-node/parent configuration, returning $P(X_i = x \mid \mathrm{Pa}(X_i) = \pi)$, which are normalized over all $x$ [2505.15918]. - **Prior elicitation:** Iterated learning alternates between sampling $\theta$ from $p(\theta|d)$ using the LLM and generating new data $d$ via $p(d|\theta)$, producing samples from the LLM's implicit prior [2406.01860]. - **Latent-state models:** LLMs' variable outputs (e.g., on repeated classification) are treated as noisy measurements of latent variables, and Bayesian latent class models are applied to calibrate and recover the ground truth [2510.23874]. ### 2.2 Active Querying and Bayesian Design Bayesian experimental design frameworks select queries or items to maximize expected information gain (EIG) about $\theta$ [2508.21184], or related mutual information criteria: - **EIG computation**: For candidate query $x$, \[ \mathrm{EIG}_\theta(x;h_t) = H[p(\theta; h_t)] - \mathbb{E}_{p(y|x;h_t)}[ H[p(\theta|y, x; h_t)] ] \] - **Practical acquisition**: Approximate EIG using Rao–Blackwellization, sample candidate $\theta$ from current belief $p_f(\theta; h_t)$, estimate predictive distributions via LLM logits or outputs [2508.21184]. For Bayesian preference elicitation, frameworks such as BAL-PM [2406.10023] and PEBOL [2405.00981] use composite acquisition functions—combining epistemic uncertainty, predictive entropy, or diversity in semantic feature space—to select the most informative queries in each batch. ## 3. Practical Implementations and Workflows ### 3.1 Elicitation Protocols - **LLM-as-expert workflows:** Chain-of-request or dual-expert strategies use one LLM to propose causal/structural elements and another to critique or verify (e.g., structure learning in BNs) [2504.10397]. - **Prompt orchestration:** Systematic prompting is used to extract conditional probabilities, feature sets, or model confidences. For example, each CPT entry in a BN is queried independently at low temperature, with responses normalized to valid distributions [2505.15918]. - **Iterated Gibbs samplers:** For prior elicitation, LLMs are alternately sampled for $\theta \sim p(\theta|d)$ and new data $d \sim p(d|\theta)$, with sufficient chains and iterations guaranteeing convergence to the implicit prior [2406.01860]. ### 3.2 Bayesian Optimization and Surrogates Frameworks such as ReElicit [2605.19093] use LLM-elicited semantic feature spaces coupled with a Gaussian process Bayesian optimizer. The LLM dynamically proposes interpretable features, maps past prompts into feature space, and GP models guide the selection of next candidate prompts, which the LLM then realizes/refines into deployable prompts. Surrogate modeling with LLMs requires careful alignment between prompt protocol (the LLM's "prior" and inference rules) and traditional Bayesian surrogate models (e.g., GPs), as different prompt designs and query protocols induce different posterior predictions and uncertainty behaviors [2605.04764]. ## 4. Empirical Findings: Calibration, Uncertainty, and Scalability ### 4.1 Calibration and Overconfidence Out-of-the-box LLM uncertainty estimates are almost universally overconfident. Across eleven contemporary LLMs, 95% credible intervals contain the true value only 9–44% of the time, far below the nominal coverage [2604.01896]. This overconfidence can be corrected post hoc using split conformal prediction, which expands intervals to restore coverage, albeit at the cost of sharpness [2604.01896]. ### 4.2 Surrogate-Specific Uncertainty and Protocol Effects LLM surrogates' uncertainty is tightly linked to prompt type and elicitation protocol. Structural prompts act as effective priors (e.g., specifying function family shape), and pointwise vs. joint query protocols induce markedly different predicted posteriors and uncertainty surfaces [2605.04764]. The uncertainty-alignment criterion, a rank correlation between surrogate uncertainty and sample-consistent reference ambiguities, diagnoses whether LLM uncertainty tracks true epistemic ambiguity. ### 4.3 Scalability and Task-Specific Observations Scalability of Bayesian elicitation with LLMs is bounded by prompt throughput, feature-space dimensionality, and LLM context limitations. In structure learning for BNs, performance and precision decrease as graph size grows [2407.09311]. Preference modeling frameworks remain sample-efficient for moderately large item sets, but require adaptation—e.g., pre-clustering, sparse updates—at scale [2405.00981]. Practical robustness varies by task: LLMs perform well and produce probabilistically meaningful priors on well-covered domains (e.g., demographic statistics, personality, standard BNs), but may display anchor and miscalibration effects in specialized or data-sparse domains [2604.01896, 2505.15918]. ## 5. Applications: Causal Modeling, Preference Elicitation, Evaluation, and More LLM-driven Bayesian elicitation is being deployed in a wide array of applications: - **Probabilistic Causal Modeling:** LLMs can generate, critique, and refine BN structures, often yielding equal or lower entropy than human-elicited or statistically induced graphs, albeit with a risk of hallucinated edges or inherited training biases [2504.10397]. - **Preference Elicitation and RLHF:** Bayesian active learning via EIG-guided selection minimizes human feedback requirements in large-scale model alignment and RLHF datasets [2406.10023, 2403.05534]. - **Automated Evaluation Calibration:** Bayesian Dawid–Skene and related models post hoc calibrate LLM-as-evaluator win-rates, improving alignment with human preferences for automatic text generation evaluation [2411.04424]. - **Measurement under Stochasticity:** Bayesian latent-state models account for LLM output variation and error, producing calibrated estimates, credible intervals for individual predictions, and base-rate uncertainty quantification [2510.23874]. ## 6. Best Practices, Limitations, and Future Directions ### Best Practices - **Explicit protocol specification:** Treat prompt wording and query protocol as part of the Bayesian model—empirically, these act as effective priors and inference operators [2605.04764]. - **Uncertainty calibration:** Apply diagnostic tools (e.g., ECE, uncertainty-alignment), and statistical recalibration (e.g., conformal prediction) before deploying LLM-derived posteriors in decision-making [2604.01896]. - **Integration with data:** Combine LLM-derived priors or CPTs with small-sample real data via Bayesian pooling or pseudo-counts for improved BNs [2505.15918]. - **Iterative design:** Use re-elicitation to adapt feature spaces in dynamic tasks (e.g., prompt optimization) [2605.19093]. ### Limitations and Open Problems - **Scalability constraints:** Context window, feature dimensionality, and token throughput can limit large-graph or large-pool elicitation [2407.09311, 2405.00981]. - **Domain generality:** LLMs' priors are strong on widely-covered topics but unreliable in specialized or emerging domains [2604.01896]. - **Bias and hallucination:** LLM-generated causal or probabilistic structures may reflect pretraining artifacts or encode implausible relations; mitigation involves cross-model validation, SEM screening, and human review [2504.10397]. - **Protocol sensitivity:** Unaccounted changes in prompt or query format may shift the induced surrogate belief and downstream acquisition trajectories [2605.04764]. - **Simulation-to-human gap:** Many protocols are tested with simulated users or evaluators; rigorous human-in-the-loop validation remains outstanding in most applications [2405.00981]. ### Future Directions Approaches under development include richer acquisition criteria (e.g., multi-step active design [2508.21184]), adaptive feature extraction, variational or neural posterior approximators for richer Bayesian representations, open-ended and multi-format feedback, and joint human–AI decision pipelines in high-stakes or data-scarce settings. --- **Table: Illustrative Bayesian Elicitation Protocols with LLMs** | Application | Elicitation Protocol | Reference | |------------------------------------------------|----------------------------------------------------|--------------| | BN parameterization | Zero-shot or few-shot node × parent CPT query | [2505.15918] | | Structure elicitation | Multi-LLM proposal + majority voting + contamination check | [2407.09311] | | Preference modeling (active) | BAL-PM: Bayesian ensemble + epistemic/diversity | [2406.10023] | | Prior extraction | Iterated in-context Gibbs sampler | [2406.01860] | | Surrogate optimization | Semantic feature elicitation + GP acquisition | [2605.19093] | | Win-rate calibration (evaluation) | Bayesian Dawid–Skene/BWRS meta-model | [2411.04424] | | Uncertainty quantification over prompts | MHLM proposal MCMC on prompt space | [2506.10060] | This table presents a concise mapping from target application to documented LLM-centric Bayesian elicitation protocol and primary reference. ## 7. Conclusion Bayesian elicitation with LLMs establishes a suite of probabilistic frameworks that leverage the semantic, generative, and adaptive capabilities of language models for structured uncertainty modeling and sample-efficient expert knowledge extraction. These paradigms blend Bayesian design and inference principles with modern large-model architectures, yielding adaptive, statistically interpretable systems that improve alignment, evaluation, and decision support across diverse domains. Researchers must attend carefully to protocol specification, robust calibration, and the integration of LLM-sourced priors with empirical data and expert review to ensure rigorous, trustworthy outcomes. Source: https://www.emergentmind.com/topics/bayesian-elicitation-with-llms