CA-BED: Conversation-Aware Bayesian Experimental Design

Abstract: LLMs excel at static reasoning tasks, yet their performance often degrades in interactive scenarios where information must be actively acquired through questioning. A key challenge lies in selecting questions that reduce uncertainty while incorporating responses that may be ambiguous or only partially informative. To address this, we propose Conversation-Aware Bayesian Experimental Design (CA-BED), an inference-time probabilistic dialog planning framework that integrates Bayesian Experimental Design with LLM-based likelihood estimation to optimize question selection over multiple conversational turns. CA-BED maintains a belief distribution over hypotheses, anticipates possible answers, and propagates expected information gain through a simulated conversation tree. Across two structured entity-deduction benchmarks, CA-BED yields an average 21.8% improvement in success rates over direct prompting, with comparable gains relative to alternative information-seeking methods. It achieves these gains with an average increase of only 1.8 conversational turns compared to direct prompting.

• The paper introduces CA-BED, which employs probabilistic belief updates and Bayesian inference to enable adaptive, multi-turn dialog planning under uncertainty.
• It features LLM-based likelihood estimation with soft belief updates and uses Shannon entropy for planning, maximizing expected information gain.
• Empirical results demonstrate significant improvements in entity-deduction tasks like 20 Questions and Detective Cases, even with ambiguous evidence.

Conversation-Aware Bayesian Experimental Design: A Probabilistic Framework for LLM-Driven Dialog

Motivation and Problem Setting

Despite strong performance of LLMs on static reasoning benchmarks, their capability degrades in settings of active, conversational information acquisition. Entity-deduction tasks (e.g., 20 Questions, diagnostic/clarification dialogues) require adaptive selection of questions based on prior responses, under significant uncertainty and ambiguity. Existing approaches either rely on supervised fine-tuning for question generation or employ planning heuristics such as the Uncertainty of Thoughts (UoT) [hu2024uncertaintythoughtsuncertaintyawareplanning], which constructs decision trees from binary question trajectories and eliminates hypotheses based on responses. However, most methods rigidly partition hypothesis spaces and fail under partial or ambiguous evidence, leading to brittle or inefficient conversations. There is a clear demand for a principled framework to support dynamic belief modeling and probabilistic conversational planning in LLM-based agents.

CA-BED: Algorithm and Theoretical Formulation

CA-BED (Conversation-Aware Bayesian Experimental Design) formulates interactive dialog as a sequential decision process under uncertainty, leveraging Bayesian Experimental Design (BED) [rainforth2023modernbayesianexperimentaldesign, choudhury2025bedllmintelligentinformationgathering]. The agent maintains a belief distribution over a discrete hypothesis set $\mathcal{H}$, updating posteriors after each exchange via Bayesian inference. Dialog is structured as an alternating tree of QuestionNodes (Q) and EvidenceNodes (E), with each Q branching over plausible user responses and each E updating the agent's belief $B_k(h_i)$ through explicit likelihood estimation.

Key design choices include:

• LLM-based Likelihood Estimation: Direct prompts obtain $\mathbb{P}(E|h_i)$ as textual probabilities from the LLM, with subsequent confidence-weighted smoothing to mitigate epistemic uncertainty or ambiguous responses.
• Soft Belief Updates: Instead of hard-pruning hypotheses on mismatch, CA-BED soft-weights hypotheses by their posteriors; this allows recovery from early ambiguous evidence and more robust inference.
• Planning via Information Gain: The expected reduction in Shannon entropy $H(B)$ (i.e., information gain) is used as a reward signal for planning, with the agent greedily selecting the question that maximizes expected downstream information gain over a fixed lookahead depth ($D_{\text{sim}}$).

This tree-search process is computationally managed via aggressive caching of likelihoods, exploiting their conditional independence on isolated questions and hypotheses. Candidate questions and answer sets ($M_q$) are also generated by LLM context.

Empirical Evaluation

The evaluation focuses on two structured entity-deduction tasks:

• 20 Questions (COMMON dataset): Diverse general knowledge targets; success defined as correctly deducing the hidden entity within allowed turns.
• Detective Cases: Complex deductive reasoning with ambiguous evidence across suspects and scenario-specific facts.

CA-BED is compared against:

• Direct Prompting;
• UoT (hard binary partitioning, $\epsilon=1.0$);
• UoT with likelihood smoothing (soft epistemic updates, various $\epsilon$ and $\tau$);
• CA-BED with categorical answer planning (multiple mutually exclusive answer choices per question).

Strong Numerical Results:

• CA-BED achieves a 21.8% absolute improvement in success over direct prompting, with comparable or larger gains over UoT in both Detective Cases (46.0% vs. 27.6–29.0%) and 20 Questions (82.9% vs. 57.7–60.4%).
• The increase in average conversation length is minor (mean increase of only 1.8 turns).
• Introducing probabilistic belief updates to UoT narrows the performance gap on simpler tasks but CA-BED sustains superior accuracy, especially with complex or ambiguous evidence.

Notably, in 20 Questions, augmenting CA-BED with answer planning (non-binary answers) increases success rate to 89.2% and reduces average conversation length, but in open, less-structured domains (Detective Cases), performance degrades, highlighting the challenge of designing mutually exclusive answer sets in unconstrained settings.

Analysis and Discussion

The experiments provide clear evidence that hard binary elimination (as in UoT) is highly sensitive to ambiguous or partially informative evidence, often prematurely discarding correct hypotheses. Likelihood smoothing partially addresses this but cannot fully resolve the challenges of distributed and overlapping evidence, especially in complex domains. CA-BED's probabilistic belief maintenance and multi-answer planning endow the agent with resilience to early errors and support robust information gathering, albeit at computational cost.

A secondary analysis using Top-3 accuracy and “post-dialog” inference by an external LLM reasoner reveals that CA-BED logs more informative conversations (Top-3 accuracy of 90% in Detective Cases), but its attribute-based dialog format can make automated downstream inference more challenging compared to direct guessing strategies.

Limitations

CA-BED's application is confined to domains with explicit, finite hypothesis sets and does not currently generalize to truly open-ended dialog where the space of possible targets is unbounded or emergent. Its effectiveness depends on reliable LLM-generated likelihoods and well-formed candidate answers, especially in categorical answer-planning settings. Furthermore, the inference-time cost is significant compared to direct prompting due to repeated candidate/question generation, likelihood estimation, and recursive simulation, though caching mitigates this to some extent.

Implications and Future Directions

On the theoretical side, CA-BED offers a unified, probabilistically principled approach for information-seeking dialog, connecting BED and LLM-driven interaction. On the practical side, its demonstrated accuracy and robustness position it as a strong candidate for real-world systems requiring adaptive information acquisition (diagnosis, support, tutoring, troubleshooting).

Moving forward, several lines of research are evident:

• Extending CA-BED to dynamically generated/discovered hypothesis spaces, leveraging large-scale LLM context and self-refinement.
• Model-free or fast-approximate variants to reduce inference cost, possibly via neural amortization of policy/value functions over dialog states.
• Richer integration between explicit answer planning and open-ended language understanding for complex dialog domains.
• Human-in-the-loop and cross-domain evaluation to test transfer and usability in unconstrained, real deployment settings.

Conclusion

CA-BED establishes a new standard for principled, probabilistic dialog planning in entity-deduction tasks using LLMs. By uniting Bayesian experimental design with explicit uncertainty modeling and multi-turn lookahead, it achieves substantially improved information acquisition over both naive prompting and prior planning-based baselines, especially under ambiguity. The work has significant implications for developing interactive, reliable LLM agents and points toward broader Bayesian and probabilistic frameworks for future interactive AI systems.

---

Principal Citation:

"CA-BED: Conversation-Aware Bayesian Experimental Design" (2606.01182)