Uncertainty as a Planning Signal: Multi-Turn Decision Making for Goal-Oriented Conversation

Abstract: Goal-oriented conversational systems require making sequential decisions under uncertainty about the user's intent, where the algorithm must balance information acquisition and target commitment over multiple turns. Existing approaches address this challenge from different perspectives: structured methods enable multi-step planning but rely on predefined schemas, while LLM-based approaches support flexible interactions but lack long-horizon decision making, resulting in poor coordination between information acquisition and target commitment. To address this limitation, we formulate goal-oriented conversation as an uncertainty-aware sequential decision problem, where uncertainty serves as a guiding signal for multi-turn decision making. We propose a Conversation Uncertainty-aware Planning framework (CUP) that integrates LLMs with structured planning: a LLM proposes feasible actions, and a planner evaluates their long-term impact on uncertainty reduction. Experiments on multiple conversational benchmarks show that CUP consistently improves success rates while requiring fewer interaction turns. Further analysis demonstrates that uncertainty-aware planning contributes to more efficient information acquisition and earlier confident commitment.

• The paper introduces the CUP framework, which uses entropy-based belief tracking to guide tree-based lookahead planning in multi-turn goal-oriented dialogues.
• It integrates SBERT semantic similarity and Monte Carlo Tree Search with expected information gain to balance exploration and commitment in decision making.
• Experimental results show CUP outperforms baselines, improving success rates by over 20 percentage points in ambiguous domains while reducing interaction turns.

Uncertainty as a Planning Signal in Multi-Turn Goal-Oriented Conversation

Problem Formulation and Motivation

This paper addresses the core challenge in goal-oriented conversational agents: efficient multi-turn decision making under uncertainty regarding user intent. Existing paradigms bifurcate into structured methods (e.g., slot-filling, RL, MDP-based) that enable long-horizon planning but demand rigid domain schemas, and LLM-centric approaches that offer flexibility but lack principled long-term reasoning. Both paradigms fail to optimally balance information acquisition (clarification queries) and target commitment (task resolution), especially in open-domain or ambiguous settings.

To resolve this deficiency, the paper formulates goal-oriented dialogue as an uncertainty-aware sequential decision process. At each turn, a belief distribution over candidate hypotheses representing possible user intents is maintained and updated. The principal innovation is to use this epistemic uncertainty—quantified entropy—as the central planning signal for tree-based lookahead over action sequences, whereby actions are chosen to maximize expected long-term reduction in uncertainty (expected information gain, EIG), in addition to optimizing task success and interaction efficiency.

Figure 1: CUP framework overview elaborating how belief monitoring, uncertainty-aware planning, and language action execution interact in CUP.

CUP Framework

Belief and Uncertainty Modeling

The agent maintains a belief state over the candidate set $C_t$ at time $t$, utilizing SBERT-based semantic similarity between the current conversation history and each candidate. Uncertainty is computed as entropy of this distribution. A commitment to a candidate (i.e., final recommendation or action) is triggered once the belief is sufficiently concentrated, with thresholds parameterized for normalized entropy and confidence.

Action Generation and Expected Information Gain

At each turn, feasible next actions (e.g., attribute queries, candidate commitments) are generated via an LLM conditioned on the ongoing dialogue and candidate set. For each candidate action, the EIG is computed as the expected decrease in entropy after receiving a user response, encompassing both the belief update and anticipated user feedback. Actions with higher EIG are prioritized; action priors are defined via a softmax over EIG, and inform downstream planning.

Uncertainty-Guided Planning: MCTS with Uncertainty Priors

The agent performs Monte Carlo Tree Search (MCTS) on the space of future action-response trajectories, guided by EIG-based priors. The edge selection in MCTS balances exploitation (maximizing expected value of actions) and exploration (favoring less-visited actions weighted by EIG), using a variant of the PUCT formula. Rewards in rollouts are a weighted sum of task success, interaction penalty, EIG bonus, and miscommitment penalty.

Natural Language Execution and State Update

The selected action is realized as a natural language utterance using the LLM. User responses (from a simulator) are observed, and history/candidate pool/beliefs are updated via Bayesian logic conditioned on new semantic evidence. This iterative belief update continues until a commit or interaction budget is exhausted.

Experimental Results

Benchmarks and Metrics

Experiments span four multi-turn benchmark datasets (Beauty, Fashion, Home, Inspired) with significant candidate ambiguity and varying candidate pool sizes. Evaluation focuses on Success Rate (SR, correct completion within allowed turns) and average number of turns (avgT), with multiple backbone LLMs (Qwen3-4B, Mistral-7B-v0.3, Llama-3.1-8B).

Comparison with Baselines

CUP robustly outperforms prior baselines, including retrieval-only, naive LLM prompting (both with and without chain-of-thought or heuristic multi-turn reasoning), and all existing uncertainty-aware planning approaches such as UoT, ATD, BED-LLM, and MISQ-HF. The most salient gains occur in the most ambiguous domains (e.g., Inspired, Beauty), with SR improvements exceeding 20 percentage points over the strongest uncertainty-aware baselines, and requiring fewer interaction turns.

Figure 2: Performance on Beauty dataset showing SR versus avgT under different planning strategies.

Performance is consistent across LLM backbones, demonstrating that the planning mechanism is backbone-agnostic. Notably, CUP nearly eliminates the dependency of success rates on LLM size, indicating that uncertainty-aware planning, rather than raw model capacity, is the key driver of sample efficiency and optimal interaction policy.

Ablation Analysis

Systematic ablation (e.g., using heuristic commitment logic, random action selection) demonstrates that both structured planning and language-based action realization contribute to final performance, with uncertainty-guided planning as the primary contributor to efficiency and outcome.

Figure 3: A success case from Fashion, illustrating sequential narrowing of candidates via attribute queries, leading to confident early commitment.

Mechanism Analysis and Case Study

Analysis of candidate dominance ratios and normalized uncertainty reduction trajectories confirms that EIG-driven planning enables rapid belief concentration on the target, especially within the early interaction turns. Where ambiguity persists till late turns, errors are attributed to insufficient semantic separation in the candidate pool, exposing a limit of current belief modeling and highlighting the importance of stronger user intent modeling for further advances.

Figure 4: "Beauty" dataset visualization of the dominance of the target candidate across dialogue turns.

Implications and Prospects

CUP demonstrates that integrating explicit uncertainty quantification and long-horizon planning substantially improves efficiency and effectiveness in goal-oriented conversation—a setting previously dominated by either rigid schema-based or locally myopic LLM-driven solutions. The proposed tree-search architecture with EIG-based priors constitutes a scalable and LLM-agnostic scaffold for controllable multi-turn decision making.

The adoption of entropy-based uncertainty as the central planning signal allows automatic adaptation of interaction depth to scenario ambiguity and supports confident early commitment, thus reducing interaction burden and miscommitment rates. This framework is extensible to more complex tasks (e.g., multi-user negotiation, non-stationary user preferences) and can integrate advanced user simulators, richer feedback, and alternative uncertainty proxies.

Potential directions include incorporating graph-structured candidate embeddings, hierarchical EIG estimation, and on-policy RL fine-tuning of the planner. The general architecture is also applicable to alignment-sensitive domains such as medical dialogue or policy negotiation, where exploration versus exploitation and transparency in decision signals are critical.

Conclusion

The integration of principled uncertainty quantification with structured lookahead planning, coupled with flexible language-grounded action execution, establishes a new, robust paradigm for goal-oriented conversation. By using uncertainty as a planning signal, CUP achieves superior performance in both effectiveness and efficiency, and substantially narrows the gap between the flexibility of LLMs and the long-term optimality afforded by structured planning. This work sets a clear trajectory for future research in modular, uncertainty-aware multi-turn conversational agents with transparent decision processes.

Figure 5: Illustration of the dominance and uncertainty reduction profile for target identification across turns on the Beauty dataset.