Uncertainty-Aware Generation and Decision-Making Under Ambiguity

Abstract: With rapidly improving capabilities, LLMs are increasingly used in many complex real-world tasks. Beyond requiring in-depth knowledge and reasoning skills, many of these tasks exhibit a high degree of subjectivity and require that the outputs of the model can be trusted. While a lot of progress has been made to train better models, decision-making algorithms have received less attention. In this work, we present and evaluate various uncertainty-aware decision-making algorithms based on Bayesian decision theory and risk-averse decision making on the tasks of tutoring and automatic peer reviewing. Concretely, we take uncertainty over tutoring strategies and review scores into account when generating a tutor response or review and use conformal prediction to provide guarantees over strategy and score. We find empirically that these algorithms can improve the utility of the generations but need to be carefully implemented when ambiguity is high. For example, risk-averse rules can degrade performance by optimizing for generic outputs, while Bayesian methods tend to perform better. Our work uses techniques from decision theory to improve LLM-based decision-making and outlines open challenges for the community.

• The paper introduces a framework that marginalizes over ambiguity using Bayesian decision theory and conformal prediction for LLM outputs.
• It integrates calibrated conformal prediction sets to provide statistical guarantees and manage risks in high-stakes applications.
• Empirical evaluations demonstrate that Bayesian strategies outperform minimax approaches in improving actionability and consistency.

Uncertainty-Aware Generation and Decision-Making Under Ambiguity: A Technical Overview

Motivation and Problem Setting

LLMs are increasingly deployed in subjective and high-stakes domains such as peer review and automated tutoring. These domains are inherently ambiguous: expert annotators and reviewers often disagree, resulting in a substantial irreducible uncertainty over intermediate decisions such as reviewer scores or pedagogical strategies. Traditional approaches to generation and decision-making with LLMs have paid limited attention to explicitly modeling such uncertainty—often collapsing intermediate decision spaces into singular predicted values—impairing the reliability and ultimate utility of system outputs, particularly under ambiguity. This paper addresses the gap by proposing principled uncertainty-aware decision-making algorithms, adapting formal methods from Bayesian decision theory and conformal prediction to the LLM generation process.

Figure 1: In review and tutoring tasks, genuine human disagreement and decision ambiguity are ubiquitous; systems for supporting humans in these tasks must account for this uncertainty to maximize utility, prevent overcommitment, and correctly represent plausible outputs.

Bayesian and Risk-Averse Decision Rules for Generation

The central proposal adopts a two-fold decision problem structure: (i) the output decision space (e.g., tutor responses or reviews), and (ii) a discrete latent state space representing ambiguous intermediate variables (e.g., possible review scores or tutoring strategies). Rather than committing to a single intermediate variable, the framework marginalizes over its uncertainty. Under Bayesian decision theory, the optimal output is that which maximizes the expected utility with respect to the posterior distribution $p(\omega \mid x)$ over latent variables, given context $x$. The utility function $u(y, \omega)$ can be tailored to enforce both response quality and consistency with the latent variable, for instance requiring pedagogical soundness and alignment with a plausible tutoring strategy.

The Bayes decision rule is executed by sampling candidate outputs, evaluating utility across sampled latent states, and averaging according to the predictive distribution. This generalizes minimum Bayes risk (MBR) methods beyond sequence-level uncertainty to structured latent-variable models. Unlike classic MBR, utility functions can be designed to encourage application-specific desiderata rather than strictly output-to-output agreement (2606.30578).

Statistical Guarantees and Risk Control: Conformal Prediction and Minimax Decision

While Bayesian averaging provides robustness against ambiguity, it supplies only the mean-case optimal decision and lacks coverage guarantees for the latent variable. To offer statistical guarantees over intermediate decisions, the paper integrates conformal prediction to generate calibrated prediction sets for $\omega$, ensuring (marginal) coverage of plausible labels at a user-defined confidence level. In scenarios where label ambiguity arises from genuine annotator disagreement, recent advances (e.g., Monte-Carlo conformal prediction) are employed, compensating for non-exchangeability and multiple ground truths.

The conformal set $C(x)$ over possible $\omega$ serves two purposes: (i) focusing utility estimation on a compressed plausible set (imperative for practical computation as the state space grows), and (ii) enabling a worst-case (risk-averse) utility maximization over $C(x)$, aligning with minimax decison rules. This is theoretically well-motivated for high-stakes applications where adverse decisions are costly and conservative strategies may be preferable [(2606.30578), kiyani2025decision].

Experimental Protocol: Review Generation and Tutoring

Comprehensive experiments cover (i) automatic review generation and scoring (benchmarked on Review-5k and NLPEER datasets), (ii) tutor response generation on MathDial, with each process involving LLM sampling, fine-tuned latent state classifiers, and state-of-the-art, rubric-aligned utility/reward models.

Review Generation

• Models: Google Gemma-3-27B, Qwen3-30B, Mistral-3.2-24B for review text; Llama-3.2-3B for score classification.
• Evaluation: Reviews are rated for actionability, verifiability, helpfulness, and groundedness, using separate LLM-based and reward model evaluators.
• Findings: The Bayesian decision rule consistently outperforms both unconstrained sampling and minimax strategies across most criteria, with notable gains in actionability and verifiability.

Tutoring

• Models: TutorRL-7B for responses, fine-tuned Llama-3.2-3B for strategy prediction.
• Evaluation: Responses are assessed with pedagogically focused reward models and (oracle/human) ground-truth comparisons.
• Findings: Only Bayesian strategies yield reliable improvements; minimax can result in undesirable degenerate outputs (e.g., generic or non-informative hints) if prediction sets are excessively broad—a consequence of ambiguous latent states and weak classifiers (2606.30578).

Analysis of Prediction Sets and Scaling Behavior

The utility of conformal prediction sets is empirically analyzed:

• In highly ambiguous domains (e.g., tutoring), prediction sets are routinely wide, resulting in fluctuating or degraded risk-averse performance.
• Compression rates are more favorable in review tasks, resulting in both computational and performance gains when used for Bayesian pruning.
• Increasing the number of output samples enhances the quality of Bayesian decisions (monotonic improvement), while minimax approaches may deteriorate due to error amplification within large prediction sets.

Theoretical and Practical Implications

• The fusion of Bayesian risk minimization and conformal prediction sets a new standard for uncertainty-aware LLM-driven generation in subjective high-value settings. This framework allows practitioners to control the risk-utility tradeoff by defining custom utility functions and desired coverage levels.
• Enforcing calibrated coverage via conformal prediction and deploying statistical guarantees is likely to increase user trust, as blind marginalization—without display of uncertainty bounds—may overstate LLM confidence and mislead stakeholders.
• However, the effectiveness of both Bayesian and minimax strategies fundamentally depends on the quality of predictive distributions over latent states. Weak classifiers can lead to excessive ambiguity, poor set compression, and suboptimal or even pathological behaviors in minimax optimization.

Open Challenges and Directions for Future Research

• Modeling annotator disagreement: Progress requires more systematic collection of datasets with multiple annotator labels per example, enabling proper estimation of label distributions and further advances in ambiguity-aware conformal predictors [plank-2022-problem, stutz2023conformal].
• Improved scoring/classification models: Further refinement of intermediate label classifiers is necessary. LLMs still struggle with high-quality categorical predictions in ambiguous domains; error propagation remains a concern.
• Advanced, user-defined utility models: Continued development of robust learned utility functions (LLM-as-verifier, reward models) will enable end-user customization for heterogeneous task requirements [kwok2026llmverifier].
• Compositional risk and utility tradeoffs: More advanced, possibly hierarchical combination of Bayesian and risk-averse strategies (e.g., combining mean, variance, and tail risks within utility) could be explored.
• Field studies and impact evaluation: Larger field experiments measuring real-world impacts in peer review and education are essential.

Conclusion

This work integrates Bayesian and frequentist (conformal) uncertainty quantification into LLM-based generation pipelines for tasks with fundamental ambiguity over intermediate decisions, proposing and validating Bayesian and risk-averse rules for optimizing utility subject to uncertainty. Empirically, Bayesian strategies offer robust and reliable utility improvement across evaluation domains. Calibrated conformal prediction sets provide essential guarantees, computational efficiency, and interpretability, although their effectiveness is bounded by classifier quality and the ambiguity structure of the task. The study identifies persistent methodological challenges and prescribes a clear research agenda centered on multi-annotator data, improved intermediate classifiers, and practical, utility-driven uncertainty quantification.

Figure 1: In review and tutoring tasks, human disagreement and fundamental ambiguity are pervasive; supporting methods must explicitly account for uncertainty to maximize output utility and provide calibrated risk guarantees.

---

Reference: "Uncertainty-Aware Generation and Decision-Making Under Ambiguity" (2606.30578)