BAS: A Decision-Theoretic Approach to Evaluating Large Language Model Confidence

Abstract: LLMs often produce confident but incorrect answers in settings where abstention would be safer. Standard evaluation protocols, however, require a response and do not account for how confidence should guide decisions under different risk preferences. To address this gap, we introduce the Behavioral Alignment Score (BAS), a decision-theoretic metric for evaluating how well LLM confidence supports abstention-aware decision making. BAS is derived from an explicit answer-or-abstain utility model and aggregates realized utility across a continuum of risk thresholds, yielding a measure of decision-level reliability that depends on both the magnitude and ordering of confidence. We show theoretically that truthful confidence estimates uniquely maximize expected BAS utility, linking calibration to decision-optimal behavior. BAS is related to proper scoring rules such as log loss, but differs structurally: log loss penalizes underconfidence and overconfidence symmetrically, whereas BAS imposes an asymmetric penalty that strongly prioritizes avoiding overconfident errors. Using BAS alongside widely used metrics such as ECE and AURC, we then construct a benchmark of self-reported confidence reliability across multiple LLMs and tasks. Our results reveal substantial variation in decision-useful confidence, and while larger and more accurate models tend to achieve higher BAS, even frontier models remain prone to severe overconfidence. Importantly, models with similar ECE or AURC can exhibit very different BAS due to highly overconfident errors, highlighting limitations of standard metrics. We further show that simple interventions, such as top-$k$ confidence elicitation and post-hoc calibration, can meaningfully improve confidence reliability. Overall, our work provides both a principled metric and a comprehensive benchmark for evaluating LLM confidence reliability.

• The paper introduces BAS, a decision-theoretic metric aligning model confidence with utility under abstention policies.
• BAS penalizes high-confidence errors asymmetrically and aggregates decision-level utility across varying risk thresholds.
• Empirical benchmarks demonstrate that models with similar accuracy can have vastly different decision reliability when evaluated with BAS.

Decision-Theoretic Evaluation of LLM Confidence: The BAS Metric

Introduction

The propensity of LLMs to yield overconfident, hallucinated outputs remains a critical safety and deployment concern, particularly in domains where erroneous predictions entail significant risk. Although numerous studies have established that LLMs lack rigorous mechanisms for expressing or calibrating uncertainty, typical evaluation metrics (e.g., accuracy, Expected Calibration Error (ECE), and Area Under the Risk-Coverage Curve (AURC)) fail to directly capture whether confidence estimates meaningfully support abstentions or risk-aware decision-making. This work introduces the Behavioral Alignment Score (BAS) as a foundational decision-theoretic metric, deriving it from explicit answer-or-abstain utility considerations. BAS addresses substantive limitations of existing metrics by penalizing high-confidence errors asymmetrically and aggregating decision-level utility across risk thresholds.

Figure 1: Introduction of BAS as a decision-theoretic metric, with substantial task- and model-dependent variation and persistent overconfidence in frontier LLMs.

The Behavioral Alignment Score: Methodological Foundations

Decision-Theoretic Framework

The BAS framework formalizes model evaluation under selective prediction with abstention. For each query $x$, the model produces a response $R$ and a confidence score $s \in [0,1)$. The true probability of correctness $p$ remains unobservable. The core abstention policy dictates that a downstream agent answers if and only if $s \ge t$, where $t \in [0,1)$ is a risk threshold reflecting the end-user’s tolerance for error.

Utility assignment is asymmetric:

• Correct answer: +1 reward
• Incorrect answer: $-\frac{t}{1-t}$ penalty (growing rapidly with more risk-averse thresholds)
• Abstention: 0 utility

The expected utility integrates over all thresholds under a uniform (or weighted, for safety-critical applications) risk profile:

$$\mathrm{BAS} = \mathbb{E}_{t \sim \mathrm{Uniform}[0,1)}\left[ \mathbb{E}_{Z} S_t(Z, \pi_s(t)) \right].$$

This yields the closed-form observable per-example utility:

$$U(s, Z) = \begin{cases} s, & Z=1 \ s + \ln(1-s), & Z=0 \end{cases}$$

Decision-theoretic optimality is proven: truthful confidence reporting ($s=p$) uniquely maximizes BAS for $R$0.

Relation to Standard Reliability Metrics

Standard proper scoring rules (log loss, Brier score) penalize under- and overconfidence symmetrically, and common calibration metrics such as ECE and AURC capture only partial aspects of decision reliability. BAS diverges critically by:

• Aggregating over all risk thresholds, not only a fixed one.
• Assigning a logarithmic penalty diverging to $R$1 for overconfident errors ($R$2, $R$3).
• Distinguishing models with identical calibration or ranking but radically different catastrophic error profiles.

Empirical Analysis: Benchmarking and Trends

Benchmark Setup: Tasks, Models, and Protocols

A comprehensive empirical benchmark encompasses several representative tasks:

• AIME: Mathematical multi-step reasoning.
• MedQA: Medical multiple-choice QA.
• SimpleQA: Open-domain factoid questions, emphasizing cases where the correct action is to abstain.

Twelve models spanning proprietary and open-weight, small to frontier scales, are compared using direct and alternative black-box confidence elicitation protocols.

Strong Empirical Observations

BAS reveals several contradictory or non-monotonic results relative to standard metrics:

• In open-ended settings (e.g., SimpleQA), even SOTA models yield negative BAS due to systematic overconfident errors, despite moderate ECE or AURC.
• On AIME, GPT-oss (75% accuracy, BAS=0.57) vastly outperforms GPT-4o (11.7% accuracy, BAS=–5.06); the latter suffers catastrophic overconfidence despite its general capability.
• Larger and more accurate models tend (but not always) to offer better BAS and reliability, but substantial outliers exist.

Figure 2: Scaling trends showing that while accuracy and reliability often improve jointly with model size, significant model-wise variance remains.

Diagnostic Power: BAS vs. Calibration and Ranking Metrics

While BAS is correlated with standard reliability metrics, numerous instances deviate sharply from these trends. Models with similar ECE/AURC can exhibit substantially different BAS:

Figure 3: BAS is only partially correlated with ECE/AURC; several model–task pairs deviate substantially.

Analysis of confidence distributions (Figure 4) identifies model behaviors driving these discrepancies. For example, Llama 3.3 frequently assigns very high confidence to incorrect responses—a property punished disproportionately by BAS but barely affecting ECE.

Figure 4: Distribution of predicted confidence highlights the criticality of tail overconfidence—rare but extreme errors that dominate BAS but have marginal ECE impact.

BAS is also highly correlated with log loss in practice due to LLMs’ tendency toward overconfidence, but it distinguishes models through its asymmetric error penalties (Figure 5).

Figure 5: BAS vs. log loss; despite high practical correlation among overconfident models, BAS structurally separates models with differing trade-offs.

Practical Interventions: Elicitation and Calibration

Confidence elicitation protocol significantly modulates reliability. Simple interventions such as top-$R$4 answer elicitation—where the model produces several candidate answers with softmaxed probabilities—reliably reduce overconfidence and improve BAS across models and settings. Post-hoc calibration via isotonic regression is also highly effective, improving ECE and shifting BAS from negative to near-zero or positive values across the board.

Figure 6: Top-$R$5 confidence elicitation strategies substantially improve BAS, outperforming direct and reflection-based elicitation.

Figure 7: Post-hoc calibration aligns reported confidence with empirical accuracy, improving BAS and reducing high-confidence errors.

Theoretical and Practical Implications

Theoretical

• BAS provides a normative, decision-theoretic foundation for abstention-aware model evaluation, defining a unique proper scoring rule for the selective prediction regime.
• It formally justifies truthful confidence reporting as a utility-maximizing policy and extends gracefully to non-uniform, application-specific risk priors.

Practical

• Models with high accuracy or standard calibration are not necessarily decision-reliable; robust deployment requires explicit evaluation under BAS, especially in open-ended or safety-critical domains.
• Practical gains can be achieved through prompt engineering (top-$R$6 protocols) and minimal post-hoc calibration, requiring only black-box model access.

Future Developments

Prospective directions include:

• Integrating BAS into in-the-loop safety systems, e.g., dynamic rejection in clinical or scientific automation pipelines.
• Training objectives that directly optimize for BAS, especially in RLHF or supervised fine-tuning with explicit abstain/reject supervision.
• Extension of BAS to structured outputs, sequence-to-sequence tasks, and agentic reasoning pipelines.

Conclusion

The introduction of BAS represents a substantial methodological advance in the selective prediction evaluation of LLMs by enforcing a rigorous, decision-theoretic alignment between reported confidence and risk-aware utility. Empirical results demonstrate that BAS reveals model failure modes masked by conventional metrics, systematically penalizes rare but catastrophic overconfidence, and motivates practical improvements in elicitation and calibration. Hence, BAS should serve as a standard benchmark for the deployment and comparison of LLMs in settings where abstention and calibrated uncertainty underwrite safe and reliable decision-making.

---

Reference: "BAS: A Decision-Theoretic Approach to Evaluating LLM Confidence" (2604.03216)