Alignment Uncertainty Quantification

• Alignment Uncertainty Quantification is a framework that evaluates uncertainty relative to explicit targets such as human judgment, value alignment, and decision utility.
• It integrates various methods including calibration metrics, Bayesian analysis, ensemble techniques, and decision-aligned evaluation to measure model performance.
• AUQ has practical applications in large language models, climate finance, and control systems, addressing challenges like overconfidence and misalignment.

Searching arXiv for the specified AUQ papers and related recent work. Alignment Uncertainty Quantification (AUQ) denotes a family of uncertainty-analysis frameworks in which the central object is an alignment relation rather than uncertainty in isolation. In LLMs, AUQ has been used to ask both whether a model’s stated confidence matches its actual accuracy and whether its uncertainty judgments mirror those of human respondents (Moore et al., 29 May 2026), and also to address the uncertainty inherent in aligning models with human values and intentions (Lu et al., 25 Jul 2025). A related evaluation program asks whether a UQ metric is itself aligned with downstream decision utility (Schneider et al., 25 Jun 2026). In climate-finance, AUQ is the systematic propagation and assessment of all sources of uncertainty—model, parameter, input-data and scenario—through a portfolio temperature alignment framework (Weichel et al., 2024).

1. Distinct senses of “alignment” in AUQ

In the literature represented here, AUQ is not a single formalism. The term is used for several technically distinct targets of alignment, each with its own observables, objectives, and guarantees.

Setting	Alignment target	Representative formalism
LLM uncertainty behavior	Model uncertainty should “mirror those of human respondents”	Correlation between $U_{\mathrm{model}}(x)$ and $U_{\mathrm{human}}(x)$
LLM value alignment	Behavior should reflect “human values and intentions”	Alignment gap $\mathcal G(x)$
UQ evaluation	Metric should rank models by downstream utility	$$M_\pi(f,y)=\int_\Theta[-U_\theta(f,y)]\,\pi(d\theta)$$
Portfolio temperature alignment	Uncertainty should be propagated through XDC and FaIR	Posterior over $\theta$ and implied temperature trajectories

The first sense is behavioral and comparative: a model is not only calibrated if its confidence tracks correctness, but aligned if its uncertainty resembles human uncertainty. The second is normative: uncertainty concerns whether an aligned policy actually reflects intended values across contexts and edge cases. The third is evaluative: a metric is useful only if it preserves the ranking induced by realized utility. The fourth is system-level and forward-propagative: uncertainty is pushed through a socio-economic and climate-model pipeline to quantify the uncertainty of a portfolio’s “implied temperature” (Moore et al., 29 May 2026).

These usages share a common structure. Each treats uncertainty as meaningful only relative to a reference object—human respondents, human values, downstream decisions, or climate-alignment trajectories. A plausible implication is that AUQ is best understood as a relational program: uncertainty estimates are evaluated by what they align to, not merely by how sharply or smoothly they are expressed.

2. Formal objectives and metrics

In the human-uncertainty setting, calibration and alignment are explicitly separated. Calibration is defined by the gap between a model’s numerical confidence and its empirical accuracy. With $\mathrm{Conf}(x)$ denoting confidence and $\mathrm{Acc}(x)\in\{0,1\}$ correctness, the expected calibration error is

$$\mathrm{ECE}=\mathbb{E}_{\text{bucket}}\bigl|\mathrm{acc}(B_m)-\mathrm{conf}(B_m)\bigr| \approx \sum_{m=1}^M \frac{|B_m|}{n}\bigl|\mathrm{acc}(B_m)-\mathrm{conf}(B_m)\bigr|.$$

To mitigate binning bias, ECESweep is used, choosing $M$ so that empirical accuracy increases monotonically with bin index. Alignment, by contrast, is the statistical association between model-derived and human-derived uncertainty: $$\rho_{\mathrm{align}} = \mathrm{Spearman}\bigl(\{U_{\mathrm{model}}(x_i)\},\{U_{\mathrm{human}}(x_i)\}\bigr),$$ with Pearson’s $U_{\mathrm{human}}(x)$0 used when appropriate (Moore et al., 29 May 2026).

In the value-alignment setting, the central object is the alignment gap. Let $U_{\mathrm{human}}(x)$1 be an aligned policy and $U_{\mathrm{human}}(x)$2 the utility of response $U_{\mathrm{human}}(x)$3 to prompt $U_{\mathrm{human}}(x)$4 under value function $U_{\mathrm{human}}(x)$5. The gap between the model’s internal value estimate $U_{\mathrm{human}}(x)$6 and true human values $U_{\mathrm{human}}(x)$7 is

$U_{\mathrm{human}}(x)$8

AUQ then seeks to characterize the distribution, magnitude, and causes of $U_{\mathrm{human}}(x)$9, so that deployers understand when—and by how much—the model may be misaligned. The same literature distinguishes model-inherent uncertainty, human feedback variability and value pluralism, and context sensitivity and distribution shift as the three main categories of alignment uncertainty (Lu et al., 25 Jul 2025).

In decision-aligned evaluation, the formal question is different: given predictor $\mathcal G(x)$0, action set $\mathcal G(x)$1, and utility family $\mathcal G(x)$2, an evaluation metric $\mathcal G(x)$3 is decision-aligned under prior $\mathcal G(x)$4 if, for each fixed $\mathcal G(x)$5, there exists a strictly increasing $\mathcal G(x)$6 such that

$\mathcal G(x)$7

The canonical choice is the prior-weighted utility metric

$\mathcal G(x)$8

which ranks models exactly by their prior-weighted downstream utility and is strictly proper when the Bayes-act under each $\mathcal G(x)$9 is unique (Schneider et al., 25 Jun 2026).

These formulations encode different notions of correctness. Calibration asks whether confidence predicts empirical success. Alignment to humans asks whether uncertainty judgments resemble human judgments. Value-alignment AUQ asks whether behavior reflects intended values. Decision-alignment asks whether the metric used to judge uncertainty is itself utility-relevant.

3. Measurement strategies and algorithmic families

In LLMs, one important methodological division is between overt and internal uncertainty signals. Overt uncertainty is derived from the output distribution. For multiple-choice question answering, representative measures include Top-1-prob,

$$M_\pi(f,y)=\int_\Theta[-U_\theta(f,y)]\,\pi(d\theta)$$0

and Total-ent,

$$M_\pi(f,y)=\int_\Theta[-U_\theta(f,y)]\,\pi(d\theta)$$1

along with focused variants over $$M_\pi(f,y)=\int_\Theta[-U_\theta(f,y)]\,\pi(d\theta)$$2, $$M_\pi(f,y)=\int_\Theta[-U_\theta(f,y)]\,\pi(d\theta)$$3, or $$M_\pi(f,y)=\int_\Theta[-U_\theta(f,y)]\,\pi(d\theta)$$4. For free-response generation, token-level uncertainties are averaged across the generated span $$M_\pi(f,y)=\int_\Theta[-U_\theta(f,y)]\,\pi(d\theta)$$5 of length $$M_\pi(f,y)=\int_\Theta[-U_\theta(f,y)]\,\pi(d\theta)$$6; sequence perplexity is also used and converted to uncertainty by $$M_\pi(f,y)=\int_\Theta[-U_\theta(f,y)]\,\pi(d\theta)$$7. For open-ended factual recall, “Bracketed FR” measures operate on the marked answer span and “1TFR” on the first generated token. Internal AUQ is obtained with a layer-wise linear readout: for each layer $$M_\pi(f,y)=\int_\Theta[-U_\theta(f,y)]\,\pi(d\theta)$$8, the final-token representation $$M_\pi(f,y)=\int_\Theta[-U_\theta(f,y)]\,\pi(d\theta)$$9 is used in a 10-fold cross-validated ridge regression predicting $\theta$0, and alignment is measured by the Pearson correlation $\theta$1 between predicted and actual human uncertainty (Moore et al., 29 May 2026).

A second LLM line formulates uncertainty as a decomposition into aleatoric and epistemic components using only black-box text generations. With reference model $\theta$2, auxiliary ensemble $\theta$3, and sequence-semantic similarity $\theta$4, the within-model similarity

$\theta$5

induces aleatoric uncertainty

$\theta$6

while the cross-model similarity

$\theta$7

yields epistemic uncertainty

$\theta$8

Total uncertainty is additive: $\theta$9 This framework uses sentence-T5-xl embeddings and cosine similarity scaled to $\mathrm{Conf}(x)$0 (Hamidieh et al., 18 Apr 2026).

The broader AUQ survey for LLM alignment organizes methods into Bayesian Reward Modeling, Ensembles, Monte Carlo Dropout, Info-Theoretic Metrics, Conformal Alignment, Uncertainty-Aware Learning, and Token/Session AUQ. Bayesian reward modeling places a posterior over reward functions $\mathrm{Conf}(x)$1. Conformal alignment adapts conformal prediction to produce p-values for candidate responses and, on held-out calibration data, guarantees a user-specified false discovery rate $\mathrm{Conf}(x)$2; response sets with $\mathrm{Conf}(x)$3 are certified as aligned. Uncertainty-Aware Learning smooths reward signals according to

$\mathrm{Conf}(x)$4

Token-level epistemic uncertainty may be modeled as mutual information $\mathrm{Conf}(x)$5, then aggregated to utterance- or session-level risk scores (Lu et al., 25 Jul 2025).

Method	Strengths	Limitations
Bayesian Reward Modeling	Principled; captures full preference post.	Very high cost; requires strong priors
Ensembles	No special assumptions; practical	Requires training multiple models
Monte Carlo Dropout	Single model; approximate Bayesian	Approximation; only epistemic uncertainty
Info-Theoretic Metrics	Model-agnostic; scalable	May conflate aleatoric/epistemic uncertainty
Conformal Alignment	Provable FDR control; distribution-free	Needs calibration set; only selection
Uncertainty-Aware Learning	Integrates into training; smooths noise	Requires entropy estimation; tuning $\mathrm{Conf}(x)$6
Token/Session AUQ (multi-scale)	Fine granularity; dialog-level safety	Complex; multi-component

4. Empirical findings in LLMs

The most explicit empirical AUQ study of human-similar uncertainty reports that calibration and alignment are only partially coupled. On Coane MCQA, most models achieve $\mathrm{Conf}(x)$7 on at least one measure; LLaMa 2 13B base with total-ent yields $\mathrm{Conf}(x)$8. Yet instruct fine-tuning systematically degrades calibration: Wilcoxon tests across measure-model pairs show significant increases in ECE with $\mathrm{Conf}(x)$9 and effect sizes $\mathrm{Acc}(x)\in\{0,1\}$0–0.5. MCQA choice-prob exhibits “anti-calibration” $\mathrm{Acc}(x)\in\{0,1\}$1 in all instruct-tuned LLaMa 2 and Mistral models. On OEQA, calibration is more robust, and on some measures such as top-k entropy in free response, instruct fine-tuning slightly improves calibration with $\mathrm{Acc}(x)\in\{0,1\}$2. Human instance-level calibration errors on Coane are reported as $\mathrm{Acc}(x)\in\{0,1\}$3 for MCQA entropy, $\mathrm{Acc}(x)\in\{0,1\}$4 for MCQA RT, and $\mathrm{Acc}(x)\in\{0,1\}$5 for OEQA RT, so LLMs frequently outperform human calibration on these benchmarks (Moore et al., 29 May 2026).

Alignment to human uncertainty is weaker in overt outputs. Peak overt alignment reaches Spearman $\mathrm{Acc}(x)\in\{0,1\}$6 for Gemma 2 9B base on Coane MCQA entropy. ProtoQA shows weak but consistent $\mathrm{Acc}(x)\in\{0,1\}$7–0.15, CamChoice sporadic significant $\mathrm{Acc}(x)\in\{0,1\}$8, and OEQA overt signals are near zero except a singular $\mathrm{Acc}(x)\in\{0,1\}$9 for Falcon 3 10B RT. Instruct fine-tuning again uniformly reduces alignment, with Wilcoxon $$\mathrm{ECE}=\mathbb{E}_{\text{bucket}}\bigl|\mathrm{acc}(B_m)-\mathrm{conf}(B_m)\bigr| \approx \sum_{m=1}^M \frac{|B_m|}{n}\bigl|\mathrm{acc}(B_m)-\mathrm{conf}(B_m)\bigr|.$$0 and effects $$\mathrm{ECE}=\mathbb{E}_{\text{bucket}}\bigl|\mathrm{acc}(B_m)-\mathrm{conf}(B_m)\bigr| \approx \sum_{m=1}^M \frac{|B_m|}{n}\bigl|\mathrm{acc}(B_m)-\mathrm{conf}(B_m)\bigr|.$$1. Hidden states behave differently: linear probes recover substantially stronger human alignment, with maximal Pearson $$\mathrm{ECE}=\mathbb{E}_{\text{bucket}}\bigl|\mathrm{acc}(B_m)-\mathrm{conf}(B_m)\bigr| \approx \sum_{m=1}^M \frac{|B_m|}{n}\bigl|\mathrm{acc}(B_m)-\mathrm{conf}(B_m)\bigr|.$$2, roughly doubling overt alignment. Group-level uncertainty peaks in middle-late layers and grows monotonically with depth, whereas response-time signals are flatter across layers. Instruct-tuned models exhibit significantly lower probe correlations than base models, with Wilcoxon $$\mathrm{ECE}=\mathbb{E}_{\text{bucket}}\bigl|\mathrm{acc}(B_m)-\mathrm{conf}(B_m)\bigr| \approx \sum_{m=1}^M \frac{|B_m|}{n}\bigl|\mathrm{acc}(B_m)-\mathrm{conf}(B_m)\bigr|.$$3 (Moore et al., 29 May 2026).

The black-box cross-model disagreement line reaches a related conclusion from a different direction. Across five 7–9B instruction-tuned models and ten long-form tasks, total uncertainty $$\mathrm{ECE}=\mathbb{E}_{\text{bucket}}\bigl|\mathrm{acc}(B_m)-\mathrm{conf}(B_m)\bigr| \approx \sum_{m=1}^M \frac{|B_m|}{n}\bigl|\mathrm{acc}(B_m)-\mathrm{conf}(B_m)\bigr|.$$4 consistently outperforms aleatoric uncertainty $$\mathrm{ECE}=\mathbb{E}_{\text{bucket}}\bigl|\mathrm{acc}(B_m)-\mathrm{conf}(B_m)\bigr| \approx \sum_{m=1}^M \frac{|B_m|}{n}\bigl|\mathrm{acc}(B_m)-\mathrm{conf}(B_m)\bigr|.$$5 alone in correctness calibration, with average $$\mathrm{ECE}=\mathbb{E}_{\text{bucket}}\bigl|\mathrm{acc}(B_m)-\mathrm{conf}(B_m)\bigr| \approx \sum_{m=1}^M \frac{|B_m|}{n}\bigl|\mathrm{acc}(B_m)-\mathrm{conf}(B_m)\bigr|.$$6–0.10 and the largest gains on HotpotQA $$\mathrm{ECE}=\mathbb{E}_{\text{bucket}}\bigl|\mathrm{acc}(B_m)-\mathrm{conf}(B_m)\bigr| \approx \sum_{m=1}^M \frac{|B_m|}{n}\bigl|\mathrm{acc}(B_m)-\mathrm{conf}(B_m)\bigr|.$$7, CoQA $$\mathrm{ECE}=\mathbb{E}_{\text{bucket}}\bigl|\mathrm{acc}(B_m)-\mathrm{conf}(B_m)\bigr| \approx \sum_{m=1}^M \frac{|B_m|}{n}\bigl|\mathrm{acc}(B_m)-\mathrm{conf}(B_m)\bigr|.$$8, and WMT16-de-en $$\mathrm{ECE}=\mathbb{E}_{\text{bucket}}\bigl|\mathrm{acc}(B_m)-\mathrm{conf}(B_m)\bigr| \approx \sum_{m=1}^M \frac{|B_m|}{n}\bigl|\mathrm{acc}(B_m)-\mathrm{conf}(B_m)\bigr|.$$9. $M$0 also improves selective abstention, lowering AURC and increasing accuracy at $M$1 and $M$2 coverage; on HotpotQA, $M$3 improves from $M$4 under AU to $M$5 under TU. Under matched token budgets, the cross-model term adds signal that oversampling a single model cannot recover. Diagnostic analyses define Jaccard redundancy $M$6 and oracle coverage gain $M$7, finding that EU AUROC correlates positively with $M$8 $M$9 and negatively with $$\rho_{\mathrm{align}} = \mathrm{Spearman}\bigl(\{U_{\mathrm{model}}(x_i)\},\{U_{\mathrm{human}}(x_i)\}\bigr),$$0 $$\rho_{\mathrm{align}} = \mathrm{Spearman}\bigl(\{U_{\mathrm{model}}(x_i)\},\{U_{\mathrm{human}}(x_i)\}\bigr),$$1 (Hamidieh et al., 18 Apr 2026).

A recurrent empirical result is that instruct fine-tuning does not uniformly improve AUQ. In one line it degrades calibration and human-uncertainty alignment; in another, uncertainty gains come from cross-model disagreement rather than further single-model self-consistency. This suggests that uncertainty-relevant information may be compressed, distorted, or hidden by post-training regimes that prioritize answer style and single-best-answer behavior.

5. Decision-aligned evaluation and prior-weighted utility

Decision-aligned evaluation begins from the observation that generic uncertainty metrics need not imply high utility in downstream decisions. The formal criterion requires that an evaluation metric preserve the ordering induced by expected negative utility under a prior over decision scenarios. From this, prior-weighted utility metrics are defined directly as

$$\rho_{\mathrm{align}} = \mathrm{Spearman}\bigl(\{U_{\mathrm{model}}(x_i)\},\{U_{\mathrm{human}}(x_i)\}\bigr),$$2

with explicit instantiations for binary decision, selective prediction with abstention cost $$\rho_{\mathrm{align}} = \mathrm{Spearman}\bigl(\{U_{\mathrm{model}}(x_i)\},\{U_{\mathrm{human}}(x_i)\}\bigr),$$3, top-$$\rho_{\mathrm{align}} = \mathrm{Spearman}\bigl(\{U_{\mathrm{model}}(x_i)\},\{U_{\mathrm{human}}(x_i)\}\bigr),$$4 selection, and top-$$\rho_{\mathrm{align}} = \mathrm{Spearman}\bigl(\{U_{\mathrm{model}}(x_i)\},\{U_{\mathrm{human}}(x_i)\}\bigr),$$5 regression with risk aversion. If the Bayes-act under each $$\rho_{\mathrm{align}} = \mathrm{Spearman}\bigl(\{U_{\mathrm{model}}(x_i)\},\{U_{\mathrm{human}}(x_i)\}\bigr),$$6 is unique, then $$\rho_{\mathrm{align}} = \mathrm{Spearman}\bigl(\{U_{\mathrm{model}}(x_i)\},\{U_{\mathrm{human}}(x_i)\}\bigr),$$7 is strictly proper (Schneider et al., 25 Jun 2026).

This framework also reinterprets conventional scores. Binary NLL is decision-aligned to binary decisions under $$\rho_{\mathrm{align}} = \mathrm{Spearman}\bigl(\{U_{\mathrm{model}}(x_i)\},\{U_{\mathrm{human}}(x_i)\}\bigr),$$8, which places unbounded mass near $$\rho_{\mathrm{align}} = \mathrm{Spearman}\bigl(\{U_{\mathrm{model}}(x_i)\},\{U_{\mathrm{human}}(x_i)\}\bigr),$$9 and $U_{\mathrm{human}}(x)$00. Brier Score is decision-aligned under a uniform prior $U_{\mathrm{human}}(x)$01. Accuracy corresponds to a point mass at $U_{\mathrm{human}}(x)$02. By contrast, ECE, MCE, R-AUC, and error-detection are coordinate-dependent and cannot be written in the decision-alignment form for any $U_{\mathrm{human}}(x)$03. The critique is therefore not only empirical but structural: some metrics encode implausible priors, while others are not decision-aligned at all (Schneider et al., 25 Jun 2026).

The empirical protocol compares ten models on five binary-classification and five univariate-regression datasets, measuring the Kendall’s $U_{\mathrm{human}}(x)$04 between metric ranking and realized utility ranking over 100 random train/test splits. Only $U_{\mathrm{human}}(x)$05 achieves consistently high positive $U_{\mathrm{human}}(x)$06 for binary decisions; conventional metrics hover around $U_{\mathrm{human}}(x)$07 or below. In top-$U_{\mathrm{human}}(x)$08 selection, only $U_{\mathrm{human}}(x)$09 shows meaningful $U_{\mathrm{human}}(x)$10, while generic metrics are near zero. In selective prediction for regression, only $U_{\mathrm{human}}(x)$11 aligns with abstention-task utility; NLL is pathological with negative $U_{\mathrm{human}}(x)$12, and MSE aligns only at $U_{\mathrm{human}}(x)$13. The same pattern appears in real-world case studies. In day-ahead electricity bidding, PWUs achieve $U_{\mathrm{human}}(x)$14–0.20, while NLL and ECE are near zero or negative. In credit approval, $U_{\mathrm{human}}(x)$15 reaches $U_{\mathrm{human}}(x)$16, whereas generic metrics are often below $U_{\mathrm{human}}(x)$17. In P2P lending, $U_{\mathrm{human}}(x)$18 and $U_{\mathrm{human}}(x)$19 yield $U_{\mathrm{human}}(x)$20–0.3, while generic metrics remain near zero (Schneider et al., 25 Jun 2026).

A common misconception is that a low NLL or low ECE automatically implies decision-relevant uncertainty. The decision-alignment results reject that equivalence: utility alignment depends on the utility family and the prior over scenarios, not on generic scoring-rule performance alone.

6. Portfolio temperature alignment and dynamical precursors

In climate-finance, AUQ is defined as the systematic propagation and assessment of all sources of uncertainty—model, parameter, input-data and scenario—through a portfolio temperature alignment framework. The method is built around a fully Bayesian calibration of the FaIR simple climate model, integration with X-Degree Compatibility (XDC), a Delayed Rejection Adaptive Metropolis sampler for posterior exploration, a deep-learning emulator to accelerate sampling, and variance-based decomposition of uncertainty contributions. With $U_{\mathrm{human}}(x)$21 for $U_{\mathrm{human}}(x)$22 FaIR parameters and hyperparameters, the forward model is

$U_{\mathrm{human}}(x)$23

with posterior

$U_{\mathrm{human}}(x)$24

XDC maps portfolio emission intensity to a global emission pathway $U_{\mathrm{human}}(x)$25, which is then fed into FaIR to compute $U_{\mathrm{human}}(x)$26, and the implied temperature alignment is

$U_{\mathrm{human}}(x)$27

The emulator reduces each posterior-sampling evaluation from $U_{\mathrm{human}}(x)$28 s per 80 year FaIR run to $U_{\mathrm{human}}(x)$29 s, with held-out performance RMSE $U_{\mathrm{human}}(x)$30C and $U_{\mathrm{human}}(x)$31, yielding a $U_{\mathrm{human}}(x)$32 speed-up in MCMC (Weichel et al., 2024).

Variance decomposition is explicit: $U_{\mathrm{human}}(x)$33 First-order Sobol indices are then computed. In practice, parameter uncertainty contributes $U_{\mathrm{human}}(x)$34 of the $U_{\mathrm{human}}(x)$35 CI width under RCP 4.5, while emission-scenario uncertainty contributes $U_{\mathrm{human}}(x)$36. In Test Case 1, parameter UQ under SSP2–RCP4.5 gives $U_{\mathrm{human}}(x)$37C with $U_{\mathrm{human}}(x)$38 CI $U_{\mathrm{human}}(x)$39C, compared to $U_{\mathrm{human}}(x)$40C from simple Monte Carlo on priors. In a counterparty example, if SSAB’s green steel method were applied sector-wide under SSP2–RCP4.5, baseline global warming of $U_{\mathrm{human}}(x)$41C would reduce to $U_{\mathrm{human}}(x)$42C, an improvement of $U_{\mathrm{human}}(x)$43C with $U_{\mathrm{human}}(x)$44 credible intervals of $U_{\mathrm{human}}(x)$45C (Weichel et al., 2024).

A mathematically distinct precursor appears in stochastic flocking control, where uncertainty in interaction parameters alters the alignment dynamics of a Cucker–Smale model. Positions and velocities evolve with a random scattering rate $U_{\mathrm{human}}(x)$46, represented through generalized polynomial chaos expansions and Galerkin projection. In the uniform-interaction case, the expected velocity contains an exponent that changes sign when

$U_{\mathrm{human}}(x)$47

causing divergence; if $U_{\mathrm{human}}(x)$48, divergence occurs for $U_{\mathrm{human}}(x)$49. A selective model predictive control acting on gPC coefficients can steer the system to a target velocity and prevent blow-up even in unstable regimes (Albi et al., 2015).

This older control-theoretic work does not use the modern AUQ terminology, but it shows that “alignment under uncertainty” has a mathematically deeper lineage: random inputs can shift alignment thresholds, and uncertainty-aware control can restore consensus.

7. Limitations, controversies, and open questions

Several limitations recur across AUQ formulations. In LLM safety, alignment uncertainty arises from model-inherent uncertainty, human feedback variability and value pluralism, and context sensitivity and distribution shift. Annotator judgments exhibit inter-annotator agreement of only $U_{\mathrm{human}}(x)$50–0.8, and there is no single “true” reward function to learn. As norms evolve, a model aligned at time $U_{\mathrm{human}}(x)$51 may become misaligned at $U_{\mathrm{human}}(x)$52. These are not merely statistical nuisances; they are structural sources of irreducible uncertainty in the target itself (Lu et al., 25 Jul 2025).

Methodological trade-offs are also explicit. Bayesian reward modeling is principled but very high cost and requires strong priors. Ensembles and Monte Carlo dropout are practical but do not eliminate approximation issues. Info-theoretic metrics are scalable but may conflate aleatoric and epistemic uncertainty. Conformal alignment offers distribution-free FDR control but only for selection. Token/session AUQ provides dialog-level safety but is complex and multi-component. In the human-uncertainty literature, overt logits often show only modest alignment while hidden states carry substantially stronger human-like uncertainty signals; this makes post-hoc readout design a substantive modeling decision rather than a purely diagnostic add-on (Lu et al., 25 Jul 2025).

There are also concrete controversies. One is whether calibration is sufficient. The evidence says no: models can be well calibrated by ECE yet only weakly aligned with human uncertainty. Another is whether instruct fine-tuning should improve uncertainty behavior. In the reported experiments, instruct fine-tuning systematically erodes AUQ across overt behavior and hidden activations, reducing both calibration in several settings and correlation with human uncertainty. The paper conjectures that additional supervised instructions designed to sharpen answer correctness and style drive models toward overconfident, less human-like uncertainty distributions (Moore et al., 29 May 2026).

Open problems are named directly. For LLM uncertainty alignment, open questions include extending AUQ to complex reasoning tasks such as planning and code generation, exploring multi-sample or Bayesian UQ methods, and identifying and manipulating the exact subspaces that encode human-like uncertainty. In the broader alignment survey, the unresolved agenda includes value pluralism, robust AUQ under severe distribution shifts, continuous or online AUQ for models that adapt in deployment, multi-modal AUQ, and mechanistic insights that connect uncertainty to circuit-level explanations (Moore et al., 29 May 2026).

Taken together, these strands define AUQ as a technical program for quantifying how uncertainty relates to an alignment target. The target may be human uncertainty, human values, downstream utility, or temperature alignment. The central lesson is consistent across domains: uncertainty estimates become scientifically meaningful only when the reference object of alignment is explicit, formally measured, and empirically validated.