Claim-level Uncertainty Quantification

• Claim-level UQ is a paradigm that quantifies uncertainty for individual atomic assertions (claims) in model outputs rather than using aggregate measures.
• It employs geometric, probabilistic, and decision-theoretic methods to produce calibrated confidence intervals, facilitating selective trust and targeted correction.
• Techniques such as token likelihood, local entropy, and graph-based approaches are used to mitigate hallucinations and enhance model interpretability.

Claim-level Uncertainty Quantification (UQ) is an advanced paradigm for estimating and validating the local confidence associated with atomic assertions—“claims”—within the predictions of complex models, particularly in domains such as LLMs, computational argumentation, and scientific inference. In contrast to traditional aggregate UQ, claim-level approaches quantify uncertainty with a granularity that aligns with sub-sentential or minimal logical units of knowledge, enabling selective trust, granular abstention, hallucination mitigation, and robust fact verification. The claim-level UQ landscape incorporates geometric, probabilistic, information-, feature-, and decision-theoretic methodologies, often tailored for specific architectures or data regimes. This article surveys the foundations, major algorithmic approaches, evaluation protocols, and contemporary benchmarks for claim-level UQ.

1. Formal Definitions and Problem Statement

A claim is defined as a minimal atomic assertion or contiguous sequence of tokens $c$ within a model output $y$, corresponding to a discrete proposition or factual statement, possibly with explicit semantic segmentation (Dentan et al., 21 Nov 2025). For an input $x$, a model $M$ produces claims $\{c_1, ..., c_N\}$ with corresponding predictions or generated outputs. The goal of claim-level UQ is to assign to each claim $c$ a quantitative uncertainty score, typically $U(c) \in [0,1]$, where higher values denote higher epistemic or predictive doubt. The precise semantics of $U$ may vary by domain: in LLMs, it often reflects hallucination risk or local factuality error (Shelmanov et al., 13 May 2025); in statistical estimation, it may represent worst-case posterior risk (Bajgiran et al., 2021).

Claims can be delineated by deterministic segmentation (e.g., via rule-based parsers) or via syntactic/semantic heuristics, allowing reproducible and efficient token-to-claim mapping (Dentan et al., 21 Nov 2025). Claim-level UQ thus refines standard UQ by focusing confidence assessment on localized, verifiable assertions, supporting selective rejection, highlighting, and targeted correction.

2. Geometric, Probabilistic, and Decision-Theoretic Foundations

The decision-theoretic framework for claim-level UQ is exemplified by “4th-kind” UQ, which synthesizes robust optimization, Bayesian, and decision risk approaches. Given observed data $x$ and a likelihood function $\ell(\theta|x)$, one defines a relative likelihood region $$L_\epsilon = \{\theta : \ell(\theta|x)/\ell(\hat\theta_{\text{ML}}|x) \geq \epsilon\}$$ with a rarity parameter $\epsilon\in[0,1]$. The worst-case posterior risk for a scalar quantity of interest $f(\theta)$ is determined by minimizing the maximum $L^1$ risk over all priors supported on $L_\epsilon$, yielding an estimator $\delta^*$ and risk $\rho^*$ that correspond to the center and radius of the minimum enclosing ball (MEB) of $f(L_\epsilon)$:

$$\delta^* = \frac{\min_{θ∈L_ε} f(θ) + \max_{θ∈L_ε} f(θ)}{2}, \quad \rho^* = \frac{\max_{θ∈L_ε} f(θ) - \min_{θ∈L_ε} f(θ)}{2},$$

resulting in the claim-level UQ interval $[\delta^*-\rho^*, \delta^*+\rho^*]$ (Bajgiran et al., 2021). In high dimensions, computation of the MEB can be done efficiently via conditional-gradient methods, avoiding the curse of dimensionality.

The rarity parameter $\epsilon$ controls the accuracy-uncertainty tradeoff: large $\epsilon$ contracts $L_\epsilon$ and downsizes UQ intervals, while small $\epsilon$ yields conservative, larger regions and more robust but less informative UQ. In canonical settings (Gaussian likelihood, linear $f$), explicit formulas for claim intervals are available. This paradigm is minimax-optimal and posterior in nature, thus unifying decision uncertainty and worst-case estimation.

3. Distributional, Information-Theoretic, and Feature-Based Methods

In generative models such as LLMs, claim-level UQ often leverages the model’s internal token probability distributions or learned features:

• Token Likelihood: For token sequence $c = (x_t)_{t\in S}$, $s_t^{\text{LLH}} = p(x_t|x_{<t})$; aggregate over tokens to yield claim-level UQ via $\prod_{t\in c} s_t$ (Dentan et al., 21 Nov 2025).
• Local Entropy: Token-entropy $s_t^H = -\sum_{k=1}^K p(k|x_{<t})\log p(k|x_{<t})$ can be aggregated (mean, max, sum) per claim (Dentan et al., 21 Nov 2025, Fadeeva et al., 27 May 2025).
• Maximum Claim Probability: The “max-claim” probability $p(c|x)$ is a core uncertainty measure in both RAG and non-RAG LLMs (Fadeeva et al., 27 May 2025).
• Mahalanobis Density Methods: Compute the Mahalanobis distance of token embeddings at each layer to the centroid of in-distribution correct claims, then aggregate via PCA and learn a linear regressor for robust per-claim uncertainty (Vazhentsev et al., 20 Feb 2025).
• Feature-Gap/Epistemic UQ: Quantify the gap between actual model hidden states $h_t$ and ideal, prompted states $h^*_t$ in terms of semantic feature directions, e.g., context-reliance, comprehension, honesty. Projecting activations along learned directions yields token-level epistemic scores, which are then aggregated claim-wise; shown to outperform classic entropy or perplexity-based UQ (Bakman et al., 3 Oct 2025).
• Pre-trained Transformer UQ Heads: Auxiliary Transformer modules (“UQ heads”) are trained on attention/probability-derived features for each claim span, producing direct claim-level hallucination probabilities that generalize across domains and languages (Shelmanov et al., 13 May 2025).
• Direct Prompting: In LLMs, directly eliciting a confidence score per claim (“verbalized UQ”) via tailored prompting often provides well-calibrated claim-level UQ at minimal compute cost (Zhou et al., 26 Sep 2025).

4. Graph-Based, Consistency, and Sample-Diversity Frameworks

Ensemble, graph-theoretic, and sample-diversity approaches construct explicit semantic graphs or sample sets to estimate uncertainty:

• Directional Entailment Graphs: Nodes correspond to claims/responses, edges are weighted by NLI-derived entailment probabilities; a random-walk Laplacian $L_{rw}$ is formed, and the sum of eigenvalue deficits $U_{EigV}^d = \sum_k \max(0,1-\lambda_k)$ yields global or per-claim uncertainty (Da et al., 1 Jul 2024).
• Semantic Entropy and Eccentricity: Multiple sampled responses are clustered via bidirectional entailment (NLI); semantic entropy or graph-based eccentricity of the sample cloud captures UQ (Zhou et al., 26 Sep 2025).
• LUQ: Average pairwise NLI-based semantic disagreement measures per-claim uncertainty; effective as a “backup” to direct prompting in argument verification pipelines (Zhou et al., 26 Sep 2025).

These frameworks can be combined (e.g., via normalization and averaging) with distributional UQ for enhanced robustness. Response augmentation—a process for expanding or clarifying vague claims—serves to improve graph connectivity and reduce artificial uncertainty (Da et al., 1 Jul 2024).

5. Faithfulness-Aware and Conditional UQ for Context-Sensitive Generation

Claim-level UQ in Retrieval-Augmented Generation (RAG) systems must distinguish between factuality (truth relative to the world) and faithfulness (consistency with retrieved context). The FRANQ method first estimates claim faithfulness using a continuous entailment score (AlignScore), and then applies different UQ estimators based on faithfulness. For claims faithful to the retrieval, NLI-based uncertainty dominates; for unfaithful claims, the model probability under parametric knowledge is used (Fadeeva et al., 27 May 2025). The score is a mixture:

$$U_{\mathrm{FRANQ}}(c) = P_{\mathrm{faithful}}(c,r) UQ_{\mathrm{faith}}(c) + (1-P_{\mathrm{faithful}}(c,r)) UQ_{\mathrm{unfaith}}(c),$$

with each UQ component calibrated on held-out data via isotonic regression for optimal detection and calibration of claim-level hallucinations.

6. Benchmarking, Evaluation Protocols, and Empirical Findings

Rigorous evaluation protocols are enabled by datasets such as MUCH (Multilingual Claim Hallucination Benchmark) (Dentan et al., 21 Nov 2025), which provide token-level logits and deterministic segmentation for >20,000 claims in four languages. Key metrics include ROC-AUC and PR-AUC; operational points include TPR at fixed FPR and recall at fixed precision. Aggregation strategies (e.g., product over token likelihoods) can dramatically affect performance. Efficiency is a practical imperative: methods such as CCP or graph-based approaches may incur 100%+ inference overhead, while token-likelihood or entropy-based baselines achieve sub-1% overhead with small accuracy penalties.

Empirical findings underscore that current state-of-the-art methods remain imperfect: high-precision regimes (low FPR) are not reliably achieved, and multilingual transfer remains challenging for NLI-dependent methods. Claim segmentation is no longer a computational bottleneck, with algorithms such as “much” operating at <1% of model generation time. Among LLM justification pipelines, prompt-based confidence and simple product-of-token probabilities deliver the strongest tradeoff between calibration, discrimination, and speed (Dentan et al., 21 Nov 2025, Zhou et al., 26 Sep 2025).

7. Calibration, Validation, and Best Practices

Best practices in claim-level UQ demand robust calibration and sharpness analysis beyond accuracy. Essential diagnostics include:

• Calibration Curves/Reliability Diagrams: Empirical coverage plotted vs. nominal probability; deviations (miscalibration area, calibration error) are critical (Pernot, 2022).
• PIT Histograms: Uniformity of predictive interval probabilities for the observed values is a hallmark of well-calibrated UQ.
• Local Calibration and Sharpness: Calibration and uncertainty scatter can be examined across bins in predicted value or uncertainty (LCP/LZV).
• Cross-validation: Rigorous cross-validation protocols are required when fitting post-hoc uncertainty models or for error-correction methods.
• Operational Reporting: Both standard ($1\sigma$) and expanded ($\sim$95%) uncertainty intervals should be reported, with explicit coverage claims (Pernot, 2022).

Claim-level UQ outputs should facilitate actionable downstream decisions, such as selective abstention or manual review, especially where low recall or adverse calibration could have high downstream cost.

---

In summary, claim-level Uncertainty Quantification constitutes a multi-faceted, rapidly maturing field with foundational connections to robust statistics, geometric risk, information theory, distributional representation, and argumentation theory. It is characterized by a proliferation of technically sophisticated algorithms, explicit benchmarking standards, and a growing awareness of the need for sharp, well-calibrated, efficiently computable uncertainty estimates on local, interpretable units of model output. The state of the art remains fluid, marked by a continuous interplay between methodological innovation, empirical benchmarking, and real-world deployment constraints.