IUQ: Interrogative Uncertainty Quantification for Long-Form Large Language Model Generation

Abstract: Despite the rapid advancement of LLMs, uncertainty quantification in LLM generation is a persistent challenge. Although recent approaches have achieved strong performance by restricting LLMs to produce short or constrained answer sets, many real-world applications require long-form and free-form text generation. A key difficulty in this setting is that LLMs often produce responses that are semantically coherent yet factually inaccurate, while the underlying semantics are multifaceted and the linguistic structure is complex. To tackle this challenge, this paper introduces Interrogative Uncertainty Quantification (IUQ), a novel framework that leverages inter-sample consistency and intra-sample faithfulness to quantify the uncertainty in long-form LLM outputs. By utilizing an interrogate-then-respond paradigm, our method provides reliable measures of claim-level uncertainty and the model's faithfulness. Experimental results across diverse model families and model sizes demonstrate the superior performance of IUQ over two widely used long-form generation datasets. The code is available at https://github.com/louisfanhz/IUQ.

• The paper introduces IUQ, a novel framework that extracts atomic claims and assesses their faithfulness to detect fabrications in LLM outputs.
• It employs targeted question generation and contradiction analysis in a context-decoupled manner to expose hidden errors in long-form texts.
• Empirical results show IUQ outperforms baseline methods by reliably identifying propagation of fabricated claims across diverse LLM models and datasets.

Interrogative Uncertainty Quantification for Long-Form LLM Outputs

Motivation and Problem Statement

LLMs have demonstrated high performance in various NLP applications, but quantifying the uncertainty of their generated long-form content remains unsolved. Existing Uncertainty Quantification (UQ) techniques are mostly confined to short or structured responses, utilizing token-level probabilities or semantic entailment to interpret confidence. However, in long-form generation, LLMs face two critical issues: (i) the need to maintain logical flow may cause semantic conditioning that masks fabrication, and (ii) cross-sample consistency is often misleading, as LLMs rarely contradict themselves even when hallucinating. This leads to the phenomenon where a model can consistently generate plausible yet factually incorrect information across samples, which standard consistency-based UQ cannot effectively expose. Figure 1 exemplifies this behavior for biographical text generation.

Figure 1: LLM-generated biographies may fabricate details for consistency, yielding similar outputs across samples, yet the model reveals uncertainty when probed out-of-context about foundational facts.

IUQ Framework

This paper proposes Interrogative Uncertainty Quantification (IUQ), a black-box evaluation method that decomposes LLM generations into atomic claims and interrogates the model on each claim in a context-decoupled fashion. This dual process involves:

1. Claim Extraction: The long-form output is segmented into minimal semantic units (atomic claims).
2. Interrogation: For each claim, targeted questions are generated by the model, and new answers are elicited in separate sessions, isolating claimed knowledge from generative context.
3. Faithfulness Assessment: Contradictory answers, as judged via the model itself or NLI probes, directly expose whether the original claim reflects the model’s actual knowledge rather than hallucinated consistency.

The IUQ pipeline (see Figure 2) integrates both inter-sample consistency (how consistently claims appear across diverse generations) and intra-sample faithfulness (the alignment of claims with independently elicited answers about the same fact), with a kernel-weighted influence function to propagate the impact of early fabrications through subsequent claims.

Figure 2: The IUQ process samples model outputs, decomposes each into atomic claims, independently probes the model on these claims without original context, and computes faithfulness by checking for intra-model contradictions.

Quantification Formalism and Metrics

For each claim $c_i$ in a response $R$, the faithfulness score $F(c_i)$ is defined as one minus the contradiction rate between freshly generated answers to targeted questions and preceding claims. The response-level faithfulness $F(R)$ is the mean across all claims. IUQ introduces an uncertainty function $U(c_i) = S(c_i) \cdot W(c_i)$, where $S(c_i)$ is the proportion of sampled generations supporting $c_i$, and $W(c_i)$ encodes an exponentially-decayed sum of prior unfaithfulness, reflecting cumulative contextual contamination.

This approach captures two key uncertainty aspects:

• Fabrication-driven consistency: Cross-sample support no longer guarantees knowledge fidelity, as contextually conditioned fabrication may reproduce the same error without contradiction.
• Context propagation: Fabricated claims may serve as premises for all subsequent content, mandating downstream uncertainty inflation.

Empirical Results

Extensive experiments cover a spectrum of LLM families (GPT-4o, LLaMA3.1, LLaMA3.3, Qwen2, Gemma-3, Mistral) and two long-form datasets: FActScore (biographies of obscure entities) and LongFact (questions from well-known domains). IUQ outperforms all baselines, including token-probability, perplexity, claim-conditioned probability, graph-based methods, and sample-level claim entailment. Figure 3 shows the claim faithfulness distribution for representative models and datasets, highlighting less faithfulness in low-exposure domains.

Figure 3: Distribution of claim faithfulness scores; lower means indicate higher rates of detected fabrication and contradiction against model’s own knowledge.

Figure 4 shows that area under ROC curves (AUROC) for IUQ grow as more response samples are used, preserving monotonicity and demonstrating more stable uncertainty signal than alternatives.

Figure 4: AUROC for IUQ and baselines as a function of sampled generations—IUQ achieves both higher and more robust performance.

Further, IUQ provides fine-grained time-series portraits of faithfulness within each response (Figure 5), revealing that fabricated premises early in context propagate uncertainty to subsequent claims.

Figure 5: Visualization of faithfulness landscapes across claims in a generation; lower faithfulness frequently emerges at the beginning for obscure topics, with error cascading downstream.

Notably, for FActScore, GPT-4o achieves a faithfulness score of 0.927 vs. 0.697 for Gemma-3, with even the best systems substantially less reliable on less-canonical entities. On popular topics (LongFact), all models are more faithful, but IUQ remains essential to distinguish unconscious fabrication.

Qualitative Analysis

IUQ’s intra-model probing is critical in domains where contextually consistent fabrication evades cross-sample support detection. As the anecdotal analysis and breakdown show, multiple generations may reinforce the same fabricated fact (e.g., incorrectly stating Rory Byrne is an architect), artificially inflating supportness even as the underlying claim is unsound. IUQ’s contradiction-based faithfulness reveals these weak points—contradiction rates surge for such contextually-conditioned mistakes.

Comparison and Ablation

Ablation studies confirm the necessity of faithfulness propagation. Removing the exponential decay kernel and propagating only local claim errors substantially limits uncertainty discrimination. Directional checks further indicate that modeling contradiction against prior context (rather than future claims) is necessary; reversing this reduces detection sensitivity and correlation with correctness.

Computational Considerations

IUQ incurs higher computational cost due to iterative claim extraction, question generation, and answer verification, elevating average token usage substantially compared to graph-based UQ. However, the increased diagnostic signal and reliability justify the cost in high-stakes or safety-critical applications.

Limitations

Primary limitations include reliance on the LLM’s own question-answering and reasoning abilities, exposing IUQ’s results to self-hallucination or paraphrase errors during claim extraction and probing. The pipeline also does not process responses that express refusal (“I don’t know”), and thus uncertainty is only attributed over affirmative outputs. The method is also computationally intensive.

Implications and Future Directions

IUQ exposes the subtle decoupling between logical consistency and factual reliability in LLMs, particularly in low-signal or long-tail environments. Practically, IUQ can serve downstream systems requiring calibrated trust or selective answer presentation (e.g., grounded generative QA, risk-sensitive summarization, agent-based reasoning with explicit risk tracking).

Theoretically, decoupling intra-sample and inter-sample signal clarifies the need for context-independent truth-reflection, and future work could implement IUQ within continual learning or retrieval-augmented generation for dynamic knowledge tracking. Further refinements could integrate stronger claim extraction, non-parametric reference cross-checking, or multi-agent debate.

Conclusion

Interrogative Uncertainty Quantification provides a rigorous framework for fine-grained uncertainty estimation on long-form LLM outputs, efficiently surfacing hallucination where context consistency and self-reinforcement would otherwise mask error. Empirically, IUQ achieves superior uncertainty discrimination across both obscure and canonical domains and highlights substantial residual risk of context-conditioned fabrication among frontier LLMs.

(2604.15109)