Reliable measurement of hallucination in long-form settings

Develop a reliable method to measure hallucination in long-form generation, where hallucination is defined as correctness of facts with respect to a model’s internal knowledge, distinct from factuality assessed against external world knowledge.

Background

The work focuses on factuality—correctness with respect to external world knowledge—using SAFE to verify individual facts via Google Search.

The authors note that this is different from hallucination—correctness relative to a model’s internal knowledge—and state that a reliable approach for measuring hallucination in long-form outputs remains unsettled.

References

Furthermore, our work considers factuality (i.e., correctness of facts with respect to world knowledge), and so it is still unclear how to reliably measure hallucination (i.e., correctness of facts with respect to a model's internal knowledge) in long-form settings.

— Long-form factuality in large language models  (2403.18802 - Wei et al., 2024) in Conclusion (sec:conclusion)

Whether these are hallucinations or genuine pre-existing defects in the source paper is unresolved; either way, an auditor whose findings require their own audit loses much of its value.

— When Auditors Fabricate: Batch-Size Degradation and Confident Hallucination in LLM Detection of Planted Document Contamination  (2609.09696 - Parekh et al., 9 Sep 2026) in Section 4.4, “False positives”

We don't know if any of these examples are exact memorized passages or hallucinated ones.

— LLMs are not stochastic parrots: Evidence for meaning-mediated abstraction from conlang-like tasks  (2609.34187 - Zimmerman et al., 28 Sep 2026) in Appendix, Section “Exploratory output modes”

It remains an open empirical question whether our methods work across frontier models (70B+) or across long-form, open-domain generation tasks (e.g., biography generation).

— External Observers May See More Clearly: Cross-Model Span-Level Hallucination Detection in Large Language Models via Hidden State Probing  (2610.02066 - Gupta et al., 1 Oct 2026) in Section 7, Limitations