The Curious Case of Hallucinatory (Un)answerability: Finding Truths in the Hidden States of Over-Confident Large Language Models (2310.11877v2)

Abstract: LLMs have been shown to possess impressive capabilities, while also raising crucial concerns about the faithfulness of their responses. A primary issue arising in this context is the management of (un)answerable queries by LLMs, which often results in hallucinatory behavior due to overconfidence. In this paper, we explore the behavior of LLMs when presented with (un)answerable queries. We ask: do models represent the fact that the question is (un)answerable when generating a hallucinatory answer? Our results show strong indications that such models encode the answerability of an input query, with the representation of the first decoded token often being a strong indicator. These findings shed new light on the spatial organization within the latent representations of LLMs, unveiling previously unexplored facets of these models. Moreover, they pave the way for the development of improved decoding techniques with better adherence to factual generation, particularly in scenarios where query (un)answerability is a concern.

• The paper demonstrates that LLMs encode query answerability in their latent representations, notably in the first decoded token.
• The paper presents refined decoding strategies that enhance factual adherence and boost performance on QA datasets by up to 80%.
• The paper introduces a selective erasure method for the answerability subspace, providing actionable insights for advanced AI development.

Exploring Hallucinatory Behavior in LLMs: The Case of (Un)answerability

The paper "The Curious Case of Hallucinatory (Un)answerability: Finding Truths in the Hidden States of Over-Confident LLMs" addresses a crucial challenge faced by LLMs, specifically their tendency to produce hallucinatory responses when confronted with (un)answerable questions. This phenomenon is attributed to the overconfidence of the models, manifesting as a lack of distinction between queries that are inherently answerable and those that are not. This research investigates whether current LLMs encode the answerability of a query and evaluates how effectively this embedded information can be leveraged to improve model outputs.

Summary of Findings

The paper presents several noteworthy findings:

1. Latent Representations of (Un)answerability: The authors demonstrate strong indications that LLMs, when generating hallucinatory answers, indeed encode the answerability of the input query within their latent representations. This is especially evident in the representation of the first decoded token, which can often serve as a reliable indicator of answerability.
2. Decoding Strategies: By exploring improved decoding strategies, the researchers highlight the potential of enhancing factual adherence in generated responses, particularly when the answerability of queries is in question.
3. Performance Analytics Using QA Datasets: Utilizing three question-answering (QA) datasets—SQuAD, NQ, and MuSiQue—the paper showcases a significant performance boost for queries flagged as (un)answerable (up to 80%) when prompts explicitly mention the potential for (un)answerability.
4. Beam Search Insights: Within the broader context of decoding using beam search, the paper identifies that (un)answerability is not only often present but is encoded in the beam responses generated for such queries. This implies that LLMs inherently possess a differentiating capacity that might not be evident in top-beam selections, unveiling a depth of latent knowledge previously underutilized.
5. Identification and Erasure of Answerability Subspace: The research introduces methods to not only identify but selectively erase the linear subspace associated with answerability, successfully demonstrating the presence and separability of this information across different datasets.

Implications and Future Directions

The findings of this research have several significant implications and open up new avenues in the theoretical and practical understanding of LLMs:

• Enhanced Decoding Techniques: By integrating prompt strategies or improved beam selection methods, systems utilizing LLMs can potentially reduce the prevalence of hallucinations, thereby yielding more reliable information retrieval and interaction capabilities.
• Foundational Insights for AI Development: The investigation into latent representations of (un)answerability contributes a foundational understanding that can inform the development of future models, particularly with respect to managing uncertainty and engendering more nuanced contextual awareness.
• Cross-Dataset Generalization Potential: Demonstrated ability to generalize the detection of (un)answerability across different datasets suggests strong potential for these methods to be extrapolated and applied in diverse AI contexts, ranging from open-domain QA to interactive AI agents.

The paper thereby provides critical insights into the internal mechanics of LLMs, revealing both their latent strengths and areas for methodological development. Future research might extend this line of inquiry by exploring non-linear methodologies for representing (un)answerability and expanding testing to broader and more diverse datasets and scenarios. This endeavor is pivotal for evolving AI capabilities towards more adaptive, intuitive, and reliable models in various real-world applications.