Hallucination is Inevitable: An Innate Limitation of Large Language Models (2401.11817v1)

Abstract: Hallucination has been widely recognized to be a significant drawback for LLMs. There have been many works that attempt to reduce the extent of hallucination. These efforts have mostly been empirical so far, which cannot answer the fundamental question whether it can be completely eliminated. In this paper, we formalize the problem and show that it is impossible to eliminate hallucination in LLMs. Specifically, we define a formal world where hallucination is defined as inconsistencies between a computable LLM and a computable ground truth function. By employing results from learning theory, we show that LLMs cannot learn all of the computable functions and will therefore always hallucinate. Since the formal world is a part of the real world which is much more complicated, hallucinations are also inevitable for real world LLMs. Furthermore, for real world LLMs constrained by provable time complexity, we describe the hallucination-prone tasks and empirically validate our claims. Finally, using the formal world framework, we discuss the possible mechanisms and efficacies of existing hallucination mitigators as well as the practical implications on the safe deployment of LLMs.

• The paper formalizes hallucination in LLMs via a diagonalization argument, proving that no provable configuration can learn all computable functions.
• It validates the theory through empirical tasks like string listing and binary order comparisons, highlighting model failures as complexity rises.
• The research underscores the need for human oversight and robust safety measures since mitigation approaches cannot completely eliminate hallucinations.

An Analysis of the Inevitability of Hallucination in LLMs

Overview

Hallucination, where a LLM generates plausible but incorrect or nonsensical information, poses a critical challenge to the deployment of LLMs in various applications. This paper formalizes hallucination in LLMs and offers a fundamental examination of whether such hallucinations can be completely eradicated. Unlike previous empirical studies, this work takes a formal approach by defining a structured world where hallucination is delineated as discrepancies between a computable LLM and a computable ground truth function. Results from learning theory are then leveraged to establish the inevitability of hallucination in LLMs.

Fundamental Results

The paper draws upon the diagonalization argument, originally used to show that some sets are larger than others, to demonstrate that hallucination is a natural consequence of the limitations in learning theory.

1. Provable Limitations:
   • If an LLM is constrained by a property-prover function $P$, then none of the LLMs that can be proved by $P$ can learn all computable functions. Therefore, hallucination is inevitable for such provable LLMs.
2. Broader Implications:
   • Even when removing the provability constraint, any LLM belonging to a computably enumerable set will hallucinate on an infinite number of inputs. This result further generalizes the inevitability of hallucination.
3. General Theorem:
   • Any computable LLM will hallucinate. This is formally shown through the diagonalization argument, reinforcing that no LLM configuration can cover all possible computable functions without introducing some form of error.

Empirical Validation

To substantiate the theoretical results, the paper presents empirical studies on tasks such as listing all strings of a given length using a specified alphabet and comparing linear orders of binary integers.

• String Listing Task ( $L(m, \{a, b\})$ ):
  • Empirical studies using state-of-the-art LLMs like GPT and Llama 2 show that these models fail to list all possible strings as $m$ grows. This aligns with the theoretical prediction that polynomial-time LLMs are insufficient for tasks requiring exhaustive combinatorial listing.
• Linear Order Task ( $\Omega(m)$ ):
  • Another task demonstrates that LLMs struggle to determine order relations between binary strings, especially as complexity increases.

Practical Implications and Future Research

The paper raises significant points about the limitations and safe deployment of LLMs:

1. Critical Decision-Making:
   • LLMs should not be solely relied upon for critical decisions without human oversight, as hallucinations can introduce significant risks.
2. Hallucination Mitigators:
   • Solutions like larger models, more training data, prompt engineering, and external knowledge integration are discussed. However, it's emphasized that none can entirely eliminate hallucinations, only mitigate them.
3. Research Directions:
   • Future research should focus on determining the exact capability boundaries of LLMs, devising benchmarks to evaluate models on tasks with varied complexities, and developing robust safety nets for areas where hallucination can have severe consequences.

Conclusion

This paper provides a theoretical framework demonstrating the inevitability of hallucination in LLMs, independent of training methods, data size, and model architecture. It posits that while practical mitigations can reduce the occurrence and impact of hallucinations, a complete elimination is impossible. The implications of these findings stress the importance of human-in-the-loop systems and rigorous safety and ethical standards when deploying LLMs in real-world applications. The insights gained here lay an important foundation for future theoretical and empirical work aimed at understanding and addressing the hallucination phenomenon in AI.