Medico: Towards Hallucination Detection and Correction with Multi-source Evidence Fusion (2410.10408v1)

Abstract: As we all know, hallucinations prevail in LLMs, where the generated content is coherent but factually incorrect, which inflicts a heavy blow on the widespread application of LLMs. Previous studies have shown that LLMs could confidently state non-existent facts rather than answering ``I don't know''. Therefore, it is necessary to resort to external knowledge to detect and correct the hallucinated content. Since manual detection and correction of factual errors is labor-intensive, developing an automatic end-to-end hallucination-checking approach is indeed a needful thing. To this end, we present Medico, a Multi-source evidence fusion enhanced hallucination detection and correction framework. It fuses diverse evidence from multiple sources, detects whether the generated content contains factual errors, provides the rationale behind the judgment, and iteratively revises the hallucinated content. Experimental results on evidence retrieval (0.964 HR@5, 0.908 MRR@5), hallucination detection (0.927-0.951 F1), and hallucination correction (0.973-0.979 approval rate) manifest the great potential of Medico. A video demo of Medico can be found at https://youtu.be/RtsO6CSesBI.

• The paper introduces a framework that employs multi-source evidence fusion to reliably detect hallucinations in LLMs.
• It uses a classification system to label content as SUPPORTED, NOT SUPPORTED, or IRRELEVANT, providing clear correction rationales.
• Experimental results on HaluEval show improved Hit Rate, MRR, and F1 scores, underscoring enhanced factual accuracy in LLM outputs.

Medico: An Insightful Overview

The paper introduces "Medico," a framework designed to address hallucinations in LLMs through a comprehensive approach incorporating multi-source evidence fusion. This approach is crucial for improving the factual accuracy of LLM-generated content, a known challenge due to the models' propensity to confidently generate incorrect information.

Methodology

Multi-source Evidence Fusion: The proposed framework gathers evidence from diverse sources, including search engines, knowledge bases, knowledge graphs, and user-uploaded files. This multi-faceted approach aims to mitigate the limitations of single-source retrieval, which often lacks comprehensive evidence. The evidence is retrieved, reranked, and fused to provide a robust basis for detecting factual errors.

Hallucination Detection: Medico employs a classification task to determine the veracity of generated content. Leveraging fused evidence, the system classifies content into categories—SUPPORTED, NOT SUPPORTED, or IRRELEVANT—while also offering the rationale behind these judgments. The incorporation of multiple sources enhances accuracy by ensuring that varied aspects of the information are covered.

Correction Mechanism: For detected hallucinations, the framework utilizes iterative correction based on the rationale provided. This step not only rectifies errors but also ensures minimal disruption to the original content structure, carefully balancing edit distance to maintain content integrity.

Experimental Findings

The framework is evaluated using the HaluEval dataset, where it demonstrates significant improvements in retrieval, detection, and correction performance metrics. Notably, multi-source evidence fusion achieves a high Hit Rate and Mean Reciprocal Rank, reflecting its effectiveness in capturing relevant information. Detection accuracy—measured through F1 scores—benefits from this comprehensive evidence base, showing superior results compared to single-source approaches.

Implications and Future Directions

Practically, Medico offers a versatile tool for enhancing the reliability of LLM outputs, applicable across numerous domains requiring factual accuracy. Theoretically, it provides a blueprint for integrating multi-source data into AI systems, paving the way for advancements in automated fact-checking.

Future research could focus on refining noise reduction techniques in evidence fusion and exploring more sophisticated models for preserving semantic integrity during correction. Additionally, addressing computational efficiency and privacy concerns associated with evidence retrieval remains paramount.

In conclusion, Medico represents a significant step forward in addressing hallucinations in LLMs, offering a framework that combines detection and correction through multi-source evidence fusion. This advancement enhances both the practical utility and theoretical understanding of reliable content generation in AI systems.