RAGEval: Scenario Specific RAG Evaluation Dataset Generation Framework (2408.01262v5)

Abstract: Retrieval-Augmented Generation (RAG) is a powerful approach that enables LLMs to incorporate external knowledge. However, evaluating the effectiveness of RAG systems in specialized scenarios remains challenging due to the high costs of data construction and the lack of suitable evaluation metrics. This paper introduces RAGEval, a framework designed to assess RAG systems across diverse scenarios by generating high-quality documents, questions, answers, and references through a schema-based pipeline. With a focus on factual accuracy, we propose three novel metrics: Completeness, Hallucination, and Irrelevance to evaluate LLM generated responses rigorously. Experimental results show that RAGEval outperforms zero-shot and one-shot methods in terms of clarity, safety, conformity, and richness of generated samples. Furthermore, the use of LLMs for scoring the proposed metrics demonstrates a high level of consistency with human evaluations. RAGEval establishes a new paradigm for evaluating RAG systems in real-world applications. The code and dataset are released at https://github.com/OpenBMB/RAGEval.

• The paper presents a novel framework that generates scenario-specific datasets using domain-tailored QA pairs for RAG evaluation.
• It integrates schema extraction, rule-based document generation, and newly proposed metrics—Completeness, Hallucination, and Irrelevance—for comprehensive performance assessment.
• Empirical results demonstrate high quality and competitive performance in finance, legal, and medical domains, with implications for both open-source and proprietary models.

Overview of "RAGEval: Scenario Specific RAG Evaluation Dataset Generation Framework"

The paper entitled “RAGEval: Scenario Specific RAG Evaluation Dataset Generation Framework” introduces a novel framework aimed at enhancing the evaluation of Retrieval-Augmented Generation (RAG) systems. Unlike traditional benchmarks that assess LLMs on general knowledge, RAGEval targets the evaluation of RAG systems within specific vertical domains by generating customized datasets that reflect the nuanced requirements of those domains.

Key Contributions

The authors present several notable contributions through the RAGEval framework:

1. Scenario-Specific Dataset Generation:
   • The framework begins by summarizing a schema from seed documents, encapsulating the essential domain-specific knowledge.
   • This schema is used to generate diverse documents by configuring languages and domain-specific rules.
   • It constructs question-answer (QA) pairs tailored to the vertical domains, enhancing the evaluation of knowledge usage across different scenarios.
2. Introduction of New Evaluation Metrics:
   • The authors propose three novel metrics for evaluation, namely Completeness, Hallucination, and Irrelevance. These metrics are designed to capture the quality of model responses more accurately than conventional methods.
   • Completeness assesses whether the generated answers cover all key points from reference answers.
   • Hallucination identifies content that contradicts the key points, addressing the model’s tendency to produce factually incorrect outputs.
   • Irrelevance quantifies the portion of unaddressed key points, highlighting gaps in the answers.

Methodological Framework

The structured methodology is perhaps the most striking feature of this paper. The authors delineate their approach into several stages:

1. Schema Summary:
   • A schema is formulated from a small set of domain-specific documents, encapsulating characteristic information prevalent within the domain.
2. Document Generation:
   • Configurations derived from the schema are employed to generate documents that contain detailed and contextually coherent information.
   • Both rule-based methods and LLMs are utilized to ensure that the generated documents accurately reflect the domain-specific details.
3. QRA (Question-Reference-Answer) Generation:
   • Generating QRA triples involves leveraging the configurations to create targeted questions and initial answers, followed by identifying relevant information fragments as references.
   • The answers are optimized to ensure they are consistent with the references, and key points are extracted to support comprehensive response evaluation.
4. Creation of DRAGONBall Dataset:
   • Utilizing the RAGEval framework, the DRAGONBall dataset is constructed, encompassing texts from finance, legal, and medical domains in both Chinese and English.
   • This dataset contains 6711 questions meticulously designed to reflect the complexity and specificity of the domains.

Evaluation and Results

The evaluation section of the paper is robust, providing empirical validation for the proposed framework and metrics:

1. Human Verification:
   • A human verification process assesses the quality of the generated datasets. The results indicate that both the QARs and generated documents maintain high-quality standards, with a human reviewer score of 4.94 (CN Finance) to 4.81 (EN Law) on average for QAR quality.
   • Additionally, the generated documents outperform baseline methods in clarity, safety, richness, and conformance.
2. Comparison with Existing Models:
   • The framework's performance is evaluated against several state-of-the-art LLMs. GPT-4o exhibits superior performance in overall metrics, yet open-source models like Qwen1.5-14B-chat and Llama3-8B-Instruct also demonstrate competitive results, closing the gap with proprietary models.
3. Hyperparameter Tuning:
   • The paper also explores the effects of different hyperparameter settings on retrieval and generation performance. Insights are provided into the optimal settings for Recall, Expected Information Retrieval (EIR), and other relevant metrics, showcasing the framework's versatility.

Implications and Future Work

The implications of this research are significant for both the theoretical and practical advancement of AI in specific domains:

• Enhanced Benchmarks: By leveraging domain-specific datasets, RAG models can be more accurately benchmarked, improving their reliability in practical applications.
• Advanced Metrics: The newly proposed metrics offer a refined approach to evaluating model responses, mitigating issues like hallucination and irrelevance.
• Potential for Open-Source Models: The results indicate that with further optimization, open-source models could attain performance levels comparable to proprietary systems like GPT-4o.

Looking ahead, the framework could be extended to encompass a broader array of domains and languages, providing a more comprehensive evaluation benchmark for RAG systems globally. Additionally, further research could focus on minimizing the performance discrepancies between open-source and proprietary models, fostering a more equitable development environment across the AI community.

In summary, the RAGEval framework presented in this paper marks a significant advancement in the evaluation of RAG systems, contributing valuable methodologies and insights to the field of natural language processing and information retrieval.