Long$^2$RAG: Evaluating Long-Context & Long-Form Retrieval-Augmented Generation with Key Point Recall (2410.23000v3)

Abstract: Retrieval-augmented generation (RAG) is a promising approach to address the limitations of fixed knowledge in LLMs. However, current benchmarks for evaluating RAG systems suffer from two key deficiencies: (1) they fail to adequately measure LLMs' capability in handling long-context retrieval due to a lack of datasets that reflect the characteristics of retrieved documents, and (2) they lack a comprehensive evaluation method for assessing LLMs' ability to generate long-form responses that effectively exploits retrieved information. To address these shortcomings, we introduce the Long$^2$RAG benchmark and the Key Point Recall (KPR) metric. Long$^2$RAG comprises 280 questions spanning 10 domains and across 8 question categories, each associated with 5 retrieved documents with an average length of 2,444 words. KPR evaluates the extent to which LLMs incorporate key points extracted from the retrieved documents into their generated responses, providing a more nuanced assessment of their ability to exploit retrieved information.

• The paper introduces the Long²RAG benchmark with the innovative Key Point Recall metric to assess how well LLMs incorporate essential information from extensive documents.
• Empirical results reveal that performance declines with longer document lengths and that effective retrieval strategies are crucial for maintaining key point accuracy.
• The study highlights marked differences between closed-source and open-source models, urging the development of new architectures to better handle long-context data.

Evaluation of Long-Context RAG in LLMs Using Long$^2$RAG and Key Point Recall

The paper presents a novel benchmark, Long$^2$RAG, accompanied by an innovative metric, Key Point Recall (KPR), to enhance the evaluation of retrieval-augmented generation (RAG) systems in the sphere of long-context, long-form responses. The focus is on addressing fundamental limitations in existing benchmarks which insufficiently assess the capabilities of LLMs in handling extensive information retrieval and generating comprehensive responses from such retrievals.

Benchmark and Metric Design

The Long$^2$RAG benchmark is meticulously designed. It comprises 280 questions spanning 10 diverse domains and across 8 distinct question categories. Each question is anchored to 5 retrieved documents with substantial average length, which demand the models to effectively exploit the vast contextual data. Concurrently, the KPR metric assesses the extent to which generated responses incorporate critical points from the retrieved documents, providing nuanced insights into the efficacy of LLMs in leveraging supplementary information sources.

Empirical Evaluation

Evaluation of 9 state-of-the-art LLMs using Long$^2$RAG reveals several key insights:

1. Closed vs. Open Source Performance: GPT-4o, representing closed-source systems, consistently outperforms leading open-source models such as Qwen2 and Mis(x)tral. Notably, among open-source models, Phi-3-mini exhibits competitive performance despite its smaller parameter size, challenging the assumption that larger models inherently offer superior results.
2. Effect of Document Length: A critical finding is the performance degradation with increasing document length. The models demonstrated a decline in their ability to recall key information as the document size increased, reflecting challenges in managing extensive contextual details.
3. Retrieval Strategy Impact: Through varied truncation strategies and summarization, the paper observes that cutting down document lengths typically reduces performance, confirming the importance of utilizing full document context within RAG settings.
4. Comparator Efficacy: The robustness of the KPR metric is demonstrated across different evaluators, with GPT-4o and Llama3-70B yielding consistent model ranking, albeit with score variations, reinforcing the metric's reliability.

Implications and Future Directions

The presented work significantly contributes to the landscape of RAG evaluation within LLMs by advancing both the methodological and practical approaches to assessment. Practically, the Long$^2$RAG benchmark and KPR metric set a new standard for measuring the integration of retrieved information into coherent, long-form responses.

Theoretically, this research paves the way for further exploration of LLMs' capacity to manage long-context retrievals, emphasizing the need for new architectures or enhanced training paradigms capable of addressing challenges in processing extensive contextual input.

Future research could expand the Long$^2$RAG dataset, incorporate multilingual assessments to augment its applicability beyond English, and refine the KPR metric to reduce dependency on extracted key points, potentially integrating them with human preference models.

In summary, the paper provides a comprehensive framework to dissect and evaluate the intricacies of RAG within LLMs, setting a foundation for future exploration and innovation in enhancing the capabilities of LLMs to handle long-contextual information effectively.