MTRAG: A Multi-Turn Conversational Benchmark for Evaluating Retrieval-Augmented Generation Systems (2501.03468v1)

Abstract: Retrieval-augmented generation (RAG) has recently become a very popular task for LLMs. Evaluating them on multi-turn RAG conversations, where the system is asked to generate a response to a question in the context of a preceding conversation is an important and often overlooked task with several additional challenges. We present MTRAG: an end-to-end human-generated multi-turn RAG benchmark that reflects several real-world properties across diverse dimensions for evaluating the full RAG pipeline. MTRAG contains 110 conversations averaging 7.7 turns each across four domains for a total of 842 tasks. We also explore automation paths via synthetic data and LLM-as-a-Judge evaluation. Our human and automatic evaluations show that even state-of-the-art LLM RAG systems struggle on MTRAG. We demonstrate the need for strong retrieval and generation systems that can handle later turns, unanswerable questions, non-standalone questions, and multiple domains. MTRAG is available at https://github.com/ibm/mt-rag-benchmark.

• The paper introduces the 'mt' benchmark with 110 multi-turn conversations designed to rigorously evaluate RAG systems.
• It reveals that state-of-the-art retrieval methods struggle with non-standalone queries and later conversation turns, highlighting the need for refined context adaptation.
• The study explores synthetic data approaches to scale benchmarking and introduces tailored metrics for assessing faithfulness, appropriateness, and completeness in multi-turn interactions.

Analyzing the "mt: A Multi-Turn Conversational Benchmark for Evaluating Retrieval-Augmented Generation Systems" Paper

The paper "mt: A Multi-Turn Conversational Benchmark for Evaluating Retrieval-Augmented Generation Systems" by Yannis Katsis et al. contributes significantly to the evaluation methodologies of Retrieval-Augmented Generation (RAG) systems. RAG has gained prominence in enhancing LLMs by augmenting their response generation capabilities with information retrieval, thus improving response reliability. The primary innovation of this work lies in its benchmark, "mt", specifically designed for multi-turn conversations, an aspect that has been largely underexplored compared to single-turn RAG.

Core Contributions

1. Multi-Turn RAG Benchmark: The paper introduces "mt", a human-curated benchmark featuring 110 conversations averaging 7.7 turns across four diverse domains. This benchmark is substantial with 842 individual tasks and emphasizes real-world intricacies such as non-standalone questions, unanswerable queries, and the dynamic variation of relevant information across turns. This escalation from single-turn to multi-turn settings poses unique challenges to both retrieval and generation engines.
2. Comprehensive Evaluation: The benchmark evaluates both retrieval and generation aspects of the RAG systems. Experiments with state-of-the-art retrieval methodologies (including lexical, sparse, and dense models) reveal substantial deficiencies, particularly in handling non-standalone queries and later conversation turns. Generative performance is measured against intelligently conditioned metrics that assess faithfulness, appropriateness, and completeness, underscoring the LLMs' ongoing struggle with nuanced contextual understanding across multiple turns.
3. Automation and Synthetic Data Exploration: Automation routes are explored via synthetic data and automated metric evaluations. While synthetic data lacks certain characteristics of human-generated conversations (such as diversity and natural conversational flow), it provides a scalable alternative that can aid continuous benchmarking evolution. The creation of "mt_s", a companion synthetic data benchmark, emphasizes the need for enhanced methodologies to approximate human conversation dynamics.

Experimental Insights

Retrieval performance illustrates the complexity of multi-turn conversations, with query rewrite strategies necessary to enhance retrieval quality for later turns and non-standalone questions. Generative evaluation points to significant challenges in answering partially or unanswerable questions, reflecting a primary frontier for current LLMs. The consistency of metrics conditioned on IDK (I-Don't-Know) signals across these experiments points to a more fine-tuned assessment approach than conventional holistic scores.

Future Implications

The "mt" benchmark paves the way for more nuanced evaluation and development of RAG systems. Insights from this research indicate several paths forward:

• Enhancement of Retrieval Mechanisms: Refining retrieval methodologies to incorporate improved context comprehension, especially for non-standalone and cross-turn dependencies, remains critical.
• Advanced Generation Evaluation: The creation of metrics that better align with human perceptions of conversational quality could direct the improvement of LLMs in multi-turn settings.
• Synthetic Data Utilization: Development of synthetic data that more closely replicates human conversations could provide an effective supplementary tool for large-scale benchmarking.
• Domain and Context Adaptation: Extending benchmarks to include adversarial, multi-domain, and multilingual contexts could offer broader insights into LLM capabilities and limitations.

By addressing the multi-turn challenges in RAG systems, the work sets a foundational framework for advancing conversational AI capabilities, with implications not only in technical development but also in practical deployment in diverse real-world applications. As LLMs become increasingly integral to interactive AI systems, benchmarks like "mt" will be vital in steering future innovations and ensuring robust, trustworthy AI interactions.