RAD-Bench: Evaluating Large Language Models Capabilities in Retrieval Augmented Dialogues (2409.12558v1)

Abstract: In real-world applications with LLMs, external retrieval mechanisms - such as Search-Augmented Generation (SAG), tool utilization, and Retrieval-Augmented Generation (RAG) - are often employed to enhance the quality of augmented generations in dialogues. These approaches often come with multi-turn dialogue, where each interaction is enriched by relevant information retrieved from external sources. Existing benchmarks either assess LLMs' chat abilities in multi-turn dialogues or their use of retrieval for augmented responses in single-turn settings. However, there is a gap in evaluating LLMs' ability to leverage retrieval for more precise responses across multiple turns. To address this limitation, we introduce RAD-Bench (Retrieval Augmented Dialogue), a benchmark designed to evaluate LLMs' capabilities in multi-turn dialogues following retrievals, essential for their deployment in context-rich applications. RAD-Bench evaluates two key abilities of LLMs: Retrieval Synthesis and Retrieval Reasoning. These are measured using discriminative questions and retrieved contexts, and corresponding reference answers, assessing how effectively LLMs integrate and reason with context to maintain and enhance conversation quality over multiple turns. Our evaluation results on commonly used LLMs reveal that model performance deteriorates as additional layers of conditions or constraints are applied across conversation turns, even when accurate retrieved contexts are provided.

Evaluating LLMs in Retrieval-Augmented Dialogues: An Overview of RAD-Bench

This essay provides an in-depth analysis of the paper titled "RAD-Bench: Evaluating LLMs' Capabilities in Retrieval-Augmented Dialogues," which presents a new benchmark designed to assess the performance of LLMs in context-rich dialogue settings. The authors, affiliated with MediaTek Research, aim to address a gap in current LLM evaluation benchmarks that predominantly focus on single-turn settings or generic multi-turn dialogue capacities without considering retrieval-augmented interactions.

Introduction

The paper introduces RAD-Bench (Retrieval Augmented Dialogue Benchmark), a novel benchmarking suite specifically designed to evaluate the capabilities of LLMs in handling multi-turn dialogues that are enhanced by retrieval mechanisms. These mechanisms include Search-Augmented Generation (SAG), tool utilization, and Retrieval-Augmented Generation (RAG). In such dialogues, each turn is augmented with relevant information retrieved from external sources, posing a unique challenge for LLMs to effectively integrate and utilize this information.

Capabilities Evaluated

RAD-Bench assesses two primary abilities of LLMs: Retrieval Synthesis and Retrieval Reasoning.

• Retrieval Synthesis refers to the ability of LLMs to progressively integrate and synthesize information from retrieved contexts over multiple dialogue turns.
• Retrieval Reasoning evaluates the LLMs' capacity to adapt and reason with changing conditions or user intents based on retrieved information across multiple turns.

Two categories of practical scenarios are employed to assess these abilities:

• Retrieval Synthesis: News TLDR, Education, and Academic Writing.
• Retrieval Reasoning: Customer Support, Finance, and Travel Planning.

In these scenarios, each benchmark sample is a three-turn dialogue where each turn provides additional retrieved context to simulate real-world applications.

Benchmark Construction

The construction of RAD-Bench involves a detailed pipeline that orchestrates the creation, selection, and synthesis of synthetic questions and retrieved contexts. Key phases include data collection, question candidate generation, retrieved context integration, question candidate selection, and reference answer creation. The benchmark comprises 89 multi-turn question samples, resulting in 267 total turns for model evaluation.

Methodology

To ensure high-quality benchmark questions and accurate evaluation, the authors utilize an LLM-as-a-Judge framework. This framework scores model responses on a scale of 1 to 10 based on scenario-specific criteria such as relevance, consistency, accuracy, informativeness, and coherence. The scoring process leverages reference-guided judges, which are prompts crafted to guide the evaluation LLM (GPT-4o) in assessing responses against reference answers.

Evaluation Results

The evaluation includes a range of both closed-source models (e.g., GPT-4o, GPT-3.5-Turbo) and open-weight models (e.g., Llama3.1, Deepseek, Mistral, BreeXe). The results reveal that closed-source models generally outperform open-weight models, with GPT-4o achieving an average score of 8.72. Among open-weight models, Llama3.1-405B and Deepseek-v2 show strong performance with averages of 7.88 and 7.86, respectively.

Key observations include:

• Scenarios in Retrieval Synthesis: BreeXe-8x7B achieves competitive performance, possibly due to its involvement in question candidate scoring.
• Travel Planning as a challenging scenario: Deepseek-v2 outperforms other models, likely benefiting from its two-stage reinforcement learning training strategy tailored for reasoning tasks.

Implications and Speculations

The development and implementation of RAD-Bench provide several key implications:

• Practical Applications: The benchmark aligns with practical applications such as chatbot interactions, customer support, and data analysis, offering a more realistic evaluation of LLMs in context-rich scenarios.
• Future Directions: Enhancements in LLM training methodologies, such as adopting chain-of-density and self-discovery frameworks, could potentially improve performance in synthesis and reasoning tasks. Additionally, expanded diversity in benchmark questions and refinements in evaluation criteria could further elevate the robustness and applicability of RAD-Bench.

Conclusion

RAD-Bench represents an essential step forward in the evaluation of LLMs, specifically addressing their capabilities in retrieval-augmented dialogues. This benchmark not only fills a significant gap in existing benchmarks but also provides a comprehensive and practical framework for assessing LLMs in real-world, multi-turn interactions. Future research will undoubtedly build upon these foundations, enhancing both LLM performance and benchmark methodologies to meet the evolving demands of AI applications.