MTRAG-UN: A Benchmark for Open Challenges in Multi-Turn RAG Conversations

Abstract: We present MTRAG-UN, a benchmark for exploring open challenges in multi-turn retrieval augmented generation, a popular use of LLMs. We release a benchmark of 666 tasks containing over 2,800 conversation turns across 6 domains with accompanying corpora. Our experiments show that retrieval and generation models continue to struggle on conversations with UNanswerable, UNderspecified, and NONstandalone questions and UNclear responses. Our benchmark is available at https://github.com/IBM/mt-rag-benchmark

• The paper introduces MTRAG-UN, a comprehensive benchmark with 666 human-annotated tasks spanning 2,800 conversation turns to assess multi-turn RAG challenges.
• It demonstrates that query rewriting greatly benefits retrieval, yet even SOTA methods struggle with long, complex documents, as shown by lower recall in Banking and Telco domains.
• The evaluation highlights persistent LLM gaps in handling unanswerable, underspecified, and non-standalone questions, urging the development of improved context modeling and clarification strategies.

MTRAG-UN: Advancing Evaluation for Multi-Turn Retrieval-Augmented Generation

Introduction and Motivation

MTRAG-UN introduces a comprehensive benchmark for investigating persistent challenges in multi-turn retrieval-augmented generation (RAG) with LLMs. Addressing critical shortcomings in current evaluation resources, MTRAG-UN targets highly challenging conversational phenomena: UNanswerable, UNderspecified, and NONstandalone user questions, as well as UNclear LLM responses. The benchmark delivers 666 human-annotated tasks spanning over 2,800 individual conversation turns and six diverse domains, including the newly introduced Banking and Telecommunications corpora, positioning it as a complementary successor to MTRAG and related datasets.

This resource is intended to facilitate granular, comparative analysis of RAG pipeline weaknesses and foster algorithmic innovation by focusing on conversational goals infeasible to achieve through single-turn exchanges.

Figure 1: Portions of three conversations highlighting the challenges in MTRAG-UN, such as unanswerable, underspecified, and non-standalone questions.

Benchmark Construction and Dataset Properties

Task and Phenomena Annotation

MTRAG-UN is curated through systematic human annotation workflows, orchestrated on the RAGaphene platform. Annotators were directed to explicitly construct question types underrepresented or inadequately studied in prior benchmarks:

• UNanswerable Questions receive no support from reference corpora, enforcing scenarios where LLM abstention or appropriate IDK behavior is required.
• UNderspecified Questions are intentionally ambiguous or broad, necessitating model-initiated clarification or enumeration of possible interpretations.
• NONstandalone Questions demand effective context fusion, since they reference conversation history, challenging both retrieval (context tracking, query rewriting) and generation components.
• UNclear Responses arise when user prompts for clarification after model responses indicate disagreement, ambiguity, or lack of understanding.

These design choices yield a high density of challenging conversation turns per evaluation task, with intentional bias against first-turn (easy) questions and a large proportion of clarification and difficult multi-turn phenomena.

Figure 2: Distribution of tasks in MTRAG-UN across various conversational and answerability dimensions.

Corpora and Domain Coverage

The benchmark encompasses six domains: CLAPNQ, FiQA, Govt, Cloud, and newly introduced Banking and Telco. Each domain's corpus contains substantial, enterprise-quality documentation with intricate hyperlink structures and high intra-corpus content similarity—imposing realistic but difficult retrieval scenarios for RAG systems. Notably, Banking and Telco collections contain longer documents and denser interconnectivity, amplifying cross-page reasoning requirements and retrieval ambiguity.

Figure 3: Distribution of tasks by turn position within conversations, highlighting bias toward non-initial turns.

Evaluation Methodology and Baseline Performance

Retrieval

Retrieval experiments utilize both traditional (BM25), sparse (Elser), and SOTA dense retrieval (Granite English R2, BGE-base 1.5) architectures. Results confirm Elser's superiority across the benchmark, with all retrieval paradigms benefitting significantly from query rewriting over naive last-turn queries—especially for non-standalone tasks, which now represent nearly half the dataset.

However, absolute recall and nDCG scores are substantially lower than in MTRAG, with Banking and Telco domains particularly underperforming (R@5: .32 and .39, respectively, versus ~.52 in other domains). These deficits are linked to increased document length, hyperlink complexity, and high content overlap—features intentionally modeled in the corpus design.

Generation

Comprehensive LLM evaluation is conducted in both reference and integrated RAG settings. A wide spectrum of open and proprietary LLMs are assessed (GPT-OSS-120B/20B, DeepSeek variants, Granite, Qwen, Llama, Mistral, Phi-4), using a prompt explicitly accommodating recognition and handling of underspecified questions.

Figure 4: Generation results disaggregated by question answerability—performance is lowest for underspecified and unanswerable categories.

Results indicate all LLMs trail best-human (target) responses across Faithfulness, Reference-Based, and RAG-based automatic evaluations. Larger LLMs systematically outperform smaller ones within architectures; however, performance degrades in the RAG setting, directly attributable to retrieval noise.

Salient findings include:

• On unanswerable and underspecified questions, most LLMs attempt to generate plausible (but unsupported) answers instead of deferring or clarifying. Models like DeepSeek-V3 and GPT-OSS, however, show relative robustness by frequently responding with idk or clarifying requests—an advancement over earlier SOTA models.
• Clarification questions provoke lower scores than follow-ups, indicating significant headroom in intent refinement and self-correction.
• Performance is relatively domain-agnostic, with notable drops only in the most structure-heavy corpus (FiQA).

Figure 5: Faithfulness (F), Appropriateness (A), and Completeness (C) breakdowns for different model and judge configurations, illustrating persistent gaps over target responses.

Novel Evaluation Protocols for Underspecified Questions

A key contribution is the development of a robust metric for underspecified question handling. This approach leverages a fine-tuned GPT-OSS-120B judge with a domain-specific prompt, validated to over 96% accuracy on 80 sampled model generations. The metric rigorously identifies whether model output acknowledges ambiguity and initiates appropriate clarification—capabilities largely absent in current LLMs.

Figure 6: Prompt structure used for the clarification judge, enabling evaluation of LLM behavior in response to underspecified inputs.

Implications, Limitations, and Future Directions

MTRAG-UN empirically reveals material deficiencies in both retrieval and generation for multi-turn, information-seeking dialogue, with pronounced failure on non-standalone, underspecified, and unanswerable inputs. The results underscore the need for systemic improvements in:

• Enhanced contextual query rewriting, especially for contextually entangled or ambiguous conversational turns.
• LLM self-awareness and introspective mechanisms to identify when clarification, hesitancy, or disambiguation is required rather than overconfident generation.
• Retriever and corpus design strategies tailored for high-overlap, long-document, and densely-hyperlinked enterprise knowledge bases.

From a theoretical perspective, these findings point to the limitations of SOTA architectures in dialog state tracking, context integration, and uncertainty modeling in open-domain, multi-turn settings. Practically, the curated corpora and protocols of MTRAG-UN will be valuable for LLM developers targeting robust, trustworthy enterprise chat systems, especially in high-risk domains like banking and telco.

Future research catalyzed by MTRAG-UN will likely extend into multilingual settings, larger and more diverse annotator pools, refined retrieval/generation hybrid models, and advanced automated judges for nuanced conversational phenomena.

Conclusion

MTRAG-UN establishes a rigorous, multi-faceted benchmark and evaluation framework for multi-turn RAG, advancing the field past single-turn or over-simplified datasets. The substantial shortfalls highlighted—especially on underspecified and non-standalone phenomena—highlight critical research frontiers in context modeling, conversational empathy, and answerability awareness. The resource will likely serve as a touchstone for methodological innovation in RAG system development and evaluation.