---
title: 'MTRAGEval: Multi-Turn RAG Evaluation'
url: https://www.emergentmind.com/topics/mtrageval
type: topic
---

# MTRAGEval: Multi-Turn RAG Evaluation

MTRAGEval most prominently denotes **SemEval-2026 Task 8**, a benchmark for **multi-turn Retrieval-Augmented Generation (RAG)** that evaluates passage retrieval, reference-grounded response generation, and end-to-end multi-turn RAG over four English-language domains: **ClapNQ**, **Cloud**, **FiQA**, and **Govt** [2603.10524]. In the provided literature, however, the same label is also used in non-equivalent senses: as the name of a **radiology-aware model-based evaluation metric** adapted from COMET for report generation [2311.16764], and as a heading for **evaluation recommendations** in mesh-to-raster multi-modal registration [1703.01972]. The dominant contemporary usage is the SemEval benchmark, but the term is not yet fully standardized across subfields.

## 1. Benchmark scope and task formulation

Within SemEval-2026 Task 8, MTRAGEval is framed as a benchmark for **multi-turn conversational retrieval and grounded generation**. The benchmark provides a document collection spanning **366 K passages in 78 K documents**, and Task A evaluates the final user turn in **110 human conversations**, totaling **777 queries**, across the four domains named above [2605.12028]. AILS-NTUA characterizes the benchmark as comprising **three subtasks**: **Task A (passage retrieval)**, **Task B (reference-grounded response generation)**, and **Task C (end-to-end RAG)** [2603.10524].

The task definition varies slightly across system papers because different teams emphasize different output granularities. Caraman describes Task A as requiring a **ranked list of 10 passages** for each final user turn, whereas H-RAG formulates Task A as a **ranked list of top-5 documents judged relevant** to the final query conditioned on dialogue history [2605.12028; 2605.00631]. This difference is operationally important: some systems optimize passage-level ranking and only later aggregate to documents, while others treat document selection as the primary retrieval target.

For Task B, RaguTeam specifies the input as a **multi-turn dialogue history** $H=(u_1,a_1,\ldots,u_{t-1},a_{t-1})$ plus a set of **reference passages** $D=\{d_1,\ldots,d_k\}$, with the output being a final-turn response that answers the last query, remains strictly grounded in $D$, and abstains when $D$ contains no relevant information [2605.04523]. H-RAG extends this to Task C by evaluating whether systems can generate a fluent conversational answer that is factually grounded in retrieved evidence and able to abstain on unanswerable queries [2605.00631].

## 2. Evaluation protocol and official metrics

Task A is ranked primarily by **normalized Discounted Cumulative Gain at rank $K$**. Caraman gives the single-query formulation
$$
\mathrm{DCG}@K=\sum_{i=1}^{K}\frac{2^{\mathrm{rel}_i}-1}{\log_2(i+1)}, \qquad
\mathrm{nDCG}@K=\frac{\mathrm{DCG}@K}{\mathrm{IDCG}@K},
$$
with **secondary metrics** of **nDCG@10** and **Recall@10** [2605.12028]. Caraman also notes that **only answerable queries** are scored in the Task A test setting [2605.12028].

For Task B and Task C, submissions report harmonic-mean objectives over three components, but the notation differs slightly across papers. RaguTeam defines the official Task B score as
$$
\mathrm{HM}_3 = \frac{3}{1/M_1 + 1/M_2 + 1/M_3},
$$
where $M_1=\mathrm{RB\_alg}$, $M_2=\mathrm{RB\_llm}$, and $M_3=\mathrm{RL\_F}$ [2605.04523]. H-RAG reports Task C with the component triple **RB_agg**, **RL_F**, and **RB_llm**, again combined by harmonic mean [2605.00631]. In practice, the common structure is stable even when the reference-based component is written as **RB_alg** or **RB_agg**.

The benchmark therefore measures three distinct axes. **Reference-based relevance** rewards overlap or agreement with gold responses; **RL_F** evaluates faithfulness to retrieved evidence; and **RB_llm** is an LLM-judge assessment of answer quality. RaguTeam further describes a **conditioned variant** for abstention, under which correctly abstaining on known unanswerable instances sets the metrics on those cases to 1 if the system outputs an IDK response [2605.04523].

A recurring implication is that MTRAGEval is not a single-metric benchmark. It is a **compound evaluation regime** in which retrieval quality, grounding fidelity, abstention behavior, and answer usefulness can diverge. That design choice is central to how teams interpret success and failure.

## 3. Retrieval architectures and dominant design patterns

Task A submissions converge on a small set of retrieval patterns: **query rewriting**, **hybrid dense-sparse retrieval**, **rank fusion**, and **reranking**.

Caraman implements a **three-stage pipeline**: **query rewriting** with a **LoRA-fine-tuned Qwen 2.5 7B Instruct** model, **hybrid BM25S and dense retrieval** fused with **Reciprocal Rank Fusion**, and **cross-encoder reranking** with **BGE-reranker-v2-m3**. The official test result is **nDCG@5 = 0.531**, ranking **8th out of 38** systems and **10.7% above the organizer baseline** [2605.12028]. A central empirical result is the effect of **domain-specific temperature tuning**: **Cloud** favors **deterministic decoding** at $t=0.0$, whereas **FiQA** and **Govt** benefit from modest randomness at $t=0.3$; by contrast, **domain-aware prompting** and **multi-query expansion** both degrade holdout performance to **0.350 nDCG@5** [2605.12028].

H-RAG departs from passage-only formulations through a **hierarchical parent-child retrieval pipeline**. Documents are segmented into overlapping **three-sentence child chunks** with **stride = 2**, retrieval is performed at the child level using a hybrid dense-sparse score
$$
S_{\mathrm{hybrid}}(Q,c)=\alpha \cdot s_{\mathrm{dense}}(Q,c)+(1-\alpha)\cdot s_{\mathrm{sparse}}(Q,c),
$$
with $\alpha \in \{0.5,0.7,0.9\}$ and **optimal $\alpha=0.7$**, and selected child evidence is then aggregated back to **parent documents** by **max-score aggregation**. The submission reaches **nDCG@5 = 0.4271** on Task A, and enabling **child-first with `rank_parents=true`** yields the best gains, reported as **+0.0197** [2605.00631].

AILS-NTUA advances a more radical thesis: **query diversity over retriever diversity**. Instead of ensembling retrievers, it issues five **DeepSeek-V3.2** reformulations—**Minimal**, **Corpus-Specific**, **HyDE**, **Chain-of-Thought**, and **Anchor-Keyword**—to a single **ELSER v1** sparse retriever, then fuses them with **variance-aware nested Reciprocal Rank Fusion**. On the official leaderboard this system ranks **1st in Task A** with **nDCG@5 = 0.5776**, reported as **+20.5% over the strongest baseline** [2603.10524]. The paper further reports that **ELSER v1** achieved the highest **R@5 = 0.483** and **nDCG@5 = 0.444** in a **9-way retriever comparison**, and attributes part of this advantage to **corpus alignment** because MTRAG’s annotation process itself used ELSER for silver labeling [2603.10524].

5ting adopts a dense-first approach built around **BGE-M3**, **FAISS IndexFlatIP**, **dual-query merged retrieval**, and **GPT-4o-mini listwise reranking**. The retriever first uses both the original last-turn query and a rewritten standalone question, merges their candidate sets, and then invokes an LLM to output a ranked permutation of passage IDs. The reported Task A result is **nDCG@5 = 0.4719**, with an ablation trajectory from **0.3454** for base dense retrieval to **0.3732** after rewriting and **0.4393** after LLM reranking [2606.28737].

Across these systems, the most stable finding is that **multi-turn retrieval quality depends heavily on rewriting and reranking rather than on first-stage retrieval alone**. The exact retrieval substrate differs—BM25S, ELSER, BGE-M3, or hierarchical child chunks—but successful systems all try to resolve context dependence before final ranking.

## 4. Grounded generation and end-to-end RAG

Task B and Task C submissions place stronger emphasis on **faithfulness control**, **answerability handling**, and **candidate selection**.

RaguTeam’s Task B winner is a **judge-orchestrated LLM ensemble**. Seven LLMs, each used under one or two prompting variants, produce up to **14 candidates per instance**, and **GPT-4o-mini** assigns a faithfulness score in **[0,1]** to each candidate relative to the reference passages. The system then selects the highest-scoring response and deterministically replaces any instance with **empty $D$** by **“I don’t have an answer”**. The official conditioned score is **HM$_3$ = 0.7827**, ranking **1st out of 26 teams** and exceeding the strongest organizer baseline by **+14.4 p.p.** [2605.04523].

AILS-NTUA uses a markedly different architecture: a **multistage grounded generation pipeline** with **answerability triage**, **evidence span extraction**, **dual candidate generation**, **calibrated multi-judge selection**, and optional **micro-adjustment**. In Task B it reports **HM = 0.7698**, ranking **2nd out of 26**, with component scores **RB_alg = 0.633**, **RL_F = 0.897**, and **RB_llm = 0.832**. In Task C the same system reaches **HM = 0.5409**, with **RB_alg = 0.400**, **RL_F = 0.729**, and **RB_llm = 0.598** [2603.10524].

H-RAG links retrieval and generation through **parent-level context reconstruction**. Retrieved child chunks are aggregated into parent documents, which are then supplied to **GPT-5** with a prompt that enforces conversational tone, no explicit citations, and graceful abstention when evidence is insufficient. The reported Task C result is **HM = 0.3241**, decomposed into **RB_agg = 0.2488**, **RL_F = 0.2703**, and **RB_llm = 0.6508** [2605.00631].

5ting emphasizes **role-separated decoding**. Its prompt distinguishes **conversation history** as context for intent resolution only, **retrieved passages** as the **sole factual authority**, and the **last-turn question** as the object to be answered. The system prompt explicitly instructs the model to use only retrieved passages and to reply that it cannot provide a complete answer if the passages are insufficient. On Task C, 5ting reports **RB_llm = 0.6784**, **RL_F = 0.7692**, **RB_alg = 0.3867**, and **Harmonic = 0.5597** [2606.28737].

The generation literature within MTRAGEval therefore spans two broad paradigms. One uses **candidate multiplicity plus judging**; the other uses **pipeline decomposition and explicit grounding constraints**. Both are responses to the same failure mode: fluent multi-turn responses can remain unfaithful unless generation is tightly coupled to retrieved evidence.

## 5. Failure modes, benchmark limitations, and disputed bottlenecks

The benchmark papers are unusually explicit about limitations. RaguTeam identifies five: **target leakage via empty $D$ for unanswerable queries**, **coverage gaps** in gold answers, **metric leakage** because **RL_F** is computable on the submission set, **annotation errors and inconsistencies**, and a **lack of unified guidelines for generation vs. evaluation** [2605.04523]. These are benchmark-design issues rather than model-specific errors.

At the system level, submissions disagree about the dominant bottleneck. H-RAG concludes that **retrieval performance remains the primary bottleneck for faithful generation**, whereas AILS-NTUA argues that **answerability calibration—rather than retrieval coverage—is the primary bottleneck in end-to-end performance** [2605.00631; 2603.10524]. AILS grounds that claim in the observed Task B to Task C drop, especially the collapse of **RB_alg** from **0.633** to **0.400**, alongside an answerability classifier with **~83.9% accuracy** but only **21.8% unanswerable recall** [2603.10524].

5ting formulates the problem space as four core challenges: **context dependency and drift**, **under-specification**, **hallucination risk**, and **retrieval–generation alignment**, explicitly noting that **strong retrieval alone does not guarantee faithful generation** [2606.28737]. Caraman’s retrieval results reinforce a parallel point on the retrieval side: more elaborate query engineering is not necessarily helpful, since **domain-aware prompting** and **multi-query expansion** can underperform simpler temperature-controlled rewriting [2605.12028].

A plausible implication is that MTRAGEval is valuable precisely because it prevents a single-axis narrative of progress. A system may improve **nDCG@5** without improving end-to-end answerability; a system may improve **RL_F** while remaining weak on reference-based relevance; and abstention can materially change rankings in tasks with nontrivial unanswerable rates.

## 6. Alternative uses of the label and relation to adjacent evaluation work

Outside SemEval-2026 Task 8, the label **MTRAGEval** appears in at least two additional senses in the provided literature. In radiology report generation, it denotes a **task-adaptation of the COMET Estimator framework** that predicts a continuous quality score from a ground-truth report $s$ and a generated report $h$ by encoding them separately, forming the joint feature vector
$$
f=[e_h; e_s; e_h \odot e_s; |e_h-e_s|],
$$
and regressing either **RadGraph F$_1$** or **RadCliQ** with **MSE** supervision [2311.16764]. That system uses **XLM-R** or **BioClinical BERT** encoders, reports strong correlations with established metrics, and attains **Spearman $\rho = 0.79$** against radiologist total-error counts on the **ReXVal** dataset for one checkpoint [2311.16764].

In a separate imaging context, the mesh-to-raster registration paper invokes **“implications for MTRAGEval”** to describe best practices for **multi-modal ROI alignment assessment**: use a **gold-standard GT from multi-rater STAPLE**, report **Dice** together with **Hausdorff** or **MSD**, employ **repeated-measures ANOVA**, and solve registration as **nonlinear least-squares via Gauss–Newton with a descending-$\lambda$ continuation** [1703.01972]. Here the term does not denote the SemEval benchmark; it denotes an evaluation perspective on registration accuracy.

The broader evaluation ecosystem surrounding multi-turn and multimodal RAG also helps contextualize MTRAGEval. **MiRAGE** proposes a **claim-centric** framework with **INFOF1** and **CITEF1** for multimodal grounding [2510.24870]. **MRAG-Suite** introduces **difficulty-based** and **ambiguity-aware** filtering together with **MM-RAGChecker** for claim-level diagnostics [2509.24253]. **MEG-RAG** defines **Multi-modal Evidence Grounding (MEG)** as a semantic-aware measure of the contribution of retrieved evidence and trains a reranker against that signal [2604.24564]. These frameworks are not MTRAGEval, but they address the same structural concerns that MTRAGEval submissions repeatedly surface: ambiguity, evidence utility, hallucination, and grounding fidelity.

Taken together, the literature presents MTRAGEval less as a single method than as a **family of evaluation problems centered on grounded multi-turn reasoning**. In current usage, its principal referent is the SemEval-2026 benchmark; yet the persistence of alternate senses in radiology and registration underscores that the name still functions as a moving label across neighboring evaluation traditions.

Source: https://www.emergentmind.com/topics/mtrageval