MTRAGEval: Multi-Turn RAG Evaluation

• MTRAGEval is a benchmarking framework for multi-turn conversational retrieval-augmented generation that defines tasks in passage retrieval, reference-grounded response generation, and end-to-end RAG.
• It employs compound evaluation metrics including nDCG@K and harmonic means to assess retrieval quality, grounding fidelity, and abstention handling across diverse domains.
• Systems using MTRAGEval range from query rewriting with dense-sparse fusion to multi-stage generation pipelines, highlighting challenges in retrieval–generation alignment and answerability calibration.

MTRAGEval most prominently denotes SemEval-2026 Task 8, a benchmark for multi-turn Retrieval-Augmented Generation (RAG) that evaluates passage retrieval, reference-grounded response generation, and end-to-end multi-turn RAG over four English-language domains: ClapNQ, Cloud, FiQA, and Govt (Athanasiou et al., 11 Mar 2026). In the provided literature, however, the same label is also used in non-equivalent senses: as the name of a radiology-aware model-based evaluation metric adapted from COMET for report generation (Calamida et al., 2023), and as a heading for evaluation recommendations in mesh-to-raster multi-modal registration (Tatano et al., 2017). The dominant contemporary usage is the SemEval benchmark, but the term is not yet fully standardized across subfields.

1. Benchmark scope and task formulation

Within SemEval-2026 Task 8, MTRAGEval is framed as a benchmark for multi-turn conversational retrieval and grounded generation. The benchmark provides a document collection spanning 366 K passages in 78 K documents, and Task A evaluates the final user turn in 110 human conversations, totaling 777 queries, across the four domains named above (Caraman et al., 12 May 2026). AILS-NTUA characterizes the benchmark as comprising three subtasks: Task A (passage retrieval), Task B (reference-grounded response generation), and Task C (end-to-end RAG) (Athanasiou et al., 11 Mar 2026).

The task definition varies slightly across system papers because different teams emphasize different output granularities. Caraman describes Task A as requiring a ranked list of 10 passages for each final user turn, whereas H-RAG formulates Task A as a ranked list of top-5 documents judged relevant to the final query conditioned on dialogue history (Caraman et al., 12 May 2026, Elchafei et al., 1 May 2026). This difference is operationally important: some systems optimize passage-level ranking and only later aggregate to documents, while others treat document selection as the primary retrieval target.

For Task B, RaguTeam specifies the input as a multi-turn dialogue history $H=(u_1,a_1,\ldots,u_{t-1},a_{t-1})$ plus a set of reference passages $D=\{d_1,\ldots,d_k\}$, with the output being a final-turn response that answers the last query, remains strictly grounded in $D$, and abstains when $D$ contains no relevant information (Bondarenko et al., 6 May 2026). H-RAG extends this to Task C by evaluating whether systems can generate a fluent conversational answer that is factually grounded in retrieved evidence and able to abstain on unanswerable queries (Elchafei et al., 1 May 2026).

2. Evaluation protocol and official metrics

Task A is ranked primarily by normalized Discounted Cumulative Gain at rank $K$. Caraman gives the single-query formulation

$$\mathrm{DCG}@K=\sum_{i=1}^{K}\frac{2^{\mathrm{rel}_i}-1}{\log_2(i+1)}, \qquad \mathrm{nDCG}@K=\frac{\mathrm{DCG}@K}{\mathrm{IDCG}@K},$$

with secondary metrics of nDCG@10 and Recall@10 (Caraman et al., 12 May 2026). Caraman also notes that only answerable queries are scored in the Task A test setting (Caraman et al., 12 May 2026).

For Task B and Task C, submissions report harmonic-mean objectives over three components, but the notation differs slightly across papers. RaguTeam defines the official Task B score as

$\mathrm{HM}_3 = \frac{3}{1/M_1 + 1/M_2 + 1/M_3},$

where $M_1=\mathrm{RB\_alg}$, $M_2=\mathrm{RB\_llm}$, and $M_3=\mathrm{RL\_F}$ (Bondarenko et al., 6 May 2026). H-RAG reports Task C with the component triple RB_agg, RL_F, and RB_llm, again combined by harmonic mean (Elchafei et al., 1 May 2026). In practice, the common structure is stable even when the reference-based component is written as RB_alg or RB_agg.

The benchmark therefore measures three distinct axes. Reference-based relevance rewards overlap or agreement with gold responses; RL_F evaluates faithfulness to retrieved evidence; and RB_llm is an LLM-judge assessment of answer quality. RaguTeam further describes a conditioned variant for abstention, under which correctly abstaining on known unanswerable instances sets the metrics on those cases to 1 if the system outputs an IDK response (Bondarenko et al., 6 May 2026).

A recurring implication is that MTRAGEval is not a single-metric benchmark. It is a compound evaluation regime in which retrieval quality, grounding fidelity, abstention behavior, and answer usefulness can diverge. That design choice is central to how teams interpret success and failure.

3. Retrieval architectures and dominant design patterns

Task A submissions converge on a small set of retrieval patterns: query rewriting, hybrid dense-sparse retrieval, rank fusion, and reranking.

Caraman implements a three-stage pipeline: query rewriting with a LoRA-fine-tuned Qwen 2.5 7B Instruct model, hybrid BM25S and dense retrieval fused with Reciprocal Rank Fusion, and cross-encoder reranking with BGE-reranker-v2-m3. The official test result is nDCG@5 = 0.531, ranking 8th out of 38 systems and 10.7% above the organizer baseline (Caraman et al., 12 May 2026). A central empirical result is the effect of domain-specific temperature tuning: Cloud favors deterministic decoding at $D=\{d_1,\ldots,d_k\}$0, whereas FiQA and Govt benefit from modest randomness at $D=\{d_1,\ldots,d_k\}$1; by contrast, domain-aware prompting and multi-query expansion both degrade holdout performance to 0.350 nDCG@5 (Caraman et al., 12 May 2026).

H-RAG departs from passage-only formulations through a hierarchical parent-child retrieval pipeline. Documents are segmented into overlapping three-sentence child chunks with stride = 2, retrieval is performed at the child level using a hybrid dense-sparse score

$D=\{d_1,\ldots,d_k\}$2

with $D=\{d_1,\ldots,d_k\}$3 and optimal $D=\{d_1,\ldots,d_k\}$4, and selected child evidence is then aggregated back to parent documents by max-score aggregation. The submission reaches nDCG@5 = 0.4271 on Task A, and enabling child-first with rank_parents=true yields the best gains, reported as +0.0197 (Elchafei et al., 1 May 2026).

AILS-NTUA advances a more radical thesis: query diversity over retriever diversity. Instead of ensembling retrievers, it issues five DeepSeek-V3.2 reformulations—Minimal, Corpus-Specific, HyDE, Chain-of-Thought, and Anchor-Keyword—to a single ELSER v1 sparse retriever, then fuses them with variance-aware nested Reciprocal Rank Fusion. On the official leaderboard this system ranks 1st in Task A with nDCG@5 = 0.5776, reported as +20.5% over the strongest baseline (Athanasiou et al., 11 Mar 2026). The paper further reports that ELSER v1 achieved the highest R@5 = 0.483 and nDCG@5 = 0.444 in a 9-way retriever comparison, and attributes part of this advantage to corpus alignment because MTRAG’s annotation process itself used ELSER for silver labeling (Athanasiou et al., 11 Mar 2026).

5ting adopts a dense-first approach built around BGE-M3, FAISS IndexFlatIP, dual-query merged retrieval, and GPT-4o-mini listwise reranking. The retriever first uses both the original last-turn query and a rewritten standalone question, merges their candidate sets, and then invokes an LLM to output a ranked permutation of passage IDs. The reported Task A result is nDCG@5 = 0.4719, with an ablation trajectory from 0.3454 for base dense retrieval to 0.3732 after rewriting and 0.4393 after LLM reranking (Thien-Qua-T-Nguyen et al., 27 Jun 2026).

Across these systems, the most stable finding is that multi-turn retrieval quality depends heavily on rewriting and reranking rather than on first-stage retrieval alone. The exact retrieval substrate differs—BM25S, ELSER, BGE-M3, or hierarchical child chunks—but successful systems all try to resolve context dependence before final ranking.

4. Grounded generation and end-to-end RAG

Task B and Task C submissions place stronger emphasis on faithfulness control, answerability handling, and candidate selection.

RaguTeam’s Task B winner is a judge-orchestrated LLM ensemble. Seven LLMs, each used under one or two prompting variants, produce up to 14 candidates per instance, and GPT-4o-mini assigns a faithfulness score in [0,1] to each candidate relative to the reference passages. The system then selects the highest-scoring response and deterministically replaces any instance with empty $D=\{d_1,\ldots,d_k\}$5 by “I don’t have an answer”. The official conditioned score is HM$D=\{d_1,\ldots,d_k\}$6 = 0.7827, ranking 1st out of 26 teams and exceeding the strongest organizer baseline by +14.4 p.p. (Bondarenko et al., 6 May 2026).

AILS-NTUA uses a markedly different architecture: a multistage grounded generation pipeline with answerability triage, evidence span extraction, dual candidate generation, calibrated multi-judge selection, and optional micro-adjustment. In Task B it reports HM = 0.7698, ranking 2nd out of 26, with component scores RB_alg = 0.633, RL_F = 0.897, and RB_llm = 0.832. In Task C the same system reaches HM = 0.5409, with RB_alg = 0.400, RL_F = 0.729, and RB_llm = 0.598 (Athanasiou et al., 11 Mar 2026).

H-RAG links retrieval and generation through parent-level context reconstruction. Retrieved child chunks are aggregated into parent documents, which are then supplied to GPT-5 with a prompt that enforces conversational tone, no explicit citations, and graceful abstention when evidence is insufficient. The reported Task C result is HM = 0.3241, decomposed into RB_agg = 0.2488, RL_F = 0.2703, and RB_llm = 0.6508 (Elchafei et al., 1 May 2026).

5ting emphasizes role-separated decoding. Its prompt distinguishes conversation history as context for intent resolution only, retrieved passages as the sole factual authority, and the last-turn question as the object to be answered. The system prompt explicitly instructs the model to use only retrieved passages and to reply that it cannot provide a complete answer if the passages are insufficient. On Task C, 5ting reports RB_llm = 0.6784, RL_F = 0.7692, RB_alg = 0.3867, and Harmonic = 0.5597 (Thien-Qua-T-Nguyen et al., 27 Jun 2026).

The generation literature within MTRAGEval therefore spans two broad paradigms. One uses candidate multiplicity plus judging; the other uses pipeline decomposition and explicit grounding constraints. Both are responses to the same failure mode: fluent multi-turn responses can remain unfaithful unless generation is tightly coupled to retrieved evidence.

5. Failure modes, benchmark limitations, and disputed bottlenecks

The benchmark papers are unusually explicit about limitations. RaguTeam identifies five: target leakage via empty $D=\{d_1,\ldots,d_k\}$7 for unanswerable queries, coverage gaps in gold answers, metric leakage because RL_F is computable on the submission set, annotation errors and inconsistencies, and a lack of unified guidelines for generation vs. evaluation (Bondarenko et al., 6 May 2026). These are benchmark-design issues rather than model-specific errors.

At the system level, submissions disagree about the dominant bottleneck. H-RAG concludes that retrieval performance remains the primary bottleneck for faithful generation, whereas AILS-NTUA argues that answerability calibration—rather than retrieval coverage—is the primary bottleneck in end-to-end performance (Elchafei et al., 1 May 2026, Athanasiou et al., 11 Mar 2026). AILS grounds that claim in the observed Task B to Task C drop, especially the collapse of RB_alg from 0.633 to 0.400, alongside an answerability classifier with ~83.9% accuracy but only 21.8% unanswerable recall (Athanasiou et al., 11 Mar 2026).

5ting formulates the problem space as four core challenges: context dependency and drift, under-specification, hallucination risk, and retrieval–generation alignment, explicitly noting that strong retrieval alone does not guarantee faithful generation (Thien-Qua-T-Nguyen et al., 27 Jun 2026). Caraman’s retrieval results reinforce a parallel point on the retrieval side: more elaborate query engineering is not necessarily helpful, since domain-aware prompting and multi-query expansion can underperform simpler temperature-controlled rewriting (Caraman et al., 12 May 2026).

A plausible implication is that MTRAGEval is valuable precisely because it prevents a single-axis narrative of progress. A system may improve nDCG@5 without improving end-to-end answerability; a system may improve RL_F while remaining weak on reference-based relevance; and abstention can materially change rankings in tasks with nontrivial unanswerable rates.

6. Alternative uses of the label and relation to adjacent evaluation work

Outside SemEval-2026 Task 8, the label MTRAGEval appears in at least two additional senses in the provided literature. In radiology report generation, it denotes a task-adaptation of the COMET Estimator framework that predicts a continuous quality score from a ground-truth report $D=\{d_1,\ldots,d_k\}$8 and a generated report $D=\{d_1,\ldots,d_k\}$9 by encoding them separately, forming the joint feature vector

$D$0

and regressing either RadGraph F$D$1 or RadCliQ with MSE supervision (Calamida et al., 2023). That system uses XLM-R or BioClinical BERT encoders, reports strong correlations with established metrics, and attains Spearman $D$2 against radiologist total-error counts on the ReXVal dataset for one checkpoint (Calamida et al., 2023).

In a separate imaging context, the mesh-to-raster registration paper invokes “implications for MTRAGEval” to describe best practices for multi-modal ROI alignment assessment: use a gold-standard GT from multi-rater STAPLE, report Dice together with Hausdorff or MSD, employ repeated-measures ANOVA, and solve registration as nonlinear least-squares via Gauss–Newton with a descending-$D$3 continuation (Tatano et al., 2017). Here the term does not denote the SemEval benchmark; it denotes an evaluation perspective on registration accuracy.

The broader evaluation ecosystem surrounding multi-turn and multimodal RAG also helps contextualize MTRAGEval. MiRAGE proposes a claim-centric framework with INFOF1 and CITEF1 for multimodal grounding (Martin et al., 28 Oct 2025). MRAG-Suite introduces difficulty-based and ambiguity-aware filtering together with MM-RAGChecker for claim-level diagnostics (Ji, 29 Sep 2025). MEG-RAG defines Multi-modal Evidence Grounding (MEG) as a semantic-aware measure of the contribution of retrieved evidence and trains a reranker against that signal (Wang et al., 27 Apr 2026). These frameworks are not MTRAGEval, but they address the same structural concerns that MTRAGEval submissions repeatedly surface: ambiguity, evidence utility, hallucination, and grounding fidelity.

Taken together, the literature presents MTRAGEval less as a single method than as a family of evaluation problems centered on grounded multi-turn reasoning. In current usage, its principal referent is the SemEval-2026 benchmark; yet the persistence of alternate senses in radiology and registration underscores that the name still functions as a moving label across neighboring evaluation traditions.