5ting at SemEval-2026 Task 8: Strong End-to-End Multi-Turn RAG via LLM-Based Reranking and Faithfulness Control

Abstract: We introduce 5ting, our system for the SemEval2026 Task 8 (MTRAGEval), which evaluates multi-turn Retrieval Augmented Generation (RAG) systems. Multi turn RAG involves context drift, under specification, and hallucination risk. Our system combines BGE-M3 dense retrieval with FAISS indexing, dual-query merged retrieval, and LLM based reranking, followed by role separated generation constrained to retrieved evidence. The retriever achieved nDCG@5 = 0.4719 in Task A, while the end to end system ranked in Task C with a harmonic score of 0.5597 and RL_F = 0.7692.

• The paper introduces a modular end-to-end RAG system that integrates dual-query retrieval with LLM-based reranking to address context drift and enhance precision.
• The system achieves strong empirical performance with a nDCG@5 of 0.4719 (98.4% of a sparse baseline) and a generation score improvement of +16.5% in faithfulness.
• The study demonstrates that rigorous query rewriting and prompt engineering can effectively mitigate hallucination risks and improve evidence-constrained answer generation.

End-to-End Multi-Turn RAG: System Design and Evaluation in 5ting at SemEval-2026 Task 8

Introduction

The 5ting system addresses the SemEval-2026 Task 8 (MTRAGEval), a rigorous benchmark for evaluating end-to-end Retrieval-Augmented Generation (RAG) in multi-turn conversational settings. Multi-turn RAG introduces unique challenges including context drift, query underspecification, and heightened hallucination risk. The objective of MTRAGEval is to assess both passage retrieval and grounded answer generation across complex, cross-domain dialog scenarios with evolving information needs. This essay critically examines the architectural decisions, methodological innovations, empirical results, and broader implications presented in "5ting at SemEval-2026 Task 8: Strong End-to-End Multi-Turn RAG via LLM-Based Reranking and Faithfulness Control" (2606.28737).

System Architecture

5ting implements a modular multi-turn RAG pipeline, comprising dense retrieval, query rewriting, LLM-based reranking, and strictly evidence-grounded generation. The core components are:

1. Query Rewriting: Given that last-turn queries often contain unresolved references or ellipsis, a GPT-4o-mini module synthesizes a self-contained, context-resolved reformulation based on the dialog history. This ensures that downstream retrieval and ranking operate on an unambiguous representation of user intent.
2. Dense Retrieval and Indexing: The system leverages BAAI/bge-m3 as the dense encoder, producing 1024-dimensional embeddings over 8K-token inputs, and utilizes FAISS IndexFlatIP per domain for efficient cosine similarity search. Both the original last-turn and rewritten queries independently retrieve passages, exploiting the complementary strengths of surface lexical match and deep contextual relevance.
3. Dual-Query Merging: The union of top-10 candidates from each query stream (last-turn and rewrite) yields a diverse pool of 10-20 unique passages. Redundant entries are consolidated by maximum score, optimizing for both recall and coverage.
4. LLM-Based Reranking: A listwise reranking phase further refines candidate order. The rewrite query and all candidate passages are fed to GPT-4o-mini, which outputs a global ranking. This strategy is computationally more expensive but demonstrably enhances ranking precision, particularly in multi-turn settings where single-point similarity is insufficient for robust selection.
5. Evidence-Constrained Generation: For answer synthesis, only the top-5 reranked passages are passed via a structured, anti-hallucination prompt that clearly separates dialogue context (intent only) from retrieved passages (exclusive factual authority). The conversation history resolves intent, while all factual claims are strictly limited to retrieved evidence.

   Figure 1: The architecture of the multi-turn RAG pipeline, illustrating query rewriting, dense retrieval, dual merging, LLM reranking, and evidence-constrained answer generation.

Empirical Results and Analysis

5ting exhibits strong empirical performance across both retrieval and generation subtasks:

• Retrieval (Subtask A): The system's $nDCG@5$ reaches 0.4719, achieving 98.4% of the highly-optimized ELSER sparse retrieval baseline. This is notable, as 5ting employs a general-purpose dense encoder without task-specific adaptation, validating the efficacy of LLM-based reranking and dual-query retrieval for domain-general applications.
• Generation (Subtask C): The system attains a combined score of 0.5597 (harmonic mean of $RB_\text{alg}$, $RB_\text{LLM}$, $RL_F$) and an $RL_F$ of 0.7692, exceeding the best baseline $RL_F$ by +16.5%. This underscores the impact of prompt engineering and evidence separation over brute model scale, as $RB_\text{LLM}$ matches that of much larger LLMs.

Strong ablation findings include:

• LLM reranking yields the largest marginal gain at $nDCG@5$ (+17.7%), especially post query rewriting.
• Query rewriting enhances recall and early precision (notably +40% at $nDCG@1$), with reranking refining these gains.
• The performance gap between retrieval and end-to-end generation suggests that precise evidence grounding can partially mitigate moderate retrieval recall.

Domain-specific results show variation: ClapNQ ($nDCG@5$ 0.5112) benefits most from dense semantic retrieval, while FiQA ($RB_\text{alg}$0 0.3872) remains challenging due to domain-specific terminology and ambiguity. The gap between $RB_\text{alg}$1 and $RB_\text{alg}$2 is small, indicating that reranking primarily boosts precision at the top ranks.

Contradictory/Notable claims: Evidence-constrained generation with a smaller model can match or exceed the faithfulness metrics of larger-scale LLMs, directly attributing improvements to prompt design rather than parameter count.

Error Analysis and Theoretical Implications

Manual analysis attributes nearly half of retrieval failures to unresolved coreferences, highlighting query rewriting as a critical bottleneck. Other failure modes include topic shifts, long-range dependencies, rare terminology, and annotation ambiguity. In generation, low-scoring cases mostly result from insufficient retrieval context or over-extraction, rather than hallucination—suggesting efficacy of anti-hallucination prompts.

This reinforces that in multi-turn RAG, query resolution and contextual alignment are prerequisites for effective retrieval, and that precision in evidence scoping is paramount for reliable generation. The analysis aligns with open challenges in the MTRAG-UN dataset (Rosenthal et al., 26 Feb 2026), particularly with regards to non-standalone and unanswerable questions.

Limitations and Prospects

The system depends on proprietary LLM APIs, imposing reproducibility and latency constraints. No task-adaptive fine-tuning or hybrid sparse-dense retrieval was explored. Potential gains remain in open-source rerankers (e.g., RankZephyr), further embedding optimization, and more granular handling of ambiguity and underspecification at the pipeline level.

Theoretically, the architecture substantiates the increasing role of LLMs in retrieval reranking and prompt-mediated generative control, suggesting future RAG systems will be less bottlenecked by retriever capacity and more by alignment of evidence selection and grounding at the interface between modules.

Conclusion

5ting delivers a high-precision, end-to-end RAG system for complex multi-turn dialogues, achieving robust retrieval and generation metrics via component modularity, dual-query retrieval, LLM reranking, and strict faithfulness prompts. Empirical superiority in $RB_\text{alg}$3 and near-parity with bespoke sparse baselines validate the design. The findings spotlight reranking and prompt design as key levers for next-generation conversational RAG systems and motivate further research into hybrid retrieval architectures and fine-grained control of context dependence and ambiguity.