RaguTeam at SemEval-2026 Task 8: Meno and Friends in a Judge-Orchestrated LLM Ensemble for Faithful Multi-Turn Response Generation

Abstract: We present our winning system for Task~B (generation with reference passages) in SemEval-2026 Task~8: MTRAGEval. Our method is a heterogeneous ensemble of seven LLMs with two prompting variants, where a GPT-4o-mini judge selects the best candidate per instance. We ranked 1st out of 26 teams, achieving a conditioned harmonic mean of 0.7827 and outperforming the strongest baseline (gpt-oss-120b, 0.6390). Ablations show that diversity in model families, scales, and prompting strategies is essential, with the ensemble consistently beating any single model. We also introduce Meno-Lite-0.1, a 7B domain-adapted model with a strong cost--performance trade-off, and analyse MTRAGEval, highlighting annotation limitations and directions for improvement. Our code is publicly available: https://github.com/RaguTeam/ragu_mtrag_semeval

• The paper introduces a heterogeneous LLM ensemble that enhances response fidelity through a judge-orchestrated selection process.
• It applies dual prompt engineering strategies and parallel multi-model candidate generation to effectively handle diverse dialogue contexts.
• Empirical results demonstrate improved HM₃ scores and robust performance, underscoring the value of model diversity and tailored prompting.

Heterogeneous LLM Ensembles with LLM-Based Judging for Faithful Multi-Turn RAG: A Critical Analysis of RaguTeam at SemEval-2026 Task 8

Introduction

The task of generating faithful, context-grounded responses in multi-turn retrieval-augmented generation (RAG) is increasingly critical, especially as dialogue systems are integrated into knowledge-intensive domains. The SemEval-2026 Task 8 (MTRAGEval) addresses this by requiring models to generate answers based on provided passages and complex conversational histories. The RaguTeam submission presents a heterogeneous ensemble strategy, leveraging a diverse set of seven LLMs and two prompting variants, with final output selection driven by a GPT-4o-mini judge. This essay provides a thorough technical synthesis of the system, results, ablations, and implications for future RAG architectures.

System Overview and Architecture

The RaguTeam pipeline is structured as a three-stage ensemble framework:

1. Prompt Engineering: Two distinct strategies are employed—an iteratively refined system prompt (Group 1), and a category-aware few-shot prompting variant (Group 2).
2. Candidate Generation: Seven LLMs (four with system-prompt-only, three with few-shot) from different families (Gemini, GLM, Llama, Qwen, Claude, and the in-house Meno-Lite-0.1) generate candidate responses in parallel.
3. Judge-Based Selection: The GPT-4o-mini model evaluates each generated candidate for faithfulness, scoring them for strict grounding in the provided passages. For cases with empty reference context, a deterministic fallback abstaining response is used.

   Figure 1: Overview of the proposed framework. The architecture illustrates prompt construction, parallel LLM candidate generation, and judge-orchestrated response selection.

This structure facilitates both high candidate diversity (critical for robust coverage of failure modes) and systematic, instance-specific output selection, exploiting the strengths—and mitigating the idiosyncratic weaknesses—of each component LLM.

Prompt Engineering and Model Diversity

Prompting is handled via two complementary paradigms:

• Iteratively Refined System Prompts (Group 1): Using LLM-based analysis (Gemini), behavioral patterns are codified into a principled system design. This prompt enforces exclusive context adherence, extractive phrasings, depersonalized synthesis, and structural fidelity.
• Category-Aware Few-Shot Exemplars (Group 2): Four exemplars, medoid-selected from the training set, represent three core structural categories (empty context, empty history, full dialogue). This empirically boosts robustness, particularly for edge cases like underspecification or answerability detection.

Model selection for the ensemble is intentionally heterogeneous: parameters range from 7B (Meno-Lite-0.1) to 357B (GLM-4.6), and providers span open and proprietary lines. Such architectural and pipeline diversity is essential for maximizing the support coverage across MTRAGEval's multi-turn, evidence-complex scenarios.

Empirical Results and Ablation Insights

The judge-based ensemble achieves a conditioned harmonic mean (HM₃) of 0.7827 on the official test set, surpassing the best single model (GLM-4.6 at 0.748) and the strongest baseline (gpt-oss-120b at 0.6390) by a wide margin. Breakdown by prompting regime demonstrates:

• Group 1 models (system prompt only): Higher overall mean HM₃ (0.734) than Group 2 (few-shot, 0.634).
• Few-shot exemplars yield significant gains on underspecified questions compared to abstract system directives (+3.7 p.p. absolute for GLM-4.6, Llama).

Ablation studies reveal several key findings:

• Judge-based selection vs. random: +2.5 p.p. on Answerable, +1.9 p.p. on Underspecified; this primarily arises from improved faithfulness (RL_F).
• Ensemble diversity: Even diverse random selection exceeds most single-model configurations, underlining diversity's intrinsic value.
• Meno-Lite-0.1 contribution: Selected infrequently but provides robust, cost-effective performance, especially on unanswerable instances (IDK rate 0.54 vs. Qwen2.5-32B-Instruct's 0.24), and demonstrates near-parity with much larger models on Russian benchmarks.
• IDK fallback: The deterministic abstention for empty context yields perfect scores for the unanswerable subset, though the shortcut is an artifact of MTRAGEval's dataset construction.

Per-Model Contribution and Selection Analysis

Model selection in the ensemble is highly concentrated—Gemini-3-Pro-Preview dominates Answerable instance selection, while the deterministic fallback handles all cases with empty passages. Claude 4.5 Haiku, though rarely selected, achieves the best partial-answer harmonic mean (HM ≈ 0.697 in selected rows on PARTIAL questions), indicating significant specialization among ensemble members.

Figure 2: Contribution of each model to the weighted harmonic mean of conditioned metrics. Gemini, deterministic fallback, and Claude 4.5 Haiku collectively dominate ensemble composition.

The ensemble's superior mean performance is not exclusively driven by exploiting the empty-context/IDK shortcut; its advantage over the best individual model persists even on Answerable instances alone (HM₃ = 0.785 vs. 0.782).

Error and Benchmark Analysis

Qualitative and quantitative analysis revealed persistent challenges:

• Underspecified turns: Models frequently answer by guessing rather than seeking clarification, leading to systematic deficits in this class.
• Pragmatic incompleteness: Faithful, grounded responses may miss the communicative or pragmatic intent present in reference standards, causing misalignment in reference-based metrics.
• Metric disagreements: Faithfulness (RL_F) is not wholly aligned with completeness or reference similarity (ROUGE/BERTScore), suggesting the need for composite or multi-objective selection.

Critical benchmark limitations were highlighted:

• Empty-context as a negative signal: All unanswerable questions have empty references, trivializing answerability detection and inflating system performance in this subset.
• Annotation incompleteness: Reference answers are not always exhaustive, possibly penalizing correct but additional information.
• Reference-less faithfulness metric: Computation on the submission set allows for partial metric leakage during development.

Theoretical and Practical Implications

The RaguTeam system underscores several broader conclusions:

• Model family and prompting diversity is more influential than raw parameter count for multi-turn, retrieval-grounded tasks.
• Behavioral exemplars (few-shot) have measurable impact on edge-case and underspecified behaviors, relative to highly abstract directives.
• Faithfulness-oriented selection ensures context adherence, but must be augmented with completeness or pragmatic criteria for optimal human-aligned outputs.
• Lightweight, domain-oriented models (e.g., Meno-Lite-0.1) can, with targeted fine-tuning, approach the practical utility of far larger general-purpose models for context-grounded applications, especially in resource-constrained deployments.

Further, the reliance on a proprietary judge (GPT-4o-mini) as arbiter introduces selection bias—open judges, multi-judge schemes, or explicit multi-objective selectors (faithfulness plus completeness/appropriateness) remain relatively unexplored.

Future Directions

Future investigation should address:

• Adaptive and dynamic routing: Classifier-driven invocation of model subsets, possibly specialized per question type/answerability profile.
• Robust benchmark design: Inclusion of distractor passages for unanswerable instances, expert-validated references, and restricted access to reference-less metrics.
• Self-consistency and multi-objective selection: Techniques such as self-consistency aggregation and multi-axis selection criteria (completeness, faithfulness, pragmatic intent) should be empirically characterized.
• Scaling domain adaptation: Extending the Meno-Lite paradigm to larger models (14B–32B) and additional languages or domains.

Conclusion

The RaguTeam system demonstrates the efficacy of LLM ensembles orchestrated by a faithfulness judge for multi-turn RAG, achieving state-of-the-art performance on a challenging multi-turn benchmark. Strong empirical results validate that prompt engineering, model diversity, and judge-based selection are individually and collectively beneficial. However, substantial challenges remain, notably in the handling of pragmatics, underspecification, and robustness under realistic retrieval conditions. The approach sets a strong precedent for future architectures in retrieval-grounded dialogue, contingent on further benchmark evolution and the rigorous development of both selection and evaluation paradigms.