Retrieval-Augmented Conversational QA

• Retrieval-Augmented Conversational QA is a task that integrates external evidence retrieval with multi-turn dialogue to generate well-grounded answers.
• It employs mechanisms such as query refinement, adaptive retrieval, and conversational context usage to mitigate coreference errors and topic drift.
• Domain adaptation and multi-modal structured retrieval further enhance performance by aligning retrieval strategies with specific dataset and dialogue characteristics.

Retrieval-Augmented Conversational Question Answering (RAG-QA) is the task of answering user questions in multi-turn dialogue while grounding responses in an external knowledge corpus via retrieval. In this setting, a LLM is augmented with a retriever over an external corpus, and each turn is answered by combining the current utterance with dialogue history and retrieved evidence rather than relying solely on parametric memory. Across recent work, the external evidence may be open-domain passages, domain-specific knowledge bases, tables, PDF elements, enterprise wiki pages, or knowledge-graph-derived representations, and the central technical problem is how to retrieve, contextualize, and use evidence while controlling coreference error, topic drift, and hallucination (Alushi et al., 10 Feb 2026, Rakin et al., 2024, Hoang et al., 22 Jun 2025, Roy et al., 2024, Roy et al., 2024).

1. Formal model and task definition

The canonical RAG formulation models the answer $y$ to a query $x$ by marginalizing over retrieved passages $z$ from an external corpus $\mathcal{Z}$: $$p(y \mid x) = \sum_{z \in \mathcal{Z}} p(z \mid x)\, p(y \mid x, z).$$ In the original RAG architecture described in domain-adaptation work, $p(z \mid x)$ is induced by a DPR/BERT-based retriever over a FAISS index, while $p(y \mid x,z)$ is produced by a BART seq2seq generator; the implementation emphasized there is the RAG-Token form, in which the generator can attend differently to passages at each output token (Rakin et al., 2024).

Conversational QA generalizes this formulation from one-shot QA to multi-turn interaction. A conversation can be represented as $C=\{(u_t,s_t)\}_{t=1}^T$, with retrieval and generation at turn $t$ written as

$$q_t = f(u_t, h_{t-1}), \qquad R(q_t)=\operatorname{TopK}\big(\text{score}(q_t,d),\, d\in\mathcal{D}\big), \qquad a_t = G(q_t, R(q_t), h_{t-1}),$$

where $x$0 is dialogue history. The practical difference from single-turn QA is that the retriever must interpret pronouns, ellipsis, and evolving intent, while the generator must remain grounded as the history grows (Alushi et al., 10 Feb 2026).

This formulation also clarifies why conversational RAG is difficult. Later turns may depend on earlier entities, assumptions, or previously retrieved passages; accumulated history can either help retrieval, when the dialogue remains on one topic, or degrade it, when topic switching and mixed initiative dominate. Recent comparative work therefore treats retrieval quality and generation quality as coupled but distinct objects of study, rather than assuming that a strong retriever or a strong generator alone suffices (Alushi et al., 10 Feb 2026).

2. Conversationalization mechanisms

A major line of work treats conversationalization as a problem of deciding when to retrieve, what to retrieve, and how to exploit dialogue history. In SELF-multi-RAG, the model extends SELF-RAG to multi-turn settings by introducing retrieval-decision tokens $x$1, $x$2, and $x$3, together with summarized conversational context used as the retrieval query. On QReCC, the resulting summaries improved retrieval over T5-based query rewriting: with BM25, R@5 rose from $x$4 for T5QR to $x$5 for SELF summaries, and with Contriever from $x$6 to $x$7; the same work reports improvements of $x$8 by human annotation across conversational datasets (Roy et al., 2024).

A second line makes the conversational query itself an explicit object of modeling. ConvRAG decomposes the problem into a Conversational Question Refiner, a Fine-Grained Retriever, and a Self-Check based Response Generator. The refiner maps $x$9 to a self-contained reformulation $z$0 and a keyword set $z$1; retrieval then proceeds at paragraph level, and the generator produces both a helpfulness assessment and an answer. The ablation results show that question refinement is the most consequential module: removing the refiner reduced ROUGE-L from $z$2 to $z$3 and METEOR from $z$4 to $z$5 on the seen test split (Ye et al., 2024).

A third line adapts the retriever rather than inserting a separate rewriting model. ChatQA introduces a dense retriever optimized for conversational QA and a two-stage instruction-tuning recipe for the generator. For retrieval, fine-tuning Dragon on multi-turn query–context pairs yielded top-1 recall around $z$6–$z$7 and top-5 recall around $z$8–$z$9, essentially comparable to GPT-3.5-Turbo-based query rewriting, which reached top-1 $\mathcal{Z}$0 and top-5 $\mathcal{Z}$1, but without the extra rewriting model in the inference path (Liu et al., 2024).

Large-scale empirical comparison across eight conversational QA datasets suggests that no single conversationalization mechanism dominates. Robust methods such as reranking, hybrid BM25, and HyDE consistently outperform vanilla RAG, while several advanced methods—including some query rewriting and summarization pipelines—can degrade performance below the No-RAG baseline. The strongest gains often come from aligning the retrieval strategy with dataset structure rather than from increasing architectural complexity (Alushi et al., 10 Feb 2026).

3. Domain adaptation, question-centric indexing, and generator tuning

Domain adaptation is one of the most established ways to convert generic RAG into effective conversational QA. In hotel-domain HotelConvQA, RAG-Original uses a Wikipedia-trained DPR retriever and a BART generator without domain-specific fine-tuning and performs poorly, with EM $\mathcal{Z}$2, F1 $\mathcal{Z}$3, Top-5 retrieval $\mathcal{Z}$4, and Top-20 $\mathcal{Z}$5. End-to-end QA fine-tuning alone improves this to EM $\mathcal{Z}$6, F1 $\mathcal{Z}$7, Top-5 $\mathcal{Z}$8, and Top-20 $\mathcal{Z}$9. A two-stage retriever adaptation followed by QA fine-tuning raises performance further to EM $$p(y \mid x) = \sum_{z \in \mathcal{Z}} p(z \mid x)\, p(y \mid x, z).$$0, F1 $$p(y \mid x) = \sum_{z \in \mathcal{Z}} p(z \mid x)\, p(y \mid x, z).$$1, Top-5 $$p(y \mid x) = \sum_{z \in \mathcal{Z}} p(z \mid x)\, p(y \mid x, z).$$2, and Top-20 $$p(y \mid x) = \sum_{z \in \mathcal{Z}} p(z \mid x)\, p(y \mid x, z).$$3. The strongest model, RAG-end2end initialized from rag-token-nq and trained with a reconstruction objective, reaches EM $$p(y \mid x) = \sum_{z \in \mathcal{Z}} p(z \mid x)\, p(y \mid x, z).$$4, F1 $$p(y \mid x) = \sum_{z \in \mathcal{Z}} p(z \mid x)\, p(y \mid x, z).$$5, Top-5 $$p(y \mid x) = \sum_{z \in \mathcal{Z}} p(z \mid x)\, p(y \mid x, z).$$6, and Top-20 $$p(y \mid x) = \sum_{z \in \mathcal{Z}} p(z \mid x)\, p(y \mid x, z).$$7 (Rakin et al., 2024).

The same paper also shows that domain adaptation operates at three levels: retriever adaptation, generator adaptation, and joint end-to-end adaptation. HotelConvQA itself is built from MultiWOZ v2.2, TripAdvisor hotel reviews, and a local hotel conversational dataset, yielding a knowledge base of $$p(y \mid x) = \sum_{z \in \mathcal{Z}} p(z \mid x)\, p(y \mid x, z).$$8 passages, $$p(y \mid x) = \sum_{z \in \mathcal{Z}} p(z \mid x)\, p(y \mid x, z).$$9 QA pairs, and $p(z \mid x)$0 reconstruction summaries. This construction is representative of domain-specific conversational RAG: passage segmentation, QA synthesis, human validation, and auxiliary reconstruction signals are treated as first-class components of the system, not mere preprocessing (Rakin et al., 2024).

A distinct but related strategy is to index a corpus not by chunks alone, but by the questions that chunks can answer. QuIM-RAG generates multiple potential questions for each chunk, embeds those questions with BAAI/bge-large-en-v1.5, quantizes them to prototype vectors, and builds an inverted index from prototypes to question–chunk pairs. The NDSU corpus in that work contains $p(z \mid x)$1 links, $p(z \mid x)$2 chunks, and $p(z \mid x)$3 generated questions; retrieval uses top $p(z \mid x)$4 semantically matched questions rather than raw chunk similarity. On the custom dataset, QuIM-RAG reaches Faithfulness $p(z \mid x)$5, Answer Relevancy $p(z \mid x)$6, Context Precision $p(z \mid x)$7, Context Recall $p(z \mid x)$8, and BERTScore F1 $p(z \mid x)$9, outperforming traditional RAG baselines on both RAGAS and BERTScore (Saha et al., 6 Jan 2025).

Generator-side adaptation is equally consequential. ChatQA attributes much of its performance to context-enhanced instruction tuning: Stage 1 provides generic dialogue alignment, while Stage 2 teaches the model to use Context for Latest Question, to answer from retrieved evidence, and to indicate when the answer cannot be found in the context. On ChatRAG Bench, ChatQA-1.0-70B scores $p(y \mid x,z)$0, slightly above GPT-4-0613 at $p(y \mid x,z)$1 and GPT-4-Turbo-2024-04-09 at $p(y \mid x,z)$2, while Llama3-ChatQA-1.5-70B reaches $p(y \mid x,z)$3 (Liu et al., 2024).

4. Heterogeneous evidence, structured retrieval, and multi-view evidence modeling

RAG-QA increasingly extends beyond plain text. PDF-focused retrieval-augmented QA is explicitly motivated by the “richness and diversity of data within PDFs—including text, images, vector diagrams, graphs, and tables,” and by the need to answer “complex multimodal questions, where several data types are combined in the query.” In this setting, the retrieval unit is no longer a passage alone, but may be a multimodal chunk that preserves layout, captions, table structure, and nearby text (Hoang et al., 22 Jun 2025).

Structured-Semantic RAG (SSRAG) systematizes this extension with three modules: Query Understanding and Augmentation, Agentic Query Routing, and Hybrid Retrieval with Context Unification. The router classifies queries as FACTUAL or TEMPORAL and sends them to a Wikipedia corpus or live web retrieval; the retriever then combines vector-based retrieval with graph-based retrieval, converts graph nodes and relations into text, embeds them into the same space, re-ranks, deduplicates, and returns a unified context. This architecture directly targets contextually relevant retrieval, multi-hop reasoning, and factual grounding across heterogeneous sources (Yang et al., 19 Jan 2026).

A more explicitly multi-hop formulation appears in ConRAG, which performs relation-view, entity-anchor-view, and text-evidence-view retrieval over an evidence-grounded knowledge graph and a textual corpus. Relation, entity, and text scores are normalized, structurally controlled, and fused through a consensus score

$p(y \mid x,z)$4

where $p(y \mid x,z)$5 rewards evidence supported by multiple views. Query-side decomposition and slot-bound execution then propagate intermediate answers into later retrieval steps. On HotpotQA, 2WikiMultiHopQA, and MuSiQue, ConRAG consistently outperforms baselines, with up to $p(y \mid x,z)$6 average performance gains over vanilla RAG (Zhu et al., 27 May 2026).

Enterprise and knowledge-graph settings push heterogeneity further. One RAGONITE system contextualizes each retrieved evidence with source metadata and surrounding text—page title, heading, preceding evidence, and succeeding evidence—before indexing and prompting; on ConfQuestions, this raises P@1 from $p(y \mid x,z)$7 with no context to $p(y \mid x,z)$8 with full contextualization, and answer relevance from $p(y \mid x,z)$9 to $C=\{(u_t,s_t)\}_{t=1}^T$0 (Roy et al., 2024). A second RAGONITE system for RDF KGs replaces brittle Text2SPARQL with a two-pronged pipeline: SQL over an automatically induced relational database and dense retrieval over verbalized RDF facts, combined through iterative retrieval. On the BMW KG benchmark, the full SQL+verbalizations configuration answers $C=\{(u_t,s_t)\}_{t=1}^T$1 questions correctly, compared with $C=\{(u_t,s_t)\}_{t=1}^T$2 for verbalization-only, $C=\{(u_t,s_t)\}_{t=1}^T$3 for SQL-only, and $C=\{(u_t,s_t)\}_{t=1}^T$4 for SPARQL-only (Roy et al., 2024).

5. Evaluation, hallucination analysis, and attribution

Evaluation in RAG-QA has moved well beyond EM and token-level F1. Comparative conversational work reports MRR@5, Recall@k, and SQuAD-style F1; question-centric systems add BERTScore and RAGAS; domain-specific systems add human hallucination annotation; diagnostic frameworks separate retrieval and generation failure modes rather than collapsing them into a single end metric (Alushi et al., 10 Feb 2026, Saha et al., 6 Jan 2025).

HotelConvQA provides one of the clearest hallucination analyses. Human annotators classify each response as Supported, Non-Supported, or Irrelevant, then further distinguish Input-conflicting and Fact-conflicting hallucinations. On the 100-question hallucination evaluation set, hallucination rates fall from $C=\{(u_t,s_t)\}_{t=1}^T$5 for RAG-Original to $C=\{(u_t,s_t)\}_{t=1}^T$6 for RAG-Finetuned-QA, $C=\{(u_t,s_t)\}_{t=1}^T$7 for RAG-DPR-adapted, and $C=\{(u_t,s_t)\}_{t=1}^T$8 for RAG-end2end; fact-conflicting hallucinations fall from $C=\{(u_t,s_t)\}_{t=1}^T$9 to $t$0 between the baseline and RAG-end2end. The same study, however, reports a trade-off: as Supported responses increase, the fraction of Supported responses labeled Correct decreases and Ambiguous or Incomplete responses increase (Rakin et al., 2024).

RAG-X makes this trade-off explicit with Context Utilization Efficiency (CUE), which partitions outcomes into Effective Use, Information Blindness, Hallucination (“Lucky Guess”), and Correct Rejection. The framework evaluates retrieval and generation independently across information extraction, short-answer generation, and MCQ answering, and introduces what it calls an “Accuracy Fallacy,” where a $t$1 gap separates perceived system success from evidence-based grounding. In the GuidelineQA case study, the best pipeline shows Effective Use $t$2, Lucky Guess $t$3, and Information Blindness $t$4, illustrating that substantial apparent success may be ungrounded (Sivakumar et al., 3 Mar 2026).

Attribution has likewise shifted from similarity-only heuristics toward intervention-based analysis. The enterprise RAGONITE system computes counterfactual attribution by removing evidence clusters, regenerating answers, and measuring similarity between the original and counterfactual answers; on questions where the gold document is retrieved in the top-10, the top-attributed evidence points to the correct document with $t$5 accuracy. This is a causal explanation strategy in the limited but operational sense that it estimates contribution by observing how the answer changes when the evidence is removed (Roy et al., 2024).

These evaluation results correct two common misconceptions. The first is that answer accuracy is sufficient evidence of grounding; RAG-X shows that it is not (Sivakumar et al., 3 Mar 2026). The second is that lower hallucination alone implies better answer quality; the HotelConvQA study shows that fewer hallucinations can coincide with more ambiguity and incompleteness among supported answers (Rakin et al., 2024).

6. Empirical patterns, trade-offs, and research directions

One broad empirical pattern is that retrieval strategy must match corpus structure and dialogue structure. Across eight conversational QA datasets, straightforward methods such as reranking, hybrid BM25, and HyDE consistently outperform vanilla RAG, but several advanced techniques fail to yield gains and can even degrade performance below the No-RAG baseline; the central conclusion is that effective conversational RAG depends less on method complexity than on alignment between the retrieval strategy and the dataset structure (Alushi et al., 10 Feb 2026).

A second pattern is that retrieval itself is becoming more adaptive and multi-source. MSRAG combines GPT-3.5-based retrieval, web retrieval after semantic partitioning into three sub-questions, and non-retrieval answering, then selects among answer-Web, answer-GPT, and answer-not. On 2WikiMultiHopQA it reaches EM $t$6 and F1 $t$7, and on StrategyQA accuracy $t$8, substantially above the listed baselines (Wu et al., 2024). DR-RAG, by contrast, treats some evidence as dynamically relevant only after the query is expanded with first-stage documents; with classifier forward selection it achieves F1 $t$9 and Acc $$q_t = f(u_t, h_{t-1}), \qquad R(q_t)=\operatorname{TopK}\big(\text{score}(q_t,d),\, d\in\mathcal{D}\big), \qquad a_t = G(q_t, R(q_t), h_{t-1}),$$0 on HotpotQA while keeping the number of LLM steps at $$q_t = f(u_t, h_{t-1}), \qquad R(q_t)=\operatorname{TopK}\big(\text{score}(q_t,d),\, d\in\mathcal{D}\big), \qquad a_t = G(q_t, R(q_t), h_{t-1}),$$1 (Hei et al., 2024).

A third pattern is the growing importance of routing and retrieval control. SSRAG routes factual and temporal queries to different sources and unifies graph- and vector-derived evidence (Yang et al., 19 Jan 2026). ConRAG performs query-side decomposition and corpus-side multi-view retrieval with consensus scoring (Zhu et al., 27 May 2026). SELF-multi-RAG learns whether to retrieve, not retrieve, or continue to use existing evidence, and uses summary-based retrieval queries rather than only standalone rewrites (Roy et al., 2024). These systems all imply that conversational RAG is increasingly less a single retrieval call than a controlled retrieval policy.

The main open directions are correspondingly structural. Recent work explicitly proposes end-to-end layout-aware RAG for multimodal documents, conversational document exploration, structured reasoning over tables and graphs, scalable indexing for massive and dynamic corpora, better user studies and interaction design, and stronger verification and grounding modules (Hoang et al., 22 Jun 2025, Yang et al., 19 Jan 2026, Zhu et al., 27 May 2026). A plausible implication is that future RAG-QA systems will combine adaptive retrieval control, heterogeneous evidence handling, and explicit grounding diagnostics, rather than treating retrieval as a static preprocessing step for an otherwise unchanged chat model.