Retrieval Hallucinations in Augmented Systems
- Retrieval hallucinations are failures in retrieval-augmented systems where generated outputs are unsupported by or inconsistent with retrieved evidence.
- They arise from an imbalance between internal parametric knowledge and external context, with factors such as retrieval order instability and attention biases contributing.
- Mitigation strategies include improved retrieval selection, adversarial query framing, and structured grounding methods that enhance factual consistency.
Retrieval hallucinations are failures in retrieval-augmented systems in which generated outputs are not supported by, or are inconsistent with, retrieved evidence. In the literature, this phenomenon appears in classical text RAG, multimodal systems, graph-augmented generation, knowledge-graph querying, and agent memory systems. Retrieval augmentation can substantially reduce hallucination relative to purely parametric generation, as shown in knowledge-grounded dialogue, but later work demonstrates that retrieval does not eliminate hallucination and can introduce distinctive failure modes tied to retrieval quality, context use, model internals, and inference-time instability (Shuster et al., 2021, Sun et al., 2024).
1. Conceptual scope and terminology
A recurring distinction is between hallucinations caused by deficient internal knowledge and hallucinations caused by retrieval itself. Rowen separates internal hallucinations, arising from the model’s own knowledge gaps, from external hallucinations, introduced when irrelevant external information contaminates the answer (Ding et al., 2024). SIRG further distinguishes factuality hallucinations, which retrieval can reduce by grounding answers in retrieved context, from faithfulness hallucinations, where responses remain inconsistent with the provided context despite retrieval (Hu et al., 6 Jan 2026). FVA-RAG identifies a more specific retrieval failure, Retrieval Sycophancy, in which vector-based retrievers fetch documents that align with a false premise or common misconception, leading the model to “hallucinate with citations” (Ravishankara, 7 Dec 2025).
These distinctions have been generalized beyond text-only QA. In medical multimodal generation, hallucination is described as the generation of clinically incorrect or unsupported information; in VQA, retrieval hallucinations are cases where models generate plausible yet incorrect answers not supported by image content or relevant factual knowledge; in graph and memory settings, hallucinations are outputs inconsistent with retrieved triples or retrieved memories, respectively (Chu et al., 20 Feb 2025, Sarwar, 25 Feb 2025, Li et al., 9 Dec 2025, Chen et al., 5 Nov 2025).
| Failure mode | Description in the literature | Representative source |
|---|---|---|
| Faithfulness hallucination | Response inconsistent with provided context | (Hu et al., 6 Jan 2026) |
| External hallucination | Irrelevant external information contaminates the answer | (Ding et al., 2024) |
| Retrieval sycophancy | Retriever fetches support for a false premise | (Ravishankara, 7 Dec 2025) |
| Permutation-induced hallucination | Answer changes across permutations of the same retrieved set | (Zhang et al., 6 Jan 2026) |
This taxonomy suggests that “retrieval hallucination” is not a single error class. It includes failures of evidence selection, failures of evidence use, and failures of stability under alternative but semantically equivalent retrieval conditions.
2. Mechanistic accounts of why retrieval hallucinations arise
Several papers attribute retrieval hallucinations to disequilibrium between external context and internal parametric knowledge. ReDeEP reports that hallucinations occur when Knowledge FFNs overemphasize parametric knowledge in the residual stream while Copying Heads fail to effectively retain or integrate retrieved content (Sun et al., 2024). SEReDeEP adopts the same basic decomposition and states that excessive reliance on parametric knowledge encoded in FFNs and insufficient utilization of external information by attention mechanisms, particularly copy heads, are the two critical contributors to hallucinations in RAG (Wang, 12 May 2025). InterpDetect reaches a closely related conclusion, finding that later-layer FFN modules disproportionately inject parametric knowledge into the residual stream while attention heads underweight retrieved context (Tan et al., 24 Oct 2025).
A second mechanistic line focuses on specialized attention circuitry. DeCoRe treats retrieval heads as the attention heads responsible for extracting relevant contextual information and shows that masking these heads can induce hallucinations; its decoding strategy then contrasts the base model with a retrieval-head-masked variant to suppress hallucinated continuations (Gema et al., 2024). In graph-augmented generation, PRD and SAS were proposed to expose two distinct failure patterns: over-reliance on shortest-path triples and weak semantic grounding to retrieved knowledge (Li et al., 9 Dec 2025).
A third line emphasizes retrieval-side instability rather than internal knowledge override. Stable-RAG shows that, under Top-5 retrieval with the gold document included, answers vary substantially across permutations of the retrieved set, even when the gold document is fixed in the first position (Zhang et al., 6 Jan 2026). The paper argues that this effect is not simply positional bias or low-quality retrieval, but reflects branching reasoning trajectories associated with different retrieval orders. DelucionQA adds a domain-specific perspective, identifying retrieval insufficiency, confounding or overload from irrelevant context, and LLM prior-knowledge override as recurrent causes of hallucination in car-manual QA (Sadat et al., 2023).
3. Retrieval-side mitigation strategies
One family of methods attempts to improve retrieval selection before generation. A comparative analysis on HaluBench evaluates sparse retrieval based on BM25, dense retrieval using Sentence Transformers, and a hybrid module with query expansion and dynamically weighted Reciprocal Rank Fusion. The hybrid retriever achieves MAP@3 of 0.897 and NDCG@3 of 0.915, outperforming both sparse and dense retrieval, and on HaluBench_small it attains 80.41% accuracy, 9.38% hallucination rate, and 10.36% rejection rate (Mala et al., 28 Feb 2025). This result directly ties better top-3 retrieval relevance to lower hallucination and lower insufficient-context failure.
A second family performs retrieval only when evidence is needed. Rowen uses a multilingual semantic-aware detection module that checks the consistency of perturbed responses across languages and triggers retrieval only when inconsistency indicates hallucination. On TruthfulQA and StrategyQA, Rowen achieves FactScore 83.80 and accuracy 74.00, and it calls retrieval 1.5 times per TruthfulQA question versus 12.5 for Factool, 7.2 for Detect-and-Mitigate, and 2.1 for FLARE (Ding et al., 2024). The method is explicitly motivated by the claim that indiscriminate retrieval can itself increase external hallucinations.
A third family reframes retrieval as adversarial falsification rather than support-seeking verification. FVA-RAG introduces an Adversarial Retrieval Policy that generates contradiction-seeking “Kill Queries,” retrieves Anti-Context, and then uses a dual-verification mechanism to weigh the draft answer against contradictory evidence. On a dataset of common misconceptions, its intervention rate is 45.0%, with 4 interventions in Health, 3 in Superstitions, and 2 in Folklore/Myths (Ravishankara, 7 Dec 2025). This directly targets sycophantic retrieval, where standard semantic retrievers reinforce the user’s false premise.
A fourth family targets retrieval structure. Hyper-RAG uses a hypergraph to represent both pairwise and beyond-pairwise correlations in domain-specific knowledge. On NeurologyCrop, it improves accuracy by an average of 12.3% over direct LLM use and outperforms Graph RAG and Light RAG by 6.3% and 6.0%, respectively; across nine datasets it shows a 35.5% performance improvement over Light RAG using a selection-based assessment, while Hyper-RAG-Lite achieves twice the retrieval speed and a 3.3% performance boost compared with Light RAG (Feng et al., 30 Mar 2025). Stable-RAG, by contrast, keeps the retrieved set fixed and addresses order sensitivity by running the generator under multiple retrieval orders, clustering hidden states, and decoding from cluster-center representations; when the gold document is in position 1, PSR drops from 50.8% for Vanilla RAG to 28.3% for Stable-RAG (Zhang et al., 6 Jan 2026).
4. Detection, interpretability, and post-hoc adjudication
Interpretability-based detection has become a major approach to retrieval hallucinations. ReDeEP quantifies external-context use through an External Context Score and parametric reliance through a Parametric Knowledge Score, then combines them in a regression model to detect hallucinations; it also introduces AARF, which mitigates hallucination by amplifying selected Copying Heads and suppressing selected Knowledge FFNs at inference time (Sun et al., 2024). SEReDeEP extends this framework by replacing token-level uncertainty with semantic entropy estimated by trained linear probes, reporting accuracy improvements of 3–10% over ReDeEP and a cost that is 2 orders of magnitude cheaper than sampling-based entropy measures (Wang, 12 May 2025).
SIRG moves from token attribution to semantic fragments. It extends layer-wise relevance propagation from tokens to sentences and clauses, constructs a semantic-level internal reasoning graph, and classifies hallucinated semantic fragments with a lightweight PLM-based classifier. On RAGTruth and Dolly-15k, SIRG reaches F1 76.61% on RAGTruth with Llama-7B, 81.84% on RAGTruth with Llama-13B, and 89.71% on Dolly-15k with Qwen2.5-7B (Hu et al., 6 Jan 2026).
GraphRAG detection introduces two lightweight interpretability metrics. Path Reliance Degree measures over-reliance on shortest-path triples, and Semantic Alignment Score measures how well answer-token representations align with retrieved triples. Their formal definitions are:
On 5,000 KBQA examples, hallucinated outputs show significantly higher PRD and lower SAS, and the resulting GGA detector attains AUC 0.834 with Class-1 F1 0.539 on Llama2-7B and AUC 0.853 with Class-1 F1 0.461 on Qwen2.5-7B (Li et al., 9 Dec 2025).
Other detection systems emphasize efficiency and deployability. InterpDetect computes ECS and PKS on Qwen3-0.6b, trains regression-based classifiers, and reports validation F1 76.6 with SVC at span level; at response level, its proxy-evaluation F1 reaches 75.36 with recall 93.98 on GPT-4.1-mini responses (Tan et al., 24 Oct 2025). HalluGuard instead formulates hallucination detection as grounded-versus-hallucinated classification for document-claim pairs with evidence-grounded justifications; on the RAGTruth subset of LLM-AggreFact it reaches 84.0% balanced accuracy, matching MiniCheck (7B; 84.0%) and exceeding Granite Guardian 3.3 (8B; 82.2%) while using roughly half their parameters (Bergeron et al., 1 Oct 2025).
5. Multimodal, structured, and memory-specific variants
In multimodal systems, retrieval hallucinations arise when textual retrieval is insufficiently anchored to visual evidence. V-RAG addresses this by retrieving both similar medical images and their associated reports, rather than only textual reports from retrieved images. On MultiCaRe entity probing, the reported F1 values are 0.432 for RadFM, 0.901 for RAT/Img2Loc, 0.920 for V-RAG, and 0.940 for fine-tuned V-RAG; in report revision, RadGraph-F1 for “partial” information extraction increases from 0.145 for the original report to 0.172 after V-RAG revision (Chu et al., 20 Feb 2025). The paper emphasizes improvements for both frequent and rare entities.
FilterRAG addresses hallucinations in knowledge-driven and out-of-distribution VQA by combining BLIP-VQA with retrieval from Wikipedia and DBpedia and then filtering top- retrievals by similarity to the joint image-question embedding. On OK-VQA it achieves 36.5% accuracy, with ID grounding 70.06%, OOD grounding 70.68%, and overall grounding 70.37% (Sarwar, 25 Feb 2025). The method defines a hallucination event whenever the semantic grounding score falls below a threshold .
Object hallucination in LVLMs has also been treated as a retrieval-conditioned decoding problem. RVCD retrieves AI-generated single-concept images corresponding to objects hallucinated in a draft caption and uses their logits as negative evidence, while also adding positive logits for grounded objects to prevent over-suppression. On MSCOCO, the reported CHAIR values are 11.3 for LLaVA-1.5 and 9.0 for MiniGPT-4, CHAIR values are 3.87–3.61, and POPE accuracy reaches 88.54% (Lee et al., 26 May 2025). A related reliability-aware framework for multimodal inference retrieves nearest-neighbor visual evidence, computes similarity strength, class-support agreement, evidence margin, entropy-based uncertainty, and an aggregate reliability score, then decides whether to accept, answer with caution, or abstain. On ImageNet-100, accepted prediction accuracy rises from 85.84% to 88.88% at 89.04% coverage, while the hallucination-like accepted wrong-answer rate drops from 14.16% to 11.12% (Hariharan et al., 14 Jun 2026).
Structured retrieval creates different hallucination surfaces. PGMR decouples semantic generation from KG grounding by having the LLM generate an intermediate SPARQL query with natural-language labels marked by starturi ... enduri, then replacing those spans with URIs retrieved from a non-parametric memory indexed by label embeddings. On the original LCQUAD 2.0 split, URI hallucination drops from 27.75% to 0.38% for T5-Small, from 89.84% to 0.01% for Llama 3.1 8B, and from 81.81% to 0% for GPT-4o (Sharma et al., 19 Feb 2025). In agent memory systems, HaluMem defines hallucinations at the level of memory extraction, memory updating, and memory question answering; its datasets contain about 15k memory points and 3.5k multi-type questions, with average dialogue lengths of 1.5k and 2.6k turns and context lengths exceeding 1M tokens (Chen et al., 5 Nov 2025). The reported conclusion is that hallucinations generated during extraction and updating accumulate and propagate into retrieval and QA.
6. Evaluation practice, benchmarks, and recurring misconceptions
Evaluation protocols vary by setting, but the literature repeatedly favors grounding-sensitive metrics over generic fluency measures. In knowledge-grounded dialogue, Knowledge F1 and Rare F1 correlate more strongly with human “Knowledgeable” and “Hallucination” judgments than standard F1 or perplexity, and retrieval-augmented models reduced hallucinations in human annotation by over 60% compared with non-retrieval models, with up to 85% improvement on out-of-distribution test data (Shuster et al., 2021). DelucionQA, a human-annotated benchmark for retrieval-augmented car-manual QA, contains 2,038 triples, of which 738 are hallucinated and 1,300 are not; its best unsupervised baseline, sentence-overlap matching, reaches 71.1% Macro F1 on the test set (Sadat et al., 2023). HaluBench links retrieval quality and answer quality at scale: the hybrid retriever reduces hallucination rate to 9.38% on HaluBench_small while simultaneously lowering rejection rate to 10.36% (Mala et al., 28 Feb 2025).
A common misconception is that better retrieval alone is sufficient. The surveyed papers consistently contradict this. Stable-RAG shows that answers can change across permutations of the same retrieved set (Zhang et al., 6 Jan 2026); FVA-RAG shows that retrievers can reinforce false premises (Ravishankara, 7 Dec 2025); ReDeEP, SEReDeEP, and InterpDetect show that internal FFN-mediated parametric knowledge can override retrieved evidence even when accurate context is present (Sun et al., 2024, Wang, 12 May 2025, Tan et al., 24 Oct 2025). Another misconception is that retrieval hallucination is confined to text QA. The evidence covers multimodal radiology and VQA, object-centric LVLM captioning, GraphRAG, SPARQL generation, and long-horizon agent memory (Chu et al., 20 Feb 2025, Sarwar, 25 Feb 2025, Lee et al., 26 May 2025, Li et al., 9 Dec 2025, Sharma et al., 19 Feb 2025, Chen et al., 5 Nov 2025).
The current literature therefore treats retrieval hallucinations as a systems problem rather than a single-model defect. Retrieval quality, retrieval order, adversarial query framing, internal circuit behavior, semantic grounding, and abstention policy all affect whether retrieval acts as a corrective mechanism or becomes the substrate for new hallucinations. This suggests that future robust RAG systems will require joint advances in retrieval design, mechanistic diagnosis, structured grounding, and decision policies that can reject or revise unsupported outputs rather than merely generate more fluent ones.