Retrieval Enhanced Generations (RAGs)

• Retrieval Enhanced Generations (RAGs) are a family of workflows that integrate external retrieval mechanisms with language models to ensure outputs are factual and current.
• The standard RAG pipeline involves indexing, vector-based retrieval, and generation, using similarity metrics like dot product and cosine similarity to balance accuracy and latency.
• Recent innovations include structured, graph-based, and adaptive memory systems that optimize evidence aggregation for diverse applications from medicine to cybersecurity.

Retrieval-augmented generation (RAG) is a hybrid paradigm that integrates an external retrieval mechanism with a generative LLM so that model outputs are grounded in factual, up-to-date, or domain-specific knowledge rather than in parametric memory alone. In the recent literature, RAG is presented as a way to address two persistent limitations of LLMs: factual inaccuracies or hallucination, and the difficulty of updating knowledge without retraining. Across surveys, application papers, and systems work, RAG is treated not as a single architecture but as a family of retrieval-and-generation workflows that differ in how they index corpora, select evidence, manage context, and evaluate downstream utility (Gupta et al., 2024, Yang et al., 2024).

1. Canonical architecture and formalization

The canonical RAG workflow is commonly described as a three-stage pipeline: indexing, retrieval, and generation. In indexing, external data such as text, images, guidelines, research, logs, or system descriptions is split into chunks, encoded into high-dimensional vector representations, and stored in a vector database. In retrieval, the user query is encoded into a vector and matched against the indexed chunks by vector similarity or nearest-neighbor search. In generation, the model consumes both the original query and the retrieved context, and produces an answer conditioned on both (Yang et al., 2024, Naikov et al., 23 Jan 2025).

A standard dense-retrieval formulation scores a query $q$ and document $d$ by a similarity function such as a dot product or cosine similarity. The survey literature states this in the form

$$\text{score}(q, d) = \langle \text{Encoder}_{\text{query}}(q), \text{Encoder}_{\text{doc}}(d) \rangle,$$

while medical and systems-oriented descriptions emphasize cosine similarity between query and chunk embeddings (Gupta et al., 2024, Yang et al., 2024). Generation is often written as evidence aggregation over retrieved documents:

$$P(y \mid q) = \sum_{k=1}^{K} P(y \mid q, d_k)\cdot P(d_k \mid q),$$

which expresses the idea that the final output is informed by both the query and a ranked set of retrieved evidential contexts (Gupta et al., 2024).

This baseline architecture is modular. A typical decomposition includes an external knowledge source, an embedding model, a retriever, a LLM or generator, and a prompt template that determines how the retrieved content is presented to the model. The same abstraction appears in enterprise documentation assistants, developer-support systems, medical QA, and web-scale reasoning benchmarks, although the granularity of the retrieved unit varies from chunks to full topics to structured graph elements (Naikov et al., 23 Jan 2025, Packowski et al., 2024).

2. Retrieval design, search trade-offs, and context efficiency

A central line of recent work studies retrieval not as a fixed front end, but as the main locus of performance, latency, and cost trade-offs. One challenge is the conventional “retrieve-always” assumption. L-RAG introduces entropy-based lazy loading through a two-tier architecture: a query is first processed with a compact document summary, and detailed chunk retrieval is triggered only when mean predictive entropy exceeds a threshold $\tau$. On SQuAD 2.0 with Phi-2, L-RAG at $\tau=0.5$ reaches 78.2% accuracy with 92% retrieval, matching Standard RAG at 77.8% with 100% retrieval; at $\tau=1.0$, retrieval drops by 26% with a modest accuracy trade-off to 76.0%; and latency savings are reported at 80–210 ms per query when retrieval latency exceeds 500 ms (Voloshyn, 10 Jan 2026).

Search itself is also being optimized. “Progressive Searching for Retrieval in RAG” proposes a hierarchy of searches that begins with low-dimensional embeddings over the entire database and progressively refines a shrinking candidate set at higher dimensionality. On a dataset of 1 million English-language documents and 2,470 queries, progressive search reaches the same top-1 retrieval accuracy as full-dimensional brute force at much lower query time; for example, with gte-Qwen2-7B-instruct, 95.02% accuracy is reached in 20.63 s rather than 99.36 s, and with text-embedding-3-large, 94.45% accuracy is reached in 20.50 s rather than 80.31 s (Jeong et al., 7 Feb 2026). A related empirical result from “Toward Optimal Search and Retrieval for RAG” is that lowering ANN search accuracy has minor implications for downstream QA and attributed QA performance while improving retrieval speed and memory efficiency, provided gold-document recall remains adequate (Leto et al., 2024).

Context selection rather than raw recall is another recurring theme. “Better RAG using Relevant Information Gain” introduces the Dartboard algorithm, which optimizes a probabilistic objective over the total information relevant to a query for a set of retrieved passages. The stated claim is that diversity emerges organically rather than through an explicit relevance–diversity trade-off. On the Retrieval Augmented Generation Benchmark, Dartboard Hybrid reports 85.6% Simple QA with NDCG 0.973 and 41% Integrated QA with NDCG 0.609, outperforming KNN and MMR baselines on the reported settings (Pickett et al., 2024). This is consistent with a broader observation that context windows are limited, so redundancy in the retrieved set can degrade end-task utility even when retrieval scores appear strong (Pickett et al., 2024, Leto et al., 2024).

Datastore construction has likewise become a first-order design variable. CompactDS is presented as a diverse, high-quality, web-scale datastore with 639M passages and 380.5B words spanning 12 data sources, including CommonCrawl extracts, textbooks, Wikipedia, Reddit, academic papers, StackExchange, books, Github, PubMed, and math datasets. Its retrieval pipeline combines in-memory approximate nearest neighbor search with on-disk exact reranking, and a minimal RAG pipeline built on it yields relative gains of 10% on MMLU, 33% on MMLU Pro, 14% on GPQA, and 19% on MATH, while maintaining sub-second latency on a single node (Lyu et al., 2 Jul 2025).

3. Evaluation, attribution, and interpretability

Evaluation has become a distinct subfield because standard retrieval labels and end-to-end scores often fail to explain why a RAG system succeeds or fails. The eRAG framework addresses this by evaluating each retrieved document in isolation: each document is passed individually to the LLM with the query, the resulting output is scored against downstream ground truth, and that task score is used as the document’s relevance label. Across multiple datasets, eRAG improves Kendall’s $\tau$ correlation with downstream RAG performance by 0.168 to 0.494 relative to baseline methods, runs 2.5× faster than end-to-end evaluation on average, and uses up to 50× less GPU memory under document-level batching (Salemi et al., 2024). The underlying claim is that retriever evaluation should reflect how useful a document is to the generator, not only whether it appears semantically relevant in isolation.

RAGTrace moves from scalar evaluation to workflow diagnosis. It is described as an interactive evaluation system with multi-level analysis over workflow, retrieval, generation, and cross-component interactions. The framework includes composite performance metrics such as BLEU, ROUGE, and GPTScore, together with granular diagnostics such as Retrieval Failure Value $\mathcal{R}_{fail}$, Generation Anomaly Value $\mathcal{A}_{gen}$, Prompt Fragility, chunk-relink graphs, heatmap visualizations of chunk distributions, force-directed graphs over failure types, and evidence traceability from answer segments back to supporting chunks. A formative study with 12 experts and a user study with 11 participants are reported, with case studies showing improved debugging and iterative optimization of real-world RAG deployments (Cheng et al., 8 Aug 2025).

Enterprise practice introduces a further complication: novel user questions often invalidate benchmark-style evaluation assumptions. In an enterprise documentation setting, Packowski et al. report that common benchmark techniques were not useful for evaluating responses to novel user questions, leading to a “human in the lead” approach. Their evaluation interface checks whether the question is valid, whether the topic exists in the knowledge base, whether search succeeded, and whether the answer is good. They also report that small changes to source content can have large effects: in one benchmark example, adding “until the pre-industrial era” to a sentence about atmospheric $d$0 shifted the system from the incorrect answer “180 ppm” to the correct answer “280 ppm,” and several such tweaks yielded 100% accuracy on a benchmark subset (Packowski et al., 2024).

These results support an increasingly common view that RAG evaluation is not reducible to a single IR metric, a single generation metric, or a single human judgment. Document-level usefulness, source attribution, failure localization, and content design all materially affect reported performance (Salemi et al., 2024, Cheng et al., 8 Aug 2025, Packowski et al., 2024).

4. Structured and graph-based forms of retrieval

A substantial strand of research replaces unstructured passage concatenation with structured evidence units. Tri-RAG transforms raw knowledge into triplets

$d$1

where Condition defines applicability and constraints, Proof contains the rationale, and Conclusion states the claim. Retrieval is performed on the Condition field alone, which is treated as a semantic anchor. The reported retrieval-key ablation gives Hit@1 of 72.3% for Condition-only retrieval versus 73.8% for retrieving over all fields, with the Condition-only variant described as much faster and smaller. On HotpotQA, Tri-RAG reports 67.8% F1 on Hotpot-Dist, retrieval precision of 31.9%, and retrieval F1 of 44.1%, outperforming the cited baselines in that table (Wang et al., 14 Apr 2026).

Graph-based RAG pushes this idea further by encoding inter-document or inter-entity relations explicitly. GFM-RAG introduces what it calls the first graph foundation model applicable to unseen datasets for retrieval without fine-tuning. Its 8M-parameter graph neural network is trained in two stages over 60 knowledge graphs containing over 14M triples and 700k documents, then evaluated on three multi-hop QA datasets and seven domain-specific RAG datasets. The reported claims are state-of-the-art retrieval and QA performance, robustness to graph noise and incompleteness, zero-shot applicability to unseen datasets, and efficiency consistent with a lightweight retriever (Luo et al., 3 Feb 2025).

CogitoRAG adopts a cognitive-memory framing. It first converts segmented passages into “Semantic Gist” memories, then builds a multi-dimensional knowledge graph over entities, memory nodes, facts, and passages. Online retrieval uses a Query Decomposition Module, an Entity Diffusion Module, and the CogniRank reranker, which fuses diffusion-derived scores with semantic similarity. For evidence delivery, the generator receives passage–memory pairs rather than passages alone. On the reported carrier ablation, Triples-as-Carrier yields EM 14.90 and F1 23.06, Passages-as-Carrier yields EM 36.50 and F1 50.23, and Passages+Memory-as-Carrier yields EM 43.20 and F1 53.95. On QA benchmarks, CogitoRAG reports NQ-EM 51.3, MuSiQue-EM 43.2, 2Wiki-EM 69.9, and HotpotQA-EM 60.7, all above the cited prior system HippoRAG2 (Zhou et al., 11 Feb 2026).

Taken together, these systems suggest that retrieval quality depends not only on which items are retrieved, but also on how external knowledge is represented before retrieval. This suggests a shift from passage-level recall toward reasoning-aligned context construction (Wang et al., 14 Apr 2026, Luo et al., 3 Feb 2025, Zhou et al., 11 Feb 2026).

5. Stateful, adaptive, and memory-centered RAG

Another major direction treats retrieval as a stateful process rather than a stateless per-query lookup. RFM-RAG replaces stateless iterative retrieval with a dynamic evidence pool that accumulates curated evidence across rounds, relational triple extraction to identify knowledge gaps, and an R-Feedback model that decides whether the evidence pool is sufficient. On three public QA benchmarks, RFM-RAG is reported to outperform No Retrieval, Vanilla RAG, Probing-RAG, Adaptive-RAG, and DRAGIN. For Mistral-7b, the reported gains are EM 32.1 versus 21.6 for Vanilla RAG and 23.2 for the best non-RFM-RAG baseline, and ACC 50.7 versus 44.8 (Li et al., 25 Aug 2025).

ERM, or Evolving Retrieval Memory, addresses a related problem on the index side. It argues that query expansion and key expansion are theoretically equivalent under standard similarity functions and uses correctness-gated feedback to convert successful query-time adaptations into persistent key updates with zero additional inference-time cost after evolution. On BEIR and BRIGHT across 13 domains, ERM reports +46% average nDCG@1 gain for BM25 and +13–15% average gains for dense retrievers, with 150–280 ms per-query latency at native retrieval speed (Hu et al., 5 Feb 2026). The paper also states that the cumulative cost of expansion grows sublinearly under a Zipfian query distribution, so per-query overhead vanishes asymptotically (Hu et al., 5 Feb 2026).

ARM, or Adaptive RAG Memory, replaces the static vector index with a dynamic memory substrate governed by selective remembrance and decay. Frequently retrieved items are consolidated when access counts cross a threshold, while stale unremembered items are decayed by

$d$2

On a lightweight retrieval benchmark, ARM reports NDCG@5 $d$3, Recall@5 $d$4, and an embedding layer of roughly 22M parameters. In a static-versus-dynamic comparison, Llama 3.1 with static RAG attains 67.2% key-term coverage, while GPT-4o with a dynamic selective retrieval policy reaches 8.2 s average latency with 58.7% coverage (Bursa, 4 Jan 2026).

These memory-centered systems do not abolish the classical RAG pipeline, but they redistribute optimization effort across time. Instead of treating every query as an isolated event, they accumulate evidence, adapt the index, or reshape retention dynamics according to usage and verified downstream benefit (Li et al., 25 Aug 2025, Hu et al., 5 Feb 2026, Bursa, 4 Jan 2026).

6. Domain-specific deployments, component sensitivity, and recurring challenges

RAG has been applied most visibly in domains where factuality, freshness, or provenance are operationally significant. In medicine, RAG is presented as a way to mitigate static and outdated knowledge, bias, lack of transparency, and unreliable outputs by retrieving current guidelines, literature, and multimodal data. The paper emphasizes equity, reliability, and personalization, with examples in drug safety workflows, region- and language-specific advice, clinical decision support, patient education, chronic disease management, and precision medicine. It also states explicit limitations: external databases may themselves be incomplete or biased, benefits are limited for underrepresented groups with sparse data, and chunk-level attribution often still requires manual checking (Yang et al., 2024).

In software engineering, component-wise evidence argues against the common assumption that the generator is the main determinant of RAG quality. The empirical study “Not All RAGs Are Created Equal” reports that retriever-side choices, particularly the retrieval algorithm, often exert a more significant influence on final system performance than the selection of the generator model. BM25 is described as exceptionally robust across code generation, summarization, and repair; on APPS code generation, the reported result is W-Pass@1 38.0 for BM25 versus 15.18 for zero-shot. The same study states that re-ranking often degrades performance, compression is useful mainly under strict context constraints, and the optimal retrieval depth is usually $d$5 (Ke et al., 14 May 2026).

Developer-facing and operational deployments emphasize knowledge flow and organizational specificity. In large-system development, RAG is framed as a knowledge transfer tool over local documents, logs, and system descriptions, with an iterative feedback loop in which failures or hallucinations trigger revisions to prompts or documentation. The same paper notes that output reliability depends heavily on the quality, completeness, and clarity of the knowledge base, and that on-premise deployment can preserve security and privacy for sensitive system knowledge (Naikov et al., 23 Jan 2025). Enterprise customer-support deployments make a closely related argument from content design: modular, model-agnostic pipelines benefit substantially from rewriting documentation, using whole topics rather than sub-chunks when topics are already short and self-contained, and maintaining human-led monitoring and regression testing (Packowski et al., 2024).

Cybersecurity deployments show a different specialization pattern. MoRSE, described as the first specialised AI chatbot for cybersecurity, uses two RAG systems in a cascade: a structured RAG with seven parallel retrievers and an unstructured fallback over raw text, papers, and code. Evaluated on 600 cybersecurity-specific questions, the paper states that relevance and correctness improve by more than 10% relative to GPT-4 and Mixtral 7x8, and reports 84% CVE identification accuracy versus 34% for GPT-4 on the cited CVE setting (Simoni et al., 2024).

Across domains, one recurring misconception is that RAG is a single fixed recipe. The literature instead shows multiple moving targets: corpus construction, retriever class, search regime, evidence structuring, statefulness, prompt design, and evaluation protocol. A second misconception is that stronger generators alone determine success. The component-wise software-engineering results, the enterprise content-design results, and document-level evaluation work all indicate that retrieval quality, corpus quality, and evidence presentation can dominate downstream behavior under practical constraints (Ke et al., 14 May 2026, Packowski et al., 2024, Salemi et al., 2024).