Zero-RAG: Adaptive Retrieval in Generation

Updated 4 July 2026

Zero-RAG is a research program that reinterprets retrieval by minimizing or conditionally activating it in language generation systems.
It explores strategies like retrieval-default-off, zero-shot/zero-finetuning setups, and corpus pruning to balance internal model knowledge with external evidence.
Empirical results demonstrate that selective retrieval can boost efficiency and accuracy, though misleading evidence may sometimes degrade performance.

Zero-RAG denotes a cluster of retrieval-augmented generation regimes in which retrieval is minimized, conditioned, or reinterpreted rather than treated as an always-on primitive. Recent work uses the term in several distinct but related ways: as a retrieval-default-off policy that approximates Never-RAG and retrieves only under uncertainty (Wang et al., 12 Nov 2025), as a zero-shot or zero-finetuning RAG setting in which retrievers and generators are composed without target-domain supervision (Sawarkar et al., 23 May 2025), as a diagnostic regime in which retrieval can perform worse than a zero-shot baseline when evidence is misleading (Zeng et al., 22 Feb 2025), and as a concrete framework for pruning “redundant knowledge” from the external corpus (Luo et al., 1 Nov 2025). This suggests that Zero-RAG is best understood as a research program about when external retrieval should be absent, selective, or structurally transformed, rather than as a single fixed architecture.

1. Terminological scope and early lineage

An early antecedent appeared in zero-shot slot filling with DPR and RAG, where a retrieval-augmented generator handled T-REx and zsRE without a bespoke information-extraction pipeline and reached the top-1 position on the KILT leaderboard on both tasks (Glass et al., 2021). Later work broadened the scope from slot filling to open-domain QA, enterprise search, biomedical QA, multimodal VQA, and time-series forecasting.

The literature uses “Zero-RAG” in more than one sense. In some papers it denotes no retrieval by default, with retrieval activated only when the model appears uncertain. In others it denotes zero-shot / zero-finetuning RAG, where the retriever and generator remain off-the-shelf and adaptation occurs through prompting, indexing, metadata generation, or corpus engineering. A third usage treats Zero-RAG as a failure boundary: retrieval causes accuracy to fall below the same model’s zero-context baseline under misleading evidence. A fourth usage is the named framework “Zero-RAG: Towards Retrieval-Augmented Generation with Zero Redundant Knowledge,” which targets corpus pruning, routing, and noise robustness (Wang et al., 12 Nov 2025).

Usage of “Zero-RAG”	Core idea	Representative papers
Retrieval-default-off	Retrieve only when uncertainty is high	(Wang et al., 12 Nov 2025)
Zero-shot / zero-finetuning RAG	No task-specific retriever or LLM tuning	(Sawarkar et al., 23 May 2025, Liu et al., 2024)
Worse-than-zero-shot regime	Retrieval harms performance under misleading evidence	(Zeng et al., 22 Feb 2025)
Zero redundant knowledge	Prune mastered corpus content and route queries	(Luo et al., 1 Nov 2025)

2. Retrieval-default-off inference and adaptive gating

The clearest retrieval-default-off formulation is "TARG: Training-Free Adaptive Retrieval Gating for Efficient RAG" (Wang et al., 12 Nov 2025). TARG starts from the contrast between Always-RAG, which retrieves for every query and may reduce accuracy through noisy or off-topic passages, and Never-RAG, which never retrieves and therefore relies only on parametric knowledge. TARG operationalizes what the paper calls “conditional / Zero-RAG”: retrieval is default-off and is enabled only when a short, no-context draft indicates uncertainty.

For a query $q$ , TARG first decodes a short prefix from the base model without retrieval, usually $k=20$ tokens, then computes an uncertainty score $U(q)$ from prefix logits and retrieves iff

$\text{retrieve}(q) \iff U(q) > \tau.$

The paper studies three signals. Mean token entropy averages

$H_t = -\sum_j \pi_{t,j}\log \pi_{t,j}$

over the draft prefix. The margin signal uses the top-1/top-2 logit gap $g_t = \ell_{t,(1)}-\ell_{t,(2)}$ and the monotone link $\phi(z)=\exp(-z/\beta)$ , giving

$U_{\text{mar}}(k;\beta)=\frac{1}{k}\sum_{t=1}^k \exp(-g_t/\beta).$

Small- $N$ variance measures disagreement across a few stochastic prefixes and aggregates

$U_{\text{var}}(k,N)=\frac{1}{k}\sum_{t=1}^k d_t.$

No main experiments use multi-stage rechecking; the gate is single-shot (Wang et al., 12 Nov 2025).

Empirically, TARG uses only a short retrieval-free draft of 10–30 tokens, achieves retrieval rates as low as $k=20$ 0– $k=20$ 1 on Llama-3.1-8B for many operating points, and keeps $k=20$ 2 latency extremely close to Never-RAG; one cited operating point on NQ-Open with Llama-3.1-8B adds only $k=20$ 3 s per query. On TriviaQA with Llama-3.1-8B, the margin gate at $k=20$ 4 reaches $k=20$ 5 EM/F1 with retrieval rate $k=20$ 6 and $k=20$ 7 s overhead, versus Never-RAG at $k=20$ 8. A central empirical finding is that the margin signal is a robust default, because entropy “compresses as backbones sharpen,” while small- $k=20$ 9 variance is a more conservative, budget-first alternative (Wang et al., 12 Nov 2025).

3. Zero-shot and zero-finetuning architectural patterns

A second major interpretation of Zero-RAG is zero-finetuning domain adaptation. "MetaGen Blended RAG" explicitly assumes no fine-tuning of the LLM, no fine-tuning of the retriever, and no labeled data for model adaptation; instead it performs index-time and query-time metadata engineering plus hybrid retrieval (Sawarkar et al., 23 May 2025). Its pipeline uses KeyBERT, YAKE, spaCy NER, and optionally LLM-generated topics, phrases, synonyms, and acronyms, followed by forward selection over metadata streams. Retrieval blends lexical and dense scores, described as a weighted fusion of normalized BM25 and cosine similarity. On PubMedQA, the full configuration—hybrid retrieval with boosted fields and existing plus enriched metadata—reaches 82.1% top-1 retrieval accuracy and 77.9% RAG accuracy, while on NQ top-5 retrieval accuracy improves from 49.99% without metadata to 60.48% with existing plus enriched metadata (Sawarkar et al., 23 May 2025).

"RAG-Instruct" addresses a different bottleneck: the lack of a general RAG instruction dataset. It synthesizes 40K Wikipedia-grounded instruction examples using five paradigms that explicitly cover retrieval usefulness and document cardinality: $U(q)$ 0 useless doc, $U(q)$ 1 single-doc support, $U(q)$ 2 multi-doc support, $U(q)$ 3 single-doc answer, and $U(q)$ 4 multi-doc answer (Liu et al., 2024). This data-centric view is important because it trains models to operate not only when context directly contains the answer, but also when retrieval is partially helpful or useless. The paper’s ablations show that removing the multi-document paradigms $U(q)$ 5 and $U(q)$ 6 most strongly hurts HotpotQA-style multi-hop behavior, while removing $U(q)$ 7 and $U(q)$ 8 weakens robustness in “mid-helpful” and “helpless” settings (Liu et al., 2024).

In enterprise retrieval, "Question-Based Retrieval using Atomic Units for Enterprise RAG" proposes a zero-shot corpus-side transformation rather than retriever training (Raina et al., 2024). Each chunk is decomposed into atomic statements, and synthetic closed-answer questions are generated over those atoms. Dense retrieval is then performed against synthetic questions instead of raw chunks. On SQuAD with all-mpnet-base-v2, chunk-level retrieval yields $U(q)$ 9 R@1, atom-unstructured text retrieval yields $\text{retrieve}(q) \iff U(q) > \tau.$ 0, and atom-unstructured question retrieval yields $\text{retrieve}(q) \iff U(q) > \tau.$ 1; on BiPaR the same progression is $\text{retrieve}(q) \iff U(q) > \tau.$ 2. The paper’s claim is narrow but consequential: higher recall at retrieval enables higher downstream enterprise RAG performance, and this gain is obtained without supervised retriever adaptation (Raina et al., 2024).

A closely related zero-shot IR lineage appears in "GAR-meets-RAG Paradigm for Zero-Shot Information Retrieval," which proposes an iterative recurrence in which GAR improves retrieval and RAG improves rewrite, with rewrite–retrieval stages improving recall and a final re-ranking stage improving precision. On BEIR and TREC-DL, it establishes a new state of the art on six of eight BEIR datasets, with up to 17% relative gains over the previous best (Arora et al., 2023).

4. Zero-RAG as a failure boundary under misleading retrieval

In "RAGuard," Zero-RAG is not a design target but a pathology: retrieval can make a system worse than zero-shot (Zeng et al., 22 Feb 2025). The benchmark is built for political fact-checking and labels retrieved evidence as supporting, misleading, or irrelevant. It contains 2,648 political claims from PolitiFact and a Reddit-derived retrieval corpus of 16,331 documents, of which 16.4% are supporting, 11.1% misleading, and 72.5% irrelevant. The benchmark’s central novelty is that misleading evidence is naturally occurring rather than synthetically fabricated.

The empirical finding is categorical: when exposed to misleading retrievals, all tested LLM-powered RAG systems perform worse than their zero-shot baselines. In standard RAG with semantic retrieval, every evaluated model loses accuracy relative to zero-context prediction at both top-1 and top-5 retrieval. Under oracle misleading-only retrieval, the collapse is much larger. Gemini 1.5 Flash falls from 61.06% in zero-context to 30.57% in misleading-only conditions; Claude 3.5 Sonnet falls from 74.51% to 37.05%; Mistral 7B falls from 63.97% to 28.22% (Zeng et al., 22 Feb 2025). The paper also shows that increasing retrieval depth raises the chance of at least one misleading document being retrieved: misleading retrieval recall rises from 21.3% at $\text{retrieve}(q) \iff U(q) > \tau.$ 3 to 44.8% at $\text{retrieve}(q) \iff U(q) > \tau.$ 4.

This usage directly challenges the common assumption that retrieval is monotonically beneficial. In RAGuard’s terminology, Zero-RAG refers to the regime in which turning retrieval on pushes performance below the same model’s zero-context baseline. The broader implication is that selective retrieval, credibility-aware retrieval, or internal–external conflict resolution is not optional in misinformation-rich settings (Zeng et al., 22 Feb 2025).

5. Zero redundant knowledge: corpus pruning, routing, and noise tolerance

"Zero-RAG: Towards Retrieval-Augmented Generation with Zero Redundant Knowledge" formalizes a different problem: as LLM internal knowledge expands, external corpora such as Wikipedia contain substantial overlap, and redundant retrieval can both waste compute and hurt answer quality (Luo et al., 1 Nov 2025). The framework has three components: Mastery-Score, Query Router, and Noise-Tolerant Tuning.

Mastery-Score is defined at the sentence level. For a sentence $\text{retrieve}(q) \iff U(q) > \tau.$ 5, an LLM first generates $\text{retrieve}(q) \iff U(q) > \tau.$ 6 QA pairs $\text{retrieve}(q) \iff U(q) > \tau.$ 7, the target model answers them, and the score is

$\text{retrieve}(q) \iff U(q) > \tau.$ 8

High $\text{retrieve}(q) \iff U(q) > \tau.$ 9 means the model already “masters” the knowledge in the sentence, so that sentence is a candidate for pruning. Because evaluating all Wikipedia sentences directly is infeasible, the paper trains a 7B regression model to predict Mastery-Score from sentence text, then prunes the high-mastery portion of the corpus. The Query Router is a binary classifier that decides whether a question is “mastered” and therefore should bypass retrieval. Noise-Tolerant Tuning trains the LLM on three input types—query only, query with relevant documents, and query with noisy documents—so that it can use internal knowledge when retrieval is unhelpful and ignore irrelevant evidence when retrieval is noisy (Luo et al., 1 Nov 2025).

The reported results are explicitly efficiency-oriented. Zero-RAG prunes the Wikipedia corpus by 30% and accelerates retrieval by 22% without compromising RAG performance. Average retrieval latency over HotpotQA, EntityQuestions, and TriviaQA drops from 12.28 s to 9.66 s; on TriviaQA the reported total processing time drops from 1655.54 s to 441.80 s. The motivating exploratory analysis is equally important: for questions the LLM answers correctly without retrieval, adding the corresponding redundant Wikipedia passage lowers accuracy by approximately 20 EM points. In this formulation, Zero-RAG is not “never retrieve”; it is retrieve against a smaller, less redundant corpus, skip retrieval for mastered queries, and train the generator to tolerate noise when retrieval still occurs (Luo et al., 1 Nov 2025).

The Zero-RAG logic has already expanded beyond text QA. "TS-RAG" transfers it to time-series forecasting by freezing a time-series foundation model and a retrieval encoder, retrieving semantically similar context–horizon pairs from a knowledge base, and mixing retrieved futures with the backbone representation through a retrieval-aware mixture-of-experts module (Ning et al., 6 Mar 2025). The abstract reports state-of-the-art zero-shot forecasting, outperforming existing TSFMs by up to 6.84% across diverse domains; the detailed results report an average reduction of 3.54% in MSE and 1.43% in MAE versus Chronos-Bolt. The underlying pattern is recognizably Zero-RAG: adaptation occurs through retrieval and lightweight mixing rather than task-specific fine-tuning (Ning et al., 6 Mar 2025).

"Med-GRIM" extends the same principle to medical VQA through prompt-embedded multimodal Graph-RAG (Madavan et al., 20 Jul 2025). It uses the BIND encoder to place images and text in a joint space, retrieves over DermaGraph—a Neo4j graph of 50 dermatological conditions, each with symptoms, treatments, preventive strategies, and 10–15 representative images—and then injects retrieved knowledge into prompts for Phi-3.8B and Mistral-7B. The hybrid similarity used in graph retrieval is

$H_t = -\sum_j \pi_{t,j}\log \pi_{t,j}$ 0

and the best reported system, Med-Grim(Graph-RAG), reaches 83.33 accuracy with 0.81 Semantic-BERT on DermaGraph, outperforming Med-Grim(RAG) and several medical VLM baselines (Madavan et al., 20 Jul 2025).

Evaluation has also acquired a zero-shot branch. "CCRS" defines five LLM-judge metrics—Contextual Coherence, Question Relevance, Information Density, Answer Correctness, and Information Recall—using a single pretrained LLM as a zero-shot, end-to-end evaluator on BioASQ (Muhamed, 25 Jun 2025). The paper reports that CCRS offers comparable or superior discriminative power to RAGChecker for key aspects such as recall and faithfulness, while being approximately 5× faster on the reported hardware setup. The evaluation trend is conceptually aligned with Zero-RAG: no specialized judge fine-tuning, minimal pipeline staging, and direct assessment of retrieval-grounded responses (Muhamed, 25 Jun 2025).

A final constraint comes from high-stakes domains. In "Optimizing Medical Question-Answering Systems," the zero-shot closed-book baseline for LLaMA 2 on PubMedQA is 55.4%, the fine-tuned-only baseline is 64.3%, and the fine-tuned plus RAG system reaches 71.8%; in a manual analysis of 100 QA pairs, factual error rate drops from 35% to 14% when RAG is added (Hassan et al., 5 Dec 2025). This does not refute Zero-RAG; rather, it indicates that retrieval-default-off or retrieval-minimized regimes are domain-dependent. In safety-critical settings, zero-shot or retrieval-free behavior may remain an efficiency baseline or a routing target, but not a sufficient end state (Hassan et al., 5 Dec 2025).