Retrieval-Augmented Policy

• Retrieval-Augmented Policy is a design principle that conditions decision-making on externally retrieved evidence rather than solely relying on learned parameters.
• It applies to diverse domains including reinforcement learning, robotics, multi-agent coordination, and legal-policy support, enhancing performance and adaptability.
• Recent architectures employ iterative, modality-specific retrieval and adaptive fusion to improve efficiency, factual grounding, and overall task effectiveness.

Retrieval-augmented policy denotes a family of architectures in which a decision rule, control policy, value function, or policy-facing generator conditions its behavior on retrieved external evidence rather than relying only on parametric weights or the immediate input. Across recent work, the retrieved evidence may be past trajectories, sub-trajectories, coordinated multi-agent behaviors, policy-memory items, text passages, graph evidence, multilingual search results, recommendation candidates, or policy documents; retrieval may occur once, iteratively during reasoning, or continuously at test time (Goyal et al., 2022, Memmel et al., 2024, Yue et al., 6 Apr 2026, Turja, 26 Feb 2026).

1. Conceptual scope and terminology

The literature uses the term in several closely related senses. In reinforcement learning and control, retrieval augmentation means that the policy or value function depends on retrieved experience, as in the replacement of a purely parametric mapping by a retrieval-conditioned computation such as $Q_\theta(s_t,g_t,a)$, where $g_t$ is selected from a dataset of past trajectories (Goyal et al., 2022). In robotics and imitation learning, retrieval augmentation typically means training or conditioning a task-specific policy on demonstrations retrieved from a large offline corpus, often at deployment time rather than during pretraining alone (Memmel et al., 2024, Kumar et al., 2 Aug 2025, Park et al., 14 Jun 2026). In LLM-based systems, retrieval itself may become the policy object: the model learns when to search, what to retrieve, which retrieval mode to use, or how to weight retrieved evidence against parametric knowledge (Guo et al., 10 Dec 2025, Zhang et al., 3 Mar 2026, Qi et al., 21 Jan 2026).

A separate but related usage appears in governance and policy-support systems. In the AMR decision-support framework, the retrieval-augmented component is explicitly not treated as a standalone chatbot; it is designed to turn surveillance forecasts and curated WHO policy text into grounded, policy-relevant guidance for decision-makers (Turja, 26 Feb 2026). In access-control generation and privacy guardrails, retrieval is used to ground structured policy outputs in organization-specific entities or to enforce privacy policy over generated answers (Jayasundara et al., 2024, Zafar et al., 16 May 2026). In legal-policy applications, HyPA-RAG treats retrieval as an adaptive, policy-aware process over dense, sparse, and knowledge-graph evidence rather than as a fixed preprocessing step (Kalra et al., 2024).

This breadth suggests that retrieval-augmented policy is less a single algorithm than a recurring design principle: the decision process is coupled to a non-parametric memory whose contents can be selected, fused, and exploited at inference time.

2. Retrieval substrates and granularity

A central design choice is the retrieval unit. In STRAP, the unit is the sub-trajectory rather than the full trajectory. A trajectory $T^i$ is decomposed into consecutive sub-trajectories $$\mathbf{t}^i_{a:b} = (s^i_a, s^i_{a+1}, \dots, s^i_b)$$, and retrieval is performed with subsequence DTW so that reusable fragments such as “open drawer” or “close drawer” can transfer across otherwise different long-horizon tasks (Memmel et al., 2024). COLLAGE adopts the same broad sub-trajectory view, but retrieves multiple modality-specific subsets—visual, motion, shape, and language—and then weights them adaptively (Kumar et al., 2 Aug 2025). Multi-Agent Behavior Retrieval stores learned coordination skill vectors and retrieves demonstrations by FastDTW with cosine similarity over sequences of those skill vectors, rather than by raw state matching (Kuroki et al., 2023).

Other systems retrieve richer structured items. RAEA stores policy-memory items $m=\{i,o,a,p\}$, consisting of instruction, observation, action, and proprioception, inside an external policy memory bank built from large-scale robotic data across multiple embodiments (Zhu et al., 2024). ReCAP retrieves a pool-side state-action chunk at every control step and treats it as a coarse prior for a residual target-side action prediction, $$\hat a^{\text{query}}_{t:t+H}=a^{\text{pool}}_{t':t'+H}+\Delta a_{t:t+H}$$ (Park et al., 14 Jun 2026). In conversational recommendation, the retrieval unit is a candidate set $C_t$ over a unified movie corpus rather than an individual passage; the retriever policy is optimized for the downstream ranking behavior of the frozen LLM (Yue et al., 6 Apr 2026).

Text-centric systems vary similarly in substrate. The AMR framework uses six manually curated WHO policy document excerpts, chunked and embedded with all-MiniLM-L6-v2 in ChromaDB, and retrieves the top 3 semantically nearest chunks by cosine similarity (Turja, 26 Feb 2026). RouteRAG retrieves passages, graph evidence, or a hybrid fusion of both, with the retrieval mode emitted as part of the generation policy itself (Guo et al., 10 Dec 2025). LcRL organizes retrieval over language-specific retrievers in a multilingual collection and uses a local-first, global-supplement, then high-resource anchoring scheme across turns (Qi et al., 21 Jan 2026). RAGent retrieves top-$k$ organization-specific subjects, actions, resources, purposes, and conditions from separate vector databases to ground access-control policy generation (Jayasundara et al., 2024).

The choice of granularity is rarely incidental. The papers repeatedly argue that full-trajectory, fixed-top-$K$, or single-shot retrieval can underuse structure already present in the memory bank. This suggests that retrieval-augmented policy is often most effective when the retrieval unit matches the transferable structure of the task: subskills for manipulation, coordination patterns for multi-agent control, candidate sets for recommendation, and clause-level or chunk-level evidence for policy interpretation.

3. Architectural patterns

One major pattern is retrieve, then train a specialist policy. STRAP segments only the target demonstrations, embeds observations with a pretrained vision foundation model such as DINOv2 or CLIP, computes subsequence DTW against the prior corpus, retrieves the top-$K$ sub-trajectories, forms $g_t$0, and trains a language-conditioned behavior-cloning policy on the augmented data (Memmel et al., 2024). COLLAGE retains the same overall few-shot imitation setting but interposes a modality-selection stage: it trains a lightweight reference BC policy on each modality-specific retrieved subset, scores each subset by log-likelihood on the target demonstrations, converts those scores into weights by a softmax, and uses the resulting weights for importance sampling during final policy training (Kumar et al., 2 Aug 2025). Multi-Agent Behavior Retrieval follows a retrieve-and-learn structure as well, with a Transformer-based skill encoder, a Multi-Agent Coordination Skill Database, top-$g_t$1 retrieval via FastDTW, and downstream centralized imitation on $g_t$2 (Kuroki et al., 2023).

A second pattern is retrieve and condition a frozen or jointly trained policy online. Retrieval-Augmented Reinforcement Learning separates a retrieval process from the agent process; the retrieval module queries a dataset of trajectories, maintains its own slot state, and passes a retrieved representation $g_t$3 back to the agent, which then computes $g_t$4 (Goyal et al., 2022). RAEA retrieves top-$g_t$5 policy-memory items with a bi-encoder dense retriever using ImageBind, tokenizes retrieved instruction-observation-action-proprioception content, and fuses it into the main branch with Transformer cross-attention (Zhu et al., 2024). ReCAP freezes a retrieval-conditioned world-action model after paired query/pool training; at deployment, new tasks are added by appending pool-side demonstrations to the retrieval pool, and the policy re-retrieves a pool chunk at every control step without per-task fine-tuning (Park et al., 14 Jun 2026).

A third pattern is interleaved reasoning and retrieval as a sequential policy. RouteRAG allows the model to think, issue search, select passage, graph, or hybrid retrieval, observe the retrieved information, continue reasoning, and then answer, all within a bounded multi-turn loop (Guo et al., 10 Dec 2025). LcRL uses >, <search>, and <information> phases and couples equivalent multilingual queries into a shared GRPO group so that retrieval behavior is optimized across languages rather than per query in isolation (Qi et al., 21 Jan 2026). PORAG and ATLAS separate “how to use retrieved evidence” from “when to retrieve and what to retrieve”: ATLAS monitors token probabilities, attention shifts, and semantic filters through MLAG and LRP, while PORAG applies GRPO-style policy optimization to retrieval-grounded generation (Srinivas et al., 2 Apr 2025).

A fourth pattern is policy interpretation rather than direct control. In the AMR framework, the RAG layer appends a structured forecast-context block—summarizing XGBoost feature importances, regional MAE values, and model-comparison results—to the retrieved WHO policy chunks, then prompts a locally deployed Phi-3 Mini model through Ollama. The system prompt explicitly prohibited fabricated citations and restricted references to retrieved chunk labels only, making the RAG layer a “policy interpretation” layer that translates epidemiological forecasts into grounded policy guidance (Turja, 26 Feb 2026).

4. Optimization and learning objectives

Behavior cloning remains a common optimization rule in robotics variants. STRAP trains a transformer-based language-conditioned visuomotor policy $g_t$6 on the union of target and retrieved data with a negative log-likelihood objective plus $g_t$7 regularization (Memmel et al., 2024). COLLAGE uses a similar language-conditioned visuomotor transformer with a GMM head and trains it by behavior cloning, again with $g_t$8 regularization, while the retrieval weights are learned indirectly through reference-policy likelihoods on target demonstrations (Kumar et al., 2 Aug 2025). RAEA also uses behavior cloning, but with mean squared loss over continuous action outputs after cross-attention fusion of retrieved policy tokens (Zhu et al., 2024).

In RL-oriented systems, the retrieval decision itself is frequently cast as a policy. In RAR, the retriever $g_t$9 selects a candidate set $T^i$0 from user history, the frozen LLM reranks that set using conversational context, and the retriever parameters are updated to maximize downstream reward, most notably NDCG, rather than standalone retrieval relevance. The paper gives both a Plackett–Luce policy over candidate sets and DPO- and GRPO-based training objectives for that retriever policy (Yue et al., 6 Apr 2026). Offline prior-authorization retrieval goes further by formulating chunk selection and stopping as a finite-horizon MDP with horizon $T^i$1, action set $T^i$2 for $T^i$3, per-step cost $T^i$4, and terminal reward $T^i$5 depending on the oracle decision; CQL, IQL, and transition-level DPO are then used to learn retrieval policies from logged trajectories (Sharifullin et al., 6 Apr 2026).

Several papers optimize how retrieved evidence is used rather than assuming retrieval should always dominate. Knowledgeable-r1 defines three policy distributions—$T^i$6 for parametric reasoning on the original prompt, $T^i$7 for contextual reasoning on the retrieval-augmented prompt, and $T^i$8 for parametric reasoning under retrieval input—and jointly optimizes them to prevent over-reliance on misleading or excessive retrieval (Lin et al., 5 Jun 2025). REX-RAG studies the opposite failure mode, namely dead ends during policy-driven trajectory sampling in search-augmented reasoning. It introduces mixed sampling with exploratory prompts and corrects the resulting distribution shift with multiple importance sampling inside a corrected GRPO objective (Jiang et al., 11 Aug 2025). RAPO injects retrieved off-policy step traces into live agent rollouts, computes a retrieval reward from entropy reduction at high-entropy states, and applies retrieval-aware policy optimization with importance shaping to stabilize learning (Zhang et al., 3 Mar 2026).

Multilingual and hybrid-evidence settings add further structure. LcRL replaces single-query GRPO with language-coupled GRPO, where one completion is sampled per language-equivalent query in a group, and uses a reward based on character 3-gram recall plus an anti-consistency penalty for clusters of similarly wrong answers (Qi et al., 21 Jan 2026). RouteRAG uses a two-stage reward schedule: stage 1 optimizes exact-match correctness, and stage 2 adds a retrieval-efficiency term $T^i$9 when the answer is correct, so that the model learns both whether and how expensively to retrieve graph and text evidence (Guo et al., 10 Dec 2025). PORAG similarly treats RAG as a policy-optimization problem, with dual learned rewards for fidelity and response quality and a GRPO-RAG objective over group-relative advantages (Srinivas et al., 2 Apr 2025).

5. Empirical regimes and reported gains

In few-shot robotics, retrieval augmentation is repeatedly reported to outperform both target-only training and monolithic multi-task learning. On LIBERO-10, STRAP with DINOv2 and best $$\mathbf{t}^i_{a:b} = (s^i_a, s^i_{a+1}, \dots, s^i_b)$$0 achieves 58.1% average success, outperforming BR by about +24.7% and FR by about +25.0%; the paper also reports that sub-trajectory retrieval improves performance by about +4.1% over full-trajectory retrieval on average (Memmel et al., 2024). COLLAGE reports 50.5 average success on LIBERO-10 versus 45.6 for STRAP, and a 45.5% average real-world success rate versus 28.9% for the visual baseline and 6.7% for BC; the paper summarizes this as +5.1% improvement over the best single-modality retrieval baseline in simulation and 16.6% over the best baseline in the real world (Kumar et al., 2 Aug 2025). RAEA reports that it outperforms baselines across Franka Kitchen, Meta-World, and Maniskill-2, with especially strong gains in low-data regimes such as 10 demonstrations (Zhu et al., 2024).

Test-time retrieval instead of retraining is a distinct empirical regime. On PushT, ReCAP improves unseen-angle performance from 6.0% without retrieval to 34.9% with the full retrieval pool, and the paper states that success improves monotonically as the retrieval pool grows at test time without retraining (Park et al., 14 Jun 2026). On RoboTwin 2.0, ReCAP reports 31.5% average success on unseen tasks versus 26.0% for the strongest baseline, and pool progression raises unseen-task success from 9.0% to 31.5% while the policy remains frozen (Park et al., 14 Jun 2026). In multi-agent manipulation, retrieval over coordination skills yields an overall success rate of 56.9 ± 1.7% versus 52.1 ± 1.7% for agent-wise trajectory matching and 30.6 ± 1.6% for few-shot imitation learning (Kuroki et al., 2023).

Reinforcement-learning variants show similar effects when retrieval is integrated into policy evaluation or action selection. Retrieval-Augmented R2D2 improves mean human-normalized score by $$\mathbf{t}^i_{a:b} = (s^i_a, s^i_{a+1}, \dots, s^i_b)$$1 over 2B environment steps on Atari (Goyal et al., 2022). In BabyAI, the reported success rates rise from 32% and 45% for the RDQN baseline at 50K and 200K data to 55% and 74% for retrieval-augmented RDQN with a multi-task retrieval buffer (Goyal et al., 2022). In CausalWorld, behavior cloning improves from 61% ± 10% to 82% ± 5% under multi-task retrieval (Goyal et al., 2022).

LLM-oriented retrieval policies report gains both in answer quality and in factuality. RAR reports an average improvement of 7.60% over the best baseline results, and the paper states that hallucination rates fall to under 1% unmatched titles across datasets because the LLM is grounded in factual movie metadata (Yue et al., 6 Apr 2026). Knowledgeable-r1 reports 17.07% gain over RAG prompting in counterfactual scenarios on ConflictQA and an average improvement of +8.39% over RAG prompting (Lin et al., 5 Jun 2025). REX-RAG reports average performance gains of 5.1% on Qwen2.5-3B and 3.6% on Qwen2.5-7B over strong baselines across seven QA benchmarks (Jiang et al., 11 Aug 2025). RouteRAG reports average EM 47.0, F1 55.7 for the 3B backbone and EM 51.9, F1 60.6 for the 7B backbone across five QA benchmarks (Guo et al., 10 Dec 2025). RAPO reports a +5.0% average gain on fourteen datasets across three agentic reasoning tasks, together with 1.2x faster training efficiency (Zhang et al., 3 Mar 2026).

Adaptive retrieval policies also show strong accuracy–efficiency trade-offs in structured decision support. In prior authorization, CQL reaches 92.0% accuracy but uses 20.0 retrieval steps every episode; IQL attains 62.5% accuracy with 3.4 steps and is the only policy with positive episodic return; transition-level DPO matches CQL at 92.0% accuracy while reducing retrieval steps to 10.6 (Sharifullin et al., 6 Apr 2026). In the AMR framework, the RAG system is evaluated on five predefined policy questions, and the authors report accurate retrieval of relevant WHO policy excerpts, correct attribution of sources without fabricated references, and coherent integration of forecast findings into policy recommendations, although the evidence is qualitative rather than benchmark-based (Turja, 26 Feb 2026). In access-control generation, RAGent reports 77.9% F1 for policy generation and 80.6% after verification-refinement, alongside 87.9% average F1 for NLACP identification (Jayasundara et al., 2024).

6. Reliability, efficiency, and unresolved issues

A persistent theme is that retrieval is useful only when it is selective, grounded, and appropriately weighted. STRAP notes that retrieval data can help or hurt depending on task overlap and identifies $$\mathbf{t}^i_{a:b} = (s^i_a, s^i_{a+1}, \dots, s^i_b)$$2 as an important control knob (Memmel et al., 2024). COLLAGE is motivated by the claim that single-feature retrieval is brittle because similar scene layout, motion, or language can retrieve demonstrations with the wrong goal, which is why it replaces uniform fusion with adaptive late fusion (Kumar et al., 2 Aug 2025). In prior authorization, the contrast between exhaustive CQL and selective DPO or IQL shows that retrieval depth should be treated as a learned decision rather than as a fixed hyperparameter (Sharifullin et al., 6 Apr 2026). This suggests that “more retrieval” and “better retrieval” are analytically distinct.

Grounding and hallucination control are equally central. The AMR policy-support system treats prompt discipline as a primary hallucination-reduction mechanism: the prompt explicitly prohibited fabricated citations and allowed source attribution only from retrieved chunk labels (Turja, 26 Feb 2026). RAR attributes reduced hallucinations to grounding the LLM in structured movie metadata (Yue et al., 6 Apr 2026). HyPA-RAG emphasizes faithfulness, contextual precision, and correctness for legal-policy questions, and reports that adaptive retrieval typically improves over fixed-$$\mathbf{t}^i_{a:b} = (s^i_a, s^i_{a+1}, \dots, s^i_b)$$3 settings, although knowledge-graph augmentation does not improve every metric uniformly (Kalra et al., 2024). RAGent adds an explicit verifier and iterative refinement loop because one-shot retrieval-augmented generation can still omit ACRs or mis-handle decisions, purposes, and conditions (Jayasundara et al., 2024).

Efficiency has become part of the policy objective rather than an afterthought. RouteRAG’s stage-2 reward penalizes unnecessary retrieval time, and the reported retrieval turns fall from 2.70 to 2.25 for the 7B model when efficiency reward is enabled (Guo et al., 10 Dec 2025). PORAG and ATLAS move efficiency into test-time control: ATLAS triggers retrieval conditionally through MLAG, LRP constructs targeted queries from relevant tokens, and CRITIC compresses KV caches by token importance. The paper reports latency reduction from 68.27 sec to 34.19 sec and throughput improvement from 120 to 242 tokens/sec with CRITIC (Srinivas et al., 2 Apr 2025). In ReCAP, replacing per-task fine-tuning with indexing new pool-side demonstrations addresses both target-side data collection and per-task compute cost, with the paper noting that teleoperated robot demonstrations are roughly 18× slower to collect than human-hand demonstrations and that Cosmos Policy fine-tuning takes about 24 GPU-hours per task (Park et al., 14 Jun 2026).

The topic also carries new attack and governance surfaces. ImageAuditor shows that image-based retrieval-augmented generation can be audited by a retrieval policy of its own: Reward-Guided Policy Optimization searches for text suffixes that cause the retriever to surface a target image, and the resulting attack exceeds 80% AUROC with only four queries per audited image (Zhang et al., 2 Jun 2026). Privacy Policy Enforcement for data-sensitive RAG argues that privacy violations often arise from quasi-identifier clusters rather than explicit PII and proposes a dual one-class density detector with an abstain region; the reported detector achieves borderline AUROC of 0.93+, reduces false positives by 44–55 percentage points, and operates at millisecond latency (Zafar et al., 16 May 2026).

Several papers explicitly note missing evaluation components. The AMR framework states that formal automated evaluation, for example with RAGAS, would be valuable, because current evidence for policy relevance and citation faithfulness is qualitative (Turja, 26 Feb 2026). ReCAP identifies under-anchoring and over-anchoring to retrieval as distinct failure modes (Park et al., 14 Jun 2026). Knowledgeable-r1 frames over-trust in retrieved passages as a policy imbalance, while REX-RAG frames dead ends as a sampling pathology of RL-trained search agents (Lin et al., 5 Jun 2025, Jiang et al., 11 Aug 2025). A plausible implication is that retrieval-augmented policy should be analyzed not only for retrieval recall or downstream accuracy, but also for calibration of reliance: when to trust retrieved evidence, when to resist it, and when to stop retrieving altogether.