---
title: 'Embodied-RAG: Hierarchical Memory for Agents'
url: https://www.emergentmind.com/topics/embodied-rag
type: topic
---

# Embodied-RAG: Hierarchical Memory for Agents

Embodied-RAG denotes retrieval-augmented generation adapted to embodied agents whose knowledge is accumulated through perception, navigation, and interaction rather than through a static text corpus alone. In the specific framework "Embodied-RAG: General Non-parametric Embodied Memory for Retrieval and Generation," the core idea is a non-parametric memory that autonomously constructs hierarchical knowledge for both navigation and language generation, organized as a semantic forest that spans a full range of spatial and semantic resolutions across diverse environments and query types [2409.18313]. In broader usage across recent literature, the term also covers related systems that retrieve from 3D scene graphs, episodic interaction logs, affordance-aware visual memories, failure databases, or multimodal knowledge bases in order to support robotic planning, embodied question answering, failure identification, and situated dialogue [2410.23968].

## 1. Conceptual foundations

Embodied-RAG emerges from a mismatch between conventional RAG assumptions and embodied operation. Traditional RAG in NLP assumes a static corpus of documents, independent samples, and purely textual data, whereas embodied memory is a stream of multimodal, highly correlated observations whose raw scale exceeds typical LLM context windows. The original Embodied-RAG framework identifies three central mismatches: multimodal versus text only, correlated structure across nearby views, and varying granularity of queries ranging from explicit object search to ambiance and global environment description [2409.18313].

This motivates a shift from document retrieval toward retrieval over embodied abstractions. In Embodied-RAG, raw visual observations are converted into language descriptions and organized hierarchically so that retrieval can match the abstraction level of the query. Explicit queries can be handled at leaf level, implicit queries can use intermediate abstractions, and global queries can use root-level summaries. This coupling of retrieval resolution to query abstraction is one of the defining features of the framework [2409.18313].

A parallel formulation appears in "EmbodiedRAG: Dynamic 3D Scene Graph Retrieval for Efficient and Scalable Robot Task Planning," which treats robotic planning as retrieval over a semantic 3D scene graph rather than direct ingestion of the full environment representation. That work defines an extraction function $f : (\mathcal{G}_t, x_t) \to \mathcal{G}'_t$ that selects a task-relevant subgraph from the current scene graph and planning history, with the explicit objective of reducing input size while preserving task-relevant information [2410.23968].

A related but distinct perspective is given by P-RAG, which treats embodied planning as sequential decision-making with a dynamic database of past interactions. Its contribution is not a static retrieval layer but an iterative retrieval-and-update loop in which the database grows after each round of interaction, allowing retrieval to reflect accumulated experience rather than only a prebuilt corpus [2409.11279].

## 2. Memory structures and retrieval units

The most explicit hierarchical memory design is the semantic forest of Embodied-RAG. The topological map stores allocentric poses, image paths, and object-level captions, with topological nodes serving as leaves. Agglomerative clustering groups nodes by Euclidean distance in $(x,y,z)$, and each non-leaf node receives an abstraction generated by prompting an LLM to summarize its children’s captions. The result is a forest in which leaf nodes correspond to detailed object-level captions, intermediate nodes to room- or area-level summaries, and root nodes to global overviews [2409.18313].

In the 3D scene-graph variant, the retrieval unit is not a caption chain but a scene subgraph. The environment is represented as $\mathcal{G}_t=\langle O_t,E_t\rangle$, where nodes carry class labels or open-vocabulary names and attributes, and edges carry relations. Each node is treated as a document in a vector store: page content is the object label, metadata are selected attributes or captions, and embeddings are indexed for similarity search. Retrieved nodes induce a subgraph $\mathcal{G}'_t=\langle O'_t,E'_t\rangle$ that is then serialized for the planner [2410.23968].

Other embodied-RAG systems use different memory substrates. Affordance RAG constructs an Affordance-Aware Embodied Memory from pre-explored images. During pre-exploration, each image is segmented into regions; a VLM predicts a descriptive instance embedding, an affordance type such as `"pick"` or `"place"`, and an affordance confidence score. Multi-level representations then combine regional/textual semantics and visual semantics, while higher levels are produced by recursive clustering and LLM summarization [2512.18987].

eMEM generalizes embodied memory beyond pure RAG by using a hybrid graph-based system searchable by meaning, space, and time. Its MemoryStore combines SQLite for structured queries, HNSWlib for approximate nearest-neighbour semantic search on 384-dimensional embeddings, and an R-tree for 3D spatial queries. Nodes include ObservationNode, EpisodeNode, GistNode, and EntityNode, and the system exposes ten recall tools, including `locate` and `recall`, as first-class operations for LLM tool calling [2606.03374]. This suggests that some later embodied-memory systems treat RAG as only one retrieval mode inside a broader multi-index memory architecture.

Task-specific variants further specialize the retrieval unit. Fail-RAG stores CLIP-encoded concatenated failure-image strips and short textual descriptions of failure modes in a failure database, while the household robotics system of "LLM-Empowered Embodied Agent for Memory-Augmented Task Planning in Household Robotics" stores episodic question-answer pairs and action summaries with timestamps in ChromaDB using BGE-M3 embeddings [2606.19598][2504.21716].

## 3. Retrieval and generation mechanisms

A characteristic retrieval mechanism in Embodied-RAG is LLM-guided tree traversal. For a query $q$, retrieval proceeds by selecting the best child at each branching point using an LLM selector, producing one or more root-to-leaf chains. These chains preserve semantic descriptions at multiple scales and are then used for either navigation or explanation. A simple similarity primitive is also defined for leaf captions,
$$
\mathrm{sim}(q,c_i)=\frac{\mathbf{t}_q\cdot \mathbf{t}_{c_i}}{\|\mathbf{t}_q\|\;\|\mathbf{t}_{c_i}\|},
$$
but the framework’s distinctive step is the hierarchical traversal rather than flat nearest-neighbour lookup [2409.18313].

The 3D scene-graph formulation decomposes retrieval into pre-retrieval, grounding, and feedback. First, an LLM is asked which objects and which attributes matter for the task. Second, each abstract entity is used to query the vector store for top-$k$ scene-graph nodes above a cosine-similarity threshold. Third, the planner’s ReAct thought can trigger self-query feedback, which adds newly relevant nodes or metadata as the plan unfolds. This makes subgraph retrieval adapt both to environment changes and to evolving task relevancy [2410.23968].

P-RAG introduces a different mechanism: progressive database growth. At time $t$, the current goal embedding and scene-graph embedding retrieve historical trajectories using a combined score
$$
s_n=\mathrm{sim}(Q_{goal},K_{goal})+\max_{t\in[1,N]}\mathrm{sim}(Q_{obs,n},K_{obs,t}),
$$
where `sim` is cosine similarity. After each round, the database is updated with the latest interaction history, so retrieval in later iterations has access to increasingly task-specific experience [2409.11279].

Dialogue-oriented systems often adopt a more classical dense-retrieval pipeline. HoverAI classifies an utterance as either a structured command or a conversational query; conversational queries are embedded and matched by cosine similarity against approximately 150 curated facts about robotics and museum artifacts. The top-$k$ passages, typically $3$–$5$, are stitched into a single prompt with a fixed instruction and the user query, and the resulting answer is passed to XTTS v2 and a projected avatar [2601.13801].

Embodied question answering and manipulation introduce additional retrieval criteria. EfficientEQA combines BLIP-based image retrieval over accumulated observations with VLM reasoning for open-vocabulary answering, while exploration is guided by Semantic-Value-Weighted Frontier Exploration and stopped when highly question-relevant observations are detected as outliers [2410.20263]. Affordance RAG first retrieves candidates by hierarchical multimodal similarity and then reranks them by affordance confidence, with multi-level fusion defined as
$$
s_{mlf}^{(3,j)}=(1-\alpha)\cdot sim_{text}+\alpha\cdot sim_{vis},
$$
followed by affordance-aware prefiltering and reranking [2512.18987].

## 4. Task domains and empirical performance

The original Embodied-RAG framework evaluates navigation and language generation across 19 environments—7 small and 12 large, including indoor, outdoor, and mixed settings—and over 250 tasks spanning explicit, implicit, and global queries. On large environments, it reports explicit-query success rate of $0.977$ versus $0.947$ for Vanilla RAG and $0.895$ for Semantic Match; implicit-query success rate of $0.914$ versus $0.695$ and $0.426$; and global-query average Likert score of $4.86$ versus $2.43$ for Vanilla RAG [2409.18313].

The dynamic 3D scene-graph planner reports efficiency gains under scale. In a full 3DSG setting with 28 attributes and 120 trials, EmbodiedRAG-strict achieved $SR=0.133$, average time per step $1.76\,\mathrm{s}$, and cumulative tokens of approximately $0.42\,\mathrm{M}$, compared with ReAct at $SR=0.125$, $2.15\,\mathrm{s}$, and approximately $1.11\,\mathrm{M}$ tokens, and Full-Mem at $SR=0.066$, $3.11\,\mathrm{s}$, and approximately $1.30\,\mathrm{M}$–$2.52\,\mathrm{M}$ tokens. Across experiments it reduced input token counts by an order of magnitude and planning time by up to $70\%$ in average time per planning step [2410.23968].

P-RAG targets embodied everyday task planning rather than navigation-style retrieval. On ALFRED under the goal-only instruction setting and without ground-truth actions, P-RAG achieved $14.11$ on Valid Unseen and $18.20$ on Valid Seen, while P-RAG (Self-Iter.) reached $\mathbf{27.40}$ on Valid Unseen and $19.05$ on Valid Seen. On MINI-BEHAVIOR, P-RAG-4 improved total success rate from $15.0\%$ for GPT-4 to $16.7\%$, and P-RAG-3.5 improved total success rate from $7.5\%$ for GPT-3.5 to $10.0\%$ [2409.11279].

Affordance RAG addresses open-vocabulary mobile manipulation. On WholeHouse-MM, a baseline Embodied-RAG without affordances obtained Target R@10 of $18.5\%$, Receptacle R@10 of $11.3\%$, and Overall R@10 of $14.9\%$, whereas Affordance RAG achieved $49.9\%$, $24.3\%$, and $37.1\%$, respectively. Task Success Rate, defined there as both object and receptacle in the top 10, was $25.2\%$ versus OSG’s $1.4\%$. In real-world experiments on a Stretch 2 robot over 40 trials, the full system reached Recall@5 of $94\%$ and success of $85\%$ [2512.18987].

EfficientEQA targets open-vocabulary embodied question answering. Its abstract reports improvement in answering accuracy by over $15\%$ and efficiency, measured in running steps, by over $20\%$ compared to state-of-the-art methods, using active exploration, BLIP retrieval, VLM reasoning, and an outlier-based stopping criterion [2410.20263].

Embodied-RAG ideas have also been applied to conversation and monitoring. HoverAI reports command-recognition F1 of $0.90$, gender F1 of $0.89$, age MAE of $5.14$ years, speech-transcription WER of $0.181$, and overall pipeline latency of approximately $950\,\mathrm{ms}$ from speech onset to avatar response [2601.13801]. Fail-RAG reports approximately $25$ percentage point higher failure-detection accuracy on average across five types of robot operations than off-the-shelf VLMs, with gains up to $40$ percentage points in some tasks [2606.19598]. In household robotics, the knowledge-base agent’s answer validity improved from $53.75\%$ without RAG to $91.3\%$ with RAG for Qwen2.5-32B, using BGE-M3 embeddings and ChromaDB [2504.21716].

## 5. Evaluation methodology, misconceptions, and limitations

A recurring misconception is that embodied RAG is simply text retrieval over robot logs. The surveyed systems contradict that view. Retrieval targets can be caption chains, scene subgraphs, image embeddings, pre-explored region descriptions, affordance-tagged instances, episodic summaries, or structured failure exemplars. Memory can be indexed by semantics alone, or jointly by semantics, space, and time, as in eMEM’s multi-index architecture [2606.03374].

Another misconception is that there is already a single standardized architecture. The literature instead shows multiple design lineages. Some systems are hierarchical and abstraction-driven, such as semantic forests and affordance hierarchies; some are flat dense-retrieval pipelines, such as HoverAI; some are progressive and self-updating, such as P-RAG; and some use RAG as a component inside a larger memory substrate, such as eMEM [2409.18313][2409.11279][2601.13801][2606.03374].

Evaluation protocols also vary substantially. Embodied-RAG uses binary success rate for navigation and average Likert score for explanatory tasks [2409.18313]. EmbodiedRAG for 3DSGs emphasizes success rate, average time per step, and cumulative tokens [2410.23968]. P-RAG uses Success Rate and Success weighted by Path Length [2409.11279]. Affordance RAG evaluates R@10, Recall@5, and task success [2512.18987]. Fail-RAG measures retrieval-only accuracy and VLM-RAG absolute classification accuracy [2606.19598]. eMEM-Bench v1 introduces 988 probes across eight cognitive-psychology paradigms and reports an overall weighted mean of $80.8$, while a pure RAG baseline, `flat_rag`, loses $30$ points on context-dependent retrieval and $29$ points on DRM lure rejection [2606.03374]. This suggests that flat semantic retrieval can be insufficient when source, context, and consolidation are integral to the memory task.

The limitations are correspondingly heterogeneous. The original Embodied-RAG notes no explicit mechanism for dynamic obstacle avoidance or moving people, and counting-precision tasks can fail because clustering ignores multi-view consistency [2409.18313]. EmbodiedRAG for scene graphs is motivated precisely by token-count limits and attentional biases when the full graph is presented to the LLM [2410.23968]. P-RAG operates in a zero-training regime and defines no auxiliary retrieval loss, so retrieval quality is assessed only by downstream success rate [2409.11279]. HoverAI does not separately report retrieval precision, recall, or MRR for its approximately 150-fact knowledge base [2601.13801].

## 6. Extensions and emerging directions

Recent work extends Embodied-RAG beyond navigation and manipulation toward broader memory and generation problems. MoK-RAG generalizes standard single-source retrieval into a mixture of knowledge paths. In its 3D extension, MoK-RAG3D partitions assets into main, paired, and other objects, organizes them into a hierarchical knowledge tree, and reports better automatic and human evaluation than HOLODECK, including overall preference of $48\%$ versus $26\%$ and reduction of main-object missing rate from $31\%$ to $21.48\%$ [2503.13882].

Embodied memory systems are also becoming more explicitly long-horizon. eMEM introduces tiered consolidation from working to short-term, long-term, and archived tiers, with gist formation by LLM summarization and search primitives such as concept-to-location resolution and cross-layer recall [2606.03374]. A plausible implication is that future embodied-RAG systems may increasingly rely on consolidation and multi-index recall rather than on a single vector store alone.

The paradigm has also migrated into adjacent embodied and immersive interfaces. In virtual archaeology, a VR avatar coupled to a RAG system uses Qdrant, FlowiseAI, Whisper, Piper, and scholarly text chunks enriched with metadata; on an expert-crafted question set, the best reported Judge score is $3.42$, and NASA-TLX total workload is $8.4$, below the midpoint of $11$ [2603.23353]. In gesture synthesis, RAG-Gesture retrieves semantically matched co-speech exemplars and injects them into a diffusion-based generator via DDIM inversion and retrieval guidance at inference time, showing that retrieval-conditioned embodied generation is not limited to robot planning or QA [2412.06786].

Across these lines of work, the unifying theme is non-parametric grounding of generation in embodied experience, situated knowledge, or structured environmental context. The specific memory object, retrieval operator, and generation target vary widely, but the central claim remains consistent: embodied agents benefit when their generative model is augmented with retrievable, task-relevant, and environment-grounded external memory [2409.18313][2410.23968].

Source: https://www.emergentmind.com/topics/embodied-rag