---
title: Coarse-to-Fine Grounded Memory
url: https://www.emergentmind.com/topics/coarse-to-fine-grounded-memory
type: topic
---

# Coarse-to-Fine Grounded Memory

Searching arXiv for the primary and closely related papers to ground the article in current literature.
arXiv search: 2508.15305
Coarse-to-Fine Grounded Memory is a memory-centric framework for LLM agent planning in which environmental information is grounded at multiple granularities and then reused across data collection, offline summarization, retrieval, and online correction. In the formulation introduced for agent planning, coarse-grained focus points guide experience collection in training tasks, actionable hybrid-grained tips are grounded from collected experiences, and fine-grained key information is grounded from the current trajectory when anomalies occur; at inference, retrieved experiences and tips condition planning, while self-QA reflection supports plan correction [2508.15305].

## 1. Conceptual definition and scope

The framework is motivated by a limitation identified in prior memory-based agent designs: many studies adopt memory mechanisms that enhance an LLM with offline experiences or online trajectory analysis, but focus on single-granularity memory derived from dynamic environmental interactions. In the reported formulation, this constrains both the diversity of knowledge and the flexibility of planning. Coarse-to-Fine Grounded Memory addresses that limitation by explicitly separating memory construction into multiple granularity levels and by assigning distinct roles to each level during training and inference [2508.15305].

Its core symbolic objects are defined at three levels. First, the environment description $\mathrm{Desc}_{\mathrm{env}}$ together with few-shot exemplars $F_{\mathrm{manual}}$ yields a set of coarse-grained focus points, denoted $\mathrm{FP}=\{p_1,\dots,p_M\}$. These focus points are high-level hints such as “look for a sink before picking up the glass.” Second, an experience pool is accumulated as
\[
\mathcal{B}=\bigl\{(t_n,\tau_{n,z})\bigr\}_{n=1..N,\;z=0..Z},
\]
where $t_n$ is a training task and $\tau_{n,z}$ is its trial trajectory. Third, a tips dictionary is built as
\[
\mathrm{TD}: t_n \mapsto \{\text{hybrid tips on } t_n\}.
\]
At inference time, when a partial trajectory
\[
\tau=\{(a_0,o_1),\dots,(a_i,o_{i+1})\}
\]
encounters an anomaly, the model extracts fine-grained key information $\mathrm{KI}=\{k_1,\dots,k_N\}$ from that trajectory [2508.15305].

A central point is that “coarse-to-fine” is not confined to retrieval alone. In this framework, coarse grounding shapes what experiences are collected, hybrid-grained summaries organize what is retained from those experiences, and fine grounding is reserved for online anomaly diagnosis and correction. This suggests a broader interpretation of grounded memory as a staged allocation of representational precision rather than a single storage format.

## 2. Three-stage architecture

The architecture comprises three modules connected by a fixed data flow:
\[
\mathrm{Desc}_{\mathrm{env}}
\rightarrow (\text{ground FP})
\rightarrow \text{guided trials}
\rightarrow \mathcal{B}
\rightarrow (\text{ground hybrid-grained tips})
\rightarrow \mathrm{TD}
\rightarrow (\text{retrieve from }\mathcal{B},\mathrm{TD};\text{ monitor anomalies; ground KI})
\rightarrow \text{adaptive planning}.
\]
This decomposition is explicit in the framework specification and determines both how memory is formed and how it is later consumed [2508.15305].

The first module is **Coarse-Grained Focus-Driven Experience Collection**. It adopts a ReAct + Reflection loop, but the loop is seeded with LLM-grounded focus points. The procedure initializes $\mathcal{B}\leftarrow\emptyset$, computes $\mathrm{FP}$ from the environment description and manual exemplars, and then iterates over training tasks and up to $Z$ retries. Within each trial, the action $a_i$ is generated by $\mathrm{LLM}_{\mathrm{ReAct}}$ conditioned on the current trajectory, the exemplars, accumulated reflection text $\nu$, and the focus points. Each completed or truncated trajectory is inserted into $\mathcal{B}$; if a trial fails and retries remain, the reflection state is updated by concatenating $\mathrm{LLM}_{\mathrm{Reflect}}(\tau)$ into $\nu$ [2508.15305].

The second module is **Hybrid-Grained Experience-Wise Tips Extraction**. The experience pool is partitioned by task into a comparison set containing both successes and failures and a success-only set. For each training task, the system invokes $\mathrm{LLM}_{\mathrm{Tips}}$ first on failure/success comparisons to derive coarse insights and then on successes alone to derive fine techniques. The resulting set $T$ is stored as $\mathrm{TD}[t]$. The paper describes these outputs as “actionable hybrid-grained tips,” indicating that the tips intentionally mix higher-level strategy with lower-level operational guidance [2508.15305].

The third module is **Fine-Grained Trajectory-Adaptive Planning**. For each evaluation task, the agent retrieves top-$k$ similar experiences, assembles their success trajectories $ST$ and tips $ET$, and uses them as context for an LLM planning policy denoted $\mathrm{LLM}_{\mathrm{AutoGuide}}$. The planning loop interleaves action generation with anomaly detection. When an anomaly is triggered, the model grounds the current situation into fine-grained key information and invokes a reflection component to produce a corrective plan, which is appended into the current step before subsequent planning proceeds [2508.15305].

## 3. Memory representation and retrieval mechanics

The memory substrate is deliberately simple. The experience pool $\mathcal{B}$ is stored as a list of $(t,\tau)$ pairs, and the tips dictionary $\mathrm{TD}$ is a map from task $t$ to a small list of hybrid-grained textual tips. There is no learned key-value store in the reported implementation; instead, retrieval at inference uses an off-the-shelf embedding model $\mathcal{E}:\mathrm{Text}\to\mathbb{R}^d$—specifically all-mpnet-base-v2—and Faiss kNN for nearest-neighbor search [2508.15305].

Similarity between two texts $x$ and $y$ is defined by cosine:
\[
s(x,y)=\cos\bigl(\mathcal{E}(x),\mathcal{E}(y)\bigr)
=\frac{\langle \mathcal{E}(x),\mathcal{E}(y)\rangle}
{\|\mathcal{E}(x)\|\;\|\mathcal{E}(y)\|}.
\]
Top-$k$ retrieval is described as
\[
\arg\max_{(t_h,\tau_h)\in\mathcal{B}} s(t_m,\tau_h)
\]
subject to returning the $k$ highest-scoring entries. The retrieved set is denoted $E_{\mathrm{sim}}=\mathrm{Faiss}(t_m,\mathcal{B},\mathcal{E},k)$ in the planning algorithm [2508.15305].

Once $E_{\mathrm{sim}}$ is obtained, the system constructs two contextual memory views. The first is $ST$, the set of retrieved success trajectories. The second is $ET$, the union of the corresponding task-indexed tips from $\mathrm{TD}$. Planning then conditions on the current trajectory together with $ET$ and $ST$. In this design, raw trajectories serve as episodic memory, while tips act as compressed, cross-trajectory semantic memory. A plausible implication is that the framework separates memory by functional role rather than by storage technology.

## 4. Fine-grained key information, self-QA, and correction

Online correction is activated when the environment produces an anomaly, including “no effect,” “That is locked,” and “Invalid command.” At that point the framework performs two sequential operations. First, it extracts fine-grained key information through $\mathrm{KI}=\mathrm{LLM}_{\mathrm{KIE}}(\tau)$, where the key information identifies the missing detail or error state in the current partial trajectory. Second, it performs self-question/answer reflection through
\[
(qa,ref)=\mathrm{LLM}_{\mathrm{KIR}}(\tau,\mathrm{KI},ST),
\]
where past successful trajectories $ST$ provide exemplars for repair [2508.15305].

The summary abstracts the self-QA coherence signal as
\[
\mathrm{score}_{\mathrm{QA}}(qa,ref)
=\cos\bigl(\mathcal{E}(qa),\mathcal{E}(ref)\bigr),
\]
with higher values interpreted as more coherent correction. The final corrective plan `ref` is the LLM output that maximizes plausibility under this internal QA metric. Operationally, the reflection text is appended into the current step and the ReAct-style loop continues from the updated context [2508.15305].

This correction mechanism is the “fine” stage in the strictest sense: it is invoked only under anomaly triggers, it grounds highly localized mismatches such as missing preconditions or invalid assumptions, and it conditions repair on both current trajectory evidence and retrieved successful traces. The framework therefore reserves its highest-granularity reasoning for cases in which coarse retrieval and generic planning are insufficient.

## 5. Empirical behavior, ablations, and limitations

The reported evaluation spans three interactive benchmarks: **AlfWorld** with 134 household tasks and maximum horizon 20, **WebShop** with 100 shopping tasks and maximum horizon 15, and **ScienceWorld** with 100 science tasks and maximum horizon 80. Focus points are zero-shot prompted from GPT-4-Turbo using the environment description and few-shot exemplars; trajectories are collected by guided ReAct + reflection with up to 3 retries; and tips are extracted offline by GPT-4o through the compare/success prompts [2508.15305].

On the main comparison, the full coarse-to-fine grounding variant, denoted **AutoGuide⁺⁺**, reports **91.0\% ±0.8** success rate on AlfWorld, **57\% ±3\%** success rate and **85.0 ±1.3** mean reward on WebShop, and **74\% ±2\%** success rate on ScienceWorld. The same table reports **ReAct** at **80.6\% ±0.7** on AlfWorld, **37\% ±2\%** success rate and **58.6 ±1.0** reward on WebShop, and **43\% ±1\%** on ScienceWorld; **ExpeL** at **81.3\% ±0.8**, **42\% ±3\%** and **62.2 ±1.3**, and **57\% ±2\%** respectively; **AutoGuide** at **83.6\% ±0.8**, **47\% ±2\%** and **73.3 ±1.4**, with no ScienceWorld result in that table; and **QuBE** at **84.3\% ±0.7** on AlfWorld. The accompanying caption states gains of **+10.4\%** on AlfWorld, **+20\% SR** on WebShop, and **+31\%** on ScienceWorld for the full coarse-to-fine grounding variant [2508.15305].

The ablations are structurally informative. Module addition is reported as strictly cumulative for focus points, experience-wise tips, and key-information reflection. For top-$k$ retrieval, $k=2$ or $3$ is reported as optimal, while too many tips hurt. For reflection style, fine-grained KI + Self-QA is reported as best. In out-of-distribution transfer from WebShop to WebArena-Shopping, tips alone yield **25.1\%** success rate versus **18.5\%** for ExpeL [2508.15305].

The limitations section identifies two boundary conditions. If only extremely few training tasks are available, even guided collection may produce insufficient diversity. Very long retrieved trajectories may introduce noise or overlap. The paper lists three extensions: a learned key-value store for continuously updated retrieval, a lightweight adapter to refine embeddings for more precise tip retrieval, and automatic curriculum creation using focus points to synthesize intermediate subtasks and bootstrap the experience pool [2508.15305].

## 6. Broader research landscape

The broader literature uses closely related coarse-to-fine grounded memory decompositions in several domains. In smart personal assistants, a grounded memory system combines Vision Language Models for image captioning and entity disambiguation with Large Language Models for consistent information extraction, stores extracted information in a knowledge graph enhanced by vector embeddings, performs coarse semantic search over MemoryNote and Image nodes, and then refines retrieval through graph expansion or LLM-generated Cypher queries executed on Neo4j [2505.06328]. In joint retrieval and classification, MemMatch implements a three-level search in which level 1 performs coarse bi-encoder retrieval, level 2 performs cross-encoder reranking, and level 3 performs cross-encoder classification; the support datastore is explicitly updateable, and level distances as well as exemplar auditing provide confidence detection and behavior modification without full retraining [2012.02287].

Document-grounded dialogue and conversational machine reading instantiate the same pattern at different textual granularities. Re3G first retrieves and reranks passages and then extracts fine-grained spans within the selected passages for answer generation [2302.11849]. EMT represents each rule sentence as an explicit memory slot, updates slot values across dialogue turns, converts sentence-level “unknown” scores into interest weights $\zeta_i$, and then uses those weights to modulate token-level span scores for clarification-question generation [2005.12484].

Embodied and multimodal systems generalize the pattern further. EvoMemNav maintains a Visual-Semantic Memory Graph with room, view, and object nodes, applies a budgeted coarse stage to compress the graph into candidate anchor and frontier views, and invokes a VLM only once for fine verification; after each subtask it performs reflection-driven write-back to update graph-attached priors without retraining [2606.03509]. In radiology report generation, S2D-Align uses a shallow-to-deep curriculum that progresses from coarse radiograph-report pairing to instance-level reference reports and then to key phrases, with a shared memory bank connecting the stages [2511.11066]. In scene graph generation, Hierarchical Memory Learning first trains on coarse predicates and then on fine predicates while reconstructing stage-1 concepts and parameter-level importance through Concept Reconstruction and Model Reconstruction constraints [2203.06907].

Taken together, these systems do not define a single canonical memory structure. Instead, the literature shows a recurring architectural principle: an inexpensive broad localization stage narrows the search or hypothesis space, and a later, more selective stage performs higher-precision grounding, reasoning, or generation. This suggests that “Coarse-to-Fine Grounded Memory” is best understood not as one storage scheme, but as a family of staged grounding strategies spanning symbolic graphs, vector stores, episodic trajectory pools, explicit dialogue memories, and shared multimodal adapters.

Source: https://www.emergentmind.com/topics/coarse-to-fine-grounded-memory