Coarse-to-Fine Grounded Memory

• The paper demonstrates a method that uses coarse focus points, hybrid tips, and fine-grained corrections to improve LLM agent planning.
• The framework separates memory into three stages, enabling guided experience collection, targeted tips extraction, and anomaly-based corrective planning.
• Empirical evaluations on benchmarks like AlfWorld and WebShop show significant success rate improvements and actionable insights for memory-centric agent design.

Searching arXiv for the primary and closely related papers to ground the article in current literature. arXiv search: (Yang et al., 21 Aug 2025) Coarse-to-Fine Grounded Memory is a memory-centric framework for LLM agent planning in which environmental information is grounded at multiple granularities and then reused across data collection, offline summarization, retrieval, and online correction. In the formulation introduced for agent planning, coarse-grained focus points guide experience collection in training tasks, actionable hybrid-grained tips are grounded from collected experiences, and fine-grained key information is grounded from the current trajectory when anomalies occur; at inference, retrieved experiences and tips condition planning, while self-QA reflection supports plan correction (Yang et al., 21 Aug 2025).

1. Conceptual definition and scope

The framework is motivated by a limitation identified in prior memory-based agent designs: many studies adopt memory mechanisms that enhance an LLM with offline experiences or online trajectory analysis, but focus on single-granularity memory derived from dynamic environmental interactions. In the reported formulation, this constrains both the diversity of knowledge and the flexibility of planning. Coarse-to-Fine Grounded Memory addresses that limitation by explicitly separating memory construction into multiple granularity levels and by assigning distinct roles to each level during training and inference (Yang et al., 21 Aug 2025).

Its core symbolic objects are defined at three levels. First, the environment description $\mathrm{Desc}_{\mathrm{env}}$ together with few-shot exemplars $F_{\mathrm{manual}}$ yields a set of coarse-grained focus points, denoted $\mathrm{FP}=\{p_1,\dots,p_M\}$. These focus points are high-level hints such as “look for a sink before picking up the glass.” Second, an experience pool is accumulated as

$$\mathcal{B}=\bigl\{(t_n,\tau_{n,z})\bigr\}_{n=1..N,\;z=0..Z},$$

where $t_n$ is a training task and $\tau_{n,z}$ is its trial trajectory. Third, a tips dictionary is built as

$$\mathrm{TD}: t_n \mapsto \{\text{hybrid tips on } t_n\}.$$

At inference time, when a partial trajectory

$\tau=\{(a_0,o_1),\dots,(a_i,o_{i+1})\}$

encounters an anomaly, the model extracts fine-grained key information $\mathrm{KI}=\{k_1,\dots,k_N\}$ from that trajectory (Yang et al., 21 Aug 2025).

A central point is that “coarse-to-fine” is not confined to retrieval alone. In this framework, coarse grounding shapes what experiences are collected, hybrid-grained summaries organize what is retained from those experiences, and fine grounding is reserved for online anomaly diagnosis and correction. This suggests a broader interpretation of grounded memory as a staged allocation of representational precision rather than a single storage format.

2. Three-stage architecture

The architecture comprises three modules connected by a fixed data flow: $$\mathrm{Desc}_{\mathrm{env}} \rightarrow (\text{ground FP}) \rightarrow \text{guided trials} \rightarrow \mathcal{B} \rightarrow (\text{ground hybrid-grained tips}) \rightarrow \mathrm{TD} \rightarrow (\text{retrieve from }\mathcal{B},\mathrm{TD};\text{ monitor anomalies; ground KI}) \rightarrow \text{adaptive planning}.$$ This decomposition is explicit in the framework specification and determines both how memory is formed and how it is later consumed (Yang et al., 21 Aug 2025).

The first module is Coarse-Grained Focus-Driven Experience Collection. It adopts a ReAct + Reflection loop, but the loop is seeded with LLM-grounded focus points. The procedure initializes $F_{\mathrm{manual}}$0, computes $F_{\mathrm{manual}}$1 from the environment description and manual exemplars, and then iterates over training tasks and up to $F_{\mathrm{manual}}$2 retries. Within each trial, the action $F_{\mathrm{manual}}$3 is generated by $F_{\mathrm{manual}}$4 conditioned on the current trajectory, the exemplars, accumulated reflection text $F_{\mathrm{manual}}$5, and the focus points. Each completed or truncated trajectory is inserted into $F_{\mathrm{manual}}$6; if a trial fails and retries remain, the reflection state is updated by concatenating $F_{\mathrm{manual}}$7 into $F_{\mathrm{manual}}$8 (Yang et al., 21 Aug 2025).

The second module is Hybrid-Grained Experience-Wise Tips Extraction. The experience pool is partitioned by task into a comparison set containing both successes and failures and a success-only set. For each training task, the system invokes $F_{\mathrm{manual}}$9 first on failure/success comparisons to derive coarse insights and then on successes alone to derive fine techniques. The resulting set $\mathrm{FP}=\{p_1,\dots,p_M\}$0 is stored as $\mathrm{FP}=\{p_1,\dots,p_M\}$1. The paper describes these outputs as “actionable hybrid-grained tips,” indicating that the tips intentionally mix higher-level strategy with lower-level operational guidance (Yang et al., 21 Aug 2025).

The third module is Fine-Grained Trajectory-Adaptive Planning. For each evaluation task, the agent retrieves top-$\mathrm{FP}=\{p_1,\dots,p_M\}$2 similar experiences, assembles their success trajectories $\mathrm{FP}=\{p_1,\dots,p_M\}$3 and tips $\mathrm{FP}=\{p_1,\dots,p_M\}$4, and uses them as context for an LLM planning policy denoted $\mathrm{FP}=\{p_1,\dots,p_M\}$5. The planning loop interleaves action generation with anomaly detection. When an anomaly is triggered, the model grounds the current situation into fine-grained key information and invokes a reflection component to produce a corrective plan, which is appended into the current step before subsequent planning proceeds (Yang et al., 21 Aug 2025).

3. Memory representation and retrieval mechanics

The memory substrate is deliberately simple. The experience pool $\mathrm{FP}=\{p_1,\dots,p_M\}$6 is stored as a list of $\mathrm{FP}=\{p_1,\dots,p_M\}$7 pairs, and the tips dictionary $\mathrm{FP}=\{p_1,\dots,p_M\}$8 is a map from task $\mathrm{FP}=\{p_1,\dots,p_M\}$9 to a small list of hybrid-grained textual tips. There is no learned key-value store in the reported implementation; instead, retrieval at inference uses an off-the-shelf embedding model $$\mathcal{B}=\bigl\{(t_n,\tau_{n,z})\bigr\}_{n=1..N,\;z=0..Z},$$0—specifically all-mpnet-base-v2—and Faiss kNN for nearest-neighbor search (Yang et al., 21 Aug 2025).

Similarity between two texts $$\mathcal{B}=\bigl\{(t_n,\tau_{n,z})\bigr\}_{n=1..N,\;z=0..Z},$$1 and $$\mathcal{B}=\bigl\{(t_n,\tau_{n,z})\bigr\}_{n=1..N,\;z=0..Z},$$2 is defined by cosine: $$\mathcal{B}=\bigl\{(t_n,\tau_{n,z})\bigr\}_{n=1..N,\;z=0..Z},$$3 Top-$$\mathcal{B}=\bigl\{(t_n,\tau_{n,z})\bigr\}_{n=1..N,\;z=0..Z},$$4 retrieval is described as

$$\mathcal{B}=\bigl\{(t_n,\tau_{n,z})\bigr\}_{n=1..N,\;z=0..Z},$$5

subject to returning the $$\mathcal{B}=\bigl\{(t_n,\tau_{n,z})\bigr\}_{n=1..N,\;z=0..Z},$$6 highest-scoring entries. The retrieved set is denoted $$\mathcal{B}=\bigl\{(t_n,\tau_{n,z})\bigr\}_{n=1..N,\;z=0..Z},$$7 in the planning algorithm (Yang et al., 21 Aug 2025).

Once $$\mathcal{B}=\bigl\{(t_n,\tau_{n,z})\bigr\}_{n=1..N,\;z=0..Z},$$8 is obtained, the system constructs two contextual memory views. The first is $$\mathcal{B}=\bigl\{(t_n,\tau_{n,z})\bigr\}_{n=1..N,\;z=0..Z},$$9, the set of retrieved success trajectories. The second is $t_n$0, the union of the corresponding task-indexed tips from $t_n$1. Planning then conditions on the current trajectory together with $t_n$2 and $t_n$3. In this design, raw trajectories serve as episodic memory, while tips act as compressed, cross-trajectory semantic memory. A plausible implication is that the framework separates memory by functional role rather than by storage technology.

4. Fine-grained key information, self-QA, and correction

Online correction is activated when the environment produces an anomaly, including “no effect,” “That is locked,” and “Invalid command.” At that point the framework performs two sequential operations. First, it extracts fine-grained key information through $t_n$4, where the key information identifies the missing detail or error state in the current partial trajectory. Second, it performs self-question/answer reflection through

$t_n$5

where past successful trajectories $t_n$6 provide exemplars for repair (Yang et al., 21 Aug 2025).

The summary abstracts the self-QA coherence signal as

$t_n$7

with higher values interpreted as more coherent correction. The final corrective plan ref is the LLM output that maximizes plausibility under this internal QA metric. Operationally, the reflection text is appended into the current step and the ReAct-style loop continues from the updated context (Yang et al., 21 Aug 2025).

This correction mechanism is the “fine” stage in the strictest sense: it is invoked only under anomaly triggers, it grounds highly localized mismatches such as missing preconditions or invalid assumptions, and it conditions repair on both current trajectory evidence and retrieved successful traces. The framework therefore reserves its highest-granularity reasoning for cases in which coarse retrieval and generic planning are insufficient.

5. Empirical behavior, ablations, and limitations

The reported evaluation spans three interactive benchmarks: AlfWorld with 134 household tasks and maximum horizon 20, WebShop with 100 shopping tasks and maximum horizon 15, and ScienceWorld with 100 science tasks and maximum horizon 80. Focus points are zero-shot prompted from GPT-4-Turbo using the environment description and few-shot exemplars; trajectories are collected by guided ReAct + reflection with up to 3 retries; and tips are extracted offline by GPT-4o through the compare/success prompts (Yang et al., 21 Aug 2025).

On the main comparison, the full coarse-to-fine grounding variant, denoted AutoGuide⁺⁺, reports 91.0\% ±0.8 success rate on AlfWorld, 57\% ±3\% success rate and 85.0 ±1.3 mean reward on WebShop, and 74\% ±2\% success rate on ScienceWorld. The same table reports ReAct at 80.6\% ±0.7 on AlfWorld, 37\% ±2\% success rate and 58.6 ±1.0 reward on WebShop, and 43\% ±1\% on ScienceWorld; ExpeL at 81.3\% ±0.8, 42\% ±3\% and 62.2 ±1.3, and 57\% ±2\% respectively; AutoGuide at 83.6\% ±0.8, 47\% ±2\% and 73.3 ±1.4, with no ScienceWorld result in that table; and QuBE at 84.3\% ±0.7 on AlfWorld. The accompanying caption states gains of +10.4\% on AlfWorld, +20\% SR on WebShop, and +31\% on ScienceWorld for the full coarse-to-fine grounding variant (Yang et al., 21 Aug 2025).

The ablations are structurally informative. Module addition is reported as strictly cumulative for focus points, experience-wise tips, and key-information reflection. For top-$t_n$8 retrieval, $t_n$9 or $\tau_{n,z}$0 is reported as optimal, while too many tips hurt. For reflection style, fine-grained KI + Self-QA is reported as best. In out-of-distribution transfer from WebShop to WebArena-Shopping, tips alone yield 25.1\% success rate versus 18.5\% for ExpeL (Yang et al., 21 Aug 2025).

The limitations section identifies two boundary conditions. If only extremely few training tasks are available, even guided collection may produce insufficient diversity. Very long retrieved trajectories may introduce noise or overlap. The paper lists three extensions: a learned key-value store for continuously updated retrieval, a lightweight adapter to refine embeddings for more precise tip retrieval, and automatic curriculum creation using focus points to synthesize intermediate subtasks and bootstrap the experience pool (Yang et al., 21 Aug 2025).

6. Broader research landscape

The broader literature uses closely related coarse-to-fine grounded memory decompositions in several domains. In smart personal assistants, a grounded memory system combines Vision LLMs for image captioning and entity disambiguation with LLMs for consistent information extraction, stores extracted information in a knowledge graph enhanced by vector embeddings, performs coarse semantic search over MemoryNote and Image nodes, and then refines retrieval through graph expansion or LLM-generated Cypher queries executed on Neo4j (Ocker et al., 9 May 2025). In joint retrieval and classification, MemMatch implements a three-level search in which level 1 performs coarse bi-encoder retrieval, level 2 performs cross-encoder reranking, and level 3 performs cross-encoder classification; the support datastore is explicitly updateable, and level distances as well as exemplar auditing provide confidence detection and behavior modification without full retraining (Schmaltz et al., 2020).

Document-grounded dialogue and conversational machine reading instantiate the same pattern at different textual granularities. Re3G first retrieves and reranks passages and then extracts fine-grained spans within the selected passages for answer generation (Zhang et al., 2023). EMT represents each rule sentence as an explicit memory slot, updates slot values across dialogue turns, converts sentence-level “unknown” scores into interest weights $\tau_{n,z}$1, and then uses those weights to modulate token-level span scores for clarification-question generation (Gao et al., 2020).

Embodied and multimodal systems generalize the pattern further. EvoMemNav maintains a Visual-Semantic Memory Graph with room, view, and object nodes, applies a budgeted coarse stage to compress the graph into candidate anchor and frontier views, and invokes a VLM only once for fine verification; after each subtask it performs reflection-driven write-back to update graph-attached priors without retraining (Ge et al., 2 Jun 2026). In radiology report generation, S2D-Align uses a shallow-to-deep curriculum that progresses from coarse radiograph-report pairing to instance-level reference reports and then to key phrases, with a shared memory bank connecting the stages (Gao et al., 14 Nov 2025). In scene graph generation, Hierarchical Memory Learning first trains on coarse predicates and then on fine predicates while reconstructing stage-1 concepts and parameter-level importance through Concept Reconstruction and Model Reconstruction constraints (Deng et al., 2022).

Taken together, these systems do not define a single canonical memory structure. Instead, the literature shows a recurring architectural principle: an inexpensive broad localization stage narrows the search or hypothesis space, and a later, more selective stage performs higher-precision grounding, reasoning, or generation. This suggests that “Coarse-to-Fine Grounded Memory” is best understood not as one storage scheme, but as a family of staged grounding strategies spanning symbolic graphs, vector stores, episodic trajectory pools, explicit dialogue memories, and shared multimodal adapters.