From Passive Retrieval to Active Memory Navigation: Learning to Use Memory as a Structured Action Space

Abstract: Long-term user memory is essential for personalized conversational agents, yet many memory systems still expose memory through passive retrieval interfaces, making the model a consumer of pre-selected evidence. We introduce NapMem, a framework for learning to use long-term user memory as a structured action space rather than passively retrieved context. NapMem organizes user history into a linked multi-granularity memory pyramid, where raw conversations, typed memory records, topic tracks, and user profiles are connected through provenance relations, and exposes these levels through memory tools. The agent is trained to select memory according to the query and intermediate evidence, allowing it to inspect different memory granularities before answering. Experiments on PersonaMem-v2, LongMemEval, and LoCoMo show that a NapMem agent trained with memory-tool reinforcement learning is competitive across diverse memory-intensive tasks, while evaluations on non-memory tasks suggest that the learned policy largely preserves general reasoning and tool-use abilities. Additional analyses examine storage, inference cost, tool-use behavior, and ablations over navigation, memory granularity, and RL training. Our results suggest that long-term user memory benefits from coupling structured storage with a learned policy for using memory at the appropriate granularity.

• The paper presents NapMem as a novel framework that reframes memory retrieval into active navigation through a multi-layered memory pyramid.
• It employs reinforcement learning (GRPO) to optimize tool-based memory access, reducing unnecessary calls while enhancing evidence acquisition.
• Empirical results demonstrate that NapMem outperforms larger untrained models on long-term recall benchmarks, balancing efficiency and accuracy.

Active Memory Navigation with NapMem: Structured Long-Term Memory as an Agentic Action Space

Introduction and Motivation

Advances in LLM-based agents have raised the requirements for long-term user memory, especially in task settings where continuity, personalization, and user-specific factual recall are essential. The predominant paradigm, however, presents memory as passive retrieval: systems construct rich storage and sophisticated retrieval mechanisms, but ultimately deliver a snapshot of pre-selected context to the agent, severely constraining its agency over evidence selection.

This paper introduces NapMem, reframing agent interaction with user memory as active navigation through a structured, multi-granularity "memory pyramid." Rather than passively receiving retrieved memory slices, agents are equipped with explicit tool interfaces—allowing them to choose what memory to access, at which abstraction level, and to iteratively refine this access based on intermediate observations. The core hypothesis is that memory use should become part of the agent's learnable policy, optimized for optimal evidence access and improved downstream performance.

Figure 1: Overview of NapMem. The agent actively navigates a multi-granularity long-term memory pyramid through memory tools, selecting abstraction levels and refining actions based on intermediate evidence.

Memory Pyramid Construction

NapMem systematically organizes interaction histories into a four-level memory pyramid:

• Raw conversations: This base layer stores verbatim user-agent exchanges, each with metadata for exact retrieval.
• Memory records: Structured records distilled from conversations, abstracting over redundancy while encoding facts, events, instructions, and preferences. They maintain strong links to their provenance in the raw layer.
• Topic tracks: Mid-range aggregations that incrementally group related records into evolving narratives or user-centric topics, supporting longitudinal understanding.
• User profile: A high-level, compact summary capturing persistent user attributes, preferences, and behavioral patterns, condensed from topic track evolution.

Memory construction is incremental and data-driven, propagating lower-layer changes upwards through dependency links while maintaining provenance for agent navigation. Lower layers are stored as structured, retrievable JSONL, while upper layers leverage human-readable Markdown.

Structured Tool-Based Navigation

NapMem operationalizes memory use through explicit tools, each corresponding to operations over specific pyramid layers: search (or exact get) over conversations and records, and file read on topic tracks or the profile. At inference time, the agent may sequentially invoke any of these tools (up to a small budget), or terminate and answer. The agent's policy is thus a mapping from queries and observed evidence to tool choices and, eventually, the answer.

This active navigation enables both top-down and bottom-up memory search. Agents can begin broadly with global summaries and descend for fine-grained evidence, or start narrow and escalate abstraction if local evidence remains insufficient.

Reinforcement Learning with Group Relative Policy Optimization

Memory navigation policy is not hand-crafted: NapMem employs RL, specifically Group Relative Policy Optimization (GRPO), to optimize agent behavior. The reward rubric incentivizes formatted, correct answers that utilize memory tools (on memory-intensive tasks) while penalizing excessive tool usage or malformed output. The policy jointly learns to select both the correct final answer and the trajectory of memory actions leading to it. The reward structure is carefully calibrated to reflect success on the task, correctness under tool-use constraints, and avoidance of unnecessary memory calls.

Empirical Evaluation

NapMem is evaluated against leading baselines (Mem0, Zep, MemOS, MemoryOS, AgeMem) across three benchmarks for long-term user memory:

• PersonaMem-v2: Personalized agent behavior based on implicit user preferences.
• LongMemEval: Long-range recall, cross-session and temporal reasoning.
• LoCoMo: Extremely long, multi-session conversational recall.

On all aggregation metrics, NapMem consistently matches or exceeds the strongest baselines; notably, a trained 9B parameter NapMem (with RL) achieves the highest average memory-intensive benchmark score (62.74), outperforming even much larger untrained LLMs (e.g., 397B) in certain tasks. This demonstrates that structured, agentic memory use can offset substantial model scaling.

Non-Memory Tasks: Preservation of Agentic Capabilities

A critical concern with specialized memory policies is negative transfer: does policy fine-tuning for memory navigation degrade general agentic skill? Experiments on GPQA-Diamond (scientific reasoning), V*Bench (visual QA tool use), and BFCL-v3 (function calling) show that RL-trained NapMem preserves, and in some cases marginally improves, accuracy while strongly suppressing unnecessary memory calls. For example, unnecessary tool-call rate on GPQA-D drops from 34.51% (no RL) to 6.90% (with RL), with no performance cost. This refutes the notion that RL will induce overuse of memory tools.

Efficiency Analysis

Storage: Despite supporting pyramidal granularity and provenance-tracking, NapMem's total storage footprint is generally smaller than most competitive baselines—often by more than 2×—demonstrating efficiency of incremental abstraction.

Inference: Active navigation allows NapMem to terminate search early when sufficient evidence is gathered, reducing completion length and response latency versus systems reliant on monolithic retrieval augmentation.

Figure 2: Latency and completion-token statistics on successful runs from 100 PersonaMem-v2 samples.

Detailed Tool-Use Behavior

Model scale affects navigation style: Large-scale variants prefer file-based topic/profile initiation, leveraging their capacity for long context; 9B models start at lower, more targeted layers. RL fine-tuning sharpens evidence acquisition: RL-trained agents show reduced tool calls, increased hit-rate on relevant evidence, and substantial increases in overall answer correctness.

Figure 3: First-step memory-tool behavior across NapMem variants.

Ablation studies confirm that each component—the multi-layer pyramid, active navigation, and RL policy learning—substantially contributes to task performance.

Implications and Future Directions

NapMem's results demonstrate that reframing long-term memory as an explicit action space yields both practical and theoretical gains. Practically, it suggests that high-performing personal assistants and collaborative agents can achieve robust task-specific memory retention without intolerable cost, avoiding both memory bloat and unnecessary retrieval. Theoretically, it reinforces the value of adaptive computation and learned control in memory-intensive reasoning—suggesting future LLM agents may further benefit from more nuanced, modular, and goal-driven external memory substrates.

Anticipated developments include adaptive privacy/forgetting policies, customization for domain-specific abstraction hierarchies, and scaling studies addressing memory-use strategies at larger model sizes.

Conclusion

NapMem presents a compelling paradigm for agentic long-term memory: learning to use explicit, structured memory resources through RL yields robust, efficient, and explainable policies that improve task performance and generalization. By disentangling memory construction, storage, and active selection, NapMem provides a foundation for future AI agents to learn not just from history, but to actively wield it as part of their cognition.