AdaMem: Learning What to Remember for Personalized Long-Horizon LLM Agents

Abstract: Long-term memory systems for LLM agents typically try to \emph{remember everything}, extracting memories uniformly to retain as many facts as possible. In production, however, inference cost and finite context budgets make this untenable: beyond consolidating raw dialogue into memory, an agent must exert \emph{write control}, efficiently keeping only the information each user actually cares about. Otherwise, long-horizon personalized interactions suffer \emph{memory bloat}, where irrelevant trivia crowds out useful information and steadily erodes question-answering (QA) accuracy. We argue that what is worth remembering is role-dependent, and propose \textbf{AdaMem} (Adaptive Memory), a method that \emph{learns what to remember} for each user from feedback. AdaMem maintains a structured, role-specific Memory Policy and refines it from weekly QA feedback through a lightweight, patch-style self-reflection step with failure rollback. To study this setting, we build \textbf{AdaMem-Bench}, a benchmark that simulates weeks of interaction with week-by-week QA. Across two extraction models and two feedback modes, AdaMem improves QA accuracy by up to \textbf{+9.0\%} over the uniform Mem0 baseline while shrinking memory volume by \textbf{9\%}.

• The paper presents AdaMem with an active write-control paradigm that learns a user-specific Memory Policy to optimize long-horizon memory retention.
• It employs a closed-loop process featuring daily extraction, weekly QA evaluation, and policy adjustments to balance relevance and efficiency.
• Empirical results show AdaMem improves QA accuracy by 9% and decreases memory volume, outperforming uniform extraction methods.

AdaMem: Adaptive Memory Extraction for Personalized Long-Horizon LLM Agents

Motivation and Problem Formulation

The deployment of LLM agents in extended personalized scenarios introduces stringent requirements on long-term memory management, constrained by inference costs and finite context budgets. Conventional systems that indiscriminately extract and retain all factual dialogue content succumb to severe memory bloat. This accumulation of irrelevant details—ranging from casual greetings to minor logistical notes—undermines retrieval quality and degrades downstream QA performance over prolonged horizons. The central flaw is a lack of role- and user-specific memory control; instead, extraction is performed under uniform, non-adaptive rules.

AdaMem addresses this with an active write-control paradigm. Rather than indiscriminately consolidating every utterance, AdaMem learns a Memory Policy tailored to each user's preferences, dynamically updated based on QA feedback. Memory extraction thus becomes a managed process aligned with user relevance, fundamentally reframing long-horizon memory as a preference-guided write-control problem.

Figure 1: Accuracy on AdaMem-Bench over 10 weeks; AdaMem and Ideal Memory avoid degradation suffered by uniform Mem0.

Methodology: Closed-Loop Preference-Guided Memory

AdaMem operates iteratively through a weekly closed-loop pipeline:

1. Daily Extraction: Memories are extracted from daily dialogues in alignment with the active Memory Policy, which maps roles to natural-language extraction preferences.
2. Weekly Evaluation: The agent's recall is probed using a battery of targeted QA tasks.
3. Policy Reflection and Patch Generation: Failures in QA are analyzed as recall (missing memory) or precision (noisy retrieval) errors. Policy patches are generated to adjust extraction rules, guarded by a rollback mechanism to prevent policy drift and maintain historical QA accuracy.
4. Rendering and Time Anchoring: Policies are compiled into extractor-specific instructions, with explicit resolution of all temporal references to absolute dates for robust timeline reasoning.

The Memory Policy itself is a structured, JSON-mapped object specifying role-dependent extraction instructions (e.g., “Keep only business decisions for Manager; ignore daily chats”). Patches are applied at the policy level (not model weights), enabling lightweight integration with existing RAG architectures without retraining.

AdaMem-Bench: Benchmark Construction and Experimental Setup

AdaMem-Bench was developed to stress test personalized memory extraction under realistic, multi-session, multi-persona conditions. The benchmark features:

• Dynamic, naturalistic dialogues embedding casual and preference-specific content, with high semantic overlap to foster distractor interference.
• Annotation of golden memories per session, defining preference-aligned, atomic facts as the retrieval targets.
• Weekly QA generation targeting only user-relevant content, coupled with explicit or implicit feedback signals.

Baselines include:

• Full Context: Ideal upper bound (entire history in context).
• Ideal Memory: Human-annotated golden memories only.
• Mem0: Uniform extraction of all facts.

AdaMem and baselines are evaluated under two extraction models (DeepSeek-V4-Flash, Gemini-3.5-Flash) and two feedback modes (Explicit and Implicit).

Empirical Analysis: Performance, Retrieval Dynamics, and Policy Bottlenecks

Overall Gains and Memory Efficiency

AdaMem consistently surpasses Mem0 across all metrics: QA accuracy, Extraction F1, Memory Efficiency Ratio (MER), and memory volume. Notably, with DeepSeek-V4-Flash and explicit feedback, AdaMem achieves +9.0% absolute improvement in QA accuracy and a 9.1% reduction in memory volume. Gains are robust across both extraction models and feedback modes. AdaMem achieves higher MER values, indicating more efficient use of each memory for accurate QA, and higher Extraction F1, particularly in preference-based precision.

Information Category Analysis

Performance gains concentrate in "soft preference" categories—emotion, schedule, decision, and promises—where uniform extraction fails due to the difficulty in filtering contextual, subjective information. For factual and numeric content, both methods perform near optimally, with AdaMem's advantage largely irrelevant. This underscores AdaMem's practical relevance in real-world deployments where user preferences extend beyond factual recall.

Retrieval Quality Over Long Horizons

Uniform memory systems (Mem0) experience degrading Recall@5 and MRR as irrelevant content accumulates; by Week 10, retrieval quality drops 18% in Recall@5. AdaMem maintains retrieval stability, significantly outperforming Mem0, due to proactive write control and memory compaction.

Figure 2: Recall@5 and MRR for DeepSeek-V4-Flash over 10 weeks; AdaMem sustains superior retrieval quality relative to Mem0.

Policy Learning Bottlenecks

LLMs reliably follow extraction instructions (policy adherence), as evidenced by high policy-following rates (>80%) regardless of the model or feedback mode. However, inferring the correct Memory Policy under implicit feedback (correctness-only signals) remains unsolved; semantic alignment with ground-truth preferences stagnates below 65%. Explicit feedback enables better convergence (>80%), highlighting policy inference from weak signals as the limiting factor in adaptive memory extraction.

Practical and Theoretical Implications

AdaMem demonstrates that adaptive, feedback-driven memory extraction is critical for sustainable, high-accuracy QA in long-horizon personalized agents. By maintaining user-aligned memory with minimal bloat, AdaMem offers direct industrial benefits: reduced inference cost, improved context utilization, and seamless integration with modular RAG stacks. Future developments should target robust policy inference with weak supervision, advanced regularization against policy drift, and extension to real-world, noisy conversational datasets. The write-control paradigm sets a foundation for mnemonic sovereignty in agentic systems, suggesting cross-domain applicability in knowledge management and security-critical settings.

Conclusion

AdaMem advances long-horizon LLM agent memory management by shifting from indiscriminate retention to adaptive, preference-guided extraction. Empirical results confirm consistent improvements across accuracy, efficiency, and retrieval quality. The core challenge remains robust policy inference under minimal feedback—a crucial direction for personalized agent research and practical LLM deployment.