Rethinking How to Remember: Beyond Atomic Facts in Lifelong LLM Agent Memory

Abstract: To enable reliable long-term interaction, LLM agents require a memory system that can faithfully store, efficiently retrieve, and deeply reason over accumulated dialogue history. Most existing methods adopt an extracted fact based paradigm: handcrafted static prompts compress raw dialogues into atomic facts, which are then stored, matched, and injected into downstream reasoning. Nevertheless, such fact-centric designs inevitably discard fine-grained details in original dialogues and fail to support deep reasoning over scattered isolated facts. Moreover, static prompts cannot maintain consistent extraction granularity across diverse dialogue styles. To address these limitations, we propose TriMem, which maintains three coexisting representation granularities, including raw dialogue segments anchored by source identifiers for storage fidelity, extracted atomic facts for efficient memory retrieval, synthesized profiles that aggregate dispersed facts into holistic semantic understanding for deep reasoning. We further adopt TextGrad-based prompt optimization, which iteratively refines extraction and profiling prompts via response quality feedback, achieving lifelong evolution without any parameter updating. Extensive experiments on LoCoMo and PerLTQA across multiple LLM backbones demonstrate that TriMem consistently outperforms strong memory baselines. The code is available at https://TMLR-TriMem.github.io .

• The paper introduces TriMem, a tri-granularity memory system that preserves raw dialogues, extracts key facts, and synthesizes entity profiles to support deep reasoning.
• It demonstrates significant performance gains, achieving 20–30% improvements in F1 and BLEU scores on multi-turn dialogue benchmarks.
• The study leverages continuous prompt evolution via TextGrad, enabling lightweight, lifelong adaptation without requiring full model updates.

TriMem: Tri-Granularity Memory Architectures for Lifelong LLM Agents

Introduction and Motivation

Lifelong memory for LLM-based agents in extended, multi-session dialogue scenarios demands a system capable of high-fidelity storage, efficient retrieval, and deep reasoning over accumulated conversational data. The prevailing paradigm has centered on extracted atomic facts using static prompts: dialogues are compressed to minimal factual representations, which serve as the sole memory unit during storage, retrieval, and reasoning. This fact-centric design, while facilitating efficient similarity matching, inherently induces information loss due to lossy compression, restricts downstream reasoning to shallow fact recall, and is unable to guarantee consistent granularity in the face of highly heterogeneous conversational data. Furthermore, the use of static prompts fails to adapt to the variable requirements of real-world dialogue, leading to unstable extraction boundaries and further degradation.

Figure 1: TriMem implements a three-level memory hierarchy—raw dialogue storage, key-fact based retrieval, and in-depth reasoning on integrated profiles, with continual prompt optimization via feedback.

Detailed analysis of existing systems demonstrates that although extracted fact-based retrieval outperforms naive chunked retrieval in terms of hit rate and retrieval precision, fine-grained details encoded in the original dialogues are frequently omitted, and the inference capabilities on multi-hop or integrative queries are markedly limited. Systematic evaluation reveals a nontrivial coverage gap—14.5% reference-answer token loss relative to retaining the full dialogue corpus. Reasoning performance is highly dependent on question type: single-evidence questions are handled reliably, but multi-evidence or high-level conceptual queries are seldom addressed correctly due to the inability to integrate scattered, atomic facts (Figure 2).

Figure 2: Extracted-fact-centric memory systems display substantial information loss and shallow reasoning paths, with static prompts failing to generalize.

TriMem: Three-Level Memory Representation

To resolve the deficiencies identified above, TriMem introduces a tri-granular architecture for agent memory:

1. Raw Dialogue Storage (Dense Fidelity): All compressed memory entries preserve traceable pointers to their original conversational segments. This enables seamless recovery of omitted details for detail-sensitive queries and supports maximal storage fidelity without incurring the prohibitive context length costs of naive full-dialogue storage.
2. Extracted Key-Fact Retrieval (Retrieval Efficiency): Dialogue history is partitioned via a sliding window, and a multi-dimensional extraction schema is applied (including lossless restatement, absolute event timestamp, entities, topics). Each fact entry is indexed by both a source identifier and, optionally, fine-grained entity or event types. Extraction schemas are domain-adaptable and extensible.
3. Synthesized Entity Profiles (Deep Reasoning):

Scattered memory entries associated with a given persona are synthesized incrementally into structured, multi-faceted profiles via profile construction prompts. These profiles support entity-level comprehension, capturing both explicit and inferred attributes (including personality, values, behavioral tendencies, relationships, key life events), enabling reasoning beyond explicit fact matching.

Figure 3: Overview of TriMem pipeline: windowed segmentation, fact extraction with traceable indices, continuous profile aggregation, and feedback-driven prompt optimization.

Crucially, prompt optimization for both fact extraction and profile construction is performed using the TextGrad mechanism—feedback-driven, LLM-judged optimization steps propagate failure signal to the prompts as natural-language edits, enabling prompt-level lifelong evolution without parameter updates and ensuring robustness to diverse conversational styles.

Empirical Evaluation

Benchmarking: QA and Model Generality

TriMem is comprehensively evaluated on LoCoMo (multi-turn, persona-grounded dialogue QA) and PerLTQA (longitudinal personalized agent QA, spanning factual, behavioral, and relational information) benchmarks. It is compared to state-of-the-art memory solutions: LightMem, Mem0, MemoryOS, A-Mem, SimpleMem, xMemory, and naive RAG.

Performance on LoCoMo (using GPT-4o, GPT-4.1-mini, GPT-5-nano, Qwen3-8B, and Llama-3.1-8B-Ins) reveals TriMem achieves the highest BLEU and F1 across all subcategories (MultiHop, Temporal, OpenDomain, SingleHop). Despite slightly increased token overhead (∼1.2k tokens avg.), this is offset by the resulting 20–30% absolute gains in F1 and BLEU versus the strongest prior methods. In PerLTQA, TriMem similarly dominates with 2–10% absolute improvements across profile, events, and relational QA, including with 8B-parameter small models, demonstrating backward compatibility and efficiency.

Ablations: Module Impact

Profile and raw dialogue modules are both shown to be critical (Figure 4): removing either causes substantial performance drops, confirming that original details and entity-level integration both underpin robust reasoning.

Figure 4: Exclusion of entity profiles or raw dialogues from TriMem significantly impairs performance, confirming the necessity of both modules.

Prompt evolution rounds further enhance performance up to a saturation threshold (four TextGrad steps optimal; Figure 5), while incorrectly tuned retrieval breadth (i.e., number of top entries) or window size degrades extraction quality or inference efficiency (Figures 6 & 8). Query analysis preceding retrieval—enabling the agent to transform ill-posed or ambiguous questions into actionable search queries—is shown to be essential in empirical analyses (Figures 7 & 11).

Figure 5: System performance peaks with four prompt evolution rounds—over-refinement can result in diminishing returns.

Figure 6: Retrieval performance peaks at 25 memory entries; fewer or more results in lost information or added noise.

Figure 7: Search query analysis before retrieval increases response latency but yields substantial performance gains and efficient memory construction.

Figure 8: Smaller window sizes enable better extraction quality but at significant computational cost.

Qualitative Impact

Prompt evolution leads to concretely more detailed, context-specific extraction and synthesis. Memory retrieval is demonstrated to be precise and contextually relevant (Figure 9), and intermediate prompt versions visibly improve in detail preservation and semantic fidelity (Figure 10).

Figure 10: Prompt evolution visualization—prompts are incrementally refined for greater detail and consistency.

Figure 9: Retrieval entry contents demonstrate high precision on diverse QA types.

Figure 11: Question decomposition into key search queries enables targeted retrieval and higher reasoning accuracy.

Figure 12: Oversized windows in dialogue segmentation merge or drop low-salience facts, degrading extraction quality.

Theoretical and Practical Implications

TriMem demonstrates that atomic-fact-centric designs, despite efficiency advantages, impose an intrinsic upper bound on reasoning depth and data retention due to their inflexible granularity and static prompt constraints. By decoupling storage, retrieval, and reasoning representations, TriMem appropriately balances information density and downstream task utility. The profile integration module supports not only behavioral or conceptual inference but also the efficient encoding of hierarchical and relational knowledge structures necessary for advanced cognitive agent architectures.

Prompt-level lifelong evolution, as realized by TextGrad, is particularly notable: it offers lightweight, continuous optimization compatible with API-only LLMs, sidestepping the need for expensive reinforcement learning or model fine-tuning pipelines. This methodology is likely to serve as a baseline for future agent systems requiring domain adaptation, user-specific customization, or evolution under heterogeneous interaction regimes.

From a practical perspective, the demonstrated performance/efficiency trade-offs offer significant implications for deployment in constrained environments and with smaller models. Integration of search query decomposition and fine-grained, multi-level memory can be extended to broader agent tasks, including long-horizon planning, continual learning, and human-AI alignment through transparent memory reconstruction.

Future Research Directions

Building on the tri-granularity foundation, expanding entity profiles to cover higher-order and relational structures (event chains, social graphs, spatiotemporal patterns) is likely to augment reasoning over complex multi-agent dialogues. Further, incorporating multi-stage feedback (including intermediate retrieval quality signals or reasoning trajectory supervision) into the prompt optimization process can yield even finer adaptation for diverse application domains.

Conclusion

TriMem establishes a systematic rethinking of memory design for LLM-powered agents: storing verbatim dialogues for fidelity, extracting atomic facts for efficiency, and synthesizing entity-level profiles for semantic integration. Employing continuous, feedback-driven prompt evolution yields robust, lifelong adaptation—without parameter updates—scaling efficiently across all major LLM architectures. Extensive quantitative and qualitative evidence confirms that TriMem substantially improves both the reliability and generality of agent memory, setting a new standard for future research and practical deployment in lifelong agent interaction settings.