In Prospect and Retrospect: Reflective Memory Management for Long-term Personalized Dialogue Agents

Abstract: LLMs have made significant progress in open-ended dialogue, yet their inability to retain and retrieve relevant information from long-term interactions limits their effectiveness in applications requiring sustained personalization. External memory mechanisms have been proposed to address this limitation, enabling LLMs to maintain conversational continuity. However, existing approaches struggle with two key challenges. First, rigid memory granularity fails to capture the natural semantic structure of conversations, leading to fragmented and incomplete representations. Second, fixed retrieval mechanisms cannot adapt to diverse dialogue contexts and user interaction patterns. In this work, we propose Reflective Memory Management (RMM), a novel mechanism for long-term dialogue agents, integrating forward- and backward-looking reflections: (1) Prospective Reflection, which dynamically summarizes interactions across granularities-utterances, turns, and sessions-into a personalized memory bank for effective future retrieval, and (2) Retrospective Reflection, which iteratively refines the retrieval in an online reinforcement learning (RL) manner based on LLMs' cited evidence. Experiments show that RMM demonstrates consistent improvement across various metrics and benchmarks. For example, RMM shows more than 10% accuracy improvement over the baseline without memory management on the LongMemEval dataset.

• The paper presents Reflective Memory Management (RMM) which integrates prospective and retrospective reflection to maintain coherent, personalized dialogue memories.
• It employs dynamic summarization and reinforcement learning to optimize memory granularity and retrieval, leading to over 10% accuracy improvement on LongMemEval.
• The approach offers practical benefits in long-term interaction consistency and scalable personalized retrieval without extensive labeled data.

In Prospect and Retrospect: Reflective Memory Management for Long-term Personalized Dialogue Agents

Introduction

The paper "In Prospect and Retrospect: Reflective Memory Management for Long-term Personalized Dialogue Agents" introduces Reflective Memory Management (RMM) as a novel approach to address the challenges faced by LLMs in maintaining coherence and personalization over long-term interactions. This approach is critical for applications requiring sustained personalization, such as customer service and education platforms, where maintaining conversational continuity can significantly enhance user experience.

Framework Overview

Reflective Memory Management (RMM) integrates two primary mechanisms: Prospective Reflection and Retrospective Reflection. These mechanisms dynamically manage memory granularity and retrieval processes.

Prospective Reflection involves summarizing interactions across various granularities, such as utterances, turns, and sessions, into a personalized memory bank optimized for future retrieval. By dynamically summarizing dialogue interactions, RMM captures the natural semantic structure of conversations, ensuring that the memory bank remains coherent and conveniently retrievable.

Retrospective Reflection refines memory retrieval using reinforcement learning (RL). This mechanism iterates the retrieval based on LLMs' cited evidence during conversations, enabling adaptive retrieval that aligns with diverse dialogue contexts and user interaction patterns.

Implementation Details

Implementing the proposed RMM involves several steps:

1. Memory Bank Setup: Conversations are decomposed into topic-based segments using an LLM, with each segment associated with a corresponding raw dialogue snippet.
2. Retrieval Optimization: The framework utilizes existing dense retrieval models, such as Contriever, Stella, and GTE, for initial semantic retrieval. A reranking process is then employed, where a lightweight model refines the initial retrieval output, selecting the most relevant memory segments.
3. Reranker Training: The reranker applies the REINFORCE algorithm for RL-based updates, leveraging citation signals generated by LLM responses as rewards.

Figure 1: Granularity analysis on randomly sampled 100 instances from LongMemEval with the GTE retriever and Gemini-1.5-Flash generator. \textcolor{bluecolor}{Turn}'' and\textcolor{orgcolor}{Session}'' indicate retrieval at a fixed granularity.

Experimental Results

Experiments conducted on the MSC and LongMemEval datasets reveal significant improvements using RMM. The framework achieved over 10% accuracy improvement compared to baselines without memory management on the LongMemEval dataset, demonstrating its effectiveness in retrieval relevance and personalized response generation.

• Prospective Reflection Performance: Enables more coherent memory structures by organizing topics rather than fixed conversational boundaries, as illustrated in Figure 1.
• Retrospective Reflection Efficiency: Adaptive retrievers utilizing RL updates outperform traditional fixed retrievers by dynamically aligning retrieval strategies with user needs.

Practical Implications

The adoption of RMM has profound implications:

• Long-term Dialogue Consistency: Enhances interaction coherence over extended periods, making it ideal for applications like virtual customer service where continuity is paramount.
• Dynamic Personalization: Adjusts retrieval mechanisms in real-time, facilitating personalization even in dynamic and diverse conversational settings.
• Operational Scalability: Leverages unsupervised citation signals for retrieval refinement, removing the need for extensive labeled data and enabling scalable deployment.

Conclusion

Reflective Memory Management represents a significant advancement in personalized dialogue systems, offering a robust solution to the persistent challenges of memory granularity and retrieval adaptation. By integrating dynamic topic summarization with reinforcement learning, RMM manages to maintain dialogue coherence while adapting to diverse use cases, paving the way for more responsive and personalized AI-driven conversations. Future work could explore extending this framework to multi-modal dialogues and further optimizing the balance between memory richness and retrieval efficiency.