StratMem-Bench: Evaluating Strategic Memory Use in Virtual Character Conversation Beyond Factual Recall

Abstract: Achieving realistic human-like conversation for virtual characters requires not only a simple memorization and recall of past events, but also the strategic utilization of memory to meet factual needs and social engagement. Current memory utilization relevant (e.g., memory-augmented generation, long-term dialogue, and etc.) benchmarks overlook this nuance, treating memory primarily as a static repository of facts rather than a dynamic resource to be strategically deployed in dialogues. To address this gap, we design StratMem-Bench, a new benchmark to evaluate strategic memory use in character-centric dialogues. This dataset comprises 657 instances where virtual characters must navigate heterogeneous memory pools containing required, supportive, and irrelevant memories. We also propose a framework with different evaluation metrics including Strict Memory Compliance, Memory Integration Quality, Proactive Enrichment Score and Conditional Irrelevance Rate, to evaluate strategic memory use capabilities of virtual characters. Experiments on StratMem-Bench which leverage the state-of-the-art LLMs as virtual characters show that all models perform well at distinguishing between required and irrelevant memories, but struggle once supportive memories are introduced into the decision process.

• The paper introduces a benchmark that distinguishes must, nice, and irrelevant memories for strategic virtual conversation.
• It employs a multi-stage annotation pipeline with high inter-annotator reliability (Fleiss’ κ=0.81) and evaluates 9 LLMs using four novel metrics.
• Results indicate robust factual recall but significant challenges in enriching responses, highlighting a trade-off between proactivity and relevance.

StratMem-Bench: A Benchmark for Strategic Memory Selection in Virtual Character Conversations

Motivation and Conceptual Framework

StratMem-Bench advances the evaluation of memory use in virtual character conversations by addressing the fundamental limitation of prevailing memory-augmented benchmarks, which conflate memory utilization with simple factual recall. Human conversation demands not only retrieval and factual grounding but strategic deployment of memories according to dynamic goals: required (“must”), supportive (“nice”), and irrelevant (“irr”). The benchmark operationalizes this distinction, formalizing strategic memory use as the selective integration of must memories to ensure correctness, judicious inclusion of nice memories for enrichment and coherence, and rigorous exclusion of irr memories to prevent topic drift and incoherence. This tripartite categorization is rooted in established pragmatic theory (Gricean maxims) and cognitive architecture research.

The implications for LLM-based agents are significant: contemporary virtual character systems demonstrate robust factual recall but lack mechanisms to strategically select and integrate heterogeneous memories, especially when supportive enrichment is contextually desirable yet not explicitly demanded.

Figure 1: An illustration of memory categorization and response exemplars, contrasting minimal, strategic, and over-inclusive memory use.

Dataset Construction and Annotation Protocol

StratMem-Bench comprises 657 instances, each featuring a synthetic dialogue history, user query, explicit persona, and a memory pool containing items from previous interactions. Dataset construction leverages LoCoMo as a foundational resource, ensuring contextually rich and temporally separated memory acquisition and query formation.

Annotation utilizes a multi-stage pipeline: GPT-5.1 produces initial labels for must/nice/irr memory categorization, followed by triadic expert review and consensus resolution, yielding a reliability of Fleiss’ $\kappa = 0.81$. Role assignment is context-dependent rather than static, requiring annotators to explicitly assess functional relevance relative to query and conversational objectives. Ambiguous cases are discarded to preserve label quality.

Figure 2: Illustration of the construction pipeline transforming LoCoMo dialogues to StratMem-Bench instances.

Evaluation Metrics: SMC, MIQ, PES, and CIR

To quantify strategic memory use, four evaluation metrics are introduced:

• Strict Memory Compliance (SMC): Binary pass/fail for each scenario—requiring all must memories, at least one nice memory when available, and strict irrelevance exclusion. This measures the agent’s ability to satisfy hard constraints for strategic memory selection.
• Memory Integration Quality (MIQ): A failure-sensitive Likert metric (1–5) scored by an LLM with explicit rubrics. This captures integration coherence, penalizing topic drift, forced over-association, factual contradiction, overexpansion, and misattribution.
• Proactive Enrichment Score (PES): Ratio of instances where nice memories are proactively but contextually incorporated, conditional on successful must memory use.
• Conditional Irrelevance Rate (CIR): Fraction of instances where irr memories are erroneously included when nice memories are present.

SMC and MIQ diagnose discrete selection/integration abilities, while PES and CIR characterize behavioral tendencies: proactivity versus risk aversion.

Model Evaluation and Empirical Findings

Nine state-of-the-art LLMs (OpenAI GPT-5.2/Chat, Claude Sonnet 4.5, Gemini 3 Pro, DeepSeek, Llama 4 Maverick, Qwen3) are evaluated under a unified template and controlled scenario composition (downsampled nice memories). Automated memory use detection and MIQ scoring utilize DeepSeek-V3.2, with substantial agreement to human judges ($\kappa = 0.96$ for memory use, $\kappa = 0.69$ for MIQ), minimizing self-evaluation bias.

Core Results

• Must-only scenarios: All models achieve robust SMC ($76\%-92\%$), evidencing reliable factual recall with minimal distractor interference.
• Nice-only and must+nice scenarios: SMC degrades sharply ($46\%-57\%$, $42\%-49\%$), revealing a bottleneck in strategic selection—LLMs systematically under-utilize supportive memories and frequently default to minimal recall or inappropriate enrichment.
• Integration Quality: MIQ for must-used responses averages $4.2$–$4.6$, while nice-used responses drop to $3.9$–$4.4$, indicating a modest but systematic “enrichment tax” attributed to increased failures in memory integration. Responses incorporating irr memories collapse further ($\kappa = 0.96$0–$\kappa = 0.96$1 MIQ), suggesting that irrelevant insertions remain exceptionally disruptive to conversational coherence.
• Proactivity vs. Risk Aversion Trade-off: Models cluster along a Pareto frontier: highly proactive (high PES) models (e.g., Gemini 3 Pro, PES $\kappa = 0.96$2, CIR $\kappa = 0.96$3) enrich responses aggressively but at the expense of increased irrelevance, while conservative (low CIR) models (e.g., GPT-5-chat, CIR $\kappa = 0.96$4) avoid enrichment but risk dryness.

  Figure 3: Scatter plot illustrating the trade-off between memory enrichment and irrelevance, mapping PES against CIR.

Qualitative Case Analysis

Case studies underscore the diversity in model behavior. Strategic responders (Qwen3-235B, Claude Sonnet 4.5) judiciously integrate both must and nice memories, enhancing informativeness. Others (Gemini 3 Pro, DeepSeek-reasoner) exhibit forced over-association or indiscriminate inclusion, undermining response focus. Even with identical memory selection, MIQ differentiates models with respect to their integration strategies and conversational coherence.

Theoretical and Practical Implications

StratMem-Bench demonstrates that current LLMs, while adept at factual recall under memory-augmented conditions, lack principled strategies for selective memory integration in open-ended, character-centric dialogue. The explicit tripartite partitioning and contextual annotation reveal vulnerabilities in adaptive memory control, relevant for multi-session, social, or empathetic agents.

Practically, the benchmark can drive new architectures and training paradigms for memory selection, including instance-specific role inference, attention routing over heterogeneous pools, and reinforcement-based tuning for conversational enrichment. The metrics offer diagnostically granular signals, enabling direct optimization toward context-sensitive memory use and balanced proactivity.

Theoretically, the results challenge assumptions underlying memory-augmented generative paradigms, suggesting that optimization for factual retrieval does not generalize to pragmatic memory deployment. The enrichment tax and trade-off frontier indicate that strategic selection is a higher-order skill distinct from recall and surface coherence.

Conclusion

StratMem-Bench establishes a principled evaluation scheme for strategic memory use in virtual character conversation, operationalizing the distinction between required, supportive, and irrelevant memories. Quantitative and qualitative results show that state-of-the-art LLMs excel in factual recall but struggle to proactively and judiciously enrich responses. By providing both rigorous annotation and multifaceted metrics, the benchmark enables research targeting next-generation agents capable of human-like, goal-sensitive memory deployment in conversational settings. Future extensions to multi-turn and multimodal scenarios are anticipated to further refine the theoretical landscape and practical toolkit for embodied agent memory control.