ES-MemEval Benchmark Suite
- ES-MemEval is a benchmark suite designed to assess conversational agents on five memory capabilities in multi-session emotional support dialogues.
- It operationalizes tasks across question answering, summarization, and dialogue generation to measure information extraction, temporal reasoning, conflict detection, abstention, and user modeling.
- Empirical findings demonstrate that full history access and retrieval-augmented generation improve factual consistency and reduce hallucinations in personalized dialogue systems.
ES-MemEval is a benchmark suite specifically designed to evaluate conversational agents on five essential memory capabilities—information extraction, temporal reasoning, conflict detection, abstention, and user modeling—in the context of long-term, personalized emotional support dialogues. Unlike prior benchmarks, ES-MemEval targets the challenges implicit in real-world emotional support: tracking evolving, often fragmented and implicit user disclosures across month-spanning multi-session dialogues, and assessing agents not just on explicit fact retrieval but on their ability to reason over, summarize, and generate supportive dialogue that leverages complex, dynamic user memory (Chen et al., 2 Feb 2026).
1. Scope, Motivation, and Capabilities
The ES-MemEval suite is motivated by limitations observed in existing long-term dialogue benchmarks (e.g., MemoryBank, PerLTQA, LongMemEval), which are primarily concerned with static, explicit personal fact retrieval in question answering (QA) format, with little evaluation of abstraction, contradiction handling, or abstention. Emotional support (ES) dialogues require agents capable of:
- Information Extraction (IE): Identifying and extracting salient user facts from long, multi-session dialogues (e.g., “User has two younger siblings”).
- Temporal Reasoning (TR): Inferring temporal order and causality of user events (e.g., “started therapy in March,” “felt better in June”).
- Conflict Detection (CD): Identifying and resolving memory contradictions (e.g., user expresses dislike for public speaking in one session and enjoyment in another).
- Abstention (Abs): Withholding answers in the absence of sufficient information (“Cannot answer” when favorite childhood toy is never mentioned).
- User Modeling (UM): Inferring and continually updating traits, background, and emotional states to support personalized, evolving responses (Chen et al., 2 Feb 2026).
2. Task Structure and Formats
ES-MemEval operationalizes these capabilities within three core tasks:
| Task | Memory Capabilities Assessed | Input Structure |
|---|---|---|
| Question Answering | IE, TR, CD, Abs, UM | All past sessions (up to 128K tokens) + question |
| Summarization | TR, UM | 2–6 grouped sessions + summarization query |
| Dialogue Generation | All five + ES best practices | Structured scenario: profile, timeline, relevant sessions, user psychological state, prompt |
- QA: Includes single/multi-hop retrieval, contradiction-detection, and abstention subtasks, requiring synthesis across evidence dispersed over long interaction histories. Example: “When did the user start feeling less anxious? Provide the month and year.”
- Summarization: Cross-session abstraction tasks such as summarizing state evolution (“Summarize career concerns from Session 1 to 10”).
- Dialogue Generation: Requires scenario-driven agent response creation, integrating past memory and emotional support principles (“Continue the conversation with empathy after the user’s promotion was delayed”).
3. EvoEmo Dataset Construction
ES-MemEval is built upon EvoEmo, a multi-session synthetic dataset designed to simulate realistic, privacy-compliant ES interaction patterns:
- User Modeling: 18 virtual users with demographic and personality attributes, and core beliefs.
- Session Statistics: Average 22.3 sessions/user, 23.4 turns/session, mean span 14.9 months, ≈13.3K tokens/conversation.
- Event Timeline: Each profile augmented with ≈24.8 temporally and causally coherent life events per user, generated by GPT-4o and curated by humans.
- Dialogue Creation: GPT-4o produces multi-turn sessions conditioned on current profile and event state; six human annotators review for implicitness, coherence, and emotional realism, annotating emotion, topic, and user observation per turn.
- Annotation Focus: Event-based labels tag fragmented self-disclosures requiring cross-session aggregation, operationalizing implicit memory challenges (Chen et al., 2 Feb 2026).
4. Evaluation Protocols and Metrics
ES-MemEval applies task-specific quantitative and LLM-based subjective metrics:
- QA:
- Precision, recall, and F1-score at token-level overlap:
- BERTScore for semantic similarity
- GPT-4o as judge (0/1/2 for wrong/partial/correct)
- Retrieval accuracy: Recall@k and nDCG@k
- Summarization:
- ROUGE-N (N=1,2), ROUGE-L
- Event-based precision/recall/F1, and LLM faithfulness/coverage (GPT-4o, 0–5 scale)
- Dialogue Generation:
- Observation recall and weighted accuracy (accounts for partial/full relevance)
- GPT-4o LLM-ratings for long-term memory, personalization, and emotional support (1–5 scale)
- Novel Metrics: Conflict detection QA (contradiction accuracy), abstention QA (abstention-F1) (Chen et al., 2 Feb 2026).
5. Experimental Protocols and Baselines
The evaluation spans multiple modeling paradigms:
- Open-Source Long-Context LLMs: Mistral-8B-Instruct (8B), Phi-3-Medium-128k (3B), Mistral-24B-Instruct (24B), with history lengths up to 128K tokens.
- Commercial LLMs: gpt-3.5-turbo (4K tokens), gpt-4o (16K tokens).
- Retrieval-Augmented Generation (RAG): Dense retriever (bge-m3) via FAISS, retrieving top-4 units (session/turn/round level) per prompt.
- Hardware: All experiments conducted on a single NVIDIA A100 80GB GPU.
- Hyperparameters and prompts: Publicly available at the project repository (Chen et al., 2 Feb 2026).
6. Major Findings and Insights
Several key empirical results are documented:
- Explicit memory reduces hallucination: Models with “no memory” fabricate personalized details (observation-based recall < 0.30); access to full histories increases recall to 0.31–0.35, and RAG raises this further to 0.34–0.38.
- RAG improves factual consistency but not temporal dynamics: QA scores—F1 up 2–5 points, BERTScore up 3–7—especially for smaller models. However, temporal reasoning F1 remains below 20%, indicating persistent challenge in ordering and causal inference.
- Personalization correlates with memory: LLM ratings for personalization and long-term memory are strongly correlated (Spearman ρ > 0.7). Emotional support scores are less dependent on explicit memory.
- Retrieval granularity effects: Session-level retrieval (k=4) is superior to turn- or round-level for QA Quality (LLM-as-Judge mean 1.27 vs. 1.15/1.20).
- Context length limitations: Mistral-8B degrades past 8K context, peaking at 2K; Mistral-24B is stable up to 20K.
- RAG narrows open-source/commercial gap: Mistral-24B+RAG approaches GPT-3.5-turbo+RAG on QA/Summarization, and gpt-3.5/gpt-4o also gain 1–2 points from RAG (Chen et al., 2 Feb 2026).
7. Limitations and Future Directions
Identified limitations include:
- Dataset scale and realism: EvoEmo’s synthetic nature and modest user count (18) limit representativeness, though its session/task length exceeds contemporary ES benchmarks.
- Topic balance and diversity: Self-growth topics are underrepresented; cultural diversity is not modeled.
- Retrieval configurations: Only dense retrieval (bge-m3) and fixed granularity were used.
- Suggested future work:
- Retrieval-aware calibration: Joint training for re-ranking/fusion to handle temporality and implicitness.
- Adaptive memory granularity: Dynamic dialogue chunking (observation summaries, compressed schemas).
- Hybrid memory-retrieval: Architectures combining explicit memory graphs with RAG for contradiction and event reasoning.
- Dataset expansion: Increase user count, topic diversity, and—pending ethics review—incorporate real anonymized ES data.
- Advanced metrics: Develop temporal consistency and conflict-resolution scores for fine-grained evaluation (Chen et al., 2 Feb 2026).
ES-MemEval, with its systematic, multi-faceted task design and the EvoEmo dataset, constitutes a reproducible and discriminative testbed for advancing memory-robust, personalized long-term emotional support dialogue systems.
