---
title: 'LongMemEval: Benchmark for LLM Memory'
url: https://www.emergentmind.com/topics/longmemeval-dataset
type: topic
---

# LongMemEval: Benchmark for LLM Memory

LongMemEval is a large-scale benchmark designed for systematic evaluation of long-term, interactive memory in large language model (LLM)-driven chat assistants. By embedding memory-intensive queries within freely scalable, task-oriented multi-session chat histories, LongMemEval enables rigorous assessment of five key long-term memory abilities: information extraction, multi-session reasoning, temporal reasoning, knowledge updates, and abstention. It establishes a challenging and realistic data regime for longitudinal memory benchmarking and provides a unified experimental framework, including optimizations for indexing, retrieval, and reading. The dataset, experimental protocols, and code are publicly available to facilitate advances in memory-augmented LLM systems.

## 1. Dataset Design and Structure

LongMemEval is constructed to emulate realistic prolonged user–assistant interaction scenarios. Each test sample comprises:

- A lengthy, ordered sequence of session tuples $(t_1, S_1), (t_2, S_2), \dots, (t_N, S_N)$, where each $S_i$ is a multi-turn, task-oriented dialogue and $t_i$ is an associated timestamp.
- One corresponding question paired with a unique answer that requires evidence from the provided history.

Key dataset properties include:

- **Scale:** The standard LongMemEval_S configuration contains histories of approximately 115,000 tokens per instance, while LongMemEval_M features up to 500 sessions and 1.5 million tokens per instance.
- **Data Generation:** Sessions are generated using a “self-chat plus human editing” pipeline. Questions are designed via an attribute-controlled curation process ensuring diverse memory requirements.
- **Needle-in-Haystack Metric:** Questions embed relevant evidence indirectly among distractor content, simulating real difficulty in memory retrieval.
- **Access:** Data and code are publicly released at https://github.com/xiaowu0162/LongMemEval.

## 2. Memory Abilities Evaluated

LongMemEval is organized explicitly to benchmark five core abilities required for robust long-term interactive memory in LLM-based chat assistants:

| Ability                | Description                                                                                             |
|------------------------|---------------------------------------------------------------------------------------------------------|
| Information Extraction (IE)    | Recall of explicit, specific details mentioned during user–assistant interactions                |
| Multi-Session Reasoning (MR)   | Aggregation and synthesis of evidence distributed across several distinct sessions               |
| Temporal Reasoning (TR)        | Interpretation and manipulation of temporal information, including timestamps and inferred times |
| Knowledge Updates (KU)         | Correct handling of updates—tracking and revising user/assistant knowledge changes over time     |
| Abstention (ABS)               | Controlled refusal to answer when necessary evidence is absent or question is ill-posed          |

These categories reflect not only low-level memory (fact recall) but also higher-level cognitive phenomena necessary for personalized and temporally coherent AI interaction.

## 3. Challenges, Baseline Performance, and Degradation

LongMemEval reveals that state-of-the-art commercial and open-source chat models—when tasked with multi-session memory recall in contexts scaled beyond prior benchmarks—suffer pronounced performance drops:

- Accuracy decreases by 30–60% vis-à-vis an oracle retrieval setting (i.e., if presented with only the evidence).
- Notable failure cases include loss of detail in the “middle” of a long chat sequence (“lost-in-the-middle”), failure to aggregate temporally distributed information, and over-reliance on shallow pattern matching.
- Empirical evaluation of models such as ChatGPT and Coze demonstrates that scaling interaction length consistently exacerbates these challenges.
- Even when advanced dense retrievers are integrated, correct synthesis of evidence remains a limiting factor on end-to-end performance.

These results document the depth of the challenge posed by realistic longitudinal memory requirements and question the sufficiency of naïve long-context capabilities for real interactive agents.

## 4. Unified Memory Framework and Optimizations

LongMemEval formalizes a modular architecture for LLM-based chat assistant memory, partitioned into three stages:

1. **Indexing:** Each session (or its sub-components) is converted into one or more key–value memory entries.
2. **Retrieval:** Given a query, a retrieval mechanism fetches the top-$k$ candidate memory items from the memory store.
3. **Reading:** The LLM generates a final answer by reasoning over the content of retrieved memory entries.

Within this framework, LongMemEval introduces and evaluates several key optimizations:

- **Session Decomposition (Value Granularity):** Granularity of storage is compared at three levels: whole session, round, and per-user-fact. Decomposing to rounds or extracting user facts notably boosts retrieval-augmented generation performance; for MR queries, further fact extraction is necessary.
- **Fact-Augmented Key Expansion (Indexing):** Rather than indexing by raw text, extracted user facts are concatenated to the value, yielding multiple key “paths” that improve retrieval recall (by ~4% recall@k) and downstream QA accuracy (by ~5%).
- **Time-Aware Query Expansion (Retrieval):** For temporally grounded queries, an LLM parses the time range from the question and restricts retrieval to relevant session blocks—enhancing recall by 7–11% on temporal reasoning tasks.
- **Reading Strategies (Synthesis):** Employing Chain-of-Note (CoN) reasoning—extracting key supporting evidence from each retrieved memory and prompting in a structured JSON format—yields an absolute improvement of up to 10 percentage points in answer accuracy.

## 5. Experimental Analysis and Results

Experiments in LongMemEval employ several model classes—commercial LLMs (e.g., GPT–4o), high-capacity open-source models (Llama 3.1-Instruct, Phi–3 Medium), and state-of-the-art dense retrievers (Stella V5 1.5B):

- **Key Results:**
    - Long-context and memory-augmented LLMs exhibit 30–60% accuracy drop relative to the oracle.
    - Granularity optimization—decomposing to rounds and using fact augmentation—improves recall and NDCG in retrieval, as well as downstream QA.
    - For temporal reasoning, time-restricted retrieval delivers measurable increases in success rate.
    - Enhanced reading (CoN & structured prompts) is essential for synthesizing scattered retrievals, raising correctness up to 10 points.
- **Ablation Studies:** Systematically demonstrate each aforementioned optimization’s contribution, enabling diagnostic understanding of failure modes.

These benchmarks quantitatively substantiate the value of architectural decisions in memory-augmented systems.

## 6. Broader Implications and Related Research

The methodology and findings of LongMemEval have substantial implications:

- **Towards Realistic Memory Benchmarks:** Unlike prior long-context QA datasets limited to passive or “needle-in-haystack” retrieval, LongMemEval introduces a realistic multi-session conversation regime, emphasizing temporal evolution, knowledge change, and abstention.
- **Unified Memory Design Schema:** The tripartite framework—indexing, retrieval, and reading—has been adopted or extended in recent works on reflective memory management [2503.08026] and continual learning evaluation [2510.17281].
- **Research Drivers:** The persistent performance gap relative to oracle retrieval underscores ongoing challenges in scaling retrieval and reasoning, and the necessity to optimize LLM-based agents for dynamic, evolving user interaction histories rather than static QA.

A plausible implication is the need for further architectural innovation in memory store representations, retrieval methods, and strategies for information synthesis under high noise and temporal drift.

## 7. Dataset Release and Future Directions

All LongMemEval data, annotation protocols, and experimental code are open source at https://github.com/xiaowu0162/LongMemEval. The benchmark’s extensible design supports ongoing research on:

- New retrieval mechanisms and hybrid memory models
- Advanced temporal and logical reasoning in long-term dialogue
- Techniques for compressive or hierarchical memory representations

LongMemEval provides a standardized, challenging testbed for the longitudinal memory dimensions of LLM-based conversational agents, catalyzing progress in both academic and applied research into long-lived, personalized AI assistants.

Source: https://www.emergentmind.com/topics/longmemeval-dataset