---
title: LongMemEval-S Memory Benchmark
url: https://www.emergentmind.com/topics/longmemeval-s-benchmark
type: topic
---

# LongMemEval-S Memory Benchmark

LongMemEval-S (Long-Term Memory Evaluation—Standard Split) is a benchmark suite designed to rigorously evaluate the long-horizon memory and retrieval capabilities of large language model (LLM)-based conversational agents. It targets tasks that require robust recall, integration, and temporal reasoning over lengthy, multi-session user–assistant dialogues. Developed as part of the LongMemEval initiative by Wu et al. (2024), LongMemEval-S encapsulates the most challenging scenario of context scale, featuring 500 synthetic conversations and 500 manually curated questions embedded in histories averaging approximately 115,000 tokens per instance. The benchmark has become a critical standard for assessing models and memory architectures that operate far beyond the training-time context window, especially for tasks like fact recall, multi-session composition, and dynamic knowledge tracking.

## 1. Benchmark Design and Objectives

LongMemEval-S is constructed to diagnose and stress-test how conversational systems manage scalable, long-term, and contextually complex memory. The benchmark simulates a realistic setting for an AI assistant: accumulating and querying online memory across multi-session dialogues, personalized to individual user personas. Each conversation aggregates a large number of timestamped exchanges, requiring systems to resolve information extraction, knowledge updating, multi-session reasoning, and temporal logic—all within histories that vastly exceed the context window of conventional LLMs [2410.10813][2601.02845][2505.03320].

Task categories covered in LongMemEval-S:

- **Knowledge Update (KU):** Detect and apply corrections or superseding information, e.g., updated addresses or preference changes.
- **Multi-Session Reasoning (MS):** Synthesize or reconcile information distributed across multiple sessions.
- **Single-Session Information Extraction:** Retrieve (i) facts stated by the user (SSU), (ii) facts referenced by the assistant (SSA), (iii) session-limited user preferences (SSP).
- **Temporal Reasoning (TR):** Perform reasoning over explicit dates, durations, or tolerate off-by-one discrepancies (e.g., "How many days since my birthday?").

The underlying motivation is to challenge memory-augmented systems on "needle-in-haystack" scenarios via tasks that mimic long-term user interactions seen in personal assistants and enterprise conversational agents [2410.10813][2501.13956].

## 2. Dataset Structure and Composition

The dataset comprises 500 synthetic conversation streams, each representing a unique user persona (sampled from a template pool covering occupation, hobbies, values, etc.). Each conversation involves multiple sessions (typically 50–60) with extensive multi-turn user-assistant exchanges. Questions are distributed by design, ensuring broad coverage across memory types and temporal axes [2601.02845].

| Task Type        | Number of Questions |
|------------------|--------------------|
| Knowledge Update | 78                 |
| Multi-Session    | 133                |
| SS Assistant     | 56                 |
| SS Preference    | 30                 |
| SS User          | 70                 |
| Temporal Reasoning| 133               |
| **Total**        | **500**            |

During benchmarking, each question is embedded at a strategic point within an exceedingly long dialogue, and evidence for answers may require integrating knowledge from multiple disparate parts of the conversational history. Distractor sessions, derived from external datasets (e.g., ShareGPT, UltraChat), are interleaved to prevent trivial retrieval [2410.10813][2601.02845].

## 3. Evaluation Protocol and Metrics

All evaluated systems are assessed using a uniform LLM-as-Judge (LLJ) pipeline. The protocol involves generating a single-turn answer string for each query, which is then automatically assessed for semantic correctness by a prompt-instructed LLM (typically GPT-4o or GPT-4o-mini), with ">97% human–LLJ agreement" reported [2601.02845].

The primary evaluation metric is accuracy:
\[
\mathrm{Accuracy} = \frac{1}{N}\sum_{i=1}^{N} \mathbf{1}\{\hat y_i = y_i\}
\]
where $N$ is the number of questions, and $\hat y_i$, $y_i$ denote model prediction and ground-truth, respectively.

For systems that expose evidence or memory retrievals (e.g., retriever–reader architectures), the following metrics are sometimes additionally reported on auxiliary slices or ablations:

- **Recall@k:** Fraction of gold evidence retrieved in the top-$k$ results.
- **nDCG@k:** Normalized Discounted Cumulative Gain for the ranking of retrieved evidence.

Latency and efficiency are measured as context size (tokens per query) and recall latency (median and tail, P50/P95), enabling direct comparison of memory consolidation and retrieval trade-offs [2511.12987][2601.02845].

## 4. Memory System Architectures and Baselines

LongMemEval-S serves as the principal arena for comparing memory architectures, including both retrieval-augmented generation (RAG)-style stacks and memory-consolidation frameworks. The following approaches have been directly evaluated:

- **Full-Context LLM (baseline):** The entire 115k-token dialogue is provided to the LLM for each question. This baseline often exhibits pronounced degradation in accuracy (<40%) and incurs prohibitive compute latency [2511.12987][2501.13956].
- **Fact Card Memory Layers (e.g., ENGRAM-R):** Compact, typed fact snippets (Fact Cards) with explicit citation control are retrieved and presented as context, demonstrating marked gains in efficiency and accuracy (+30.1% on MS, +13.5% on TR) [2511.12987].
- **Temporal-Hierarchical Frameworks (e.g., TiMem):** Memory is structured in a temporal memory tree, enabling consolidation from fine-grained episodic details up to high-level persona summaries, and balancing recall precision with efficiency. TiMem achieves 76.88% ± 0.30 overall accuracy and reduces memory context by ≈22.8% compared to strong RAG baselines [2601.02845].
- **Temporal Knowledge Graphs (e.g., Zep/Graphiti):** Episodes and facts are organized in bi-temporal, fact-entity graphs with dynamic timestamp-based validity windows. This enables accurate support for Knowledge Update and Temporal Reasoning queries while reducing average context to ≈1,600 tokens (vs. 115k) and latency by 90% [2501.13956].

Other notable baselines include A-MEM (agentic self-organizing memory), Mem0 (semantic clustering), MemoryBank (preference-weighted), MemoryOS, and MemOS (multi-tier/page-swap memory) [2601.02845].

## 5. Empirical Results Across Systems

Selected results illustrate both the magnitude of the long-context memory challenge and the progress enabled by structured memory frameworks.

| Memory System  | Answer Model           | Accuracy (%)      | Context Tokens  | Latency (s, P50/P95) |
|----------------|-----------------------|-------------------|-----------------|----------------------|
| Full-Context   | GPT-4o-mini           | 55.4              | 115,000         | 31.3 / 8.76          |
| Zep/Graphiti   | GPT-4o-mini           | 63.8              | 1,600           | 3.20 / 1.31          |
| TiMem          | GPT-4o-mini           | 76.88 ± 0.30      | 1,270.62        | 1.76 / 4.48          |
| ENGRAM-R       | gpt-oss-20b           | 59.8              | 54,301 (COG)    | 1.88 / 5.54          |
| MemOS          | GPT-4o-mini           | 68.68 ± 0.16      | 1,091.51        | 1.64 / 2.70          |

When focusing on the two hardest categories, ENGRAM-R demonstrates a 30.1% absolute accuracy gain for multi-session reasoning and a 13.5% gain for temporal reasoning, while cutting input tokens by ≈96% and median latency by ≈81% [2511.12987]. TiMem establishes state-of-the-art (<1.3k memory tokens/query; 76.88% accuracy), outperforming baselines by up to 12.9 percentage points [2601.02845].

Sustained improvements are achieved on temporal reasoning and multi-session logic, but retrieval and recall remain challenging, particularly with assistant-centric queries or under evidence budget constraints [2511.12987][2501.13956].

## 6. Analytical Findings, Ablations, and Limitations

Analysis across evaluated systems surfaces several recurring findings:

- **Signal-to-noise challenge:** Full-context LLMs struggle to isolate relevant segments in ultra-long input, suffering from context overload and “dilution” of evidence signal [2511.12987].
- **Temporal organization:** Explicit modeling of fact validity intervals and temporal hierarchy (memory trees, knowledge graphs) proves critical for knowledge update and time-based inference [2601.02845][2501.13956].
- **Recall and consolidation:** Fact granularity, session decomposition, semantic-guided consolidation, and context gating are essential for bridging precision (avoiding distractors) and recall (capturing all salient evidence) [2410.10813][2601.02845].
- **LLM-as-Judge reliability:** LLM-based meta-evaluation is robust with >97% human alignment, but subject to residual bias—especially salient for high-stakes applications [2511.12987][2601.02845].
- **Resource-efficiency trade-offs:** Retrieval budgets ($k$) present an explicit trade-off; setting $k$ too low may omit critical evidence, while too high increases compute [2511.12987].

Category-level breakdowns indicate that accurate updating, multi-session synthesis, and single-session user recalls are the main axes of progress for leading systems, while temporal-reasoning and assistant-centric queries still invite further refinement [2501.13956][2601.02845]. Manifold analyses of memory representations via distance metrics confirm that semantic consolidation mitigates noise and enables stable persona tracking in the embedding space [2601.02845].

## 7. Broader Impact and Future Directions

LongMemEval-S is now a de facto benchmark for research and development of memory architectures for LLM-based agents in both academic and industrial contexts. Its structure directly supports innovations in:

- **Retrieval-augmented LLMs:** Guiding retrieval and reading policy design under ultra-long context constraints.
- **Temporal reasoning:** Informing fact validity modeling, date arithmetic, and temporal slotting in agent memories.
- **Personalization and long-horizon learning:** Stress-testing approaches to persona stabilization, evolving preference recall, and user identity continuity.

A plausible implication is that further advances will require not only architectural improvements (e.g., adaptive memory budget allocation, distributed memory consolidation) but also more challenging variants (LongMemEval-M), human-in-the-loop judging, and richer evaluation of abstention/uncertainty alongside accuracy [2410.10813][2601.02845]. The emergence of LongMemEval-S as a shared yardstick will continue to sharpen competitive benchmarks and drive the development of lifelong conversational reasoning in AI systems.

Source: https://www.emergentmind.com/topics/longmemeval-s-benchmark