---
title: LoCoMo and LongMemEval_S Benchmarks
url: https://www.emergentmind.com/topics/locomo-and-longmemeval-_s-benchmarks
type: topic
---

# LoCoMo and LongMemEval_S Benchmarks

LoCoMo and LongMemEval$_S$ are leading benchmarks for evaluating the long-term memory and reasoning capabilities of large language models (LLMs) in extended, multi-session conversational settings. Both benchmarks probe not only factual retention but complex temporal, multi-hop, and cross-session inference in configurations that stress current memory architectures, retrieval schemes, and answer generation protocols. They have become reference datasets in the study of persistent, agentic memory for LLM-based dialog agents.

## 1. Dataset Construction and Composition

**LoCoMo** was initially introduced to address the challenge of evaluating LLMs on *very long-term, persona-grounded, multi-modal* dialogues [2402.17753]. Dialogues feature two generative agents with explicit personas and temporal event graphs, spanning up to 35 sessions, 300+ turns (≈9,000 tokens) per conversation, and include text plus image exchanges. Human annotators edit for consistency, grounding, and event coherence. LoCoMo’s fully constructed dataset contains 50 dialogues, averaging 19.3 sessions each.

The associated *LoCoMo benchmark* (as used in subsequent works) typically comprises between 1,500 and ~2,000 question–answer (QA) pairs [2511.01448, 2511.17208], evenly split among reasoning categories:

- **Single-hop**: direct factual retrieval.
- **Multi-hop**: chaining two or more events/facts across sessions.
- **Temporal**: ordinal or precedence questions, e.g., “When did X do Y?”
- **Open-domain**: requiring external/world knowledge integrated with dialogue content.
- **Adversarial/unanswerable**: designed to probe consistency under misleading/distracting context [2402.17753].

**LongMemEval$_S$** is a widely adopted variant of the LongMemEval benchmark [2410.10813], intended as the “standard” (hence the subscript "S") 500-question suite for scalable multi-session dialogue memory assessment [2511.01448, 2511.17208]. Each instance comprises ≈115,000 tokens of user–assistant chat history over ∼50 sessions. Questions are distributed across six types: single-session-user, single-session-assistant, single-session-preference, multi-session (cross-session reasoning), temporal reasoning, and knowledge update.

There is no distinct “LongMemEval_S” slice in core benchmark sources; rather, LongMemEval$_S$ denotes the standard, full 500-item evaluation used in almost all recent experiments [2511.01448, 2511.17208, 2511.12987].

## 2. Evaluation Protocols and Key Metrics

Benchmarks use LLM-judged answer quality and retrieval metrics tailored to long-horizon dialogue:

**Primary metrics** [2511.01448, 2410.10813, 2511.17208]:

- **QA Accuracy (Acc.)**: fraction of correct answers as determined by a strong LLM grader (e.g., GPT-4o), formalized as $\mathrm{Accuracy} = \frac{1}{N}\sum_{i=1}^N \delta_i$, where $\delta_i=1$ iff the $i$th answer is correct.
- **Recall@k**: fraction where the ground-truth evidence appears in the top-$k$ retrieved memory units, typically $k=15$ for LoCoMo/LongMemEval, $k=5$/$10$ for ablations.
- **QA F1**: harmonic mean of accuracy and recall, or token-level overlap for partial-match evaluation (formally $F_1 = \frac{2\,P\,R}{P+R}$ using answer span tokens).
- **BLEU-1** and **ROUGE**: for summarization and multi-modal tasks (LoCoMo only).
- **Efficiency**: total input tokens per query ($K$), reasoning tokens (chain-of-thought length), and wall-clock query latency ($T_R$).

**Category-specific metrics** address long-horizon reasoning by isolating multi-session, temporal, and knowledge-update tasks [2410.10813, 2511.01448].

## 3. Memory and Retrieval Frameworks

The benchmarks drive research on memory-centric LLM architectures, retrieval, and update methods:

- **Flat RAG**: Dense retrieval over raw session or chunk memories.
- **Hierarchical/Graph-based**: Multi-level memory (LiCoMemory’s CogniGraph), event-centric graphs of elementary discourse units (EMem/EMem-G) [2511.01448, 2511.17208].
- **Temporal-aware filters**: Query expansion, temporal decay weighting, and context compression (e.g., Weibull decay $w(\Delta\tau) = \exp[-(\Delta\tau/\hat\tau)^k]$ for recency modulation).
- **Fact card construction and controlled citation**: Inline justification linking in model output to specific retrieved memory units (ENGRAM-R, [2511.12987]).

Optimizations include session decomposition for retrieval granularity, fact-augmented key expansion (fact lists as retrieval keys), time-aware query expansion (temporal constraints), and structured reading (chain-of-note, JSON augmentation) [2410.10813].

## 4. Benchmark Results and Baseline Comparison

**Result trends across LoCoMo and LongMemEval$_S$**:

- **Event-centric memory (EMem, EMem-G)** [2511.17208]: QA accuracy of 76–85% (gpt-4o-mini, gpt-4.1-mini backbones), with especially strong gains in temporal and multi-session categories.
- **LiCoMemory** [2511.01448]: Outperforms other graph and fact-extraction baselines (Mem0, Zep, A-Mem, LoCoMo-RAG) across both benchmarks, achieving 73.8% accuracy/76.6% recall on LongMemEval (GPT-4o-mini backbone) and consistently reducing latency by 10–40%. Largest margin improvements are in multi-session (+26.6 pp Acc.) and temporal (+15.9 pp Acc.) subsets.
- **ENGRAM-R** [2511.12987]: On LongMemEval$_S$, reduces input token budget by 95.5%, reasoning tokens by 77.8%, and improves accuracy by +21.8 pp overall (notably +30.1 pp in multi-session, +13.5 pp in temporal).
- **Full-context LLMs**: Strong accuracy drop (∼30–55%) when scaling to full dialogue history. RAG over observations lifts QA F$_1$ but saturates quickly with context dilution [2410.10813, 2402.17753].
- **Best-practice findings** from ablations: recall-oriented LLM filters, event-centric memory, hierarchical graph structures, and fact-card citation yield the greatest robustness on long-horizon benchmarks.

| Model/Method     | LoCoMo Accuracy | LongMemEval$_S$ Accuracy | Retrieval Efficiency | Temporal/Multi-Session Gains |
|------------------|----------------|--------------------------|---------------------|------------------------------|
| Full-context LLM | ∼30–37%        | ∼38–40%                  | N/A                 | Poor                         |
| Hierarchical (e.g., LiCoMemory) | ∼67–73%     | ∼69–76%         | ✓                   | +15–26 pp                    |
| Event-centric (EMem-G)           | 78–85%       | 77–85%                   | ✓                   | ✓                            |
| ENGRAM-R         | 76%            | 60% (overall slice)      | Superior            | +13–30 pp                    |

*All numbers drawn explicitly from benchmark leaderboards in [2511.01448, 2511.12987, 2511.17208, 2410.10813].*

## 5. Core Challenges and Error Modes

Analysis across studies highlights persistent challenges:

- **Temporal Reasoning**: LLMs are consistently weakest when temporal disambiguation, event ordering, or recency are required. Even with optimized retrieval or long context, temporal accuracy lags by ≥50 pp versus humans [2402.17753, 2410.10813].
- **Multi-session Aggregation**: Chaining facts or resolving state changes over distant sessions results in substantial error, especially for generic RAG or full-history approaches.
- **Update and Consistency**: Detecting entity attribute changes and resisting distraction from irrelevant, stale, or contradictory evidence is nontrivial. Memory frameworks with hierarchical and update-tracking logic (LiCoMemory, EMem-G) outperform flat or purely dense retrieval approaches.
- **Context “noise” and dilution**: Larger retrieval windows or chunk-based RAG lead to accuracy saturation or degradation as more unrelated content is introduced [2410.10813].
- **Efficiency vs. Quality trade-off**: Practical inference requires compact, information-rich evidence pools; best results use ≤1,000 tokens for QA input from memory, far below full-history context [2511.12987, 2511.17208].

## 6. Practical Implications and Methodological Trends

Long-term conversational memory benchmarks have catalyzed methodological advances:

- **Memory modularization**: Three-stage frameworks splitting indexing, retrieval, and reading stages with explicit control points for value chunking, keying, and query transformation are now standard [2410.10813].
- **Recall-biasing**: LLM-based filters tuned for high recall outperform manual or score-based pruning.
- **Temporal and event representations**: Structured memory arranges atomic facts/events with timestamps and causal/participant links to enable associative and time-sensitive retrieval.
- **Structured prompts and answer generation**: Forcing citation, structured answer formats (JSON, chain-of-note), and answer rationales improves answer faithfulness and efficiency.
- **Ablation studies**: Removal of LLM-based filters or structured event representations causes 4–8 percentage point drops in QA accuracy, underscoring their necessity [2511.17208].

A plausible implication is that future systems will require highly structured, timestamped, event-oriented persistent memories tightly integrated with dense and symbolic retrieval, along with adaptive answer construction modules.

## 7. Benchmark Evolution and Limitations

LoCoMo and LongMemEval$_S$ continue to evolve:

- **Multi-modal and world knowledge integration**: LoCoMo explicitly incorporates images and open-domain reasoning, but persistent performance gaps remain in multi-modal and adversarial categories [2402.17753].
- **No true “LongMemEval_S” split**: The nomenclature “LongMemEval_S” is best interpreted as the canonical 500-question suite, not a distinct subset or configuration [2511.01448].
- **Benchmark saturation**: Human-level performance remains a distant target, especially in temporal/multi-session categories (temporal: 20.3 for GPT-3.5-turbo-16K vs 92.6 for humans; adversarial: 2.1 for LLMs vs 89.4 for humans [2402.17753]).
- **Cross-paper consistency**: Recent studies have synchronized protocols and categories, ensuring comparability across graph/event-centric and fact-extraction methods.
- **Enduring error lines**: Long-horizon dialogue reasoning, state tracking, and multi-modal integration are ongoing bottlenecks for current architectures.

These benchmarks are now indispensable for research into scalable, reliable, and temporally sophisticated conversational agents and memory-augmented LLMs. Their impact is evident in the rapid dissemination and benchmarking of agentic memory architectures in the dialog systems literature [2511.01448, 2511.17208, 2410.10813, 2511.12987, 2402.17753].

Source: https://www.emergentmind.com/topics/locomo-and-longmemeval-_s-benchmarks