---
title: 'LOCOMO: Long-Term Conversational Memory Benchmark'
url: https://www.emergentmind.com/topics/locomo-1b7af604-e910-443f-a64e-87a74b0b1021
type: topic
---

# LOCOMO: Long-Term Conversational Memory Benchmark

LoCoMo—also written as LOCOMO in parts of the later literature—is a benchmark and dataset family for evaluating very long-term conversational memory in LLM agents. It was introduced to measure whether models can retain and use information across multi-session dialogues grounded in persistent personas, temporal event structure, and multimodal interaction, rather than merely handling short-context dialogue coherence. In the agent-memory literature, LoCoMo functions both as a dataset of long conversations and as an evaluation suite spanning question answering, event summarization, and multimodal dialogue generation; later work also uses LoCoMo as a substrate for retrieval studies, paging systems, contradiction resolution, state-aware memory, and cognitive-memory extensions [2402.17753].

## 1. Origin, construction, and underlying formalism

The original LoCoMo resource was created through a machine-human pipeline designed to generate very long-term conversations with explicit longitudinal structure. Each virtual speaker is assigned a persona statement \(p\), beginning from short persona descriptions \(p_c\) from the MSC dataset and expanded with **gpt-3.5-turbo** into fuller profiles. Each speaker is also grounded in a temporal event graph \(\mathcal{G}\) whose nodes are events \(e_i\) associated with dates \(t_i\), with causal edges written as \(l=(e_i,e_j)\). The graph-generation process starts with \(k=3\) events and can produce **up to 25 events** distributed across a **6–12 month** period, so that later dialogue can remain grounded in evolving personal history rather than only recent context [2402.17753].

The generative-agent architecture used in the dataset construction separates short-term and long-term memory. Short-term memory \(\mathcal{H}_s\) stores session summaries \(w_k\), while long-term memory \(\mathcal{H}_l\) stores observations \(o_{k_j}\) extracted from dialogue turns. After each session \(k\), the agent generates a summary \(w_k\) conditioned on current history \(h_k\) and the previous summary \(w_{k-1}\). When responding in session \(k+1\), the agent conditions on the latest summary, retrieved observations, persona \(p\), current-session history, and the subset of event-graph events satisfying
\[
\{e \in \mathcal{G}\mid t_k^s < t_i^e < t_{k+1}^s\}.
\]
This design makes temporal continuity and causal linkage part of the conversation-generation process rather than post hoc annotations [2402.17753].

LoCoMo is multimodal in its original form. Agents can share images by first generating a caption \(c\), converting it to keywords \(w\), retrieving an image via \(WEB(k)\), and then sharing the selected image. For image reaction, **BLIP-2** is used to caption the image before the LLM generates a reaction grounded in both the image and the speakers’ personas. Human annotators subsequently edit the generated conversations for long-range consistency and grounding accuracy; the original paper reports edits to **nearly 15% of dialogue turns** and removal or substitution of about **19% of images** [2402.17753].

The original dataset statistics emphasize scale. The paper reports **50 conversations**, **average 19.3 sessions per conversation**, **average 15.8 turns per session**, **average 300.9 turns**, **average 9,209.2 tokens per conversation**, **up to 35 sessions**, **average 24.2 ground-truth events per conversation**, and **average 32.3 images per conversation**. Additional statistics include **30.2 tokens per turn**, **18.2 tokens per observation**, **127.4 tokens per session summary**, and **896.5 tokens** for average event-summary length [2402.17753].

## 2. Benchmark structure and evaluation regimes

LoCoMo was introduced with three task families: **question answering**, **event summarization**, and **multimodal dialogue generation**. In the QA task, questions are grouped into five reasoning types: **single-hop**, **multi-hop**, **temporal reasoning**, **open-domain knowledge**, and **adversarial**. Event summarization is evaluated against the underlying temporal event graph \(\mathcal{G}\), with a FactScore-style decomposition into precision, recall, and \(F1\). Multimodal dialogue generation is evaluated with **MMRelevance**, alongside standard NLG metrics in appendix experiments [2402.17753].

In later agent-memory work, LoCoMo is often used in a narrower operational form as a long-term conversational memory QA benchmark. One common description is a benchmark over **10 conversations**, **19–32 sessions**, about **9K tokens per dialogue**, and **1,986 QA pairs**, with five question types: **Single-hop (SH)**, **Multi-hop (MH)**, **Temporal (T)**, **Open-ended (O)**, and **Adversarial (A)**. In that setting, the primary official metric is **token-level F1 with Porter stemming**, described as the standard token-overlap F1 used in QA evaluation [2604.01007].

Other papers evaluate specific slices rather than the full benchmark. Some report the **1,540-question** four-category retrieval subset that excludes adversarial questions, while others use an **answerable-factual pool** of **1,444** LoCoMo questions restricted to **single-hop**, **temporal**, and **open-domain factual** items. Probe-based studies sometimes evaluate only **98** or **100** questions from a **LoCoMo-10 subset**, and local-memory studies sometimes use **304 QA pairs** from **2 of 10 LoCoMo conversations** [2603.17244] [2606.06240] [2604.12376] [2604.04514].

| Evaluation regime | Reported scope | Primary metric(s) |
|---|---|---|
| Original LoCoMo | QA, event summarization, multimodal generation | F1, FactScore-style precision/recall/F1, MMRelevance |
| Official QA usage in later memory papers | 10 conversations, 1,986 QA pairs | Token-level F1 with Porter stemming |
| Retrieval-focused subset | 1,540 questions | Token-level F1, refusal accuracy |
| Answerable-factual slice | 1,444 questions | Accuracy |
| Probe subsets | 98–100 probes or 304 QA pairs | LLM-judged 1–5 scores or judged accuracy |

This variation in setup is methodologically important. Later papers evaluate LoCoMo with **token-level F1**, **BLEU-1**, **judged QA accuracy**, **LLM-as-a-Judge accuracy**, and retrieval metrics such as **Hit@1**, **MRR**, and **NDCG@5**. The literature therefore uses LoCoMo both as an end-to-end memory benchmark and as a retrieval-stage benchmark, with metric choice shaping what is being measured [2603.17244] [2507.07957] [2606.10842] [2606.04194].

## 3. What LoCoMo measures and the failure modes it exposes

LoCoMo was created to test memory over spans much longer than the dialogue benchmarks that preceded it. The original paper argues that existing long-term dialogue datasets were still too short to stress memory over weeks or months, and its experimental results show that LLMs struggle with lengthy conversations and with long-range temporal and causal dynamics. Human performance on the original QA evaluation is reported as **87.9** overall F1, while the best base-model results remain substantially lower, even when long-context or retrieval-augmented methods are used [2402.17753].

A consistent finding across later papers is that LoCoMo stresses more than surface retrieval. The benchmark repeatedly exposes failures involving **temporal markers**, **implicit coreference**, **ephemeral updates**, and answer formatting that is mismatched to the evaluation protocol. One cost-performance study attributes the large gap between full-history long-context inference and a fact-based memory system on LoCoMo to information lost during compression, especially **temporal markers**, **implicit coreference**, and **ephemeral updates** [2603.04814]. Omni-SimpleMem further reports that verbose natural-language outputs can “destroy” token-level F1, so benchmark-specific answer style and formatting materially affect results [2604.01007].

The original LoCoMo paper already showed that retrieval quality depends strongly on the unit being retrieved. Retrieving compact **observations** rather than raw dialogue or summaries improved GPT-3.5 QA by about **5%** over raw dialogue retrieval, while retrieving too many items degraded performance by lowering the signal-to-noise ratio. Session summaries were less effective than observations because summarization lost details, and open-domain questions could be harmed by distracting retrieved context [2402.17753].

Later work has sharpened the benchmark’s diagnostic role. A state-aware memory paper argues that LoCoMo is useful as an external generalization check but is **not designed to isolate old/current state coordination** in the way its conflict-heavy extension LTP is. A plausible implication is that LoCoMo is strong at exposing broad long-horizon memory failures, but weaker at disentangling bank-level, retrieval-level, and answer-level errors unless additional instrumentation is added [2607.01935].

## 4. Systems evaluated on LoCoMo and reported performance

The LoCoMo literature contains a large number of systems, but their numbers are not directly interchangeable because papers report different metrics and sometimes exclude categories such as adversarial questions. The table below therefore lists representative results together with the protocol used, rather than treating them as a single leaderboard.

| System | Reported LoCoMo protocol | Reported result |
|---|---|---|
| Omni-SimpleMem | Official token-level F1 | **0.598** final, from **0.117** baseline |
| Kumiho | Token-level F1 on retrieval subset; refusal on adversarial | **0.447** four-category F1; **0.565** overall F1; **97.5%** refusal accuracy |
| TiMem | LLM-as-a-Judge accuracy; also F1 and ROUGE-L | **75.30%** accuracy; **F1 = 54.40**; **ROUGE-L = 54.68** |
| MIRIX | GPT-4.1 judge accuracy, adversarial excluded | **85.38%** overall |
| D-Mem | F1 with GPT-4o-mini | **53.5** for Quality Gating; **55.3** for Full Deliberation |
| Synthius-Mem | GPT-4.1-mini binary judging | **94.37%** overall; **99.55%** adversarial robustness |

Omni-SimpleMem is notable because the paper documents the optimization path rather than only the final score. Starting from a **naive SimpleMem baseline** at **F1 = 0.117**, the autonomous research loop reached **0.598** on LoCoMo after nine successful iterations and two reverted experiments. The most consequential changes were a missing `response_format` fix (**0.117 \(\to\) 0.322**), hybrid **BM25 + FAISS** retrieval with set-union merging (**0.322 \(\to\) 0.464**), anti-hallucination prompting (**0.516**), evaluation-format alignment (**0.543**), and repair of **4,277 MAU timestamps** corrupted to the ingestion date, which were corrected at **99.98%** accuracy without re-ingestion (**0.543 \(\to\) 0.580**). The final gain to **0.598** came from adaptive top-\(k\) and metadata [2604.01007].

Kumiho frames LoCoMo as a retrieval benchmark rather than a generative benchmark. It reports **0.447** four-category token-level F1 over **\(n=1{,}540\)** retrieval questions, per-category scores of **0.462** single-hop, **0.355** multi-hop, **0.533** temporal, and **0.290** open-domain, plus **97.5%** adversarial refusal accuracy on **\(n=446\)** adversarial questions. Including the adversarial portion, the paper reports **0.565 overall F1** on the full **\(n=1{,}986\)** set. The system attributes this performance to a dual-store **Redis + Neo4j** graph-native architecture with immutable revisions, mutable tag pointers, typed provenance edges, and hybrid BM25/vector retrieval fused as
\[
S(q,m)=w(m)\cdot \max\!\bigl(\mathrm{BM25}(q,m),\ \beta\cos(\mathbf e(q),\mathbf e(m))\bigr),
\]
with \(\beta=0.85\) [2603.17244].

TiMem organizes memory as a five-level **Temporal Memory Tree** and reports **75.30% \(\pm 0.16\%\)** LLM-as-a-Judge accuracy on LoCoMo, together with **Single-Hop 81.43**, **Temporal 77.63**, **Open-Domain 52.08**, **Multi-Hop 62.20**, **F1 = 54.40**, and **ROUGE-L = 54.68**. The paper also reports a **52.20%** reduction in recalled memory length on LoCoMo, from **1,070.10** tokens per query for Mem0 to **511.25** for TiMem [2601.02845].

MIRIX evaluates LoCoMo under a memory-only protocol: the conversation is injected into memory, the chat agent does not receive the original transcript at query time, and answers are scored by **GPT-4.1** as an LLM judge. In that setup MIRIX reports **85.38% overall accuracy**, with **85.11** single-hop, **83.70** multi-hop, **65.62** open-domain, and **88.39** temporal. The paper excludes the adversarial category for fairness with earlier work [2507.07957].

D-Mem treats LoCoMo as a test of whether a fast retrieval path should be augmented with an exhaustive fallback. On **GPT-4o-mini**, the paper reports **Mem0\(^\ast\) = 51.2 F1**, **Full Deliberation = 55.3**, and **Quality Gating = 53.5**, while using **12,681** tokens for Quality Gating versus **35,435** for Full Deliberation. The paper describes Quality Gating as recovering **96.7%** of Full Deliberation’s performance at substantially lower cost [2603.18631].

Synthius-Mem reports the highest figure in the provided literature, but under a different protocol. Using **GPT-4.1-mini** as both answer model and judge on the full LoCoMo dataset of **10 conversations**, **20 participants**, and **1,813 questions**, it reports **94.37% overall accuracy**, **96.73%** single-hop, **94.34%** multi-hop, **89.32%** temporal, **77.33%** open-domain, **99.55%** adversarial robustness, **98.64%** core memory fact accuracy, and **94.40%** temporal precision. The paper explicitly replaces token-overlap F1 with binary LLM judging, so these results should be read as protocol-specific rather than directly comparable with official F1-based scores [2604.11563].

## 5. Extensions, diagnostic variants, and retrieval-focused uses

LoCoMo has also served as the basis for several extensions that move beyond direct factual recall. LoCoMo-Plus redefines the task from explicit answer recovery toward what it calls **beyond-factual cognitive memory**. It introduces **cue–trigger semantic disconnect**, where a later query depends on an earlier latent constraint but is not semantically similar to the original cue. The formal distinction is between Level-1 factual memory and Level-2 cognitive memory, with valid outputs defined as
\[
\mathcal{A}_c = \{ a \mid a \text{ is consistent with } c \}.
\]
This reframes memory evaluation around **constraint consistency** rather than string overlap, and the paper argues that conventional task disclosure and string-matching metrics are misaligned with such cases [2602.10715].

A-TMA uses LoCoMo as an external generalization benchmark while introducing **LTP (LoCoMo Temporal Plus)** as a conflict-heavy benchmark for **ghost memory**. LTP is built from **10 profiles**, with **40 mutable slot instances per profile**, **400 state units**, **800 cue events**, and **800 evaluation probes**. On LoCoMo itself, the headline result is for **Graphiti/Zep + A-TMA**, where **temporal F1** rises from **0.0295** to **0.1705** and **average F1** from **0.0809** to **0.1556**. The paper emphasizes that these gains are **host- and metric-dependent** [2607.01935].

Paging and memory-virtualization studies use LoCoMo differently. Cooperative paging compresses evicted conversation segments into minimal keyword bookmarks such as \([pN:\text{keywords}]\) and exposes a `recall()` tool. On the **LoCoMo-10 subset**, **Bookmark+Recall** on **GPT-4o-mini** scores **2.18** on a 1–5 judged scale, above **Full context 2.02**, **Search-tool baseline 1.90**, **Word-overlap retrieval 1.88**, **BM25 retrieval 1.86**, and **Truncation 1.64**. In the multi-judge evaluation, **Bookmark+Recall vs. BM25** yields \(\Delta=+0.34\) with \(p=0.017\), and **vs. Full Context** yields \(\Delta=+0.27\) with \(p=0.047\) [2604.12376].

Write-time memory-correctness research also uses LoCoMo as a natural-workload slice. TOKI evaluates a typed contradiction-resolution layer with an audit-row schema and reports that the **audit-row defense** moves LoCoMo by **\(\Delta = +0.86\)** with CI **[0.76, 0.94]** on a matched mechanism slice. In the paired memory ablation on **1,444 answerable LoCoMo questions**, performance drops from **0.540** with the typed memory layer to **0.048** without it, for a paired delta of **+0.492**, **95% CI [+\(0.465\), +\(0.519\)]**, and **McNemar \(p<10^{-4}\)** [2606.06240].

LoCoMo is also widely used as a pure retrieval-stage benchmark. A training-free lexical-dense fusion study formulates it as a session-ranking problem and reports that fusing late-interaction dense retrieval with BM25 yields **Hit@1 = 0.752** and **NDCG@5 = 0.829** with **e5-large-v2**, which is **+11.2 pp** over BM25 and **+8.8 pp** over dense late interaction alone. A separate reranking study, ConvMemory v2, keeps the top-10 candidate set fixed and reorders only that protected prefix; on **5 seeds** and **\(n=4955\)** test rows it raises **FULL MRR** from **0.5824** to **0.6560** and **H@1** from **0.4440** to **0.5474**, with **Recall@10** preserved by construction [2606.04194] [2606.10842].

## 6. Terminological ambiguity: LoCoMo as “Local Contact Moment”

Outside conversational-memory research, **LoCoMo** also denotes **Local Contact Moment** in robot grasp planning. In “Dual Quaternion-Based Visual Servoing for Grasping Moving Objects,” LoCoMo is a **model-free, learning-free grasp planner** that scores candidate grasps directly from an object point cloud. It produces a ranked list of **grasp poses** and **pre-grasp poses**, with ranking based on contact compatibility across fingers. The paper gives the overall LoCoMo grasp score in the form
\[
\mathcal{R} = \gamma \prod_{i=1}^{N_{f}} \mathcal{C}_{i}^{\omega_{i}},
\]
where \(\mathcal{C}_i\) is the contact-moment score for finger \(i\) [2107.08149].

In that robotics usage, LoCoMo is not a dialogue benchmark at all. It is the grasp-generation module in a pipeline that couples LoCoMo candidate generation with dual-quaternion pose-based visual servoing and dynamic grasp re-ranking. The system tracks the object pose \(\dq x_o\), uses a **vantage-point tree (vp-tree)** to find the \(\mathcal{K}\) nearest feasible grasps, ranks them by dual-quaternion spatial distance, and applies hysteresis to prevent chattering. The reported simulation success rates for the full method are **80%, 80%, 70%** across three objects [2107.08149].

The coexistence of these two meanings makes explicit disambiguation useful in technical writing. In current arXiv literature on LLM agents, “LoCoMo” almost always refers to the long-term conversational memory benchmark introduced in 2024, whereas in the cited robotics line it refers to Local Contact Moment grasp planning. The two usages are unrelated except for the acronym [2402.17753] [2107.08149].

Source: https://www.emergentmind.com/topics/locomo-1b7af604-e910-443f-a64e-87a74b0b1021