LOCOMO: Long-Term Conversational Memory Benchmark

Updated 5 July 2026

LoCoMo is a benchmark dataset designed to assess long-term conversational memory in LLM agents through multi-session dialogues with persistent personas and structured temporal events.
It employs a machine-human pipeline that expands short persona descriptions and generates temporal event graphs to separate short-term from long-term memory.
The benchmark challenges models with tasks such as question answering, event summarization, and multimodal dialogue generation, exposing failures in temporal reasoning and memory retention.

LoCoMo—also written as LOCOMO in parts of the later literature—is a benchmark and dataset family for evaluating very long-term conversational memory in LLM agents. It was introduced to measure whether models can retain and use information across multi-session dialogues grounded in persistent personas, temporal event structure, and multimodal interaction, rather than merely handling short-context dialogue coherence. In the agent-memory literature, LoCoMo functions both as a dataset of long conversations and as an evaluation suite spanning question answering, event summarization, and multimodal dialogue generation; later work also uses LoCoMo as a substrate for retrieval studies, paging systems, contradiction resolution, state-aware memory, and cognitive-memory extensions (Maharana et al., 2024).

1. Origin, construction, and underlying formalism

The original LoCoMo resource was created through a machine-human pipeline designed to generate very long-term conversations with explicit longitudinal structure. Each virtual speaker is assigned a persona statement $p$ , beginning from short persona descriptions $p_c$ from the MSC dataset and expanded with gpt-3.5-turbo into fuller profiles. Each speaker is also grounded in a temporal event graph $\mathcal{G}$ whose nodes are events $e_i$ associated with dates $t_i$ , with causal edges written as $l=(e_i,e_j)$ . The graph-generation process starts with $k=3$ events and can produce up to 25 events distributed across a 6–12 month period, so that later dialogue can remain grounded in evolving personal history rather than only recent context (Maharana et al., 2024).

The generative-agent architecture used in the dataset construction separates short-term and long-term memory. Short-term memory $\mathcal{H}_s$ stores session summaries $w_k$ , while long-term memory $\mathcal{H}_l$ stores observations $p_c$ 0 extracted from dialogue turns. After each session $p_c$ 1, the agent generates a summary $p_c$ 2 conditioned on current history $p_c$ 3 and the previous summary $p_c$ 4. When responding in session $p_c$ 5, the agent conditions on the latest summary, retrieved observations, persona $p_c$ 6, current-session history, and the subset of event-graph events satisfying

$p_c$ 7

This design makes temporal continuity and causal linkage part of the conversation-generation process rather than post hoc annotations (Maharana et al., 2024).

LoCoMo is multimodal in its original form. Agents can share images by first generating a caption $p_c$ 8, converting it to keywords $p_c$ 9, retrieving an image via $\mathcal{G}$ 0, and then sharing the selected image. For image reaction, BLIP-2 is used to caption the image before the LLM generates a reaction grounded in both the image and the speakers’ personas. Human annotators subsequently edit the generated conversations for long-range consistency and grounding accuracy; the original paper reports edits to nearly 15% of dialogue turns and removal or substitution of about 19% of images (Maharana et al., 2024).

The original dataset statistics emphasize scale. The paper reports 50 conversations, average 19.3 sessions per conversation, average 15.8 turns per session, average 300.9 turns, average 9,209.2 tokens per conversation, up to 35 sessions, average 24.2 ground-truth events per conversation, and average 32.3 images per conversation. Additional statistics include 30.2 tokens per turn, 18.2 tokens per observation, 127.4 tokens per session summary, and 896.5 tokens for average event-summary length (Maharana et al., 2024).

2. Benchmark structure and evaluation regimes

LoCoMo was introduced with three task families: question answering, event summarization, and multimodal dialogue generation. In the QA task, questions are grouped into five reasoning types: single-hop, multi-hop, temporal reasoning, open-domain knowledge, and adversarial. Event summarization is evaluated against the underlying temporal event graph $\mathcal{G}$ 1, with a FactScore-style decomposition into precision, recall, and $\mathcal{G}$ 2. Multimodal dialogue generation is evaluated with MMRelevance, alongside standard NLG metrics in appendix experiments (Maharana et al., 2024).

In later agent-memory work, LoCoMo is often used in a narrower operational form as a long-term conversational memory QA benchmark. One common description is a benchmark over 10 conversations, 19–32 sessions, about 9K tokens per dialogue, and 1,986 QA pairs, with five question types: Single-hop (SH), Multi-hop (MH), Temporal (T), Open-ended (O), and Adversarial (A). In that setting, the primary official metric is token-level F1 with Porter stemming, described as the standard token-overlap F1 used in QA evaluation (Liu et al., 1 Apr 2026).

Other papers evaluate specific slices rather than the full benchmark. Some report the 1,540-question four-category retrieval subset that excludes adversarial questions, while others use an answerable-factual pool of 1,444 LoCoMo questions restricted to single-hop, temporal, and open-domain factual items. Probe-based studies sometimes evaluate only 98 or 100 questions from a LoCoMo-10 subset, and local-memory studies sometimes use 304 QA pairs from 2 of 10 LoCoMo conversations (Park, 18 Mar 2026, Wang, 4 Jun 2026, Liu, 14 Apr 2026, Bhardwaj, 6 Apr 2026).

Evaluation regime	Reported scope	Primary metric(s)
Original LoCoMo	QA, event summarization, multimodal generation	F1, FactScore-style precision/recall/F1, MMRelevance
Official QA usage in later memory papers	10 conversations, 1,986 QA pairs	Token-level F1 with Porter stemming
Retrieval-focused subset	1,540 questions	Token-level F1, refusal accuracy
Answerable-factual slice	1,444 questions	Accuracy
Probe subsets	98–100 probes or 304 QA pairs	LLM-judged 1–5 scores or judged accuracy

This variation in setup is methodologically important. Later papers evaluate LoCoMo with token-level F1, BLEU-1, judged QA accuracy, LLM-as-a-Judge accuracy, and retrieval metrics such as Hit@1, MRR, and NDCG@5. The literature therefore uses LoCoMo both as an end-to-end memory benchmark and as a retrieval-stage benchmark, with metric choice shaping what is being measured (Park, 18 Mar 2026, Wang et al., 10 Jul 2025, Pan, 9 Jun 2026, Lysenstøen, 2 Jun 2026).

3. What LoCoMo measures and the failure modes it exposes

LoCoMo was created to test memory over spans much longer than the dialogue benchmarks that preceded it. The original paper argues that existing long-term dialogue datasets were still too short to stress memory over weeks or months, and its experimental results show that LLMs struggle with lengthy conversations and with long-range temporal and causal dynamics. Human performance on the original QA evaluation is reported as 87.9 overall F1, while the best base-model results remain substantially lower, even when long-context or retrieval-augmented methods are used (Maharana et al., 2024).

A consistent finding across later papers is that LoCoMo stresses more than surface retrieval. The benchmark repeatedly exposes failures involving temporal markers, implicit coreference, ephemeral updates, and answer formatting that is mismatched to the evaluation protocol. One cost-performance study attributes the large gap between full-history long-context inference and a fact-based memory system on LoCoMo to information lost during compression, especially temporal markers, implicit coreference, and ephemeral updates (Pollertlam et al., 5 Mar 2026). Omni-SimpleMem further reports that verbose natural-language outputs can “destroy” token-level F1, so benchmark-specific answer style and formatting materially affect results (Liu et al., 1 Apr 2026).

The original LoCoMo paper already showed that retrieval quality depends strongly on the unit being retrieved. Retrieving compact observations rather than raw dialogue or summaries improved GPT-3.5 QA by about 5% over raw dialogue retrieval, while retrieving too many items degraded performance by lowering the signal-to-noise ratio. Session summaries were less effective than observations because summarization lost details, and open-domain questions could be harmed by distracting retrieved context (Maharana et al., 2024).

Later work has sharpened the benchmark’s diagnostic role. A state-aware memory paper argues that LoCoMo is useful as an external generalization check but is not designed to isolate old/current state coordination in the way its conflict-heavy extension LTP is. A plausible implication is that LoCoMo is strong at exposing broad long-horizon memory failures, but weaker at disentangling bank-level, retrieval-level, and answer-level errors unless additional instrumentation is added (Shi et al., 2 Jul 2026).

4. Systems evaluated on LoCoMo and reported performance

The LoCoMo literature contains a large number of systems, but their numbers are not directly interchangeable because papers report different metrics and sometimes exclude categories such as adversarial questions. The table below therefore lists representative results together with the protocol used, rather than treating them as a single leaderboard.

System	Reported LoCoMo protocol	Reported result
Omni-SimpleMem	Official token-level F1	0.598 final, from 0.117 baseline
Kumiho	Token-level F1 on retrieval subset; refusal on adversarial	0.447 four-category F1; 0.565 overall F1; 97.5% refusal accuracy
TiMem	LLM-as-a-Judge accuracy; also F1 and ROUGE-L	75.30% accuracy; F1 = 54.40; ROUGE-L = 54.68
MIRIX	GPT-4.1 judge accuracy, adversarial excluded	85.38% overall
D-Mem	F1 with GPT-4o-mini	53.5 for Quality Gating; 55.3 for Full Deliberation
Synthius-Mem	GPT-4.1-mini binary judging	94.37% overall; 99.55% adversarial robustness

Omni-SimpleMem is notable because the paper documents the optimization path rather than only the final score. Starting from a naive SimpleMem baseline at F1 = 0.117, the autonomous research loop reached 0.598 on LoCoMo after nine successful iterations and two reverted experiments. The most consequential changes were a missing response_format fix (0.117 $\mathcal{G}$ 3 0.322), hybrid BM25 + FAISS retrieval with set-union merging (0.322 $\mathcal{G}$ 4 0.464), anti-hallucination prompting (0.516), evaluation-format alignment (0.543), and repair of 4,277 MAU timestamps corrupted to the ingestion date, which were corrected at 99.98% accuracy without re-ingestion (0.543 $\mathcal{G}$ 5 0.580). The final gain to 0.598 came from adaptive top- $\mathcal{G}$ 6 and metadata (Liu et al., 1 Apr 2026).

Kumiho frames LoCoMo as a retrieval benchmark rather than a generative benchmark. It reports 0.447 four-category token-level F1 over $\mathcal{G}$ 7 retrieval questions, per-category scores of 0.462 single-hop, 0.355 multi-hop, 0.533 temporal, and 0.290 open-domain, plus 97.5% adversarial refusal accuracy on $\mathcal{G}$ 8 adversarial questions. Including the adversarial portion, the paper reports 0.565 overall F1 on the full $\mathcal{G}$ 9 set. The system attributes this performance to a dual-store Redis + Neo4j graph-native architecture with immutable revisions, mutable tag pointers, typed provenance edges, and hybrid BM25/vector retrieval fused as

$e_i$ 0

with $e_i$ 1 (Park, 18 Mar 2026).

TiMem organizes memory as a five-level Temporal Memory Tree and reports 75.30% $e_i$ 2 LLM-as-a-Judge accuracy on LoCoMo, together with Single-Hop 81.43, Temporal 77.63, Open-Domain 52.08, Multi-Hop 62.20, F1 = 54.40, and ROUGE-L = 54.68. The paper also reports a 52.20% reduction in recalled memory length on LoCoMo, from 1,070.10 tokens per query for Mem0 to 511.25 for TiMem (Li et al., 6 Jan 2026).

MIRIX evaluates LoCoMo under a memory-only protocol: the conversation is injected into memory, the chat agent does not receive the original transcript at query time, and answers are scored by GPT-4.1 as an LLM judge. In that setup MIRIX reports 85.38% overall accuracy, with 85.11 single-hop, 83.70 multi-hop, 65.62 open-domain, and 88.39 temporal. The paper excludes the adversarial category for fairness with earlier work (Wang et al., 10 Jul 2025).

D-Mem treats LoCoMo as a test of whether a fast retrieval path should be augmented with an exhaustive fallback. On GPT-4o-mini, the paper reports Mem0 $e_i$ 3 = 51.2 F1, Full Deliberation = 55.3, and Quality Gating = 53.5, while using 12,681 tokens for Quality Gating versus 35,435 for Full Deliberation. The paper describes Quality Gating as recovering 96.7% of Full Deliberation’s performance at substantially lower cost (You et al., 19 Mar 2026).

Synthius-Mem reports the highest figure in the provided literature, but under a different protocol. Using GPT-4.1-mini as both answer model and judge on the full LoCoMo dataset of 10 conversations, 20 participants, and 1,813 questions, it reports 94.37% overall accuracy, 96.73% single-hop, 94.34% multi-hop, 89.32% temporal, 77.33% open-domain, 99.55% adversarial robustness, 98.64% core memory fact accuracy, and 94.40% temporal precision. The paper explicitly replaces token-overlap F1 with binary LLM judging, so these results should be read as protocol-specific rather than directly comparable with official F1-based scores (Gadzhiev et al., 13 Apr 2026).

5. Extensions, diagnostic variants, and retrieval-focused uses

LoCoMo has also served as the basis for several extensions that move beyond direct factual recall. LoCoMo-Plus redefines the task from explicit answer recovery toward what it calls beyond-factual cognitive memory. It introduces cue–trigger semantic disconnect, where a later query depends on an earlier latent constraint but is not semantically similar to the original cue. The formal distinction is between Level-1 factual memory and Level-2 cognitive memory, with valid outputs defined as

$e_i$ 4

This reframes memory evaluation around constraint consistency rather than string overlap, and the paper argues that conventional task disclosure and string-matching metrics are misaligned with such cases (Li et al., 11 Feb 2026).

A-TMA uses LoCoMo as an external generalization benchmark while introducing LTP (LoCoMo Temporal Plus) as a conflict-heavy benchmark for ghost memory. LTP is built from 10 profiles, with 40 mutable slot instances per profile, 400 state units, 800 cue events, and 800 evaluation probes. On LoCoMo itself, the headline result is for Graphiti/Zep + A-TMA, where temporal F1 rises from 0.0295 to 0.1705 and average F1 from 0.0809 to 0.1556. The paper emphasizes that these gains are host- and metric-dependent (Shi et al., 2 Jul 2026).

Paging and memory-virtualization studies use LoCoMo differently. Cooperative paging compresses evicted conversation segments into minimal keyword bookmarks such as $e_i$ 5 and exposes a recall() tool. On the LoCoMo-10 subset, Bookmark+Recall on GPT-4o-mini scores 2.18 on a 1–5 judged scale, above Full context 2.02, Search-tool baseline 1.90, Word-overlap retrieval 1.88, BM25 retrieval 1.86, and Truncation 1.64. In the multi-judge evaluation, Bookmark+Recall vs. BM25 yields $e_i$ 6 with $e_i$ 7, and vs. Full Context yields $e_i$ 8 with $e_i$ 9 (Liu, 14 Apr 2026).

Write-time memory-correctness research also uses LoCoMo as a natural-workload slice. TOKI evaluates a typed contradiction-resolution layer with an audit-row schema and reports that the audit-row defense moves LoCoMo by $t_i$ 0 with CI [0.76, 0.94] on a matched mechanism slice. In the paired memory ablation on 1,444 answerable LoCoMo questions, performance drops from 0.540 with the typed memory layer to 0.048 without it, for a paired delta of +0.492, 95% CI [+ $t_i$ 1, + $t_i$ 2], and McNemar $t_i$ 3 (Wang, 4 Jun 2026).

LoCoMo is also widely used as a pure retrieval-stage benchmark. A training-free lexical-dense fusion study formulates it as a session-ranking problem and reports that fusing late-interaction dense retrieval with BM25 yields Hit@1 = 0.752 and NDCG@5 = 0.829 with e5-large-v2, which is +11.2 pp over BM25 and +8.8 pp over dense late interaction alone. A separate reranking study, ConvMemory v2, keeps the top-10 candidate set fixed and reorders only that protected prefix; on 5 seeds and $t_i$ 4 test rows it raises FULL MRR from 0.5824 to 0.6560 and H@1 from 0.4440 to 0.5474, with Recall@10 preserved by construction (Lysenstøen, 2 Jun 2026, Pan, 9 Jun 2026).

6. Terminological ambiguity: LoCoMo as “Local Contact Moment”

Outside conversational-memory research, LoCoMo also denotes Local Contact Moment in robot grasp planning. In “Dual Quaternion-Based Visual Servoing for Grasping Moving Objects,” LoCoMo is a model-free, learning-free grasp planner that scores candidate grasps directly from an object point cloud. It produces a ranked list of grasp poses and pre-grasp poses, with ranking based on contact compatibility across fingers. The paper gives the overall LoCoMo grasp score in the form

$t_i$ 5

where $t_i$ 6 is the contact-moment score for finger $t_i$ 7 (Farias et al., 2021).

In that robotics usage, LoCoMo is not a dialogue benchmark at all. It is the grasp-generation module in a pipeline that couples LoCoMo candidate generation with dual-quaternion pose-based visual servoing and dynamic grasp re-ranking. The system tracks the object pose $t_i$ 8, uses a vantage-point tree (vp-tree) to find the $t_i$ 9 nearest feasible grasps, ranks them by dual-quaternion spatial distance, and applies hysteresis to prevent chattering. The reported simulation success rates for the full method are 80%, 80%, 70% across three objects (Farias et al., 2021).

The coexistence of these two meanings makes explicit disambiguation useful in technical writing. In current arXiv literature on LLM agents, “LoCoMo” almost always refers to the long-term conversational memory benchmark introduced in 2024, whereas in the cited robotics line it refers to Local Contact Moment grasp planning. The two usages are unrelated except for the acronym (Maharana et al., 2024, Farias et al., 2021).