LoCoMo: Conversational Memory Benchmark

• LoCoMo is a benchmark suite for assessing long-range conversational memory, reasoning, and multi-modal dialogue using extensive, multi-session datasets.
• It comprises detailed datasets with hundreds of dialogue turns, event graphs, and tasks including QA, summative event extraction, and next-turn generation.
• LoCoMo drives research into memory-augmented architectures and dynamic context pruning, highlighting the limitations and potential optimizations for modern LLMs.

LoCoMo (Long-term Conversational Memory) is a benchmark suite designed to rigorously evaluate LLMs' (LLMs) capacity for long-range memory, contextual reasoning, and multi-modal dialogue understanding in extremely long, multi-session conversational settings. Originating with Maharana et al. (ACL 2024), LoCoMo was motivated by the observation that human conversations span many sessions and months, and truly robust dialogue agents must remember, retrieve, and integrate information scattered across such temporal horizons (Maharana et al., 2024). LoCoMo has become a primary evaluation framework for long-horizon agent memory, driving research around memory-augmented LLMs, retrieval architectures, inference efficiency, and multi-modal context fusion.

1. Scope, Structure, and Objectives

LoCoMo’s objective is to stress-test LLM agents on their ability to recall and utilize information from hundreds of conversational turns, addressing memory, retrieval, and reasoning at scales where context windows and simple retrieval strategies fail. The benchmark encompasses the following defining features:

• Dataset Size and Composition: The full LoCoMo suite comprises 50 (human–human or LLM–human) dialogues, each averaging 300–600 turns, 9k–26k input tokens, and spanning 19–35 sessions. Each session is grounded in speaker personas and temporally ordered event graphs, with every turn consisting of (user query, agent response) pairs. LoCoMo includes multimodal elements, with ∼32 images per dialogue.
• Temporal and Topical Scope: Dialogues are constructed over imagined timelines of 6–12 months, with frequent topic shifts. Each agent is seeded with a rich persona and an event graph of up to 25 causally connected life events (Maharana et al., 2024).
• Annotation Protocols: For question answering (QA) tasks, dialogues are annotated with factual gold answers and “relevant turn” indices for retrieval evaluation. Event graphs and persona statements are human-verified for temporally coherent, causally plausible interactions.
• Task Suite: LoCoMo defines several tasks: factual question answering (single-hop, multi-hop, temporal, open-domain), event summarization (graph reconstruction), and multi-modal next-turn generation.

The core thesis is that models should recall and reason over previously established information—tracking facts, intentions, events, and personalities—despite large dialogue length, inter-session gaps, and shifting conversational focus (Maharana et al., 2024).

2. Task Definitions and Evaluation Metrics

LoCoMo’s evaluation methodology is centered on three primary task types and associated metrics:

• Question Answering (QA/Memory Recall): Over 7,500 human-authored QA pairs probe five question categories:
  • Single-hop: Fact local to one session.
  • Multi-hop: Composition of facts across sessions.
  • Temporal: Date arithmetic, event ordering.
  • Open-domain: Persona/world knowledge, preference.
  • Adversarial: Unanswerable distractors.
  • Evaluation employs token-level F₁, exact match, ROUGE-1/2/L, LLM-as-judge accuracy, and, for multimodal QA, VQA exact match (Choi et al., 12 Jan 2026, Maharana et al., 2024, Bini et al., 4 Dec 2025).
• Event Summarization (Graph Reconstruction): Models output a chronological list of discussed events. FactScore (F₁ on event graph), ROUGE, and BLEU metrics quantify alignment to ground-truth event graphs (Maharana et al., 2024).
• Multi-Modal Dialogue Generation: Next-turn prediction is scored with BLEU, ROUGE, BERTScore, and MM-Relevance for image-grounded response quality (Maharana et al., 2024, Bini et al., 4 Dec 2025).

Advanced Metrics:

• Retrieval accuracy: Hit@k, Recall@k, Precision@k versus annotated “relevant turns” (for models with explicit retrieval modules) (Choi et al., 12 Jan 2026).
• Latency: Streaming first-token latency, wall-clock response time.
• Token efficiency: Context window reduction, average tokens per query.

The evaluation protocol is LLM-centric: for QA, GPT-4, GPT-OSS-120B, or gpt-4.1 act as independent judges, comparing model predictions to gold answers and returning integer/boolean ratings (Choi et al., 12 Jan 2026, Bini et al., 4 Dec 2025, Wang et al., 10 Jul 2025).

3. Benchmark Construction Methodology

LoCoMo’s dialogue generation and annotation process is both scalable and methodologically rigorous:

• Dialogue Simulation: LLM-based agents, each initialized with expanded persona profiles and temporal event graphs, engage in alternating conversations (Reflect-and-Respond paradigm). Inputs include session histories, persona, and event context. An image-sharing module produces and exchanges image captions and visual reactions (Maharana et al., 2024).
• Human Verification: Human annotators review and edit up to 19% of turns and images, correcting factual inconsistencies and ensuring adherence to persona and event timelines (Maharana et al., 2024).
• Annotation for QA and Summarization: Each dialogue is retro-annotated with 200-300 probe questions (QA), gold answers, event-graph alignments, and image references.
• Multi-Modal Extension: LoCoMo-V extends the QA task to visual question answering, generating one-word VQA pairs (e.g., object counting, color identification, binary object detection) for each image, with test splits and correctness judged by exact match (Bini et al., 4 Dec 2025).

This two-stage machine–human pipeline ensures long-range narrative and factual consistency, greatly exceeding the session length and context scope of prior conversational benchmarks (Maharana et al., 2024).

4. Research Insights and Model Performance

LoCoMo has driven key findings about memory, context, and model limitations:

• Human–Model Gap: Human QA overall F₁ ≈ 88 vs. best LLM baselines ≈ 37–42 (GPT-3.5/4-Turbo), with RAG/observation-augmented variants moderately higher (up to F₁ ≈ 41) (Maharana et al., 2024). Temporal and adversarial questions are particularly challenging: models score 20–30 F₁ on temporals versus human 92.6, and adversarial QA can drop to ≈2 F₁.
• Context Window Effects: Longer context windows (e.g., 16K, 128K tokens) do not guarantee improved factual extraction beyond a few sessions—some models display recency bias and input bloat, degrading QA performance as dialogue length grows (Choi et al., 12 Jan 2026, Maharana et al., 2024).
• Memory-Augmented Architectures: Retrieval-augmented generation, hybrid structured memory, and dynamic context pruning yield efficiency gains (token reduction 90–97%) and quality improvements, especially for temporal/multi-hop reasoning (Choi et al., 12 Jan 2026, Bini et al., 4 Dec 2025, Wang et al., 10 Jul 2025). Novel memory frameworks such as TiMem (Li et al., 6 Jan 2026), ENGRAM-R (Patel et al., 17 Nov 2025), Amory (Zhou et al., 9 Jan 2026), O-Mem (Wang et al., 17 Nov 2025), MemWeaver (Ye et al., 26 Jan 2026), Hindsight (Latimer et al., 14 Dec 2025), and MIRIX (Wang et al., 10 Jul 2025) have all been benchmarked on LoCoMo, reporting substantial boosts over naive RAG or flattened memory.
• Latency vs. Quality Trade-offs: Dynamic context selection (e.g., DyCP (Choi et al., 12 Jan 2026)) reduces latency by 53% with increased answer quality, confirming that pruning and segment selection are essential under long dialogue conditions.
• Ablation and Bottlenecks: High recall in retrieval is more critical than precision for answer quality; ablation studies suggest that continuity-preserving weakly relevant context aids performance even if not directly factual (Choi et al., 12 Jan 2026). Multi-strategy entity and graph-based retrieval (e.g., Hindsight (Latimer et al., 14 Dec 2025)) recovers more multi-hop/temporal evidence than vector retrieval alone.

Results Table: Sample Overall Accuracy and Efficiency on LoCoMo

Model/Method	Overall Accuracy (%)	Token Reduction	Key Feature
Full Context	87.5	0%	All history in prompt
DyCP (GPT-4o)	+8.1 GPT4Score	–81%	Dynamic segment retrieval/pruning
Hindsight (OSS-20B)	83.2	—	Four-network memory, graph+entity recall
MemWeaver	+1.65 F₁	>95%	Graph + experience + passage memory
MIRIX	85.4	—	Multi-agent, 6 memory types
O-Mem	+3.6 F₁	–94%	Persona profiling, hierarchical retrieval
TiMem	75.3	–52%	Temporal-hierarchical consolidation
ENGRAM-R	–1.9 pts vs FullCtxt	–88%	Fact-cards, typed retrieval, citation

Exact reported numbers depend on task split, backbone, and specific experimental protocol; see (Choi et al., 12 Jan 2026Latimer et al., 14 Dec 2025Wang et al., 10 Jul 2025Ye et al., 26 Jan 2026Bini et al., 4 Dec 2025Patel et al., 17 Nov 2025Wang et al., 17 Nov 2025Li et al., 6 Jan 2026) for detailed breakdowns.

5. Memory Architectures and Advances Enabled by LoCoMo

The breadth of tasks and the scale of LoCoMo have catalyzed diverse architectural advances:

• Dynamic Context Pruning: LoCoMo has motivated span-level, chronologically-ordered retrieval algorithms (DyCP) that leverage bi-encoder scoring, continuity preservation, and adaptive thresholds for context selection (Choi et al., 12 Jan 2026).
• Temporal-Hierarchical Memory: Frameworks such as TiMem (Li et al., 6 Jan 2026) consolidate memory into multi-scale temporal hierarchies (turn → session → day → week → month), employing semantic abstraction and complexity-aware hierarchical recall.
• Typed and Structured Retrieval: Explicit division of fact memories (episodic, semantic, procedural), fact-card reformatting (ENGRAM-R (Patel et al., 17 Nov 2025)), and explicit citation protocols constrain model reasoning and enhance auditability.
• Agentic and Narrative Memory: Systems like Amory (Zhou et al., 9 Jan 2026) and Hindsight (Latimer et al., 14 Dec 2025) incorporate agent-driven narrative binding and consolidation, coherence-based retrieval, and multi-phase retain/recall/reflect loops.
• Personalization and Multi-Agent Coordination: MIRIX (Wang et al., 10 Jul 2025) and O-Mem (Wang et al., 17 Nov 2025) integrate user profiling, meta memory management, and multi-agent routing to support adaptive, lifelong memory across modalities.
• Multimodality: LoCoMo-V (Bini et al., 4 Dec 2025) and MIRIX (Wang et al., 10 Jul 2025) drive vision-language retrieval, native VQA, and integrated visual evidence in memory retrieval and answer generation.

6. Limitations, Insights, and Future Directions

While LoCoMo has set a new benchmark for research in long-range conversational memory, several limitations and areas for future work are evident:

• Human–Model Gap: Even the strongest models and memory systems exhibit a ∼40–50 percentage point gap to human factual recall, especially on temporal and multi-hop reasoning (Maharana et al., 2024).
• Open-Domain and Hypothetical Queries: Models continue to underperform on open-domain, preference, and hypothetical questions where retrieving explicit supporting evidence is difficult (MIRIX: –8.34 pp vs. full context on open-domain) (Wang et al., 10 Jul 2025).
• Abstraction and Summarization: Overreliance on naive session summaries can degrade factual recall; advanced adaptive summarization and entity graph synthesis are suggested directions (Maharana et al., 2024, Latimer et al., 14 Dec 2025).
• Cost–Accuracy Trade-offs: Distributed systems studies using LoCoMo demonstrate that leaner vector-based retrieval systems can achieve comparable accuracy to graph-based ones with much lower computational, network, and storage cost (Wolff et al., 12 Jan 2026).
• Memory Refresh and Staleness: Most current retrievers lack policies for memory staleness or provenance tracking, risking outdated or inconsistent citations (Patel et al., 17 Nov 2025).
• Evaluation Bias: LLM-as-judge protocols may under-detect errors, motivating future work on human verification and more granular evaluation metrics (Patel et al., 17 Nov 2025, Wang et al., 10 Jul 2025).
• Multimodal Expansion: Integration of off-the-shelf vision-LLMs (e.g., MemLoRA-V (Bini et al., 4 Dec 2025)) and memory-augmented VQA indicates significant performance boosts, but these systems remain dependent on sufficiently annotated multimodal dialogues.

LoCoMo’s continued evolution, including releases under open licensing and the expansion into multimodal and distributed evaluation, is expected to further accelerate research toward endowing dialogue agents with durable, functionally human-like memory (Maharana et al., 2024, Bini et al., 4 Dec 2025, Wang et al., 10 Jul 2025).