WorldMemArena: Multi-Session Multimodal Memory Benchmark

• WorldMemArena is a benchmark that defines agent memory through four stages (Write, Maintain, Retrieve, Use) within an action–world interaction loop.
• It organizes tasks into multi-session trajectories with detailed annotations and metrics, providing granular evaluation of memory performance.
• The suite compares strategies such as retrieval-augmented generation, engineered external memory, and harness-based agents to highlight practical trade-offs.

WorldMemArena is a benchmark and evaluation suite designed to rigorously analyze and compare the memory capabilities of multimodal agents engaged in long-horizon, real-world‐like tasks. It addresses critical gaps in existing methodologies by decomposing memory into fine-grained, observable stages, structuring tasks across multiple sessions with true multimodal (text and visual) inputs, and providing exhaustive supervision at every step. WorldMemArena operationalizes agent memory as a process embedded in an Action–World Interaction Loop, supporting head-to-head comparisons across retrieval-augmented generation (RAG), engineered external memory, and autonomous harness-based agent memory strategies (Liu et al., 28 May 2026).

1. Action–World Interaction Loop and Four-Stage Memory Lifecycle

WorldMemArena formalizes agent memory within an “action–world interaction loop,” in which the agent at each timestep observes a partially revealed world ($o_t = \Omega(z_t)$), acts based on its observation and current memory ($a_t = \pi(o_t, m_t)$), receives environmental feedback ($f_t$), and updates its memory accordingly. Memory $m_t$ persists and evolves outside of any individual session. This process is segmented into $S$ sessions $\tau^{(1)}, \dots, \tau^{(S)}$.

Agent memory is dissected into four distinct lifecycle stages with explicit interfaces and metrics:

1. Write (Observe → Write): Produces a session-specific memory delta, $\Delta_s = \mathrm{Write}(m_{s-1}, \tau^{(s)})$, selecting only potentially useful observations for later retrieval.
2. Maintain (Update & Consolidate): Integrates $\Delta_s$ with existing memory, $m_s = \mathrm{Maintain}(m_{s-1}, \Delta_s)$, handling revisions and deletions (i.e., overwriting or expunging obsolete facts).
3. Retrieve (for Decision): On demand, retrieves relevant evidence for queries or decisions, $R_{s,q} = \mathrm{Retrieve}(m_s, q)$, aligning retrieved material with current decision requirements.
4. Use (Answer / Act): Generates responses or actions, $a_t = \pi(o_t, m_t)$0, predicated strictly on retrieved evidence, with fidelity to fact and avoidance of hallucination or omission.

The interaction loop and memory lifecycle are directly mapped in the benchmark workflow, enabling targeted assessment of each phase.

2. Benchmark Composition and Task Structure

WorldMemArena comprises 461 multi-session tasks (~18.4 sessions per task, ~9k tokens/task), distributed across two complementary regimes:

• Lifelong Evolution: 38 trajectories tracking evolving states in professional and whole-life scenarios, emphasizing longitudinal change.
• Agentic Execution: 423 trajectories emulating realistic agent operations within GUI and embodied environments (6 GUI subcategories, 4 embodied).

Each task includes rich textual and visual modalities (~15k images/screenshots overall), with agents interacting through partially observed worlds, tool calls, and dynamic environment feedback.

Extensive per-session annotations enable granular evaluation:

Annotation Type	Description	Example Use
Gold memory points ($a_t = \pi(o_t, m_t)$1)	Ground-truth facts to be written per session	Recall (Write) assessment
State updates	Flags indicating obsolescence and need for revision	Maintenance (Update) accuracy
Distractors ($a_t = \pi(o_t, m_t)$2)	Plausible but irrelevant or outdated facts	Interference rejection
Evidence chains	Gold memory points required to answer each checkpoint question	Retrieval (decision relevance)
QA checkpoints	Average of ~5 per trajectory; cover 11 question types (recall, update, visual, etc.)	Use (QA correctness, error localization)

This dense annotation schema supports multi-level analysis from memory writing to end-task action.

3. Evaluation Methodology and Metrics

WorldMemArena introduces stage-specific metrics using a mix of gold alignment, LLM-based grading, and deterministic ranking measures.

Write Stage:

• Memory Recall: Coverage of gold points, weighted by importance:

$a_t = \pi(o_t, m_t)$3

• Correctness, Hallucination, Irrelevance: LLM-graded fractions for items stored in $a_t = \pi(o_t, m_t)$4.

Maintain Stage:

• Update Handling: Fraction of correctly updated, coexisting, or obsolete facts, with graded credit.
• Interference Rejection: Fraction of distractors not stored.

Retrieve Stage:

• Retrieval Coverage (RC): Mean coverage of gold evidence by retrieval.
• Recall@K: Fraction of gold evidence among top-K retrieved.
• NDCG@K: Graded ranking quality using matched relevance vectors.

Use Stage:

• QA Correctness (QA-C), Hallucination (QA-H), Omission (QA-O): LLM-labeled answer quality.
• Surface-overlap metrics: Token-level F$a_t = \pi(o_t, m_t)$5, BLEU-1 (canonicalized via stemming and stopword removal).

This structured metric suite localizes system failures to distinct memory management processes.

4. Empirical Paradigm Comparison and Findings

WorldMemArena enables systematic comparison across three agent memory paradigms:

A. Long-context agents: No explicit memory, full history appended in prompt. B. Manually engineered pipelines: RAG systems and external memory agents. C. Harness-based agents: Agents managing memory via harnesses (e.g., OpenClaw, Codex).

Key empirical observations include:

• Memory writing/storage quality does not predict QA utility: Qwen3-VL-Embedding achieves ~86% memory recall and 98% precision but only ~52% QA-C; A-Mem achieves lower recall (52%) yet higher QA utility (54.6%). This underscores that retrieval usability at decision time is more predictive of task performance than sheer recall coverage.
• Multimodal memory is a bottleneck: ViLoMem and MIRIX (with image support) yield high visual memory recall (~85.9%) but only marginal QA gains (QA-C = 49.8%). In contrast, text-only MemoryGPT achieves 85.2% recall and markedly higher QA-C (57.8%). Current approaches tend to flatten images into captions, erasing crucial spatial and procedural context.
• Performance degrades under Agentic Execution: Average QA-C falls from ~55% (Lifelong Evolution) to ~45% (Agentic Execution) as agentic scenarios diffuse evidence across actions, tool outputs, and dynamic states.
• Harness-based memory is adaptable yet costly: Harnesses (OpenClaw, Codex) often outperform fixed pipelines (Codex-GPT5.4-nano QA-C = 53.6%; MemoryGPT = 57.8%) and demonstrate flexible memory use but at the expense of higher latency (50–200 s/task) and greater variability (higher hallucination and omission rates).

5. Limitations and Prospective Directions

WorldMemArena’s task suite is limited by reliance on simulated environments and a fixed scope (11 QA types). Four principal research avenues are delineated:

• End-to-end interactional memory: Transition from isolated memory modules to architectures where memory emerges from joint optimization with policy learning.
• State mutability and consistency: Enable architectures to revise, delete, merge, or resolve conflicting memories, beyond append-only strategies.
• True preservation of multimodality: Advance memory systems that encode, store, and utilize visual and spatial representations natively, rather than reducing them to textual summaries.
• Experience-driven evaluation: Develop metrics that measure policy improvement or error reduction through memory, not just retrospective QA accuracy.

A plausible implication is that future system development will require both architectural innovation and enhanced multimodal support to close the observed gaps under realistic agentic conditions.

6. Significance within Agent Memory Evaluation

WorldMemArena establishes the first fine-grained, multimodal, and multi-session benchmark for diagnosing not only where agent memories succeed or fail, but why. Its action–world interaction formulation, exhaustive task annotation, and decoupled stage-wise evaluation framework provide the methodological backbone and terminology for systematic, reproducible comparison of a wide spectrum of agent memory architectures (Liu et al., 28 May 2026).