MemoBench: Multifaceted Memory Benchmark

Updated 6 July 2026

MemoBench is a polysemous term that refers to distinct memory benchmarks across video world modeling, diffusion-model memorization, and LLM agent evaluations.
The video MemoBench evaluates disappear-and-reappear paradigms using high-resolution dynamic datasets and multi-phase annotations to measure object memory consistency.
In diffusion and LLM contexts, related benchmarks test unintended memorization, factual and reflective memory, and continual adaptation from user feedback.

Searching arXiv for papers explicitly named “MemoBench” and closely related “MemBench/MemoryBench” benchmarks to ground the article. “MemoBench” is an ambiguous designation in the arXiv literature. In the strictest sense, it denotes the benchmark “MemoBench: Benchmarking World Modeling in Dynamically Changing Environments,” which evaluates disappear-and-reappear world modeling in video generation under dynamic camera motion and evolving off-screen state (Chen et al., 25 Jun 2026). In adjacent literatures, however, “MemoBench” is also used informally or mistakenly for several distinct benchmarks, especially “MemBench” for diffusion-model memorization mitigation (Hong et al., 2024), “MemBench” for memory in LLM-based agents (Tan et al., 20 Jun 2025), and “MemoryBench” for continual learning from user feedback in LLM systems (Ai et al., 20 Oct 2025). This usage suggests that the term functions less as a single canonical benchmark name than as a polysemous label for benchmark suites concerned with memory, persistence, retrieval, and state evolution.

1. Nomenclature and scope

The naming landscape is unusually heterogeneous. Some works use “MemBench,” some “MemoryBench,” and one uses “MemoBench” as the official title. Several papers explicitly note that community references to “MemoBench” often point to differently named benchmarks.

Name	Domain	Official status of “MemoBench”
MemoBench	Video world modeling	Official title
MemBench	Diffusion-model memorization	“MemoBench” described as an informal or mistaken reference
MemBench	LLM-agent memory evaluation	Paper and GitHub use “MemBench”
MemoryBench	Continual learning in LLM systems	“MemoBench” stated to refer to this same benchmark in the summary
AdaptMemBench	Scientific memory-subsystem benchmarking	Distinct HPC benchmark, not an LLM or generative-model memory benchmark

In the diffusion-model benchmark, the authors consistently use “MemBench,” and state that “MemoBench” is most likely an informal or mistaken reference to the same work (Hong et al., 2024). The LLM-agent benchmark is officially named “MemBench: Towards More Comprehensive Evaluation on the Memory of LLM-based Agents” (Tan et al., 20 Jun 2025). By contrast, “MemoryBench: A Benchmark for Memory and Continual Learning in LLM Systems” is a separate benchmark centered on continual adaptation from user feedback, and its summary explicitly equates “MemoBench” with “MemoryBench” (Ai et al., 20 Oct 2025). A still earlier naming precursor is “AdaptMemBench,” a configurable framework for application-specific memory-subsystem benchmarking in scientific computing, where “memory” refers to hardware memory behavior rather than model memory (Lakshminarasimhan et al., 2018).

2. MemoBench in video world modeling

The officially titled “MemoBench” addresses generative video world modeling through a disappear-and-reappear paradigm: a target object undergoes a physical process, leaves the field of view because of camera motion, and must be recovered in its updated state when the camera returns (Chen et al., 25 Jun 2026). The benchmark is motivated by the observation that prior consistency benchmarks usually test visible content, static occlusions, or task-level success, but do not jointly evaluate dynamic camera viewpoints and dynamic scene content where an object evolves while off-screen.

The dataset comprises 360 ground-truth clips at $1920 \times 1080$ resolution, split into 196 synthetic clips created in Unreal Engine 5 at 60 FPS and 164 real-world clips recorded indoors. The synthetic subset covers 14 scene subdomains across five environment categories, while the real-world subset covers 30 physical-state-change processes across seven categories. Annotated phases divide each clip into Visible $V$ , Disappeared $D$ , and Reappear $R$ , using keyframes $d_{\text{start}}$ and $r_{\text{start}}$ . The physical processes include dissolution, melting, burning or combustion, oxidation, foaming, deformation, walking pedestrians, pouring or dripping viscous fluids, diffusion or absorption, and animated robots or characters.

Evaluation is organized around four diagnostic pillars: Instruction Following, Object and Background Consistency, Continuity of Memory, and Physics Adherence. Automated metrics include Visual Quality, Motion Smoothness, Object Identity Consistency, Geo3D Consistency, Object Reappearance Score, Camera Controllability, and prompt fidelity via ImageReward. The benchmark also uses a VQA-based protocol in which a vision-LLM generates 24 polarity-balanced Yes/No questions per clip, six per pillar. Human–VLM agreement on ground-truth videos is reported as $92.9\%$ with Cohen’s $\kappa = 0.85$ .

The reported results emphasize that memory consistency under disappear-and-reappear remains unsolved. No model exceeds approximately $0.6$ Object Reappearance Score, and camera inactivity can artificially inflate both automated and VQA consistency measures. HunyuanWorldPlay is the best model on ORS at $0.582$, while LTX-Video often scores highly on smoothness and consistency because it tends to keep objects continuously in frame. The benchmark therefore treats static-camera behavior as a confound rather than a sign of genuine memory. One notable bibliographic detail is that the abstract states evaluation of eight state-of-the-art models, whereas the detailed description enumerates ten models across camera-instructed image-to-video, novel-view-synthesis, and image-to-video families; the paper’s detailed model list is the more specific account.

3. MemBench for diffusion-model memorization

In diffusion-model safety and evaluation, “MemBench” names a benchmark for measuring memorized image trigger prompts in text-to-image diffusion systems (Hong et al., 2024). The benchmark is framed around memorization as reproduction of training-set images when triggered by specific prompts, independent of random seed or initial noise. The formal definition uses a thresholded self-supervised copy-detection score: $V$ 0

The benchmark’s principal contribution is to evaluate mitigation methods on both trigger prompts and general prompts. For Stable Diffusion 1, it contains 3,000 trigger prompts and 154 verified memorized images; for Stable Diffusion 2, it contains 1,500 trigger prompts and 55 verified memorized images. For comparison, Webster (2023) released 325 and 210 prompts for SD1 and SD2, respectively, so MemBench is approximately $V$ 1 larger in total number of trigger prompts. The verified memorized images include commercial products, human images, artwork, and brand logos.

Trigger discovery is based on an MCMC search over prompt space using a prompt-level memorization predictor

$V$ 2

with sampling from a Boltzmann target distribution

$V$ 3

The reported AUC for classifying triggers is $V$ 4 with one noise sample and $V$ 5 with four. On $V$ 6A100 GPU, the method found memorized images at approximately $V$ 7 hours per image, compared with $V$ 8 hours for a greedy baseline and $V$ 9 hours for ConZIC.

Evaluation on trigger prompts reports Top-1 SSCD, Top-3 SSCD, replication rate with $D$ 0, CLIPScore, and Aesthetic Score. General-prompt evaluation uses COCO validation captions and reports average CLIPScore and Aesthetic Score. The benchmark’s core empirical result is that existing memorization-mitigation methods remain insufficient for deployment: reductions in SSCD typically come with declines in text–image alignment or image quality, and even the best settings do not reach the proposed API-based reference level of approximately $D$ 1 with $D$ 2. This benchmark is therefore not a long-context memory benchmark but a memorization benchmark in the sense of unintended training-data replication.

4. MemBench and MemoryBench for LLM-based agents

A different “MemBench” evaluates the memory capability of LLM-based agents across both factual memory and reflective memory, and across participation and observation scenarios (Tan et al., 20 Jun 2025). Factual memory covers explicitly stated attributes and event details, whereas reflective memory covers higher-level inferred properties such as preferences and emotional state. Participation scenarios use predefined assistant responses in multi-turn, multi-session dialogues; observation scenarios provide only time-ordered user messages. The benchmark is explicitly time-aware, with timestamps, sessionization, evidence-bearing turns distributed across session positions, and inserted noise sessions to modulate difficulty.

Its dataset is built from 500 user relation graphs and includes multiple-choice questions grounded in one or more evidence messages. Global statistics include PS-RM with 3.5k sessions and TPT of approximately 2,195, PS-FM with 51k sessions and TPT of approximately 10,285, OS-RM with 2k message lists and TPT of approximately 745, and OS-FM with 8.5k message lists and TPT of approximately 617. Metrics span effectiveness, efficiency, and capacity: accuracy,

$D$ 3

retrieval Recall@k,

$D$ 4

read and write time per operation, and accuracy-versus-length analysis. In the reported experiments, RetrievalMemory is the most robust mechanism at long horizons, maintaining high performance on the $D$ 5-token participation and $D$ 6-token observation sub-datasets.

“MemoryBench,” by contrast, is a continual-learning benchmark for LLM systems that explicitly models service-time feedback and adaptation (Ai et al., 20 Oct 2025). It decomposes an LLM system into parametric memory $D$ 7 and non-parametric memory $D$ 8, and evaluates how systems update memory from explicit and implicit feedback over time. The benchmark spans 11 public datasets, three domains, two languages, and four task formats: LiSo, SiLo, LiLo, and SiSo. It supports off-policy, on-policy, and stepwise off-policy schedules, with a user simulator that emits verbose critique, like or dislike actions, copy behavior, and continue or end behavior. Across these settings, the reported conclusion is that state-of-the-art memory frameworks are neither consistently effective nor efficient enough, and that simple RAG baselines such as BM25 or embedding retrieval often match or exceed more complex agentic memory systems.

5. Interactive and executable extensions of MemoBench-style evaluation

Several later benchmarks reposition “MemoBench”-like evaluation away from static question answering and toward interactive, executable, or self-evolving memory assessment. AMemGym introduces an on-policy environment for memory-driven personalization in long-horizon assistant conversations (Jiayang et al., 2 Mar 2026). It formalizes the assistant as receiving observation $D$ 9 and memory $R$ 0, then producing action $R$ 1 and updated memory $R$ 2. Its key technical device is structured data sampling over user profiles, global state schemas, and state-evolution trajectories. Evaluation includes overall multiple-choice accuracy, a normalized memory score

$R$ 3

and a diagnostic decomposition into write, read, and utilization failures. The paper explicitly contrasts this with static, off-policy memory benchmarks and argues that on-policy interaction eliminates reuse bias.

EvoMemBench generalizes agent memory benchmarking along two orthogonal axes: memory scope, distinguishing in-episode from cross-episode memory, and memory content, distinguishing knowledge-oriented from execution-oriented memory (Wang et al., 18 May 2026). It covers six datasets and 15 memory systems under a standardized utilize/update interface, with metrics including answer accuracy, success rate, progress score, average steps, and token usage. Its main empirical claim is that no single memory form works consistently across all settings: long-context baselines remain competitive when raw evidence fits the context window, retrieval-based methods are strong for knowledge-intensive settings, and procedural or long-term memory methods are more effective for execution-oriented tasks when stored experience matches task structure.

Mem2ActBench shifts attention from passive retrieval to active memory utilization in tool-grounded agents (Shen et al., 13 Jan 2026). The task is to generate a tool invocation $R$ 4 from long-term memory and an underspecified query, with parameter grounding strictly supported by the memory chain. The dataset contains 2,029 long conversational sessions and 400 memory-dependent tool-use tasks. Main metrics are Parameter F1, BLEU-1, Tool Accuracy, Tool Selection Accuracy, and end-to-end exact match. The reported central bottleneck is retrieval: passive retrieval at hybrid@5 yields approximately $R$ 5 F1, whereas oracle memory yields approximately $R$ 6 F1.

VehicleMemBench extends the executable approach to multi-user in-vehicle agents (Chen et al., 25 Mar 2026). Built on an in-vehicle simulation environment with 23 tool modules and 111 executable APIs, it provides 50 multi-user scenarios, 500 executable queries, and instances averaging 81.78 historical events, 2,690 dialogue lines, and 92,819 tokens. The benchmark evaluates Exact State Match, field-level and value-level precision, recall, and F1, plus efficiency measures such as tool-call count and memory tokens. With Gold Memory, Gemini-3-Pro-Preview reaches $R$ 7 ESM, but under autonomous memory even strong systems drop sharply; the paper identifies memory error as $R$ 8 of all failures.

6. Cross-cutting methodology, limitations, and significance

Across these works, memory is operationalized in markedly different ways: as off-screen state persistence in video world modeling, as unintended replication in diffusion models, as factual and reflective retention in LLM agents, as procedural adaptation from feedback, or as executable state transformation in tool-using systems (Chen et al., 25 Jun 2026). This suggests that “MemoBench” is best treated as a family resemblance term rather than a single benchmark specification.

A consistent methodological trend is movement from static recall toward action-conditioned or state-conditioned evaluation. The video MemoBench penalizes camera inactivity because it can masquerade as consistency; diffusion MemBench evaluates both trigger suppression and general-prompt fidelity; agent-memory MemBench adds efficiency and capacity; MemoryBench adds service-time feedback loops; AMemGym introduces on-policy interaction and write/read/utilization diagnosis; Mem2ActBench and VehicleMemBench require memory to drive executable decisions rather than merely answer questions (Hong et al., 2024). A plausible implication is that benchmark difficulty increasingly comes not from storing facts alone but from revising, grounding, and applying state under partial observability, long delays, conflicting evidence, or tool constraints.

The limitations are correspondingly domain-specific. The video MemoBench notes that 360 clips are diagnostically substantial but still modest, that VQA remains dependent on an LLM judge despite polarity balancing and filtering, and that no unified inactivity-aware composite score is defined (Chen et al., 25 Jun 2026). Diffusion MemBench states that its verified memorized images are not exhaustive, that thresholding with SSCD can capture semantic similarity rather than only structural copying, and that search remains compute-intensive (Hong et al., 2024). The LLM-agent MemBench emphasizes that it focuses on structured memory and does not yet cover broader reflective dimensions such as long-term emotions exhaustively (Tan et al., 20 Jun 2025). MemoryBench acknowledges that feedback and judging are simulated by strong LLMs, and that some on-policy experiments cannot finish in reasonable time because of memory-system overheads (Ai et al., 20 Oct 2025). AMemGym similarly relies on LLM-simulated users, even though it reports strong meta-evaluation reliability (Jiayang et al., 2 Mar 2026). VehicleMemBench is intentionally domain-specific, which sharpens executable validity but limits immediate generalization (Chen et al., 25 Mar 2026).

Taken together, these benchmarks establish that contemporary memory evaluation is no longer confined to long-context reading comprehension. It now encompasses world-state continuity, memorization mitigation, multi-session personalization, continual learning from feedback, tool-grounded parameter grounding, and executable state control. In that broader scholarly sense, “MemoBench” names an evolving research program: the attempt to measure whether AI systems preserve, update, retrieve, and operationalize memory under realistic temporal and interactional constraints.